Mediapipe 通过未检测到的手势识别与 GUI 程序交互

Question

我目前有一个自定义 haar 级联对象检测的工作示例，其中用户的拳头将用于在菜单页面中使用箭头形状浏览编程的 GUI 组件，但是对于一项任务，我必须将其转换为 mediapipe 示例将检测每个索引点上的拳头绘制圆圈并以相同的方式浏览菜单。我目前对如何调用手势识别来与菜单交互感到困惑，并且不太确定在几次迭代后我需要在程序中添加或删除什么。抱歉，如果这是一个真正的新示例，我对 mediapipe、python 和 opencv 完全陌生，只是将它们用于一个任务。

# Fist hand function
def detect_fist(image):
    # Image convert to RGB
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    results = hands.process(image)

    # Are hands in the image
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Check if the index and middle fingers are extended
            if (hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP].y <
                    hand_landmarks.landmark[mp_hands.HandLandmark.MIDDLE_FINGER_TIP].y):
                return True
    return False


while camera.isOpened():
    # Main Camera
    ret, frame = camera.read()

    if ret:
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)


        # Draws circle landmarks onto the videocapture
        def draw_landmarks(image, landmarks):
            for landmark in landmarks.landmark:
                x, y = int(landmark.x * image.shape[1]), int(landmark.y * image.shape[0])
                cv2.circle(image, (x, y), 5, (0, 255, 0), -1)


        # Draw landmarks on the image
        results = hands.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
        if results.multi_hand_landmarks:
            for hand_landmarks in results.multi_hand_landmarks:
                draw_landmarks(frame, hand_landmarks)

                fist_pos = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
                thumb_pos = hand_landmarks.landmark[mp_hands.HandLandmark.MIDDLE_FINGER_TIP]

if detect_fist(frame):
            for (fist_pos, thumb_pos) in landmarks:
                cv2.putText(frame, "Fist detected!", (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 0, 0), 2)
                dx = (fist_pos.x - thumb_pos.x)
                dy = (fist_pos.y - thumb_pos.y)
                if change_menu == "" or change_menu == "variable":

                    for i in range(len(menu_components)):
                        x_shape_start = menu_components[i]['startEnd'][0]
                        y_shape_start = menu_components[i]['startEnd'][1]
                        x_shape_end = menu_components[i]['startEnd'][2]
                        y_shape_end = menu_components[i]['startEnd'][3]
                        if (x_shape_start < dx < x_shape_end) and \
                                (y_shape_start < dy < y_shape_end):
                            if i < 6:
                                act = i
                            else:
                                if i == 6:
                                    act = "back"
                                elif i == 7:
                                    act = "next"
                        elif change_menu == "number":
                            for i in range(10):
                                x_shape_start = numbers_menu[i][0]
                                y_shape_start = numbers_menu[i][1]
                                x_shape_end = numbers_menu[i][2]
                                y_shape_end = numbers_menu[i][3]
                                if (x_shape_start < dx < x_shape_end) and (
                                        y_shape_start < dy < y_shape_end):
                                    # Number Acknowledgement
                                    # menusound.play(1)
                                    # menusound.wait()
                                    temp_code += str(i)
                                    change_menu = ""
                                    selected_bool = True

我还将在此处包含指向完整代码的 pastebin 链接：https://pastebin.com/PjEGhmQ5

非常感谢任何指导，如果真的很明显，请随时称我为新手！

提前致谢并保重:)

我已经尝试使用 mediapipe 拳头识别来被相机检测到并通过定制的 GUI 导航，它在拳头周围绘制圆形地标但是根本不与菜单项交互。