使用 pyautogui 查找浏览器中单击的点的坐标

Question

我有一个脚本可以根据网页的屏幕截图查找彩色框。 我想单击我找到的框，但 pyautogui 无法从浏览器获取点并将其直接转换为右侧可单击的点。

有没有办法告诉 pyautogui 在浏览器窗口内查找坐标，而不必转换它们？

我的脚本的点击部分供参考：

def click_correct_box(driver):
    time.sleep(4)
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".block2"))
    )
    element.screenshot('elem.png')
    element_image = Image.open('elem.png')
    ocr_data = pytes`your text`seract.image_to_data(element_image, output_type=pytesseract.Output.DICT)
    text_bounding_box = find_text_bounding_box(ocr_data)
    reference_color = find_reference_color(element_image, text_bounding_box)
    visualize_ocr_data(element_image, ocr_data)
    if reference_color is None:
        print("Failed to analyze reference color")
        return
    time.sleep(2)
    driver.save_screenshot('element_with_boxes.png')
    element_with_boxes_image = Image.open('element_with_boxes.png')
    closest_box = find_color_boxes(element_with_boxes_image, reference_color, text_bounding_box)
    if closest_box:
        x1, y1, x2, y2 = closest_box
        click_x = (x1 + x2) / 2
        click_y = (y1 + y2) / 2
        print(f"Click coordinates: ({click_x}, {click_y})")

        driver.execute_script("window.scrollTo(0, arguments[0]);", y1 - 50)
        time.sleep(1)

        location = element.location
        size = element.size

        # Adjust click_x and click_y to absolute coordinates relative to the browser window
        click_x_absolute = location['x'] + click_x
        click_y_absolute = location['y'] + click_y

        print(f"Absolute click coordinates: ({click_x_absolute}, {click_y_absolute})")

        # Directly use the absolute coordinates relative to the browser window
        pyautogui.moveTo(click_x_absolute, click_y_absolute, duration=1)
        pyautogui.click()
        time.sleep(2)
    else:
        print("No suitable box found")

我尝试过直接使用js的另一种点击方法，但没有成功。 几个小时的调试和微调，我还在这里

Answer 1

据我所知，您正在页面的屏幕截图中找到该元素。这意味着永远不需要滚动，因为屏幕截图仅包含当前可见的内容。您没有提供足够的信息，以便有人可以找出问题所在，但我的猜测是它与坐标数学有关。试试这个：

driver.execute_script(f"document.elementFromPoint({click_x}, {click_y}).click();")