使用 Selenium 和 Python 抓取 Javascript 页面

Question

我正在寻找从网站上抓取列表，您必须选择“是”，选中“按州显示”框，单击“提交并查找医生”，然后从下拉列表中选择一个州，然后记下医生的名字。

网站：

https://www.inspiresleep.com/en-us/find-a-doctor/

我卡在“按州显示”按钮上。使用此代码：

<input id="show-by-state-checkbox" type="checkbox">
<span class="checkbox"></span>
<span class="label-text">Show by State</span>

这是我到目前为止所拥有的：

from selenium import webdriver 
from selenium.webdriver.common.by import By
from selenium.webdriver import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

import time
import requests

doctor_dict = {}

#configure webdriver

options = webdriver.ChromeOptions()

driver = webdriver.Chrome(options = options)

driver.get("https://www.inspiresleep.com/en-us/find-a-doctor/")
time.sleep(3)
driver.find_element(By.XPATH,"//button[text()='Yes']").click()
time.sleep(5)
driver.find_element(By.CLASS_NAME,"value = 'checkbox'").click()
time.sleep(2)
driver.find_element(By.XPATH,"//button[text()='Submit & Find a Doctor']")

有人可以帮我解决选择复选框然后单击“选择并查找医生”的错误吗？是否可以从此页面后的下拉列表中选择一个州？或者是否有 API 或其他可以用来不必使用 Selenium 的东西？

Answer 1

尝试：

import requests

api_url = "https://www.inspiresleep.com/api/crm/get-clinics/"

payload = {
    "lat": 45.23312,
    "lng": -93.29134,
    "osa": True,
    "radius": 5000,
    "stateCode": "TX",  # <-- texas
}

data = requests.post(api_url, json=payload).json()

for c in data["clinics"]:
    print(c["Name"])
    # ... other data

打印：

Allergy & ENT Associates - Sterling Ridge
Allergy and ENT Associates - Town Center
Allergy ENT Clinic of Northeast Texas
Allergy, Ear, Nose & Throat Clinic of Northeast Texas, Rockwall
Associates of Ear, Nose & Throat Surgery - Flower Mound
Associates of Ear, Nose & Throat Surgery - Lewisville
Austin Association of Otolaryngologists, P.A.
Austin Ear Nose & Throat Clinic -  South Austin
Austin Ear, Nose & Throat Clinic - Medical Parkway
Austin Ear, Nose & Throat Clinic - Seton Center Parkway

...