我编写了一个Python程序来从网站上抓取数据
https://www.kijijiautos.ca/cars/gmc/canyon/#ml=%3A&ms=9900%3B13&od=up&p=5000%3A&sb=p&yc=2024% 3A2024。我想使用“设置位置”按钮输入我的邮政编码(加拿大的)。我正在尝试使用 Chromedriver 命令来模拟我手动执行的操作。我会手动单击“设置位置”按钮,这会弹出一个新的小窗口。然后我会输入我的邮政编码并按 Enter 键以启用它。接下来,我单击“半径”字段并选择“1000”选项。然后我会单击“限制半径到省份”复选框来启用它。接下来,我将单击“nnn 结果”字段,然后输入数据,窗口将消失,网页将更新我的搜索结果。
我试图在我的编程中模拟这一点,但我根本看不到我的数据正在更新网页。当我在更新之前和之后转储元素数据时,数据永远不会改变。这是因为它失败并显示“半径”字段的“您不能选择禁用选项”,因为在数据成功输入“邮政编码”字段之前它被禁用。我不知道我做错了什么。任何意见将不胜感激。请注意,我的程序保存了 Webdriver 执行的日志,因此我可以看到其中的任何错误 - 没有任何错误表明我可以看到。
程序代码:
# type: ignore
print('\nStarting program ...')
# Bring in basic libraries
import os
import time
# File names to be created
fnames = {
'web': 'logging-web.txt', # Webdriver console messages
}
# Check if the output files are in use - remove file if it exists and is not open
for fn in fnames:
if os.path.isfile(fnames[fn]):
try:
os.remove(fnames[fn])
except Exception as err:
print(f"\nERROR: {err} - ABORTING ...",flush=True)
exit(False)
# Bring in additional libraries
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Set up ChromeDriver options
chrome_options = Options()
chrome_options.add_argument('--remote-debugging-pipe')
chrome_options.add_argument('--incognito')
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--accept-all-cookies')
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--ignore-ssl-errors')
chrome_options.add_argument('--disable-infobars')
chrome_options.add_argument('--disable-notifications')
# DEBUG chrome_options.add_argument('--disable-blink-features=AutomationControlled')
# DEBUG chrome_options.add_argument('--enable-popup-blocking')
chrome_options.add_argument('--log-level=3')
chrome_options.page_load_strategy = 'normal' # 'eager' does not always get all of the data on a page
chrome_options.add_experimental_option('excludeSwitches',['enable-logging','enable-automation'])
chrome_options.add_experimental_option('prefs',{'profile.managed_default_content_settings.images':2}) # Don't render images
# Set up ChromeDriver logging
wservice = webdriver.ChromeService(service_args=['--append-log','--readable-timestamp'],log_output=f"{fnames['web']}")
# Start the webdriver
driver = webdriver.Chrome(options=chrome_options,service=wservice)
driver.maximize_window()
driver.set_script_timeout(30)
winhancurr = driver.current_window_handle
url = 'https://www.kijijiautos.ca/cars/gmc/canyon/#ml=%3A&ms=9900%3B13&od=up&p=5000%3A&sb=p&yc=2024%3A2024'
print(f"\nGetting webpage {url} ...")
print(f"\nINFO: original window handle={driver.current_window_handle}")
driver.get(url)
WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.TAG_NAME,'body')))
# Find "Set location" button and click on it. Note that there are 2 identical fields and only the 2nd one works.
# Also note that clicking this button opens another window on top of the current webpage.
elem = WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH,'//*[@id="root"]/div[3]/div/section[5]/div/div/div[1]/div/div[1]/div/button')))
print(f"\nINFO: elem.before={elem.get_attribute("outerHTML")}")
elem.click()
print(f"\nINFO: elem.after={driver.find_element(By.XPATH,'//*[@id="root"]/div[3]/div/section[5]/div/div/div[1]/div/div[1]/div/button').get_attribute("outerHTML")}")
# Find new window and move to it
for winhan in driver.window_handles:
print(f"\nINFO: window handles={winhan}")
if winhan != winhancurr:
driver.switch_to.window(winhan)
print(f"\nINFO: new window handle={driver.current_window_handle}")
break
if driver.current_window_handle == winhancurr:
print(f"\nERROR: No new window created ...")
# Put postalcode data into field and end data input
elem2 = WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.ID,'LocationAutosuggest')))
print(f"\nINFO: elem2.before={driver.find_element(By.ID,'LocationAutosuggest').get_attribute("outerHTML")}")
elem2.send_keys('T1J 4C8')
elem2.send_keys(Keys.ENTER)
print(f"\nINFO: elem2.after={driver.find_element(By.ID,'LocationAutosuggest').get_attribute("outerHTML")}")
# Find "Radius" field and enter data
elem3 = WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.ID,'rd')))
print(f"\nINFO: elem3.before={elem3.get_attribute("outerHTML")}")
drop = Select(elem3)
drop.select_by_value('1000')
print(f"\nINFO: elem3.after={elem3.get_attribute("outerHTML")}")
# Click on "Limit radius to province" checkbox
elem4 = WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.NAME,'limitedToProvince')))
print(f"\nINFO: elem4={elem4.get_attribute("outerHTML")}")
elem4.click()
# Apply all changes
elem5 = WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.LINK_TEXT,'Apply')))
print(f"\nINFO: elem5={elem5.get_attribute("outerHTML")}")
elem5.click()
# Switch back to original window
driver.switch_to.window(winhancurr)
# End the driver
driver.quit()
# End of main
启动程序...
获取网页
https://www.kijijiautos.ca/cars/gmc/canyon/#ml=%3A&ms=9900%3B13&od=up&p=5000%3A&sb=p&yc=2024%3A2024 ...
信息:原始窗口句柄=5DCA8B95024DB03D50D0A6302BA77DE3信息:elem.before=设置位置
信息:elem.after=设置位置
信息:窗口句柄=5DCA8B95024DB03D50D0A6302BA77DE3
错误:未创建新窗口...
信息:elem2.before=
信息:elem2.after=
信息:elem3.before=+ 5公里+ 10公里+ 15公里+ 25公里+ 50公里+ 100公里+ 150公里+ 250公里+ 500公里+ 750公里+ 1,000公里 回溯(最近一次调用最后一次): 文件“C:\Users\david.DESKTOP-IH066BB\Documents\Software\CarSearch estx.py”,第 95 行,位于 drop.select_by_value('1000') ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ 文件“C:\Users\david.DESKTOP-IH066BB\AppData\Local\Programs\Python\Python313\Lib\site-packages\selenium\webdriver\support\select.py”,第 79 行,在 select_by_value 中 self._set_selected(选择) ~~~~~~~~~~~~~~~~~~^^^^^ 文件“C:\ Users \ david.DESKTOP-IH066BB \ AppData \ Local \ Programs \ Python \ Python313 \ Lib \ site-packages \ selenium \ webdriver \ support \ select.py”,第213行,在_set_selected中 raise NotImplementedError("您不能选择禁用的选项") NotImplementedError:您不能选择禁用的选项
import os
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Set up ChromeDriver options
chrome_options = Options()
chrome_options.add_argument('--remote-debugging-pipe')
chrome_options.add_argument('--incognito')
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--accept-all-cookies')
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--ignore-ssl-errors')
chrome_options.add_argument('--disable-infobars')
chrome_options.add_argument('--disable-notifications')
chrome_options.add_argument('--log-level=3')
chrome_options.page_load_strategy = 'normal'
chrome_options.add_experimental_option('excludeSwitches', ['enable-logging', 'enable-automation'])
chrome_options.add_experimental_option('prefs', {'profile.managed_default_content_settings.images': 2})
# Set up ChromeDriver logging
wservice = webdriver.ChromeService(service_args=['--append-log', '--readable-timestamp'], log_output='logging-web.txt')
# Start the webdriver
driver = webdriver.Chrome(options=chrome_options, service=wservice)
driver.maximize_window()
driver.set_script_timeout(30)
url = 'https://www.kijijiautos.ca/cars/gmc/canyon/#ml=%3A&ms=9900%3B13&od=up&p=5000%3A&sb=p&yc=2024%3A2024'
print(f"\nGetting webpage {url} ...")
driver.get(url)
WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.TAG_NAME, 'body')))
# Find "Set location" button and click on it
elem = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="root"]/div[3]/div/section[5]/div/div/div[1]/div/div[1]/div/button')))
elem.click()
# Wait for the modal to appear and interact with it
modal = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.ID, 'LocationAutosuggest')))
modal.send_keys('T1J 4C8')
modal.send_keys(Keys.ENTER)
# Wait for the radius dropdown to become enabled and select a value
radius_dropdown = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.ID, 'rd')))
drop = Select(radius_dropdown)
drop.select_by_value('1000')
# Click on "Limit radius to province" checkbox
checkbox = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.NAME, 'limitedToProvince')))
checkbox.click()
# Apply all changes
apply_button = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.LINK_TEXT, 'Apply')))
apply_button.click()
# End the driver
driver.quit()