如何使用 selenium 滚动浏览并打印网站的结果。我正在尝试抓取一个网站，但它没有打印所有结果[重复]

Question

我正在使用 selenium 扫描 booking.com，它有 600 多个结果，但 selenium 只扫描 51 个结果。我不知道我做错了什么。这是我的代码。

from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd

url = 'https://www.booking.com/searchresults.html?aid=397645&label=bin859jc-1DCAEoggI46AdIM1gDaKcBiAEBmAExuAEXyAEM2AED6AEB-AECiAIBqAIDuAK-xqeyBsACAdICJGEwYTAzM2QwLTQ1MjEtNGIzNi04NTYyLTcxMjY5NzJlMGNjNtgCBOACAQ&sid=968beab70293559758c39b5f6e59b26e&checkin=2024-05-20&checkout=2024-05-23&region=5140&'
driver = webdriver.Edge()
driver.get(url)

name = []
location = []

container = driver.find_element(By.CLASS_NAME, "d4924c9e74")
apartment = container.find_elements(By.XPATH, './/div[contains(@role, "group")]')

for x in apartment:
    name.append(x.find_element(By.XPATH, './/a[contains(@data-testid, "title")]').text)
    location.append(x.find_element(By.XPATH, './/span[contains(@data-testid, "address")]').text)
    # print(x.text)

hotels = {
    'name' : name,
    'location' : location
}
print(hotels)

这是我的结果

{'name': ['The Mora Zanzibar\nOpens in new window', 'Zanziblue\nOpens in new window', 'Maisha Nungwi\nOpens in new window', 'Baladin Zanzibar Beach Hotel\nOpens in new window', 'Be Zanzibar Boutique Hotel\nOpens in new window', 'Promised Land Lodge\nOpens in new window', 'Kwanza Resort by SUNRISE\nOpens in new window', 'Karafuu Beach Resort & Spa\nOpens in new window', 'Hekaya Zanzibar\nOpens in new window', 'Paradise Beach Resort & Spa\nOpens in new window', 'Zuri Zanzibar\nOpens in new window', 'Tembo House Hotel\nOpens in new window', 'Sunset Kendwa Beach Hotel\nOpens in new window', 'Villa Nyota Zanzibar\nOpens in new window', 'Kizikula\nOpens in new window', 'Hotel Riu Jambo - All Inclusive\nOpens in new window', 'Chumbe Island Coral Park\nOpens in new window', 'Gold Zanzibar Beach House & Spa\nOpens in new window', 'Mtende Beach Bungalow océan view\nOpens in new window', 'Le Mersenne Zanzibar, Autograph Collection\nOpens in new window', 'Pongwe Bay Resort\nOpens in new window', 'Aurelia Zanzibar\nOpens in new window', 'Aya Beach Resort\nOpens in new window', 'Z-Lodge Zanzibar\nOpens in new window', 'Mayai Ocean Resort\nOpens in new window'], 'location': ['Matemwe', 'Matemwe', 'Nungwi Beach, Nungwi', 'Michamvi', 'Mfumbwi', 'Kizimkazi Beach, Kizimkazi', 'Kizimkazi', 'Pingwe', 'Pingwe', 'Uroa', 'Kendwa Beach, Kendwa', 'Zanzibar City', 'Kendwa Beach, Kendwa', 'Kidenga', 'Kizimkazi', 'Nungwi', 'Mbweni', 'Kendwa Beach, Kendwa', 'Mtende', 'Michamvi', 'Pongwe', 'Ngambo', 'Kizimkazi Beach, Kizimkazi', 'Kiwengwa', 'Bwejuu Beach, Bwejuu']}

Answer 1

当您滚动时，结果会动态加载。您需要使用selenium向下滚动页面才能加载所有结果。

参见这里例如