Python Selenium 脚本无法从网页检索产品价格

Question

我正在尝试使用 Python 和 Selenium 从网站 Ultra Liquors 抓取产品价格，但尽管 HTML 包含预期元素，但我无法检索价格。我的目标是比较几家商店的价格，以找到最优惠的价格或我们场地的任何正在进行的特价商品。

这是我正在使用的代码：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

# Set up Chrome options
options = Options()
options.add_argument('--headless')  # Run in headless mode
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

# Initialize Chrome driver
service = Service('path_to_chromedriver')  # Replace with your path to chromedriver
driver = webdriver.Chrome(service=service, options=options)

# Open the product page
driver.get('https://george.ultraliquors.co.za/olof-bergh-olof-bergh-brandy-750ml')

try:
    # Attempt to retrieve product name
    product_name = driver.find_element(By.XPATH, '//h1[@class="product-title"]').text
    print(f"Product Name: {product_name}")
except Exception as e:
    print(f"Could not locate product name: {e}")

try:
    # Attempt to retrieve price
    price_element = driver.find_element(By.CLASS_NAME, 'price-value-10677')
    price = price_element.text
    print(f"Price: {price}")
except Exception as e:
    print(f"Could not locate price: {e}")

# Close the driver
driver.quit()

我希望获得价格值“R169.99”，但脚本找不到它并返回错误消息。我尝试过使用不同的元素定位器并检查元素是否动态加载。

我使用的是 Python 3.12、Selenium 4.8 和 ChromeDriver。任何帮助将不胜感激！

Answer 1

你应该使用bs4，因为bs4比selenium更快（如果你不需要处理机器人保护），

我使用了这个端点

https://george.ultraliquors.co.za/getFilteredProducts

，它接受带有JSON正文的

POST

请求，您可以获取

SPIRITS

-->

BRANDY

类别下的所有产品价格，并且您的

catagoryId

是

4

有您的目标应用程序中有很多类别，其中您的目标类别编号是

4

，这里是带有

bs4

和

requests

库的示例代码，

示例代码：

import requests
from bs4 import BeautifulSoup

url = "https://george.ultraliquors.co.za/getFilteredProducts"
page = 0
while True:
    page = page + 1
    data = {"categoryId":"3","manufacturerId":"0","vendorId":"0","pageNumber":page,"orderby":"5","viewmode":None,"pagesize":0,"queryString":"","shouldNotStartFromFirstPage":True,"keyword":"","searchCategoryId":"0","searchManufacturerId":"0","searchVendorId":"0","priceFrom":"","priceTo":"","includeSubcategories":"False","searchInProductDescriptions":"False","advancedSearch":"False","isOnSearchPage":"False","inStockFilterModel":None}
    header = {
        "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
    }

    r = requests.post(url, json=data, headers=header).text
    soup = BeautifulSoup(r, 'lxml')
    html_page = soup.findAll('div', class_='product-item product-box-product-item product-box-grid')
    if html_page:
        for i in html_page:
            product = i.find(class_="product-title-product-box").text.strip()
            price = f"{i.find(class_='price actual-price').text.strip()}.{i.find(class_='price actual-price-cents').text.strip()}"
            size = i.find(class_='desktop-product-box-pack-size').text.strip()
            product_url = f"https://george.ultraliquors.co.za{i.find(class_='product-title-product-box')['href']}"
            all_details = [product, price, size, product_url]
            print(all_details)
    else:
        break

输出示例：

['1000 POUNDER RUM', 'R249.99', '1 x 750ML', 'https://george.ultraliquors.co.za/1000-pounder-1000-pounder-rum-750ml-2']
['1000 POUNDER RUM', 'R1479.00', '6 x 750ML', 'https://george.ultraliquors.co.za/1000-pounder-1000-pounder-rum-750ml-x-6']
['ABERLOUR 12YR MALT TIN', 'R809.99', '1 x 750ML', 'https://george.ultraliquors.co.za/aberlour-aberlour-12yr-malt-tin-750ml-3']
['ABERLOUR 12YR MALT TIN', 'R4779.00', '6 x 750ML', 'https://george.ultraliquors.co.za/aberlour-aberlour-12yr-malt-tin-750ml-x-6']
['ABERLOUR 16YR MALT TIN', 'R1539.00', '1 x 750ML', 'https://george.ultraliquors.co.za/aberlour-aberlour-16yr-malt-tin-750ml']
['ABERLOUR 16YR MALT TIN', 'R8419.00', '6 x 750ML', 'https://george.ultraliquors.co.za/aberlour-aberlour-16yr-malt-tin-750ml-x-6']
['ABSOLUT VODKA BLUE', 'R249.99', '1 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-blue-750ml']
['ABSOLUT VODKA BLUE', 'R2949.00', '12 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-blue-750ml-x-12']
['ABSOLUT VODKA GRAPEFRUIT', 'R249.99', '1 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-grapefruit-750ml']
['ABSOLUT VODKA GRAPEFRUIT', 'R2949.00', '12 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-grapefruit-750ml-x-12']
['ABSOLUT VODKA LIME', 'R249.99', '1 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-lime-750ml']
['ABSOLUT VODKA LIME', 'R2949.00', '12 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-lime-750ml-x-12']
['ABSOLUT VODKA RASPBERRI', 'R249.99', '1 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-raspberri-750ml-2']
['ABSOLUT VODKA RASPBERRI', 'R2949.00', '12 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-raspberri-750ml-x-12-2']
['ABSOLUT VODKA WATERMELON', 'R249.99', '1 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-watermelon-750ml']
['ABSOLUT VODKA WATERMELON', 'R2949.00', '12 x 750ML', 'https://george.ultraliquors.co.za/absolut-absolut-vodka-watermelon-750ml-x-12']
['AERSTONE SINGLE MALT LAND CASK', 'R444.99', '1 x 750ML', 'https://george.ultraliquors.co.za/aerstone-aerstone-single-malt-land-cask-750ml-2']
['AERSTONE SINGLE MALT LAND CASK', 'R2619.00', '6 x 750ML', 'https://george.ultraliquors.co.za/aerstone-aerstone-single-malt-land-cask-750ml-x-6']

请告诉我这是否适合您