使用 BS4 和 python 抓取曼城网站

Question

所以基本上，我想从 https://www.mancity.com/fixtures 网站中提取有关即将到来的比赛日期的所有文本并将其放入列表中。 问题是，当我给出 div 标签的类名并提取所有数据时，我无法提取任何内容，它只返回一个空列表 这可能是我这边的错误，但经过大量故障排除后我仍然无法解决它:(

请帮忙..

代码：

from bs4 import BeautifulSoup
import requests

url = 'https://www.mancity.com/fixtures'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
dates_tag = soup.find_all("div", class_="fixture-header__competition")
dates = []

for date in dates_tag:
    dates.append(date)

print(dates)

输出：

[]

版本： 请求==2.32.3 美丽汤4==4.12.3

Answer 1

Cloudflare 保护您的目标应用程序（如果您尝试

403 Forbidden

并且没有旁路工作，您将看到

response.status_code

错误（就我而言，我尝试了

cookie

和

user-agent

），因此唯一的选择是使用硒与 bs4，

这是带有 bs4 的 Selenium chrome 驱动程序的代码，

示例代码：

from bs4 import BeautifulSoup
from selenium import webdriver

chrome_options = webdriver.ChromeOptions()

chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--window-size=1920x1080")

driver = webdriver.Chrome(options=chrome_options)
driver.get('https://www.mancity.com/fixtures')

html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
content = soup.find_all("li", class_="grid grid-bleed fixture fixtures-list-item")
for i in content:
    match = i['aria-label']
    time = i.find('time')['datetime']
    print(f"Match: {match} | Time: {time}")

输出：

Match: Liverpool versus Manchester City | Time: 2024-12-01T16:00:00.0000000+00:00
Match: Manchester City versus Nottingham Forest | Time: 2024-12-04T19:30:00.0000000+00:00
Match: Crystal Palace versus Manchester City | Time: 2024-12-07T15:00:00.0000000+00:00
Match: Juventus versus Manchester City | Time: 2024-12-11T20:00:00.0000000+00:00
Match: Manchester City versus Manchester United | Time: 2024-12-15T16:30:00.0000000+00:00
Match: Aston Villa versus Manchester City | Time: 2024-12-21T12:30:00.0000000+00:00
Match: Manchester City versus Everton | Time: 2024-12-26T12:30:00.0000000+00:00
Match: Leicester City versus Manchester City | Time: 2024-12-29T14:30:00.0000000+00:00
Match: Manchester City versus West Ham United | Time: 2025-01-04T15:00:00.0000000+00:00
Match: Brentford versus Manchester City | Time: 2025-01-14T19:30:00.0000000+00:00
Match: Ipswich Town versus Manchester City | Time: 2025-01-19T16:30:00.0000000+00:00
Match: Paris Saint-Germain versus Manchester City | Time: 2025-01-22T20:00:00.0000000+00:00
Match: Manchester City versus Chelsea | Time: 2025-01-25T17:30:00.0000000+00:00
Match: Manchester City versus Club Brugge | Time: 2025-01-29T20:00:00.0000000+00:00
Match: Arsenal versus Manchester City | Time: 2025-02-01T15:00:00.0000000+00:00
Match: Manchester City versus Newcastle United | Time: 2025-02-15T15:00:00.0000000+00:00
Match: Manchester City versus Liverpool | Time: 2025-02-22T15:00:00.0000000+00:00
Match: Tottenham Hotspur versus Manchester City | Time: 2025-02-25T19:45:00.0000000+00:00
Match: Nottingham Forest versus Manchester City | Time: 2025-03-08T15:00:00.0000000+00:00
Match: Manchester City versus Brighton and Hove Albion | Time: 2025-03-15T15:00:00.0000000+00:00
Match: Manchester City versus Leicester City | Time: 2025-04-02T18:45:00.0000000+00:00
Match: Manchester United versus Manchester City | Time: 2025-04-05T14:00:00.0000000+00:00
Match: Manchester City versus Crystal Palace | Time: 2025-04-12T14:00:00.0000000+00:00
Match: Everton versus Manchester City | Time: 2025-04-19T14:00:00.0000000+00:00
Match: Manchester City versus Aston Villa | Time: 2025-04-26T14:00:00.0000000+00:00
Match: Manchester City versus Wolves | Time: 2025-05-03T14:00:00.0000000+00:00
Match: Southampton versus Manchester City | Time: 2025-05-10T14:00:00.0000000+00:00
Match: Manchester City versus Bournemouth | Time: 2025-05-18T14:00:00.0000000+00:00
Match: Fulham versus Manchester City | Time: 2025-05-25T15:00:00.0000000+00:00