我正在尝试使用 Selenium 和 BeautifulSoup 从 https://yellowpages.com.eg/en/category/abrasives 中抓取元数据。我可以成功提取一些数据，但无法从嵌套在循环内的 div 内的标签获取文本。这是我当前的代码：
pagecount = 1
driver = webdriver.Chrome()
page_url = f"{base_url}/en/category/abrasives/p{pagecount}"
driver.get(page_url)
driver.implicitly_wait(10)
page_source = driver.page_source
time.sleep(1)
bs = BeautifulSoup(page_source, 'html.parser')
divs = bs.find_all('div', class_ = 'col-xs-12 item-details')
for div in divs:
img_tag = div.find('img')
if(img_tag):
img_src = img_tag['data-src']
print(img_src)
else:
# print("i provided no tag be off stupid")
pass
title = div.find('a', class_ = 'item-title').text.strip()
print(title)
address = div.find('a', class_ = 'address-text').find('span').text.strip()
print(address)
# description = div.find('div', class_ = 'item-aboutUs' )
descriptions = div.find_all('div', class_='item-aboutUs')
print(descriptions)
问题： 我想确保我正确地从 item-aboutUs div 内的 a 标签中提取文本。有没有更好的方法来处理这个问题，特别是在有多个 item-aboutUs div 的情况下？
注意：您的目标站点受 Cloudflare 保护，因此在运行脚本时不要忘记使用
user-agentcookieimport requests
from bs4 import BeautifulSoup
url = "https://yellowpages.com.eg/en/category/abrasives"
header = {
"User-Agent": "YOUR USER AGENT HEADER",
"Cookie": "YOUR COOKIE IS HERE"
}
resp = requests.get(url, headers=header, verify=False)
soup = BeautifulSoup(resp.text, 'lxml')
get_html = soup.findAll('div', class_='item-top-details')
for i in get_html:
about_us = ""
if i.find('div', class_='item-aboutUs'):
about_us = i.find('div', class_='item-aboutUs').text.strip()
else:
about_us = None
address = i.find('a', class_='address-text').text.strip()
company = i.find('a', class_='item-title').text.strip()
id = i.find('button', class_='add-wish-company')['data-wishid']
get_number = requests.post(f'https://yellowpages.com.eg/en/getPhones/{id}/false', headers=header, verify=False).json()
print(f"ID: {id} | Company: {company} | Address: {address} | Number: {get_number[0]} | About Us: {about_us}")
ID: 695543 | Company: Al Wasila Import & Export - BLASK | Address: 21 Emad El Din St., Floor 5, Downtown, Cairo. | Number: ['0101-0296-629'] | About Us: We specialize in surface treatment using abrasive products. We can offer you consultations on how to use the product and what is the best system to achieve (matte/mirror) finish
ID: 23510 | Company: The Egyptian Center For General Trade | Address: 7 Mohamed El Mowelhy St., Off Salah El Din St., Al laban, Alexandria. | Number: ['03-3917-689'] | About Us: We are leading importers & distributors of industrial chemicals used for polishing marble, granite & metals through international agencies since 1969. We satisfy our customers’ demand with excellent products & consistent quality, as well as providing them with technical guidance for the optimal use of these product.
ID: 451553 | Company: 123 For Abrasive | Address: 58 El Gomhoureya St., Downtown, Cairo. | Number: ['0100-0488-203', '0100-0311-871', '0120-0821-234', '0111-9108-474', '0111-1144-745'] | About Us: None
ID: 267655 | Company: 3M | Address: Darb El Qotta, Ataba, Cairo. | Number: ['0122-4625-762'] | About Us: None
ID: 12074 | Company: 3M Egypt Trading Ltd | Address: Holiday Inn Tower Corniche El Nil, Corniche El Maadi, Maadi, Cairo. | Number: ['02-2525-9004'] | About Us: None
ID: 429916 | Company: Alwasl General Supplies | Address: El Ordonya, 10th of ramadan, El sharkeya. | Number: ['055-4382-182', '0127-3730-310'] | About Us: None
ID: 267444 | Company: Amin Samir For Trade | Address: 25 Darb El Ibrahimy, Ataba, Cairo. | Number: ['0111-1833-352'] | About Us: None
ID: 568175 | Company: Demo | Address: El Gomhoureya St., El Madina Tower, El shark district, Port said. | Number: ['066-3222-992', '066-3231-314'] | About Us: None
ID: 20313 | Company: EACO - Egyptian Abrasives Co. | Address: 2 Mohamed Masoud St., Floor 1, Off Wabor El Meyah St., Bab shark, Alexandria. | Number: ['03-4255-001', '0127-3666-625'] | About Us: None
ID: 699064 | Company: Eissa Group For Abrasives | Address: Corniche El Nil St., El Shoaraa District, Damietta, Damietta. | Number: ['0120-1518-333'] | About Us: None
ID: 694330 | Company: El Asil For Abrasives | Address: 43 Ahmed Badawy St., El moneeb, Giza. | Number: ['0111-0505-255'] | About Us: None
ID: 267751 | Company: El Asly For Abrasives | Address: Darb Rayash, Ataba, Cairo. | Number: ['0100-7400-927'] | About Us: None
ID: 267399 | Company: El Fath | Address: 19 darb el qotta, Ataba, Cairo. | Number: ['0100-1805-942'] | About Us: None
ID: 419955 | Company: El Horreya For Abrasive | Address: 14 El Khalig El Masry St., Hadayek el kobba, Cairo. | Number: ['0127-5542-213'] | About Us: None
ID: 267401 | Company: El Tawakol | Address: 16 darb el qotta, Ataba, Cairo. | Number: ['0100-5791-922'] | About Us: None
ID: 699018 | Company: Elseady & Eissa For Abrasives | Address: 112 Teraet El Shoaraa St., El Shoaraa District, Damietta, Damietta. | Number: ['0122-4984-436', '0102-0425-625'] | About Us: None
ID: 679322 | Company: International Co. For Abrasives | Address: El Nasr Rd., Nasr city, Cairo. | Number: ['0111-9920-403'] | About Us: None
ID: 267657 | Company: Mohamed For Abrasives | Address: darb el qotta, Ataba, Cairo. | Number: ['0100-4360-754'] | About Us: None
ID: 267745 | Company: Sayed For Abrasives | Address: clot Bey st., darb el qotta, Ataba, Cairo. | Number: ['0122-4239-291'] | About Us: None
ID: 61861 | Company: Sultan Grinding Wheels Manufacturing Co. - SGWMCO | Address: 17 Ishaq Yaaqub St., Almaza, Heliopolis, Cairo. | Number: ['02-2775-8404', '02-2418-4036', '02-2775-8403'] | About Us: None
如果我错过了什么，请告诉我