我不确定如何选择表 class="table-info" 中的以下项目
使用python和beautifulsoup,我想提取:
电话
电子邮件
网站
主要活动(不含 div 的 li 元素文本) “计算机咨询活动”。
<table class="table-info">
<tbody>
<tr>
<td class="col-1">
<div class="col-1-text">Business name</div>
</td>
<td class="col-2">
<div class="col-2-text">Company XYZ</div>
</td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">Register code:</div>
</td>
<td class="col-2">
<div class="col-2-text">112233558</div>
</td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">Operating address:</div>
</td>
<td class="col-2">
<div class="col-2-text"><a target="googlemaps" href="https://www.google.com/maps/place/Some-location"
class="link-location">Some location strt. 233</a></div>
</td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">Legal address</div>
</td>
<td class="col-2">
<div class="col-2-text">
<a class="link-location" href="https://www.google.com/maps/place/Some-location" target="_new">Some
location
</a>
</div>
</td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">VAT No:</div>
</td>
<td class="col-2">
<div class="col-2-text"><a href="javascript:void(0)" onclick="return getVAT(this, '12345678')">Get VAT
liability</a></div>
</td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">Age:</div>
</td>
<td class="col-2">
<div class="col-2-text">1 year 3 months</div>
</td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">Founded:</div>
</td>
<td class="col-2">
<div class="col-2-text">20/09/2019</div>
</td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">Capital:</div>
</td>
<td class="col-2">
<div class="col-2-text">2000 USD</div>
</td>
</tr>
<tr>
<td colspan="2" class="sep"></td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">Phone:</div>
</td>
<td class="col-2">
<div class="col-2-text">123456789</div>
</td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">E-mail:</div>
</td>
<td class="col-2">
<div class="col-2-text"><a href="mailto:[email protected]">[email protected]</a></div>
</td>
</tr>
<tr>
<td colspan="2" class="sep"></td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">Representatives:</div>
</td>
<td class="col-2">
<div class="col-2-text">
<div class="box-message">
<p class="desc">To access information, please</p>
<p>
<a href="#" onclick="return loginClicked(this, '#');"
class="btn btn-small btn-purple link-login">Log in</a>
</p>
</div>
</div>
</td>
</tr>
<tr>
<td colspan="2" class="sep"></td>
</tr>
<tr>
<td class="col-1">
<div class="col-1-text">
Main activity:
<span class="tip info" title=""
data-original-title="Activities are classified according to EMTAK 2008"></span>
</div>
</td>
<td class="col-2">
<div class="col-2-text" id="activity_top5ffe2eab23d13">
<ul>
<li>
Computer consultancy activities
<div class="main_activities_top_link_wrapper">
<a href="https://www.somesite.com/" target="_blank"
onclick="ga('send', 'event', 'check', 'top_btn', 'Anonym');"
class="btn btn-simple btn-open-graph">
<span>Open TOP 20</span> </a>
</div>
</li>
</ul>
</div>
</td>
</tr>
</tbody>
注意:以上代码是一个查询结果/html示例,但有时查询结果/公司没有电子邮件或网站/反之亦然。因此,如果代码找不到所需的 html 内容,那么代码不会出错,这一点很重要。我发现最好遵循类名或 id,而不是计算表/div 嵌套的深度(xpath)。
我的代码不能很好地工作:
import csv
import requests
import datetime
import time
from requests import get
from bs4 import BeautifulSoup
with open('data.csv', encoding='utf8') as csvfile:
reader = csv.reader(csvfile, delimiter=';')
next(reader)
count = 0
for row in reader:
timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
url = f'https://www.somedomain.com/result?country=en&q={row[1]}'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
cookies = {'__test': '1bb6e881021f013463740eeb74840b18'}
content = get(url, headers=headers, cookies=cookies).content
soup = BeautifulSoup(content, "lxml")
table_info = soup.select_one('.table-info')
mail = table_info.select_one('.col-2 a[href^=mailto]')
mail = mail.get('href')
mail_clean = mail.split(':')[1]
website = soup.find(text='Website:')
website = table_info.select_one('.col-2 a[target^=_blank]')
website = website.get('href')
collected_data = row[1], mail_clean, website, timestamp
data_list = [["Regcode", "Email", "Website", "Timestamp"],collected_data]
with open('extracted.csv', 'w', newline='') as file:
writer = csv.writer(file, delimiter=';')
writer.writerows(data_list)
print(row[1], "|", mail_clean,"|", website,"|", timestamp)
#print("Waiting 3 seconds...")
#time.sleep(3)
count+=1
您是否考虑过使用 css 选择器来计算表的子项数量?如果您的表始终反映示例代码,则使用
nth-child
属性可能会更容易。
tr:nth-child(10) .col-2-text
tr:nth-child(11) a
span
li
我使用 Selector Gadget 来抓取这些标签。您可能想直接在您的页面上运行它,看看是否有其他更容易实现的。