从 python 中查找或选择元素以使用 beautifulsoup 进行抓取

问题描述 投票:0回答:1

我不确定如何选择表 class="table-info" 中的以下项目

使用python和beautifulsoup,我想提取:

  1. 电话

  2. 电子邮件

  3. 网站

  4. 主要活动(不含 div 的 li 元素文本) “计算机咨询活动”。

     <table class="table-info">
     <tbody>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Business name</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text">Company XYZ</div>
             </td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Register code:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text">112233558</div>
             </td>
         </tr>
    
    
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Operating address:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text"><a target="googlemaps" href="https://www.google.com/maps/place/Some-location"
                         class="link-location">Some location strt. 233</a></div>
             </td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Legal address</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text">
                     <a class="link-location" href="https://www.google.com/maps/place/Some-location" target="_new">Some
                         location
                     </a>
                 </div>
             </td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">VAT No:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text"><a href="javascript:void(0)" onclick="return getVAT(this, '12345678')">Get VAT
                         liability</a></div>
             </td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Age:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text">1 year&nbsp;3 months</div>
             </td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Founded:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text">20/09/2019</div>
             </td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Capital:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text">2000 USD</div>
             </td>
         </tr>
         <tr>
             <td colspan="2" class="sep"></td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Phone:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text">123456789</div>
             </td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">E-mail:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text"><a href="mailto:[email protected]">[email protected]</a></div>
             </td>
         </tr>
         <tr>
             <td colspan="2" class="sep"></td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">Representatives:</div>
             </td>
             <td class="col-2">
                 <div class="col-2-text">
                     <div class="box-message">
                         <p class="desc">To access information, please</p>
                         <p>
                             <a href="#" onclick="return loginClicked(this, '#');"
                                 class="btn btn-small btn-purple link-login">Log in</a>
                         </p>
                     </div>
                 </div>
             </td>
         </tr>
         <tr>
             <td colspan="2" class="sep"></td>
         </tr>
         <tr>
             <td class="col-1">
                 <div class="col-1-text">
                     Main activity:
                     <span class="tip info" title=""
                         data-original-title="Activities are classified according to EMTAK 2008"></span>
                 </div>
             </td>
             <td class="col-2">
                 <div class="col-2-text" id="activity_top5ffe2eab23d13">
                     <ul>
                         <li>
                             Computer consultancy activities
                             <div class="main_activities_top_link_wrapper">
                                 <a href="https://www.somesite.com/" target="_blank"
                                     onclick="ga('send', 'event', 'check', 'top_btn', 'Anonym');"
                                     class="btn btn-simple btn-open-graph">
                                     <span>Open TOP 20</span> </a>
                             </div>
                         </li>
                     </ul>
    
                 </div>
             </td>
         </tr>
    
    
     </tbody>
    

注意:以上代码是一个查询结果/html示例,但有时查询结果/公司没有电子邮件或网站/反之亦然。因此,如果代码找不到所需的 html 内容,那么代码不会出错,这一点很重要。我发现最好遵循类名或 id,而不是计算表/div 嵌套的深度(xpath)。

我的代码不能很好地工作:

import csv
import requests
import datetime
import time
 
from requests import get
from bs4 import BeautifulSoup
 
 
with open('data.csv', encoding='utf8') as csvfile:
    reader = csv.reader(csvfile, delimiter=';')
    next(reader)
 
    count = 0
     
    for row in reader:
         
        timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
 
        url = f'https://www.somedomain.com/result?country=en&q={row[1]}'
         
        headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
        cookies = {'__test': '1bb6e881021f013463740eeb74840b18'}
        content = get(url, headers=headers,  cookies=cookies).content
        soup = BeautifulSoup(content, "lxml")
 
        table_info = soup.select_one('.table-info')
 
        mail = table_info.select_one('.col-2 a[href^=mailto]')
        mail = mail.get('href')
        mail_clean = mail.split(':')[1]
 
        website = soup.find(text='Website:')
        website = table_info.select_one('.col-2 a[target^=_blank]')
        website = website.get('href') 
         
        collected_data = row[1], mail_clean, website, timestamp
 
        data_list = [["Regcode", "Email", "Website", "Timestamp"],collected_data]
        with open('extracted.csv', 'w', newline='') as file:
            writer = csv.writer(file, delimiter=';')
            writer.writerows(data_list)
 
        print(row[1], "|", mail_clean,"|", website,"|", timestamp)
        #print("Waiting 3 seconds...")
        #time.sleep(3)
        count+=1
     
  
python web-scraping beautifulsoup
1个回答
0
投票

您是否考虑过使用 css 选择器来计算表的子项数量?如果您的表始终反映示例代码,则使用

nth-child
属性可能会更容易。

  • 电话:
    tr:nth-child(10) .col-2-text
  • 邮箱:
    tr:nth-child(11) a
  • 网站:
    span
  • 主要活动:
    li

我使用 Selector Gadget 来抓取这些标签。您可能想直接在您的页面上运行它,看看是否有其他更容易实现的。

© www.soinside.com 2019 - 2024. All rights reserved.