我应该如何使用python和BS4从网页中正确提取和解析主题数据？

Question

我正在寻找一个网络爬虫，从论坛收集主题行。一旦有了，我想将每个主题显示为一个新行，每行前面都有一个[*]。

使用BeautifulSoup，我可以抓取一个页面并提取span类“subject”。然而，从那里，我不确定如何解析主题文本，然后按照我正在尝试的方式对其进行排序。

import requests

from bs4 import BeautifulSoup

url = "https://boards.4channel.org/sci/"

#send the HTTP request
response = requests.get(url)

if response.status_code == 200:

    #pull the content
    html_content = response.content

    #send the page to BeautifulSoup
    html_doc     = BeautifulSoup(html_content, "html.parser")

    #extract topic data
    topic_spider = html_doc.find_all("span",{"class":"subject"})

    print topic_spider

爬虫的当前结果如下所示：

[<span class="subject"></span>, <span class="subject"></span>, <span class="subject">Cigarettes vs. Cannabis</span>, <span class="subject">Cigarettes vs. Cannabis</span>, <span class="subject"></span>, <span class="subject"></span>, <span class="subject"></span>, <span class="subject"></span>, <span class="subject"></span>...

我试图像这样订购它们：

[*] Topic 1
[*] Topic 2
[*] Topic 3

Answer 1

检查元素的文本是否为null然后删除重复项并对列表进行排序，然后遍历并将[*]添加到字符串中。希望你这一个。如果没有让我知道你的预期输出。

import requests
from bs4 import BeautifulSoup

url = "https://boards.4channel.org/sci/"

#send the HTTP request
response = requests.get(url)

if response.status_code == 200:

    #pull the content
    html_content = response.content

    #send the page to BeautifulSoup
    html_doc     = BeautifulSoup(html_content, "html.parser")

    #extract topic data
    topic_spider = html_doc.find_all("span",{"class":"subject"})
    data=[]
    for topic in topic_spider:
        if topic.text!='':
             data.append(topic.text)
    mylist = list(dict.fromkeys(data)) #Remove the duplicates here
    mylist.sort(reverse=False) #sort here
    for d in mylist:
        print ('[*]' + d)

Answer 2

使用:not(:empty) css伪类的集合理解。输出已按字母顺序排列，但您始终可以调用sort方法

import requests
from bs4 import BeautifulSoup as bs

url = "https://boards.4channel.org/sci/"
r = requests.get(url)
soup = bs(r.content, "lxml")
data = {"[*]" + item.text for item in soup.select('.subject:not(:empty)')}
#data.sort()