我正在尝试抓取数学/科学/经济学页面下所有可汗学院页面的标题和 URL。然而,目前它只输出一个开括号,在此之前它只会抓取起始 URL。
from openbar_index.items import OpenBarIndexItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
class OpenBarSpider(CrawlSpider):
"""
scrapes website URLs from educational websites and commits urls/webpage names/text to a document
"""
name = 'openbar'
allowed_domains = 'khanacademy.org'
start_urls = [
"https://www.khanacademy.org"
]
rules = [
Rule(SgmlLinkExtractor(allow = ['/math/']), callback='parse_item', follow = True),
Rule(SgmlLinkExtractor(allow = ['/science/']), callback='parse_item', follow=True),
Rule(SgmlLinkExtractor(allow = ['/economics-finance-domain/']), callback='parse_item', follow=True)
]
def parse_item(self, response):
item = OpenBarIndexItem()
url = response.url
item['url'] = url
item['title'] = response.xpath('/html/head/title/text()').extract()
yield item
有谁知道为什么会发生这种情况或如何解决它的提示?
问题在于对
allowed_domains
的分配。根据 documentation,这不能是
string
,而是 list
。使用该字符串,潜在结果会被 scrapy 过滤为异地请求,因为没有有效的域。
因此像下一行一样添加方括号应该可以修复它
allowed_domains = ['khanacademy.org']