Scrapy 仅输出开括号

问题描述 投票:0回答:1

我正在尝试抓取数学/科学/经济学页面下所有可汗学院页面的标题和 URL。然而,目前它只输出一个开括号,在此之前它只会抓取起始 URL。

from openbar_index.items import OpenBarIndexItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor


class OpenBarSpider(CrawlSpider):
    """
    scrapes website URLs from educational websites and commits urls/webpage names/text to a document
    """

    name = 'openbar'
    allowed_domains = 'khanacademy.org'
    start_urls = [

        "https://www.khanacademy.org"

    ]

     rules = [

            Rule(SgmlLinkExtractor(allow = ['/math/']), callback='parse_item', follow = True),
             Rule(SgmlLinkExtractor(allow = ['/science/']), callback='parse_item', follow=True),
             Rule(SgmlLinkExtractor(allow = ['/economics-finance-domain/']), callback='parse_item', follow=True)
    ]

    def parse_item(self, response):

         item = OpenBarIndexItem()
         url = response.url
         item['url'] = url
         item['title'] = response.xpath('/html/head/title/text()').extract()
         yield item

有谁知道为什么会发生这种情况或如何解决它的提示?

python web-scraping url scrapy web-crawler
1个回答
0
投票

问题在于对

allowed_domains
的分配。根据
documentation
,这不能是
string
,而是 list。使用该字符串,潜在结果会被 scrapy 过滤为异地请求,因为没有有效的域。

因此像下一行一样添加方括号应该可以修复它

    allowed_domains = ['khanacademy.org']
© www.soinside.com 2019 - 2024. All rights reserved.