如何使用scrapy刮取XML。
我的XML看起来像这样:
<rss xmlns:media="http://search.yahoo.com/mrss/" version="2.0">
<channel>
<generator>NFE/5.0</generator>
<title>"python" - Google News</title>
<link>
https://news.google.com/search?q=python&hl=en-IN&gl=IN&ceid=IN:en
</link>
<language>en-IN</language>
<webMaster>[email protected]</webMaster>
<copyright>2019 Google Inc.</copyright>
<lastBuildDate>Thu, 07 Mar 2019 16:48:55 GMT</lastBuildDate>
<description>Google News</description>
<item>
<title>
Brown snake attacks python eating a rat - NEWS.com.au
</title>
</channel>
</rss>
我的代码看起来像这样:
from scrapy.spiders import XMLFeedSpider
from scrapy.http import HtmlResponse
from scrapy.selector import Selector
response = HtmlResponse(url='https://news.google.com/rss/search?q=python&hl=en-IN&gl=IN&ceid=IN:en')
xxs = Selector(response)
obj = xxs.xpath('//title/text()').extract()
我想在标题标签中获取文本。但在这里我得到一个空列表。请帮帮我。这一点很重要。非常感谢
你被robots.txt禁止了。您需要在settings.py
中更改此行为并更改ROBOTSTXT_OBEY=True
to ROBOTSTXT_OBEY=False
。