如何从新标签页打开爬取数据

Question

我正在尝试通过scrapy-selenium抓取此网页https://www.goo-net.com/php/search/summary.php的产品详细信息。

因为我想爬取每个产品的详细信息，所以我从页面中爬取了产品的所有url。然后我使用回调方法将其解析为另一个def以爬取该url的所有信息。

但是我尝试了很多解决方案，但我的输出总是没有显示任何内容

这是我的代码

import scrapy
import selenium
from scrapy_selenium import SeleniumRequest
from selenium.webdriver.common.keys import Keys


class Goonet1Spider(scrapy.Spider):
    name = 'goonet1'

    def start_requests(self):
        yield SeleniumRequest (
            url='https://www.goo-net.com/php/search/summary.php',
            wait_time=4,
            callback=self.parse
        )

    def parse(self, response):
        links = response.xpath("//*[@class='heading_inner']/h3/a")
        url_detail = []
        for link in links:
            url = response.urljoin(link.xpath(".//@href").get())
            url_detail.append(url)
        for i in url_detail:
            yield SeleniumRequest (
                url=i,
                wait_time=4,
                callback=self.parse_item
            )

    def parse_item(self,response):
        base_price = response.xpath("//table[@class='mainData']/tbody/tr[2]/td[1]/span/text()").get()
        yield {
            'base_price': base_price
        }

这是我的设置.py

DOWNLOADER_MIDDLEWARES = {
    'scrapy_selenium.SeleniumMiddleware': 800
}

#SELENIUM
from shutil import which
SELENIUM_DRIVER_NAME = 'chrome'
SELENIUM_DRIVER_EXECUTABLE_PATH = which('chromedriver')
SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox

请帮助我

Answer 1

将 BaseURL 添加到 url_detail 以完成您的链接：

def parse(self, response):
        links = response.xpath("//*[@class='heading_inner']/h3/a")
        url_detail = []
        for link in links:
            url = response.urljoin(link.xpath(".//@href").get())
            url_detail.append(url)
        for i in url_detail:
            link = "https://www.goo-net.com" + i
            yield SeleniumRequest (
                url=link,
                wait_time=4,
                callback=self.parse_item
            )