如何使用Web抓取来废弃视图页面源上的Inspect元素中的数据

Question

我尝试在特定网站上进行webscraping。但是我无法获得标签。我在Inspect元素和viewpage源中看到了标签。如何获取标签可以请你给我任何建议。

Web scrap Y.朋友

from bs4 import BeautifulSoup
from urllib.request import urlopen
import html5lib
import urllib
import pandas as pd
import xlsxwriter
from docx import Document
from docx.shared import Inches

document = Document()

url = "https://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Order=BESTMATCH&Description="
Remaining_url="&ignorear=0&N=-1&isNodeId=1"
product_name = 'Seagate 80GB 7200 RPM SATA 3.0Gb/s Internal Hard Drive (IMSourcing) Bare Drive'
p = document.add_paragraph("Product_name " +":"+"  "+product_name)

search_words = {'text': product_name}
search_url = urllib.parse.urlencode(search_words).split("=")[1]
product_url = url + search_url + Remaining_url
content = urlopen(product_url).read()
soup = BeautifulSoup(content, "html5lib")
print(soup.find_all("div", class_="list-wrap"))  

我运行程序它抛出空列表。如何解决它任何人都可以给出任何解决方案。

Answer 1

是的，这是正确的，结果列表是空的。

        <div class="result-message">
         <p class="result-message-title">
          <span class="result-message-error">
           We have found 0 items that match "Seagate 80GB 7200 RPM SATA 3.0Gb/s Internal Hard Drive (IMSourcing) Bare Drive".
          </span>
         </p>
        </div>

您可以使用sleep()在GET请求之间暂停：

time.sleep(1.5)