无法刮擦这个网站。如何从这个网站抓取数据？

Question

我不能从这个网站上抓取数据。

我尝试过其他网站，但其他网站也没问题......

from bs4 import BeautifulSoup
from urllib.request import urlopen

response = urlopen("https://www.daraz.com.np/catalog/?spm=a2a0e.searchlistcategory.search.2.3eac4b8amQJ0zd&q=samsung%20m20&_keyori=ss&from=suggest_normal&sugg=samsung%20m20_1_1")

html = response.read()

parsed_html = BeautifulSoup(html, "html.parser")

containers = parsed_html.find_all("div", {"class" : "c2prKC"})

print(len(containers))

Answer 1

加载后看起来像JS渲染到页面。你可以使用Selenium来渲染页面和漂亮的汤来获取元素。

from bs4 import BeautifulSoup
from selenium import webdriver
import time
driver = webdriver.Chrome()
driver.get("https://www.daraz.com.np/catalog/?spm=a2a0e.searchlistcategory.search.2.3eac4b8amQJ0zd&q=samsung%20m20&_keyori=ss&from=suggest_normal&sugg=samsung%20m20_1_1")
time.sleep(5)

html = driver.page_source

parsed_html = BeautifulSoup(html, "html.parser")

containers = parsed_html.find_all("div", {"class" : "c2prKC"})

print(len(containers))

Answer 2

您想要的信息是脚本标记。您可以使用正则表达式或循环脚本标记来获取正确的字符串作为json进行解析（使用小修改）

import requests
import json
from bs4 import BeautifulSoup as bs
import pandas as pd

headers = {
    'User-Agent' : 'Mozilla/5.0'
}
res = requests.get('https://www.daraz.com.np/catalog/?spm=a2a0e.searchlistcategory.search.2.3eac4b8amQJ0zd&q=samsung%20m20&_keyori=ss&from=suggest_normal&sugg=samsung%20m20_1_1', headers = headers)
soup = bs(res.content, 'lxml')
for script in soup.select('script'):
    if 'window.pageData=' in script.text:
        script = script.text.replace('window.pageData=','')
        break
items = json.loads(script)['mods']['listItems']
results = []

for item in items:
    #print(item)
    #extract other info you want
    row = [item['name'], item['priceShow'], item['productUrl'], item['ratingScore']]
    results.append(row)

df = pd.DataFrame(results, columns = ['Name', 'Price', 'ProductUrl', 'Rating'])

print(df.head())

---

正则表达式版本：

import requests
import json
from bs4 import BeautifulSoup as bs
import pandas as pd

headers = {
    'User-Agent' : 'Mozilla/5.0'
}
res = requests.get('https://www.daraz.com.np/catalog/?spm=a2a0e.searchlistcategory.search.2.3eac4b8amQJ0zd&q=samsung%20m20&_keyori=ss&from=suggest_normal&sugg=samsung%20m20_1_1', headers = headers)
soup = bs(res.content, 'lxml')
r = re.compile(r'window.pageData=(.*)')
data = soup.find('script', text=r).text
script = r.findall(data)[0]
items = json.loads(script)['mods']['listItems']
results = []

for item in items:
    row = [item['name'], item['priceShow'], item['productUrl'], item['ratingScore']]
    results.append(row)

df = pd.DataFrame(results, columns = ['Name', 'Price', 'ProductUrl', 'Rating'])

print(df.head())