图像“src”文本废料和使用beautifulsoup的网页中的tablescrap

Question

我正在尝试网络刮这个page

它有两个问题：

1）我试图从表格中获取数据，这是包裹详细信息选项卡，但我没有得到任何结果。我的选择器路径是正确的，但没有输出。所需的输出如下：

2）虽然我得到图像“src”文本但我没有得到用于图像的所需文本。所需输出如下。

import requests
from bs4 import BeautifulSoup


result = []

response = requests.get("https://www.ikea.com/sa/en/catalog/products/00361049/")
assert response.ok
page = BeautifulSoup(response.text, "html.parser")

for record in page.find_all('.packages-specification-table tr:last-child'):
   for data in record.find_all('td'):
      print(data.text)

for record1 in page.find_all('.packages-specification-table tr:first-child'):
   for data1 in record1.find_all('th'): 
      print(data1)  
for des in page.find_all('img'):
   image= des.get('src')
   print(image)

必需的表输出：

商品编号00361049

套餐1

宽74厘米

身高48厘米

长106厘米

直径 -

重量30.00公斤

所需的图像输出src：

/PIA images/0618875_PE688687_S1.jpg

/PIA images/0325432_PE517964_S1.jpg

/PIA images/0690287_PE723209_S1.jpg

/PIA images/0513996_PE639275_S1.jpg

/PIA images/0325450_PE517970_S1.jpg

Answer 1

此页面使用JavaScript加载数据。

此代码获取图像的URL。

import requests

url = 'https://www.ikea.com/sa/en/iows/catalog/products/?catalog=departments&category=10687&type=json&dataset=small,allImages,prices&count=11&sort=relevance&sortorder=ascending&startIndex=0'

r = requests.get(url)

data = r.json()

for item in data['products']:
    print(item['item']['name'])
    for image in item['item']['images']['large']:
        print(image)

其他信息可以在JavaScript加载的其他文件中。

您可以在Chrome / Firefox的DevTools中找到它们 - 标签：网络，过滤器：XHR

---

编辑：

此页面使用JavaScript，但BS不运行JavaScript。

当我在Web浏览器中关闭JavaScript时，我会在您的代码中看到不同标签中的元素。

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.ikea.com/sa/en/catalog/products/00361049/")

soup = BeautifulSoup(r.text, "html.parser")

html = soup.select('div#productDimensionsContainer div#metric')[0].encode_contents().decode().strip()
data = list(filter(None, html.split('<br/>')))
print(data)

# ['Width: 82 cm', 'Depth: 96 cm', 'Height: 101 cm', 'Seat width: 49 cm', 'Seat depth: 54 cm', 'Seat height: 45 cm']

html = soup.select('div#custMaterials')[0].encode_contents().decode().strip()
data = list(filter(None, html.split('<br/>')))
print(data)

# ['Total composition: 100% polyester', 'Frame: Solid wood, Plywood, Particleboard, Polyurethane foam 25 kg/cu.m., Polyurethane foam 35 kg/cu.m., Polyester wadding', 'Seat cushion: Polyurethane foam 35 kg/cu.m., Polyester wadding', 'Leg: Solid beech, Clear lacquer']

---

编辑：

还有<script>与var jProductData=...and有来自表的信息。

import requests
from bs4 import BeautifulSoup
import json

r = requests.get("https://www.ikea.com/sa/en/catalog/products/00361049/")
soup = BeautifulSoup(r.text, "html.parser")

# var jProductData = {"product":{"items": ... }};

all_scripts = soup.select('script')

for script in all_scripts:
    script = script.encode_contents().decode().strip()
    if 'var jProductData' in script:
        for row in script.split('\n'):
            if 'var jProductData' in row:
                data = json.loads(row.strip()[19:-1])
                for item in data['product']['items']:
                    #print(item['pkgInfoArr'][0])
                    print('articleNumber:', item['pkgInfoArr'][0]['articleNumber'])
                    print('weightMet:', item['pkgInfoArr'][0]['pkgInfo'][0]['weightMet'])
                    print('widthMet:', item['pkgInfoArr'][0]['pkgInfo'][0]['widthMet'])
                    print('quantity:', item['pkgInfoArr'][0]['pkgInfo'][0]['quantity'])
                    print('consumerPackNo:', item['pkgInfoArr'][0]['pkgInfo'][0]['consumerPackNo'])
                    print('lengthMet:', item['pkgInfoArr'][0]['pkgInfo'][0]['lengthMet'])
                    print('heightMet:', item['pkgInfoArr'][0]['pkgInfo'][0]['heightMet'])
                    print('---')

结果：

articleNumber: 20343224
weightMet: 30.40 kg
widthMet: 74 cm
quantity: 1
consumerPackNo: 1
lengthMet: 106 cm
heightMet: 48 cm
---
articleNumber: 00361049
weightMet: 30.00 kg
widthMet: 74 cm
quantity: 1
consumerPackNo: 1
lengthMet: 106 cm
heightMet: 48 cm
---
articleNumber: 90361894
weightMet: 29.70 kg
widthMet: 74 cm
quantity: 1
consumerPackNo: 1
lengthMet: 106 cm
heightMet: 47 cm
---
articleNumber: 80359844
weightMet: 30.00 kg
widthMet: 74 cm
quantity: 1
consumerPackNo: 1
lengthMet: 106 cm
heightMet: 53 cm
---
articleNumber: 40359855
weightMet: 31.00 kg
widthMet: 75 cm
quantity: 1
consumerPackNo: 1
lengthMet: 106 cm
heightMet: 49 cm
---
articleNumber: 10413953
weightMet: 29.90 kg
widthMet: 74 cm
quantity: 1
consumerPackNo: 1
lengthMet: 106 cm
heightMet: 47 cm
---
articleNumber: 40433247
weightMet: 29.90 kg
widthMet: 74 cm
quantity: 1
consumerPackNo: 1
lengthMet: 106 cm
heightMet: 47 cm
---

可能还有其他信息，如图片的网址，但我没有挖var jProductData找到它。