如何处理python硒中的延迟加载？

Question

在程序的此部分中，我试图获取网页上图像的所有链接。但是，这些图片是延迟加载的。即使不是这样：向下滚动时出现图片，但这些图片已经在这里。每页上有30个产品。向下滚动页面也不会起到神奇作用。我该如何处理？

from selenium import webdriver
import os 
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.common.action_chains import ActionChains


url= 'https://www.bookdepository.com/category/2/Art-Photography/browse/viewmode/all'

driver = webdriver.Chrome('blabla')

driver.get(url)


a = 1
while a != 100:
    try:                                                                                                                                                                   
        link_picture = WebDriverWait(driver, 2).until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[4]/div[5]/div[2]/div[3]/div/div/div/div/div["+ str(a) +"]/div[1]/a/img"))).get_attribute('src')                                                                                                    
        print(link_picture)
    except:
        print("\nno products left")
        #e = a - 1
        #print(a)
        #print(e)
        break
    a = a + 1

Answer 1

实际上页面是通过JavaScript加载的，一旦页面加载，页面就会在internally本身上呈现为host，因此Lazy-Load是通过selenium处理Implicit, Explicit, & Fluent的非常不好的方法。在Selenium WebDriver中，这将花费大量时间。

我们只能以一种复杂的方式使用requests和bs4。我们将收集img IDs#，然后将它们匹配为通常的站点上的那个。

注意：我已经手动检查过，我可以为您确认该站点的token的CloudFront ID是static，即d1w7fb2mkkr3kw。

from bs4 import BeautifulSoup
import requests

r = requests.get(
    "https://www.bookdepository.com/category/2/Art-Photography/browse/viewmode/all")

soup = BeautifulSoup(r.text, 'html.parser')

url = "https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/"

for item in soup.findAll("meta", itemprop="isbn"):
    item = item.get("content")
    print(
        f"{url}{item[:4]}/{item[4:8]}/{item}.jpg")

输出：

https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4087/9781408708989.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/5098/9781509853311.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/7522/9780752265629.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/1410/9780141014081.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/2500/9781250038821.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/1410/9780141035796.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8499/9781849941679.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/0995/9780099539551.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4087/9781408711705.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9788/8837/9788883701153.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/7475/9780747568766.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8609/9781860969423.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4711/9781471157790.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/5098/9781509829477.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/7515/9780751535662.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/5005/9780500513606.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4088/9781408890769.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/8771/9780877180128.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9788/8837/9788883705601.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4722/9781472200341.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8478/9781847807717.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8609/9781860969430.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/3409/9780340936177.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/4862/9780486254500.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/7160/9780716022237.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/0994/9780099457046.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4521/9781452106557.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/2411/9780241184837.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/7423/9781742372389.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8499/9781849942850.jpg

更新：

import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://www.bookdepository.com/category/2/Art-Photography/browse/viewmode/all")
soup = BeautifulSoup(r.text, 'html.parser')

for item in soup.findAll("img", class_="lazy"):
    print(item.get("data-lazy"))