由NewConnectionError引起的url超出了最大重试次数

Question

我正在尝试抓取一个产品网站，其中包含一个类别中超过2000种产品的详细信息，例如名称等。然而，随着时间的推移，随着上述错误随机链接不同的链接。这里有追溯：

Traceback (most recent call last):
File "crawler1.py", line 103, in <module>
crawler(25)
File "crawler1.py", line 35, in crawler get_single_data(href)
File "crawler1.py", line 57, in get_single_data source_code = requests.get(item_url, timeout=335)
File "/Library/Python/2.7/site-packages/requests/api.py", line 71, in get return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 475, in request resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 585, in send r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 467, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.therealreal.com', port=443): Max retries exceeded with url: /products/women/handbags/handle-bags/chanel-lax-handle-bag-4 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10d8de190>: Failed to establish a new connection: [Errno 60] Operation timed out',))

通过捕获所有错误，我在每个可以想到使用睡眠的地方都添加了延迟。有没有办法避免这种情况，我可以一次性提取所有2000个产品数据。或者任何人都可以建议解决问题。请帮忙。

这是代码：

try:
    source_code = requests.get(item_url, timeout=335)
    sleep(.3)
except requests.exceptions.ReadTimeout:
    print("1")
    sleep(30) 
    source_code = requests.get(item_url, timeout=335)
except requests.exceptions.Timeout:
    print("2")
    sleep(30)
    source_code = requests.get(item_url, timeout=335)
except ConnectionError:
    print("3")
    sleep(30)
    source_code = requests.get(item_url, timeout=335)
except socket.error:
    sleep(30)
    source_code = requests.get(item_url, timeout=335)
plain_text  = source_code.text
temp = BeautifulSoup(plain_text)

p.s你可以忽略我甚至没有尝试过很多值的超时。没有帮助。出了什么问题？

Answer 1

尝试捕获requests.exceptions.ConnectionError