python网页抓取请求错误(model security)

问题描述 投票:0回答:2

我是新来的,我想抓取一个网页的源代码作为教程,我得到了beautifulsoup安装,请求安装。一开始我想抓取源码,我做的这个抓取工作是从"https:/pythonhow.comexample.html。".我没有做任何违法的事情,我想这个网站也是为了这个目的而建立的.这是我的代码。

import requests
from bs4 import BeautifulSoup

r=requests.get("http://pythonhow.com/example.html")
c=r.content
c

我得到了mod安全错误。

b'<head><title>Not Acceptable!</title></head><body><h1>Not Acceptable!</h1><p>An appropriate representation of the requested resource could not be found on this server. This error was generated by Mod_Security.</p></body></html>'

谢谢所有和我打交道的人,尊敬的各位。

python python-requests mod-security
2个回答
0
投票

你可以通过为请求提供一个用户代理来轻松解决这个问题。通过这样做,网站会认为有人在使用浏览器访问网站。

下面是你要使用的代码。

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:55.0) Gecko/20100101 Firefox/55.0',
}

r = requests.get("http://pythonhow.com/example.html", headers=headers)
c = r.content

print(c)

这将给你带来预期的输出

b'<!DOCTYPE html>\n<html>\n<head>\n<style>\ndiv.cities {\n    background-color:black;\n    color:white;\n    margin:20px;\n    padding:20px;\n} \n</style>\n</head>\n<body>\n<h1 align="center"> Here are three big cities </h1>\n<div class="cities">\n<h2>London</h2>\n<p>London is the capital of England and it\'s been a British settlement since 2000 years ago. </p>\n</div>\n<div class="cities">\n<h2>Paris</h2>\n<p>Paris is the capital city of France. It was declared capital since 508.</p>\n</div>\n<div class="cities">\n<h2>Tokyo</h2>\n<p>Tokyo is the capital of Japan and one of the most populated cities in the world.</p>\n</div>\n</body>\n</html>'

0
投票

而不是请求,尝试使用 url module

from urllib.request import urlopen

page = urlopen("http://pythonhow.com/example.html")
page.read()
© www.soinside.com 2019 - 2024. All rights reserved.