使用Beautifulsoup抓取网站时编码错误

Question

我试图从this网站上删除文本。它返回如下文本：

डा. भà¥à¤·à¤¬à¤¹à¤¾à¤¦à¥à¤° थापालाठपà¥à¤¤à¥à¤°à¥à¤¶à¥à¤, à¤à¤®à¥à¤°à¤¿à¤à¤¾à¤®à¤¾ तà¥à¤à¤¶à¥à¤°à¥à¤à¥ निधन

代替：

भारतीय विदेश सचिव गोखले आज नेपाल आउँदै.

现行代码：

headers = {
        'Connection': 'close',
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    }

def get_url_soup(url):
        url_request = requests.get(url, headers=headers, allow_redirects=True)
        soup = BeautifulSoup(url_request.text, 'lxml')
        return soup

soup = get_url_soup('https://www.onlinekhabar.com/2019/03/753522')
title_card = soup.find('div', {'class': 'nws__title--card'})

Answer 1

使用EncodingDetector：

from bs4.dammit import EncodingDetector

headers = {
        'Connection': 'close',
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    }

def get_url_soup(url):
        url_request = requests.get(url, headers=headers, allow_redirects=True)
        http_encoding = url_request.encoding if 'charset' in url_request.headers.get('content-type', '').lower() else None
        html_encoding = EncodingDetector.find_declared_encoding(url_request.content, is_html=True)
        encoding = html_encoding or http_encoding
        soup = BeautifulSoup(url_request.content, 'lxml', from_encoding=encoding)
        return soup

soup = get_url_soup('https://www.onlinekhabar.com/2019/03/753522')
title_card = soup.find('div', {'class': 'nws__title--card'})

print(title_card.text)

OUTPUT：

होमपेज / 
समाचार / 
राष्ट्रिय समाचार

भारतीय विदेश सचिव गोखले आज नेपाल आउँदै
प्रधानमन्त्रीलगायत शीर्ष नेतासँग भेट्ने 
.
.
.