python中的Web Scraping调用返回空值

Question

我试图从python 2.0中的MCX网站https://www.mcxindia.com/market-data/market-watch获得不同商品的最后交易价格（LTP）。以下是我正在使用的代码。

import requests
from bs4 import BeautifulSoup
url = 'https://www.mcxindia.com/market-data/market-watch'
page = requests.get(url)
soup = BeautifulSoup(page.text, 'html.parser')
soup.findAll('div',attrs={'class':'ltp green ltpcenter'})

但是当我运行代码时，我得到空值。我怀疑网站查询其他一些网络服务器的值，因为当我查看网页的来源时，我没有看到最后的交易价格。任何人都可以帮我如何将价格数据导入python？

Answer 1

下面的代码获取该页面上显示的所有市场数据，从json响应中提取您想要的任何内容。

import requests
url = "https://www.mcxindia.com/backpage.aspx/GetMarketWatch"

headers = {
    "Host": "www.mcxindia.com",
    "Origin": "https://www.mcxindia.com",
    "X-Requested-With": "XMLHttpRequest",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36",
    "Content-Type": "application/json",
"Referer": "https://www.mcxindia.com/market-data/market-watch",
"Accept": "application/json, text/javascript, */*; q=0.01",  
}


resp  = requests.post(url, headers = headers)
market_data = resp.json()

Answer 2

你必须处理JS，你可以使用selenium来加载JS，参见下面的代码。

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
driver.get("https://www.mcxindia.com/market-data/market-watch")
wait(driver, 10).until(EC.visibility_of_element_located(
        (By.XPATH, '//*[@class="symbol chnge-perc right5"]')))
source = driver.page_source
soup = BeautifulSoup(source, 'html.parser')
soup.findAll('div',attrs={'class':'ltp green ltpcenter'})
print soup