Selenium XPath：如何单击“加载更多”按钮

Question

我正在尝试在职业页面上抓取职位列表。我试图单击“加载更多”按钮，但似乎无法弄清楚。我想知道是否有人可以帮助我 - 我不断收到错误消息“元素在 x 点不可点击。其他元素将收到点击”。

这是链接：https://www.bain.com/careers/find-a-role/

def button_company_name(driver, page, outer_loop_break, all_links):
    """
    Handles pagination by clicking the "next page"/"load more" button.

    Parameters:
    driver (WebDriver): The Selenium WebDriver instance.
    page (int): The current page number. (optional)
    outer_loop_break (bool): Flag to indicate when to stop scraping.
    all_links (list): List of all links on the current page.

    Returns:
    tuple: Updated page number and outer_loop_break flag.
    """

    try:
        # Define the XPath
        load_more_button_xpath = '//*[@id="role-search-page-react"]/div/div/div[3]/a'

        # Wait for the load more button to be clickable
        load_more_button = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.XPATH, load_more_button_xpath))
        )

        # Trying to bring button into view before clicking on it
        driver.execute_script("arguments[0].scrollIntoView();", load_more_button)
        load_more_button.click()

        # Wait for the page to load
        time.sleep(5)


    except Exception as e:
        print(f"Error occurred while trying to click the 'next page' button: {e}")
        outer_loop_break = True
    return page + 1, outer_loop_break

Answer 1

您可以使用他们的分页 API 来获取更多结果，例如：

import requests

url = "https://www.bain.com/en/api/jobsearch/keyword/get?start={}&results=10&filters=&searchValue="

for page in range(0, 3): # <-- increase number of pages here
    data = requests.get(url.format(page)).json()
    for r in data["results"]:
        print(r["JobTitle"])

打印：

Expert Senior Manager, Machine Learning Engineer
Expert Manager, Machine Learning Engineer
Lead, Machine Learning Engineer
Senior Machine Learning Engineer
Expert Senior Manager, Machine Learning Engineer
Senior Machine Learning Engineer
Senior Software Engineer
Director, Global Financial Accounting
Director, Global Financial Accounting
Analyst /Sr. Analyst, APAC Finance & Staffing (FP&A)
Analyst, IT Support
Assistant, Office Services (m/w/d) in München
Assistant, Office Services (m/w/d) in Wien (Teilzeit)
Facilities Assistant (Contract/Temporary Full-Time)
Associate - Data Engineer
Associate - Data Science
Associate - ENR
Associate - Human Resources Business Partner
Associate - S&M Transformation CoE
Associate - Tools Specialist, Financial Services COE
Associate – Advanced Manufacturing & Industrial Services
Associate (B2C) - Pricing CoE
Associate (Data Engineering) – Pyxis, Data Business CoE
Associate (Supply Chain) - Performance Improvement CoE
Associate / Coordinator Payroll
Associate Consultant
Associate Consultant Intern
Associate Consultant Trainee
Business Presentation Designer (Associate), Global Business Services KL
IT Help Desk Support Associate