python 网站-使用文本文件中的变量进行抓取

Question

有人知道我如何抓取网站从 .txt 中读取 URL 列表 IE，然后使用 .txt 中的名称将每个 url 结果写入 .txt 中。因此，代码将读取 URL 和名称文件，并使用 .txt 文件中的相应行写入正文。我发现的最接近的是下面的代码，但是它将所有内容保存到一个 .txt 文件中，该文件是一个固定名称而不是变量；并从列表中读取 URL。我猜循环将是最好的方法，但是我还没有看到代码或对此类任务有太多帮助。

import requests
from bs4 import BeautifulSoup
from collections import Counter
urls = ["http://en.wikipedia.org/wiki/Wolfgang_Amadeus_Mozart","http://en.wikipedia.org/wiki/Golf"]

with open('thisisanew.txt', 'w', encoding='utf-8') as outfile:
     for url in urls:
     website = requests.get(url)
     soup = BeautifulSoup(website.content)
     text = [''.join(s.findAll(text=True))for s in soup.findAll('p')]
     for item in text:
            print(item ,file=outfile,)

提前感谢您的帮助！

Answer 1

import requests
from bs4 import BeautifulSoup
from pathlib import Path
urls = ["http://en.wikipedia.org/wiki/Wolfgang_Amadeus_Mozart","http://en.wikipedia.org/wiki/Golf"]
for url in urls:
    # make requests
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'lxml')
    text = '\n'.join(s.get_text() for s in soup.findAll('p'))  # add '\n' betweent each p
    # write to file 
    filename = Path(url).name   # get the last part of url
    out_file = Path(filename).with_suffix('.txt')  # add.txt to filename
    with out_file.open(mode='w') as f:
        f.writelines(text)

输出：

高尔夫.txt：

Golf is a club and ball sport in which players use various clubs to hit balls into a series of holes on a course in as few strokes as possible.
Golf, unlike most ball games, does not require a standardized playing area. The game is played on a course with an arranged progression of either nine or 18 holes. Each hole on the course must contain a tee box to start from, and a putting green containing the actual hole or cup (4.25 inches in width). There are other standard forms of terrain in between, such as the fairway, rough (long grass), sand traps, and hazards (water, rocks, fescue) but each hole on a course is unique in its specific layout and arrangement.
Golf is played for the lowest number of strokes by an individual, known as stroke play, or the lowest score on the most individual holes in a complete round by an individual or team, known as match play. Stroke play is the most commonly seen format at all levels.

Wolfgang_Amadeus_Mozart.txt

Wolfgang Amadeus Mozart (/ˈwʊlfɡæŋ æməˈdeɪəs ˈmoʊtsɑːrt/; MOHT-sart;[1] German: [ˈvɔlfɡaŋ amaˈdeːʊs ˈmoːtsaʁt]; 27 January 1756 – 5 December 1791), baptised as Johannes Chrysostomus Wolfgangus Theophilus Mozart,[2] was a prolific and influential composer of the Classical era.
Born in Salzburg, he showed prodigious ability from his earliest childhood. Already competent on keyboard and violin, he composed from the age of five and performed before European royalty. At 17, Mozart was engaged as a musician at the Salzburg court, but grew restless and traveled in search of a better position. While visiting Vienna in 1781, he was dismissed from his Salzburg position. He chose to stay in the capital, where he achieved fame but little financial security. During his final years in Vienna, he composed many of his best-known symphonies, concertos, and operas, and portions of the Requiem, which was largely unfinished at the time of his death. The circumstances of his early death have been much mythologized. He was survived by his wife Constanze and two sons.