有没有办法使用BeautifulSoup将数据从列表正确转换为CSV文件？

Question

我正在尝试为网站创建一个webscraper。问题是收集的数据存储在列表中后，我无法正确地将其写入csv文件。我已经被这个问题困扰了很长时间，希望有人知道如何解决这个问题！

从网页获取数据的循环：

import csv
from htmlrequest import simple_get
from htmlrequest import BeautifulSoup

# Define variables
listData = ['Companies', 'Locations', 'Descriptions']

plus = 15
max = 30
count = 0

# while loop to repeat process till max is reached
while (count <= max):
        start = 'https://www.companiesintheuk.co.uk/find?q=Activities+of+sport+clubs&start=' + str(count) + '&s=h&t=SicCodeSearch&location=&sicCode=93120'
        raw_html = simple_get(start)
        soup = BeautifulSoup(raw_html, 'html.parser')
        for i, div in enumerate(soup.find_all('div', class_="search_result_title")):
            listData[0] = listData[0].strip() + div.text 

        for i, div2 in enumerate(soup.find_all('div', class_="searchAddress")):
             listData[1] = listData[1].strip() + div2.text  

# This is extra information
#        for i, div3 in enumerate(soup.find_all('div', class_="searchSicCode")):
#            listData[2] = listData[2].strip() + div3.text 

        count = count + plus

输出示例如果打印：

Companies
(AMG) AGILITY MANAGEMENT GROUP LTD
(KLA) LIONS/LIONESS FOOTBALL TEAMS WORLD CUP LTD
(Dissolved)
1 SPORT ORGANISATION LIMITED
100UK LTD
1066 GYMNASTICS
1066 SPECIALS
10COACHING LIMITED
147 LOUNGE LTD
147 SNOOKER AND POOL CLUB (LEICESTER) LIMITED

Locations
ENGLAND, BH8 9PS
LONDON, EC2M 2PL
ENGLAND, LS7 3JB
ENGLAND, LE2 8FN
UNITED KINGDOM, N18 2QX
AVON, BS5 0JH
UNITED KINGDOM, WC2H 9JQ
UNITED KINGDOM, SE18 5SZ
UNITED KINGDOM, EC1V 2NX

我试图通过使用此代码将其变为CSV文件，但我无法弄清楚如何正确格式化我的输出！欢迎任何建议。

# writing to csv
with open('test.csv', 'w') as csvfile:
            write = csv.writer(csvfile, delimiter=',')
            write.writerow(['Name','Location'])
            write.writerow([listData[0],listData[1]])

print("Writing has been done!")

我希望代码能够在csv文件中正确格式化，以便能够导入数据库中的两行。

This is the output when I write the data on 'test.csv'

which will result into this when opened up

The expected outcome would be something like this!

Answer 1

我不确定它是如何格式不正确的，但也许你只需要用with open('test.csv', 'w')替换with open('test.csv', 'w+', newline='')

我已经结合你的代码（为htmlrequests和requests模块取出bs4，也没有使用listData，而是创建我自己的列表。我已经离开了你的列表，但他们什么都不做）：

import csv
import bs4
import requests

# Define variables
listData = ['Companies', 'Locations', 'Descriptions']
company_list = []
locations_list = []
plus = 15
max = 30
count = 0

# while loop to repeat process till max is reached

while count <= max:
    start = 'https://www.companiesintheuk.co.uk/find?q=Activities+of+sport+clubs&start={}&s=h&t=SicCodeSearch&location=&sicCode=93120'.format(count)
    res = requests.get(start)
    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    for i, div in enumerate(soup.find_all('div', class_="search_result_title")):
        listData[0] = listData[0].strip() + div.text
        company_list.append(div.text.strip())
    for i, div2 in enumerate(soup.find_all('div', class_="searchAddress")):
        listData[1] = listData[1].strip() + div2.text
        locations_list.append(div2.text.strip())
# This is extra information
#        for i, div3 in enumerate(soup.find_all('div', class_="searchSicCode")):
#            listData[2] = listData[2].strip() + div3.text
    count = count + plus

if len(company_list) == len(locations_list):
    with open('test.csv', 'w+', newline='') as csvfile:
        writer = csv.writer(csvfile, delimiter=',')
        writer.writerow(['Name', 'Location'])
        for i in range(len(company_list)):
            writer.writerow([company_list[i], locations_list[i]])

这会生成一个csv文件，如：

Name,Location
(AMG) AGILITY MANAGEMENT GROUP LTD,"UNITED KINGDOM, M6 6DE"
"(KLA) LIONS/LIONESS FOOTBALL TEAMS WORLD CUP LTD
(Dissolved)","ENGLAND, BD1 2PX"
0161 STUDIOS LTD,"UNITED KINGDOM, HD6 3AX"
1 CLICK SPORTS MANAGEMENT LIMITED,"ENGLAND, E10 5PW"
1 SPORT ORGANISATION LIMITED,"UNITED KINGDOM, CR2 6NF"
100UK LTD,"UNITED KINGDOM, BN14 9EJ"
1066 GYMNASTICS,"EAST SUSSEX, BN21 4PT"
1066 SPECIALS,"EAST SUSSEX, TN40 1HE"
10COACHING LIMITED,"UNITED KINGDOM, SW6 6LR"
10IS ACADEMY LIMITED,"ENGLAND, PE15 9PS"
"10TH MAN LIMITED
(Dissolved)","GLASGOW, G3 6AN"
12 GAUGE EAST MANCHESTER COMMUNITY MMA LTD,"ENGLAND, OL9 8DQ"
121 MAKING WAVES LIMITED,"TYNE AND WEAR, NE30 1AR"
121 WAVES LTD,"TYNE AND WEAR, NE30 1AR"
1-2-KICK LTD,"ENGLAND, BH8 9PS"
"147 HAVANA LIMITED
(Liquidation)","LONDON, EC2M 2PL"
147 LOUNGE LTD,"ENGLAND, LS7 3JB"
147 SNOOKER AND POOL CLUB (LEICESTER) LIMITED,"ENGLAND, LE2 8FN"
1ACTIVE LTD,"UNITED KINGDOM, N18 2QX"
1ON1 KING LTD,"AVON, BS5 0JH"
1PUTT LTD,"UNITED KINGDOM, WC2H 9JQ"
1ST SPORTS LTD,"UNITED KINGDOM, SE18 5SZ"
2 BRO PRO EVENTS LTD,"UNITED KINGDOM, EC1V 2NX"
2 SPLASH SWIM SCHOOL LTD,"ENGLAND, B36 0EY"
2 STEPPERS C.I.C.,"SURREY, CR0 6BX"
2017 MOTO LIMITED,"UNITED KINGDOM, ME2 4NW"
2020 ARCHERY LTD,"LONDON, SE16 6SS"
21 LEISURE LIMITED,"LONDON, EC4M 7WS"
261 FEARLESS CLUB UNITED KINGDOM CIC,"LANCASHIRE, LA2 8RF"
2AIM4 LIMITED,"HERTFORDSHIRE, SG2 0JD"
2POINT4 FM LTD,"LONDON, NW10 8LW"
3 LIONS SCHOOL OF SPORT LTD,"BRISTOL, BS20 8BU"
3 PT LTD,"ANTRIM, BT40 2FB"
3 PUTT LIFE LTD,"UNITED KINGDOM, LU3 2DP"
3 THIRTY SEVEN LTD,"KENT, DA9 9RS"
3:30 SOCCER SCHOOL LTD,"UNITED KINGDOM, EH6 7JB"
30 MINUTE WORKOUT (LLANISHEN) LTD,"PONTYCLUN, CF72 9UA"
321 RELAX LTD,"MID GLAMORGAN, CF83 3HL"
360 MOTOR RACING CLUB LTD,"HALSTEAD, CO9 2ET"
3LIONSATHLETICS LIMITED,"ENGLAND, S3 8DB"
3S SWIM ROMFORD LTD,"UNITED KINGDOM, DA9 9DR"
3XL EVENT MANAGEMENT LIMITED,"KENT, BR3 4NW"
3XL MOTORSPORT MANAGEMENT LIMITED,"KENT, BR3 4NW"
4 CORNER FOOTBALL LTD,"BROMLEY, BR1 5DD"
4 PRO LTD,"UNITED KINGDOM, FY5 5HT"

这对我来说似乎很好，但你的帖子很清楚你是如何预期它的格式化所以我真的不知道