我有一个带有以下示例行的文件:
(22642441022L, u'<a href="http://example.com">Click</a>', u'fox, dog, cat are examples http://example.com')
(1153634043, u'<a href="http://example.com">Click</a>', u"I learned so much from my mistakes, I think I'm gonna make some more")
我正在尝试使用此代码将其解析为对象列表:
import csv
file_path = 'Data/example.txt'
data = []
with open(file_path, 'r') as f:
reader = csv.reader(f, skipinitialspace=True)
for row in reader:
data.append({'id' : row[0], 'source' : row[1], 'content' : row[2]})
正如所料,由于内容列中的“,”,内容会被截断。是否有任何包可以帮我解析这个开箱即用?
查看您的数据,有人使用python2将str
版本的列表原样转储到文件中。
有一件事是肯定的 - 你不能使用CSV阅读器来获取这些数据。你甚至不能使用JSON解析器(这将是下一个最好的东西)。
你能做什么,是使用ast.literal_eval
。使用python2,这开箱即用。
import ast
data = []
with open('file.txt') as f:
for line in f:
try:
data.append(ast.literal_eval(line))
except (SyntaxError, ValueError):
pass
data
应该看起来像这样 -
[(22642441022L,
'<a href="http://example.com">Click</a>',
'fox, dog, cat are examples http://example.com'),
(1153634043,
'<a href="http://example.com">Click</a>',
"I learned so much from my mistakes, I think I'm gonna make some more")]
然后你可以按原样将data
传递给DataFrame
-
df = pd.DataFrame(data, columns=['A', 'B', 'C'])
df
A B \
0 22642441022 <a href="http://example.com">Click</a>
1 1153634043 <a href="http://example.com">Click</a>
C
0 fox, dog, cat are examples http://example.com
1 I learned so much from my mistakes, I think I'...
如果你想让它与python3一起工作,你需要摆脱长后缀L
和unicode前缀u
。您可以使用re.sub
模块中的re
执行此操作。
import re
for line in f:
try:
i = re.sub('(\d+)L', r'\1', line) # remove L suffix
j = re.sub('(?<=,\s)u(?=\')', '', i) # remove u prefix
data.append(ast.literal_eval(j))
except (SyntaxError, ValueError):
pass
注意添加的re.sub('(\d+)L', r'\1', line)
,它删除了一串数字末尾的L
后缀。
所以它看起来像是这样生成的文件(Python str()
或print
的纯转储):
data_list = [
(22642441022L, u'<a href="http://example.com">Click</a>', u'fox, dog, cat are examples http://example.com'),
(1153634043, u'<a href="http://example.com">Click</a>', u"I learned so much from my mistakes, I think I'm gonna make some more")
] # List of tuples
with open('./stack_084.txt', 'w') as f:
f.write('\n'.join([str(data) for data in data_list]))
我们会想到正则表达式(假设第二个“列”的值)始终以<a
开头并以a>
结尾:
import pprint
import re
line_re = re.compile(
r'\('
r'(?P<num>\d+)L{0,1}.'
r'+?'
r'[\'\"](?P<source>\<a.+?a\>)[\"\']'
r'.+?'
r'[\'\"](?P<content>.+?)[\"\']'
r'\)'
)
data = []
with open('./stack_084.txt', 'r') as f:
for line in f:
match = line_re.match(line)
if match:
data.append({
'id': int(match.groupdict()['num']),
'source': match.groupdict()['source'],
'content': match.groupdict()['content']
})
# You should see parsed data here:
print(pprint.pformat(data))
这输出:
[{'content': 'fox, dog, cat are examples http://example.com',
'id': 22642441022,
'source': '<a href="http://example.com">Click</a>'},
{'content': "I learned so much from my mistakes, I think I'm gonna make some "
'more',
'id': 1153634043,
'source': '<a href="http://example.com">Click</a>'}]