计数矢量化器和拟合函数的Python列表错误

Question

请告诉我有什么问题以及如何纠正。

data = open(r"C:\Users\HS\Desktop\WORK\R\R DATA\g textonly2.txt").read()
labels, texts = [], []
#print(data)
for i, line in enumerate(data.split("\n")):
    content = line.split()
    #print(content)
    if len(content) is not 0:
        labels.append(content[0])
        texts.append(content[1:])


# create a dataframe using texts and lables
trainDF = pandas.DataFrame()
trainDF['text'] = texts
trainDF['label'] = labels

# split the dataset into training and validation datasets
train_x, valid_x, train_y, valid_y = model_selection.train_test_split(trainDF['text'], trainDF['label'])

# label encode the target variable
encoder = preprocessing.LabelEncoder()
train_y = encoder.fit_transform(train_y)
valid_y = encoder.fit_transform(valid_y)

# create a count vectorizer object
count_vect = CountVectorizer(analyzer='word', token_pattern=r'\w{1,}')
count_vect.fit(trainDF['text'])

数据文件包含如下数据：

0 #\xdaltimahora Es tracta d'un aparell de Germanwings amb 152 passatgers a bord
0 Route map now being shared by http:
0 Pray for #4U9525 http:
0 Airbus A320 #4U9525 crash: \nFlight tracking data here: \nhttp

错误：

Traceback:
"C:\Program Files\Python36\python.exe" "C:/Users/HS/PycharmProjects/R/C/Text classification1.py"
Using TensorFlow backend.
Traceback (most recent call last):
  File "C:/Users/HS/PycharmProjects/R/C/Text classification1.py", line 38, in <module>
    count_vect.fit(trainDF['text'])
  File "C:\Program Files\Python36\lib\site-packages\sklearn\feature_extraction\text.py", line 836, in fit
    self.fit_transform(raw_documents)
  File "C:\Program Files\Python36\lib\site-packages\sklearn\feature_extraction\text.py", line 869, in fit_transform
    self.fixed_vocabulary_)
  File "C:\Program Files\Python36\lib\site-packages\sklearn\feature_extraction\text.py", line 792, in _count_vocab
    for feature in analyze(doc):
  File "C:\Program Files\Python36\lib\site-packages\sklearn\feature_extraction\text.py", line 266, in <lambda>
    tokenize(preprocess(self.decode(doc))), stop_words)
  File "C:\Program Files\Python36\lib\site-packages\sklearn\feature_extraction\text.py", line 232, in <lambda>
    return lambda x: strip_accents(x.lower())
AttributeError: 'list' object has no attribute 'lower'

Process finished with exit code 1

Answer 1

来自documentation：

fit（raw_documents，y = None）[source]学习原始文档中所有标记的词汇表。

参数：raw_documents：iterable

可迭代产生str，unicode或文件对象。

返回：self：

你得到错误AttributeError: 'list' object has no attribute 'lower'，因为你给它一个可迭代的（在这种情况下是pd.Series）列表对象，而不是可迭代的字符串。

你应该能够通过使用texts.append(' '.join(content[1:]))而不是texts.append(content[1:])解决这个问题：

for i, line in enumerate(data.split("\n")):
    content = line.split()
    #print(content)
    if len(content) is not 0:
        labels.append(content[0])
        #texts.append(content[1:])
        texts.append(' '.join(content[1:]))