修复单词并使用python删除分割单词之间不需要的空格？

Question

我从文档中提取了句子列表。我正在预处理这个句子列表，使其更加明智。我遇到了以下问题

我有像Java is a prog rammng lan guage. C is a gen eral purpose la nguage.这样的句子

我想用查找词典来纠正这些句子？删除不需要的空格。

最终输出应该是Java is a programmng language. C is a general purpose language.

我需要一些帮助来寻找这些方法。如何解决上述问题？

我想用python代码解决上面的问题。谢谢。

Answer 1

这是一个适用于您的示例的简单脚本。显然你需要更大的有效词汇。此外，你可能想要一个elif分支，如果加入下一个单词无法修复非单词，则会回顾上一个单词。

from string import punctuation

word_list = "big list of words including a programming language is general purpose"
valid_words = set(word_list.split())

bad = "Java is a prog ramming lan guage. C is a gen eral purpose la nguage."
words = bad.split()

out_words = []
i = 0
while i < len(words):
    word = words[i]
    if word not in valid_words and i+1 < len(words):
        next_word = words[i+1]
        joined = word + next_word
        if joined.strip(punctuation) in valid_words:
            word = joined
            i += 1
    out_words.append(word)
    i += 1

good = " ".join(out_words)
print(good)

Answer 2

如果你想要拼写和正确解析单词，你需要拼写检查。这是一个拼写检查器，通过导入“re”命名空间，使用完整的文章here ...

import re
from collections import Counter

def words(text): return re.findall(r'\w+', text.lower())

WORDS = Counter(words(open('big.txt').read()))

def P(word, N=sum(WORDS.values())): 
    "Probability of `word`."
    return WORDS[word] / N

def correction(word): 
    "Most probable spelling correction for `word`."
    return max(candidates(word), key=P)

def candidates(word): 
    "Generate possible spelling corrections for `word`."
    return (known([word]) or known(edits1(word)) or known(edits2(word)) or [word])

def known(words): 
    "The subset of `words` that appear in the dictionary of WORDS."
    return set(w for w in words if w in WORDS)

def edits1(word):
    "All edits that are one edit away from `word`."
    letters    = 'abcdefghijklmnopqrstuvwxyz'
    splits     = [(word[:i], word[i:])    for i in range(len(word) + 1)]
    deletes    = [L + R[1:]               for L, R in splits if R]
    transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]
    replaces   = [L + c + R[1:]           for L, R in splits if R for c in letters]
    inserts    = [L + c + R               for L, R in splits for c in letters]
    return set(deletes + transposes + replaces + inserts)

def edits2(word): 
    "All edits that are two edits away from `word`."
    return (e2 for e1 in edits1(word) for e2 in edits1(e1))`

它不仅可以修复分词，还可以删除，转置和插入不规则的单词以“纠正”它们。您可以将“big.txt”文件替换为您在Counter构造函数中使用的文档，并希望一切都可以在那里工作。