将数据帧转换为语料库

Question

我使用[这里]的例子：https://tutorials.quanteda.io/machine-learning/topicmodel/

`require(quanteda)
require(quanteda.corpora)
require(lubridate)
require(topicmodels)
corp_news <- download('data_corpus_guardian')`

`corp_news_subset <- corpus_subset(corp_news, 'date' >= 2016)
dfmat_news <- dfm(corp_news, remove_punct = TRUE, remove = stopwords('en')) %>% 
     dfm_remove(c('*-time', '*-timeUpdated', 'GMT', 'BST')) %>% 
     dfm_trim(min_termfreq = 0.95, termfreq_type = "quantile", 
              max_docfreq = 0.1, docfreq_type = "prop")
 dfmat_news
Document-feature matrix of: 6,000 documents, 4,534 features (97.2% sparse).
 str(corp_news)
List of 4
 $ documents:'data.frame':  6000 obs. of  10 variables:
  ..$ texts  : chr [1:6000] "London masterclass on climate change | Do you want to understand more about climate change? On 14 March the Gua"| __truncated__ "As colourful fish were swimming past him off the Greek coast, Cathal Redmond was convinced he had taken some gr"| __truncated__ "FTSE 100 | -101.35 | 6708.35 | FTSE All Share | -58.11 | 3608.55 | Early Dow Indl | -201.40 | 16120.31 | Early "| __truncated__ "Australia's education minister, Christopher Pyne, has vowed to find another university to host the Bjorn Lombor"| __truncated__ …`

正如我们所看到的，稀疏水平为97.2％。另外，corp_news $ documents $ texts的结构包含不同级别的文档。

在我的情况下，我有一个数据帧（每行是一个文档）：

`df <- data.frame(text = c("Lorem Ipsum is simply dummy text of the printing and typesetting industry. <code> ste </code> Lorem Ipsum is simply dummy text of the printing and typesetting industry.", "<code> teft </code> Lorem Ipsum has been the industry's standard dummy text ever since the 1500s", "when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electro <code> type sth but you can't see it </code>"), stringsAsFactors = FALSE)`

我用它来消除一些噪音：

`mytext <- paste(unlist(df$text), collapse =" ")
mytext2 <- gsub("<code>.+?</code>", "", mytext)
cleanFun <- function(htmlString) {
    return(gsub("<.*?>", "", htmlString))
}
mytext3 <- cleanFun(mytext2)
df2 <- gsub("\n", "", mytext3)`

然而，该文档是不公开的，我收到稀疏的0.0％

myDfm <- dfm(df2, remove_punct = TRUE, remove = stopwords('en')) 
myDfm
Document-feature matrix of: 1 document, 28 features (0.0% sparse).

怎么可能让df2像df一样具有每一行的结构？

Answer 1

不完全确定问题是什么，但如果你想清理df中的文本，然后将其转换为语料库，这将是要走的路：

df$text <- gsub("<.*?>", "", df$text)
corp <- corpus(df, text_field = "text")
dfm <- dfm(corp, remove_punct = TRUE, remove = stopwords('en'))
> dfm
Document-feature matrix of: 3 documents, 32 features (62.5% sparse).