特定的停用词列表quanteda

Question

我想使用quanteda删除一个带有停用词的特定列表。

我使用的是：

df <- data.frame(data = c("Here is an example text and why I write it", "I can explain and here you but I can help as I would like to help"))
mystopwords <- c("is","an")
corpus<- dfm(tokens_remove(tokens(df$data, remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE), remove = c(stopwords(language = "el", source = "misc"), mystopwords), ngrams = c(4,6)))

但是我收到了这个错误：

> Error in tokens_select(x, ..., selection = "remove") : 
  unused arguments (remove = c(stopwords(language = "en", source = "misc"), stopwords1), ngrams = c(4, 6))

如何在quanteda中正确使用mystopwords列表？

Answer 1

基于@phiver的回答，这是在quanteda中删除特定标记的标准方法。不需要使用stopwords()，因为您提供了要删除的令牌的向量，并且patterns参数可以接受向量，但请使用valuetype = 'fixed'。

为了代码的可读性，我使用了dplyr，但你不必这样做。

library(quanteda)
library(dplyr)
df <- data.frame(data = c("Here is an example text and why I write it", 
                          "I can explain and here you but I can help as I would like to help"),
                 stringsAsFactors = FALSE)

mystopwords <- c("is","an")
corpus <- 
  tokens(df$data,
         remove_punct = TRUE, 
         remove_numbers = TRUE, 
         remove_symbols = TRUE) %>%
  tokens_remove(pattern = mystopwords,
                valuetype = 'fixed') %>%
  dfm(ngrams = c(4,6))

Answer 2

这将会起作用。首先，我将stringAsFactors = FALSE添加到data.frame中。提供给tokens的文本需要是一个字符矢量，而不是一个因子。接下来，我更改了代码中的remove =，因为它需要为pattern =。最后，ngram部分需要在dfm函数中，而不是在token_remove函数中。

在嵌套函数时，最好对代码进行更多的格式化。它更好地显示了可能会犯错误的地方。

library(quanteda)
df <- data.frame(data = c("Here is an example text and why I write it", 
                          "I can explain and here you but I can help as I would like to help"),
                 stringsAsFactors = FALSE)

mystopwords <- c("is","an")
corpus <- dfm(tokens_remove(tokens(df$data, 
                                   remove_punct = TRUE, 
                                   remove_numbers = TRUE, 
                                   remove_symbols = TRUE), 
                            pattern = c(stopwords(language = "el", source = "misc"), 
                                       mystopwords) 
                            ), 
              ngrams = c(4,6)
              )