带quanteda的茎

Question

我使用以下命令使用quanteda进行词干分析

myDfm <- dfm(tokens_remove(tokens(df2, remove_punct = TRUE, stem = TRUE, remove_numbers = TRUE, remove_symbols = TRUE), stopwords(source = "smart")), 
                          ngrams = c(1,2))

然而，我在最后收到以下警告：

Warning message:
Argument stem not used. 

是否有任何不同的选项来使用quanteda实现词干分析

Answer 1

是的，你想要tokens_wordstem()。在您的示例中，您将stem = TRUE提供给tokens()参数，而不是dfm()调用。tokens()没有将stem作为参数(如警告所述)。

为了清楚起见，我建议使用管道操作符%>%来更清楚地查看操作的顺序。

library("quanteda")
## Package version: 1.4.0
## Parallel computing: 2 of 12 threads used.
## See https://quanteda.io for tutorials and examples.
## 
## Attaching package: 'quanteda'
## The following object is masked from 'package:utils':
## 
##     View

df2 <- data_char_sampletext
quanteda_options(verbose = TRUE)

df2 %>%
  tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE) %>%
  tokens_remove(stopwords(source = "smart")) %>%
  tokens_wordstem() %>%
  tokens_ngrams(n = c(1, 2)) %>%
  dfm()
## removed 0 features
## 
## removed 72 features
## Creating a dfm from a tokens input...
##    ... lowercasing
##    ... found 1 document, 375 features
##    ... created a 1 x 375 sparse dfm
##    ... complete. 
## Elapsed time: 0.038 seconds.
## Document-feature matrix of: 1 document, 375 features (0.0% sparse).