用于预测分析的R软件包。如何对新文档进行分类？

Question

这是一个关于文本挖掘程序的一般性问题。假设一个人有一个分类为垃圾邮件/无垃圾邮件的文件语料库。作为标准程序，我们对数据进行预处理、删除标点符号、停止单词等。在将其转换为DocumentTermMatrix后，可以建立一些模型来预测垃圾邮件/垃圾邮件。这是我的问题。现在我想使用为新文档到达而构建的模型。为了检查单个文档，我必须构建一个DocumentTerm*Vector？因此，它可以用来预测垃圾邮件/无垃圾邮件。在tm的文档中，我发现一个使用tfidf权重将完整的语料库转换成一个矩阵。然后，我如何从语料库中转换一个单一的向量？我是否每次都要改变我的语料库并建立一个新的DocumentTermMatrix？我处理了我的语料库，把它转换成一个矩阵，然后把它分割成一个训练和测试集。但是在这里，测试集构建在与完整集的文档矩阵相同的行中。我可以检查精度等，但不知道什么是新的文本分类的最佳程序。

本，假设我有一个预处理的DocumentTextMatrix，我把它转换成一个data.frame。

dtm <- DocumentTermMatrix(CorpusProc,control = list(weighting =function(x) weightTfIdf(x, normalize =FALSE),stopwords = TRUE, wordLengths=c(3, Inf), bounds = list(global = c(4,Inf))))

dtmDataFrame <- as.data.frame(inspect(dtm))

添加了一个因子变量并建立了一个模型。

Corpus.svm<-svm(Risk_Category~.,data=dtmDataFrame)

现在，假设我给您一个新的文档d(以前不在您的语料库中)，并且您想知道垃圾邮件/垃圾邮件的模型预测。你是怎么做到的？

Ok让我们根据这里使用的代码创建一个示例。

examp1 <- "When discussing performance with colleagues, teaching, sending a bug report or searching for guidance on mailing lists and here on SO, a reproducible example is often asked and always helpful. What are your tips for creating an excellent example? How do you paste data structures from r in a text format? What other information should you include? Are there other tricks in addition to using dput(), dump() or structure()? When should you include library() or require() statements? Which reserved words should one avoid, in addition to c, df, data, etc? How does one make a great r reproducible example?"
examp2 <- "Sometimes the problem really isn't reproducible with a smaller piece of data, no matter how hard you try, and doesn't happen with synthetic data (although it's useful to show how you produced synthetic data sets that did not reproduce the problem, because it rules out some hypotheses). Posting the data to the web somewhere and providing a URL may be necessary. If the data can't be released to the public at large but could be shared at all, then you may be able to offer to e-mail it to interested parties (although this will cut down the number of people who will bother to work on it). I haven't actually seen this done, because people who can't release their data are sensitive about releasing it any form, but it would seem plausible that in some cases one could still post data if it were sufficiently anonymized/scrambled/corrupted slightly in some way. If you can't do either of these then you probably need to hire a consultant to solve your problem" 
examp3 <- "You are most likely to get good help with your R problem if you provide a reproducible example. A reproducible example allows someone else to recreate your problem by just copying and pasting R code. There are four things you need to include to make your example reproducible: required packages, data, code, and a description of your R environment. Packages should be loaded at the top of the script, so it's easy to see which ones the example needs. The easiest way to include data in an email is to use dput() to generate the R code to recreate it. For example, to recreate the mtcars dataset in R, I'd perform the following steps: Run dput(mtcars) in R Copy the output In my reproducible script, type mtcars <- then paste. Spend a little bit of time ensuring that your code is easy for others to read: make sure you've used spaces and your variable names are concise, but informative, use comments to indicate where your problem lies, do your best to remove everything that is not related to the problem. The shorter your code is, the easier it is to understand. Include the output of sessionInfo() as a comment. This summarises your R environment and makes it easy to check if you're using an out-of-date package. You can check you have actually made a reproducible example by starting up a fresh R session and pasting your script in. Before putting all of your code in an email, consider putting it on http://gist.github.com/. It will give your code nice syntax highlighting, and you don't have to worry about anything getting mangled by the email system."
examp4 <- "Do your homework before posting: If it is clear that you have done basic background research, you are far more likely to get an informative response. See also Further Resources further down this page. Do help.search(keyword) and apropos(keyword) with different keywords (type this at the R prompt). Do RSiteSearch(keyword) with different keywords (at the R prompt) to search R functions, contributed packages and R-Help postings. See ?RSiteSearch for further options and to restrict searches. Read the online help for relevant functions (type ?functionname, e.g., ?prod, at the R prompt) If something seems to have changed in R, look in the latest NEWS file on CRAN for information about it. Search the R-faq and the R-windows-faq if it might be relevant (http://cran.r-project.org/faqs.html) Read at least the relevant section in An Introduction to R If the function is from a package accompanying a book, e.g., the MASS package, consult the book before posting. The R Wiki has a section on finding functions and documentation"
examp5 <- "Before asking a technical question by e-mail, or in a newsgroup, or on a website chat board, do the following:  Try to find an answer by searching the archives of the forum you plan to post to. Try to find an answer by searching the Web. Try to find an answer by reading the manual. Try to find an answer by reading a FAQ. Try to find an answer by inspection or experimentation. Try to find an answer by asking a skilled friend. If you're a programmer, try to find an answer by reading the source code. When you ask your question, display the fact that you have done these things first; this will help establish that you're not being a lazy sponge and wasting people's time. Better yet, display what you have learned from doing these things. We like answering questions for people who have demonstrated they can learn from the answers. Use tactics like doing a Google search on the text of whatever error message you get (searching Google groups as well as Web pages). This might well take you straight to fix documentation or a mailing list thread answering your question. Even if it doesn't, saying “I googled on the following phrase but didn't get anything that looked promising” is a good thing to do in e-mail or news postings requesting help, if only because it records what searches won't help. It will also help to direct other people with similar problems to your thread by linking the search terms to what will hopefully be your problem and resolution thread. Take your time. Do not expect to be able to solve a complicated problem with a few seconds of Googling. Read and understand the FAQs, sit back, relax and give the problem some thought before approaching experts. Trust us, they will be able to tell from your questions how much reading and thinking you did, and will be more willing to help if you come prepared. Don't instantly fire your whole arsenal of questions just because your first search turned up no answers (or too many). Prepare your question. Think it through. Hasty-sounding questions get hasty answers, or none at all. The more you do to demonstrate that having put thought and effort into solving your problem before seeking help, the more likely you are to actually get help. Beware of asking the wrong question. If you ask one that is based on faulty assumptions, J. Random Hacker is quite likely to reply with a uselessly literal answer while thinking Stupid question..., and hoping the experience of getting what you asked for rather than what you needed will teach you a lesson."



corpus2 <- Corpus(VectorSource(c(examp1, examp2, examp3, examp4)))

注:我拿出了示例5。

skipWords <- function(x) removeWords(x, stopwords("english"))
funcs <- list(tolower, removePunctuation, removeNumbers, stripWhitespace, skipWords)
corpus2.proc <- tm_map(corpus2, FUN = tm_reduce, tmFuns = funcs)

corpus2a.dtm <- DocumentTermMatrix(corpus2.proc, control = list(wordLengths = c(3,10)))
dtmDataFrame <- as.data.frame(inspect(corpus2a.dtm)) 

添加了一个因子变量Spam_Classification 2级垃圾邮件/无垃圾邮件

dtmFinal<-cbind(dtmDataFrame,Spam_Classification)

我建立了一个模型支持向量机语料库(Spam_Category~.，data=dtmFinal)

现在假设我的示例5是一个新文档(电子邮件)，我是如何生成垃圾邮件/No_垃圾邮件值的？

Answer 1

谢谢你提出这个有趣的问题。我已经想了一段时间了。太简短了，我的研究结果的精髓是:对于加权方法，除了tf之外，没有办法绕过费力的工作或重新计算整个DTM (并且可能会重新运行您的svm)。

只有tf加权，我才能找到一个简单的过程分类新的内容。您必须将新文档(当然)转换为DTM。在转换过程中，您必须添加一个包含用于在旧语料库上训练分类器的所有术语的dictionary。然后，您可以像通常一样使用predict()。对于tf部分，这里有一个非常小的样本和一种分类新文档的方法：

### I) Data

texts <- c("foo bar spam",
           "bar baz ham",
           "baz qux spam",
           "qux quux ham")

categories <- c("Spam", "Ham", "Spam", "Ham")

new <- "quux quuux ham"

### II) Building Model on Existing Documents „texts“

library(tm)  # text mining package for R
library(e1071)  # package with various machine-learning libraries

## creating DTM for texts
dtm <- DocumentTermMatrix(Corpus(VectorSource(texts)))

## making DTM a data.frame and adding variable categories
df <- data.frame(categories, as.data.frame(inspect(dtm)))

model <- svm(categories~., data=df)

### III) Predicting class of new

## creating dtm for new
dtm_n <- DocumentTermMatrix(Corpus(VectorSource(new)),
                            ## without this line predict won't work
                            control=list(dictionary=names(df)))
## creating data.frame for new
df_n <- as.data.frame(inspect(dtm_n))

predict(model, df_n)

## > 1 
## > Ham 
## > Levels: Ham Spam

Answer 2

现在还不清楚你的问题是什么，也不清楚你在寻找什么样的答案。

假设您在问‘如何才能让'DocumentTermVector’传递到其他函数呢？‘，这里有一个方法。

一些可复制的数据：

examp1 <- "When discussing performance with colleagues, teaching, sending a bug report or searching for guidance on mailing lists and here on SO, a reproducible example is often asked and always helpful. What are your tips for creating an excellent example? How do you paste data structures from r in a text format? What other information should you include? Are there other tricks in addition to using dput(), dump() or structure()? When should you include library() or require() statements? Which reserved words should one avoid, in addition to c, df, data, etc? How does one make a great r reproducible example?"
examp2 <- "Sometimes the problem really isn't reproducible with a smaller piece of data, no matter how hard you try, and doesn't happen with synthetic data (although it's useful to show how you produced synthetic data sets that did not reproduce the problem, because it rules out some hypotheses). Posting the data to the web somewhere and providing a URL may be necessary. If the data can't be released to the public at large but could be shared at all, then you may be able to offer to e-mail it to interested parties (although this will cut down the number of people who will bother to work on it). I haven't actually seen this done, because people who can't release their data are sensitive about releasing it any form, but it would seem plausible that in some cases one could still post data if it were sufficiently anonymized/scrambled/corrupted slightly in some way. If you can't do either of these then you probably need to hire a consultant to solve your problem" 
examp3 <- "You are most likely to get good help with your R problem if you provide a reproducible example. A reproducible example allows someone else to recreate your problem by just copying and pasting R code. There are four things you need to include to make your example reproducible: required packages, data, code, and a description of your R environment. Packages should be loaded at the top of the script, so it's easy to see which ones the example needs. The easiest way to include data in an email is to use dput() to generate the R code to recreate it. For example, to recreate the mtcars dataset in R, I'd perform the following steps: Run dput(mtcars) in R Copy the output In my reproducible script, type mtcars <- then paste. Spend a little bit of time ensuring that your code is easy for others to read: make sure you've used spaces and your variable names are concise, but informative, use comments to indicate where your problem lies, do your best to remove everything that is not related to the problem. The shorter your code is, the easier it is to understand. Include the output of sessionInfo() as a comment. This summarises your R environment and makes it easy to check if you're using an out-of-date package. You can check you have actually made a reproducible example by starting up a fresh R session and pasting your script in. Before putting all of your code in an email, consider putting it on http://gist.github.com/. It will give your code nice syntax highlighting, and you don't have to worry about anything getting mangled by the email system."
examp4 <- "Do your homework before posting: If it is clear that you have done basic background research, you are far more likely to get an informative response. See also Further Resources further down this page. Do help.search(keyword) and apropos(keyword) with different keywords (type this at the R prompt). Do RSiteSearch(keyword) with different keywords (at the R prompt) to search R functions, contributed packages and R-Help postings. See ?RSiteSearch for further options and to restrict searches. Read the online help for relevant functions (type ?functionname, e.g., ?prod, at the R prompt) If something seems to have changed in R, look in the latest NEWS file on CRAN for information about it. Search the R-faq and the R-windows-faq if it might be relevant (http://cran.r-project.org/faqs.html) Read at least the relevant section in An Introduction to R If the function is from a package accompanying a book, e.g., the MASS package, consult the book before posting. The R Wiki has a section on finding functions and documentation"
examp5 <- "Before asking a technical question by e-mail, or in a newsgroup, or on a website chat board, do the following:  Try to find an answer by searching the archives of the forum you plan to post to. Try to find an answer by searching the Web. Try to find an answer by reading the manual. Try to find an answer by reading a FAQ. Try to find an answer by inspection or experimentation. Try to find an answer by asking a skilled friend. If you're a programmer, try to find an answer by reading the source code. When you ask your question, display the fact that you have done these things first; this will help establish that you're not being a lazy sponge and wasting people's time. Better yet, display what you have learned from doing these things. We like answering questions for people who have demonstrated they can learn from the answers. Use tactics like doing a Google search on the text of whatever error message you get (searching Google groups as well as Web pages). This might well take you straight to fix documentation or a mailing list thread answering your question. Even if it doesn't, saying “I googled on the following phrase but didn't get anything that looked promising” is a good thing to do in e-mail or news postings requesting help, if only because it records what searches won't help. It will also help to direct other people with similar problems to your thread by linking the search terms to what will hopefully be your problem and resolution thread. Take your time. Do not expect to be able to solve a complicated problem with a few seconds of Googling. Read and understand the FAQs, sit back, relax and give the problem some thought before approaching experts. Trust us, they will be able to tell from your questions how much reading and thinking you did, and will be more willing to help if you come prepared. Don't instantly fire your whole arsenal of questions just because your first search turned up no answers (or too many). Prepare your question. Think it through. Hasty-sounding questions get hasty answers, or none at all. The more you do to demonstrate that having put thought and effort into solving your problem before seeking help, the more likely you are to actually get help. Beware of asking the wrong question. If you ask one that is based on faulty assumptions, J. Random Hacker is quite likely to reply with a uselessly literal answer while thinking Stupid question..., and hoping the experience of getting what you asked for rather than what you needed will teach you a lesson."

根据这些文本创建一个语料库：

corpus2 <- Corpus(VectorSource(c(examp1, examp2, examp3, examp4, examp5)))

处理文本：

skipWords <- function(x) removeWords(x, stopwords("english"))
funcs <- list(tolower, removePunctuation, removeNumbers, stripWhitespace, skipWords)
corpus2.proc <- tm_map(corpus2, FUN = tm_reduce, tmFuns = funcs)

将已处理的语料库转换为术语文档矩阵：

corpus2a.dtm <- DocumentTermMatrix(corpus2.proc, control = list(wordLengths = c(3,10)))
inspect(corpus2a.dtm)

A document-term matrix (5 documents, 273 terms)

Non-/sparse entries: 314/1051
Sparsity           : 77%
Maximal term length: 10 
Weighting          : term frequency (tf)

    Terms
Docs able actually addition allows answer answering answers archives are arsenal avoid background based
   1    0        0        2      0      0         0       0        0   1       0     1          0     0
   2    1        1        0      0      0         0       0        0   0       0     0          0     0
   3    0        1        0      1      0         0       0        0   0       0     0          0     0
   4    0        0        0      0      0         0       0        0   0       0     0          1     0
   5    2        1        0      0      8         2       3        1   0       1     0          0     1

--这是您所指的"DocumentTerm*Vector*“的关键行：

# access vector of first document in the dtm
as.matrix(corpus2a.dtm)[1,]

able   actually   addition     allows     answer  answering    answers   archives        are 
         0          0          2          0          0          0          0          0          1 
   arsenal      avoid background      based      basic     before     better     beware        bit 
         0          1          0          0          0          0          0          0          0 
     board       book     bother        bug    changed       chat      check       

实际上，它是一个命名的数字，对于传递到其他函数等应该是有用的，这似乎是您想要做的事情：

str(as.matrix(corpus2a.dtm)[1,])
 Named num [1:273] 0 0 2 0 0 0 0 0 1 0 ...

如果您只想要一个数字向量，请尝试as.numeric(as.matrix(corpus2a.dtm)[1,]))

这就是你想做的吗？