基于n元文法的NLTK分类概率估计

Question

我有以下代码，用于估计文本字符串属于特定类(正数或负数)的概率。

import pickle
from nltk.util import ngrams
classifier0 = open("C:/Users/ned/Desktop/gherkin.pickle","rb")
classifier = pickle.load(classifier0)
words = ['boring', 'and', 'stupid', 'movie']
feats = dict([(word, True) for word in words])
classifier.classify(feats)
probs = classifier.prob_classify(feats)
for sample in ('neg', 'pos'):
    print('%s probability: %s' % (sample, probs.prob(sample)))

它产生以下结果：

neg probability: 0.944
pos probability: 0.055
[Finished in 24.7s]

我正在加载的酸洗分类器已经使用了n-gram。

我的问题是：

如何编辑此代码以将n元语法合并到概率估计中？

Answer 1

将ngram添加到您的特征字典中...

import pickle                                                                                                                                                                                              
from nltk.util import ngrams                                                                                                                                                                                   
fin = open("C:/Users/ned/Desktop/gherkin.pickle","rb")                                                                                                                                                        
classifier = pickle.load(fin)                                                                                                                                                                                 
words = ['boring', 'and', 'stupid', 'movie']                                                                                                                                                                   
ngram_list = words + list(ngrams(words, 2)) + list(ngrams(words, 3))                                                                                                                                           
feats = dict([(word, True) for word in ngram_list])                                                                                                                                                          
dist = classifier.prob_classify(feats)                                                                                                                                                                         
for sample in dist.samples():                                                                                                                                                                                  
    print("%s probability: %f" % (sample, dist.prob(sample))) 

示例输出...

$ python movie-classifer-example.py 
neg probability: 0.999138
pos probability: 0.000862

Answer 2

根据N-Gram分类器(n用于训练)，您可以生成n-Gram并使用分类器对其进行分类，从而获得这些概率。

要生成新实例，请使用以下示例：(仅适用于二元和三元)。

import nltk
words = nltk.word_tokenize(text) # or your list
bigrams = nltk.bigrams(words)
trigrams = nltk.trigrams(words)