NLTK困惑错误

Question

我收到了这个错误消息

Traceback (most recent call last):
      File "C:/Users/shengrong/Desktop/bigram", line 55, in <module>
        bg = bigram(file)
      File "C:/Users/shengrong/Desktop/bigram", line 43, in bigram
        return tt1.perplexity(my_bigrams)
      File "C:\Python27\lib\site-packages\nltk\model\ngram.py", line 217, in perplexity
        return pow(2.0, self.entropy(text))
      File "C:\Python27\lib\site-packages\nltk\model\ngram.py", line 205, in entropy
        e += self.logprob(token, context)
      File "C:\Python27\lib\site-packages\nltk\model\ngram.py", line 151, in logprob
        return -log(self.prob(word, context), 2)
    ValueError: math domain error




import os,csv,nltk
from nltk.model.ngram import NgramModel
from nltk.probability import LidstoneProbDist

fout = open("/Users/shengrong/Documents/personal/WN1.data.csv", "w")

outfilehandle = csv.writer(fout,
                           delimiter=",",
                           quotechar='"',
                           quoting=csv.QUOTE_NONNUMERIC)  


localrow = []
localrow.append("File name")
localrow.append("Perplexity for unigram")
localrow.append("Perplexity for bigram")
localrow.append("Perplexity for trigram")
outfilehandle.writerow(localrow)

def bigram(file):
   file_object = open(file)
   ln=file_object.read()

   words = nltk.word_tokenize(ln)
   my_bigrams = nltk.bigrams(words)
   my_trigrams = nltk.trigrams(words)



   tt1=NgramModel(2, my_bigrams, estimator = None)


   return tt1.perplexity(my_bigrams)    




#set the path of the folder 
os.chdir("/Users/shengrong/Documents/A") 
s = os.getcwd() 
#search files in the folder 
files = os.listdir(s) 

for file in files:
    bg = bigram(file)
    localrow= []
    localrow.append(file)
    localrow.append(bg)

    outfilehandle.writerow(localrow)

fout.close()

如何解决这个问题？在不使用循环读取文件夹的情况下，我的代码运行良好。

谢谢你们。

Answer 1

您得到的错误是一个数学域错误。由于它在没有循环的情况下工作，因此一个或多个文件可能包含一些无法由Math包处理的数据。

请确认文件夹中的所有文件都包含正确格式和预期值的数据。