nltk语料库不包含“not”？

Question

NLTK单词语料库没有"ok"，“ok”，“ok”？

> from nltk.corpus import words
> words.words().__contains__("check")
> True

> words.words().__contains__("okay")
> False

> len(words.words())
> 236736

知道为什么吗？

Answer 1

TL;DR

from nltk.corpus import words
from nltk.corpus import wordnet 

manywords = words.words() + wordnet.words() 

长时间

在文档中，nltk.corpus.words是单词-- "(Unix)“中的单词列表

在Unix中，您可以这样做：

ls /usr/share/dict/

阅读自述：

$ cd /usr/share/dict/
/usr/share/dict$ cat README
#   @(#)README  8.1 (Berkeley) 6/5/93
# $FreeBSD$

WEB ---- (introduction provided by jaw@riacs) -------------------------

Welcome to web2 (Webster's Second International) all 234,936 words worth.
The 1934 copyright has lapsed, according to the supplier.  The
supplemental 'web2a' list contains hyphenated terms as well as assorted
noun and adverbial phrases.  The wordlist makes a dandy 'grep' victim.

     -- James A. Woods    {ihnp4,hplabs}!ames!jaw    (or jaw@riacs)

Country names are stored in the file /usr/share/misc/iso3166.


FreeBSD Maintenance Notes ---------------------------------------------

Note that FreeBSD is not maintaining a historical document, we're
maintaining a list of current [American] English spellings.

A few words have been removed because their spellings have depreciated.
This list of words includes:
    corelation (and its derivatives)    "correlation" is the preferred spelling
    freen               typographical error in original file
    freend              archaic spelling no longer in use;
                    masks common typo in modern text

--

A list of technical terms has been added in the file 'freebsd'.  This
word list contains FreeBSD/Unix lexicon that is used by the system
documentation.  It makes a great ispell(1) personal dictionary to
supplement the standard English language dictionary.

因为它是234,936的固定列表，所以该列表中肯定有不存在的单词。

如果需要扩展单词列表，可以使用WordNet中使用nltk.corpus.wordnet.words()的单词添加到列表中。

最有可能的是，你只需要一个足够大的文本语料库，例如维基百科转储，然后标记它并提取所有唯一的单词。

Answer 2

由于声誉低，我无法发表评论，但我可以提供一些东西。我在数据问题与此相关中发布了一个zip文件，其中包含一组更全面的单词，这些词来自Ubuntu18.04 /usr/share/dict/american

原始/usr/share/dict文件中有一些严重缺失的单词，如“failings”和“failings”。不幸的是，使用wordnet并不能真正解决这个问题；它添加了“故障安全”和几种类型的失败，如“equipment_failure”和“renal_failure”，但它没有添加基本单词。希望所提供的use文件有一定的用处。