无法从NLTK库导入二元语法

Question

一个让我迷惑的小问题。我安装了NLTK，它一直工作得很好。然而，我正在尝试获取语料库的二元语法，并希望基本上使用二元语法(语料库)。但是它说当我“从nltk导入二元语法”时，并没有定义二元语法。

三文法也是如此。我是不是遗漏了什么？另外，我如何才能从语料库中手动获取二元语法。

我也在寻找计算二元组三元和四元组的频率，但不确定如何确切地进行这一点。

我将语料库在开头和结尾适当地标记为"<s>"和"</s>"。到目前为止，程序如下：

 #!/usr/bin/env python
import re
import nltk
import nltk.corpus as corpus
import tokenize
from nltk.corpus import brown

def alter_list(row):
    if row[-1] == '.':
        row[-1] = '</s>'
    else:
        row.append('</s>')
    return ['<s>'] + row

news = corpus.brown.sents(categories = 'editorial')
print len(news),'\n'

x = len(news)
for row in news[:x]:
    print(alter_list(row))

Answer 1

我在一个虚拟环境中测试了它，它工作正常：

In [20]: from nltk import bigrams

In [21]: bigrams('This is a test')
Out[21]: 
[('T', 'h'),
 ('h', 'i'),
 ('i', 's'),
 ('s', ' '),
 (' ', 'i'),
 ('i', 's'),
 ('s', ' '),
 (' ', 'a'),
 ('a', ' '),
 (' ', 't'),
 ('t', 'e'),
 ('e', 's'),
 ('s', 't')]

这是您得到的唯一错误吗？

顺便说一下，关于你的第二个问题：

from collections import Counter
In [44]: b = bigrams('This is a test')

In [45]: Counter(b)
Out[45]: Counter({('i', 's'): 2, ('s', ' '): 2, ('a', ' '): 1, (' ', 't'): 1, ('e', 's'): 1, ('h', 'i'): 1, ('t', 'e'): 1, ('T', 'h'): 1, (' ', 'i'): 1, (' ', 'a'): 1, ('s', 't'): 1})

对于单词：

In [49]: b = bigrams("This is a test".split(' '))

In [50]: b
Out[50]: [('This', 'is'), ('is', 'a'), ('a', 'test')]

In [51]: Counter(b)
Out[51]: Counter({('is', 'a'): 1, ('a', 'test'): 1, ('This', 'is'): 1})

这种按单词拆分显然是非常肤浅的，但根据您的应用程序，它可能就足够了。显然，您可以使用nltk的tokenize，它要复杂得多。

为了实现你的最终目标，你可以这样做：

In [56]: d = "Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum."

In [56]: from nltk import trigrams
In [57]: tri = trigrams(d.split(' '))

In [60]: counter = Counter(tri)

In [61]: import random

In [62]: random.sample(counter, 5)
Out[62]: 
[('Ipsum', 'has', 'been'),
 ('industry.', 'Lorem', 'Ipsum'),
 ('Ipsum', 'passages,', 'and'),
 ('was', 'popularised', 'in'),
 ('galley', 'of', 'type')]

我修剪了输出，因为它不必要地大，但你明白我的意思了。