打印使用gensim学习的二元语法

Question

我想使用gensim从语料库中学习二元语法，然后只打印学习到的二元语法。我还没有见过这样的例子。感谢您的帮助

from gensim.models import Phrases
documents = ["the mayor of new york was there", "human computer interaction and machine learning has now become a trending research area","human computer interaction is interesting","human computer interaction is a pretty interesting subject", "human computer interaction is a great and new subject", "machine learning can be useful sometimes","new york mayor was present", "I love machine learning because it is a new subject area", "human computer interaction helps people to get user friendly applications"]
sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream)

# how can I print all bigrams learned and just the bigrams, including "new_york" and "human computer" ?enter code here

Answer 1

如果您使用前面提到的类Phrases训练模型，并且在不持久模型的情况下打印二元语法，那么OP的答案将会起作用。当您保存模型并在将来再次加载它时，它将不起作用。在保存模型后加载模型时，需要使用Phraser类，如下所示：

from gensim.models.phrases import Phraser

然后加载模型：

bigram_model = Phraser.load('../../whatever_bigram_model')

然后，如果您确实使用以下方法作为上述OP的答案，即

OP提供的应答

import operator
sorted(
    {k:v for k,v in bigram_model.vocab.items() if b'_' in k if v>=bigram_model.min_count}.items(),
    key=operator.itemgetter(1),
    reverse=True)

您将收到一条错误消息，指出：

AttributeError: 'Phraser' object has no attribute 'vocab'

解决方案

解决这个问题的方法是使用以下代码：

for bigram in bigram_model.phrasegrams.keys():
    print(bigram)

输出：

(b'word1', b'word2')
(b'word3', b'word4')

这个解决方案在两种情况下都有效，对于持久化和非持久化模型，在OP给出的示例中，我的解决方案的修改版本是：

for ngrams, _ in bigram.vocab.items():
    unicode_ngrams = ngrams.decode('utf-8')
    if '_' in unicode_ngrams:
        print(unicode_ngrams)

提供：

the_mayor
mayor_of
of_new
new_york
york_was
was_there
human_computer
computer_interaction
interaction_and
and_machine
machine_learning
learning_has
has_now
now_become

输出中还有更多内容，但为了这个答案的长度，我把它截断了

我希望我的回答有助于增加清晰度。

Answer 2

import operator
sorted(
    {k:v for k,v in bigram.vocab.items() if b'_' in k if v>=bigram.min_count}.items(),
    key=operator.itemgetter(1),
    reverse=True)