用Gensim训练Word2vec模型

Question

这是我的code.it从excel文件(rev列)读取评论并列出列表。

xp是这样的

["['intrepid', 'bumbling', 'duo', 'deliver', 'good', 'one'],['better', 'offering', 'considerable', 'cv', 'freshly', 'qualified', 'private', 'investigator', 'thrust', 'murder', 'investigation', 'invisible'],[ 'man', 'alone', 'tell', 'fun', 'flow', 'decent', 'clip', 'need', 'say', 'sequence', 'comedy', 'gold', 'like', 'scene', 'restaurant', 'excellent', 'costello', 'pretending', 'work', 'ball', 'gym', 'final', 'reel']"]

但是当使用list作为模型时，它给出了错误“TypeError：'float‘对象是不可迭代的”.i不知道我的问题在哪里。谢谢。

xp=[]
import gensim 
import logging
import pandas as pd 
file = r'FileNamelast.xlsx'
df = pd.read_excel(file,sheet_name='FileNamex')
pages = [i for i in range(0,1000)]


for page in  pages:

 text =df.loc[page,["rev"]]
 xp.append(text[0])


model = gensim.models.Word2Vec (xp, size=150, window=10, min_count=2, 
workers=10)
model.train(xp,total_examples=len(xp),epochs=10)

这就是我的got.TypeError：'float‘对象是不可迭代的

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-32-aa34c0e432bf> in <module>()
     14 
     15 
---> 16 model = gensim.models.Word2Vec (xp, size=150, window=10, min_count=2, workers=10)
     17 model.train(xp,total_examples=len(xp),epochs=10)

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\word2vec.py in __init__(self, sentences, corpus_file, size, alpha, window, min_count, max_vocab_size, sample, seed, workers, min_alpha, sg, hs, negative, ns_exponent, cbow_mean, hashfxn, iter, null_word, trim_rule, sorted_vocab, batch_words, compute_loss, callbacks, max_final_vocab)
    765             callbacks=callbacks, batch_words=batch_words, trim_rule=trim_rule, sg=sg, alpha=alpha, window=window,
    766             seed=seed, hs=hs, negative=negative, cbow_mean=cbow_mean, min_alpha=min_alpha, compute_loss=compute_loss,
--> 767             fast_version=FAST_VERSION)
    768 
    769     def _do_train_epoch(self, corpus_file, thread_id, offset, cython_vocab, thread_private_mem, cur_epoch,

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\base_any2vec.py in __init__(self, sentences, corpus_file, workers, vector_size, epochs, callbacks, batch_words, trim_rule, sg, alpha, window, seed, hs, negative, ns_exponent, cbow_mean, min_alpha, compute_loss, fast_version, **kwargs)
    757                 raise TypeError("You can't pass a generator as the sentences argument. Try an iterator.")
    758 
--> 759             self.build_vocab(sentences=sentences, corpus_file=corpus_file, trim_rule=trim_rule)
    760             self.train(
    761                 sentences=sentences, corpus_file=corpus_file, total_examples=self.corpus_count,

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\base_any2vec.py in build_vocab(self, sentences, corpus_file, update, progress_per, keep_raw_vocab, trim_rule, **kwargs)
    934         """
    935         total_words, corpus_count = self.vocabulary.scan_vocab(
--> 936             sentences=sentences, corpus_file=corpus_file, progress_per=progress_per, trim_rule=trim_rule)
    937         self.corpus_count = corpus_count
    938         self.corpus_total_words = total_words

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\word2vec.py in scan_vocab(self, sentences, corpus_file, progress_per, workers, trim_rule)
   1569             sentences = LineSentence(corpus_file)
   1570 
-> 1571         total_words, corpus_count = self._scan_vocab(sentences, progress_per, trim_rule)
   1572 
   1573         logger.info(

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\word2vec.py in _scan_vocab(self, sentences, progress_per, trim_rule)
   1552                     sentence_no, total_words, len(vocab)
   1553                 )
-> 1554             for word in sentence:
   1555                 vocab[word] += 1
   1556             total_words += len(sentence)

TypeError: 'float' object is not iterable

Answer 1

sentences Word2Vec的语料库参数应该是单词列表标记的可迭代序列。

xp的报告值实际上是一个包含一个长字符串的列表：

[
  "['intrepid', 'bumbling', 'duo', 'deliver', 'good', 'one'],['better', 'offering', 'considerable', 'cv', 'freshly', 'qualified', 'private', 'investigator', 'thrust', 'murder', 'investigation', 'invisible'],[ 'man', 'alone', 'tell', 'fun', 'flow', 'decent', 'clip', 'need', 'say', 'sequence', 'comedy', 'gold', 'like', 'scene', 'restaurant', 'excellent', 'costello', 'pretending', 'work', 'ball', 'gym', 'final', 'reel']"
]

我不认为这会给你报告的错误，但它绝对是错误的，所以应该修复。您也许应该在实例化xp之前打印Word2Vec，以确保您知道它包含了什么。

一个真正的列表，每一项都是一个字符串标记的列表，就可以了。因此，如果xp是这样的，那么这是正确的：

    [
      ['intrepid', 'bumbling', 'duo', 'deliver', 'good', 'one'],
      ['better', 'offering', 'considerable', 'cv', 'freshly', 'qualified', 'private', 'investigator', 'thrust', 'murder', 'investigation', 'invisible'],
      [ 'man', 'alone', 'tell', 'fun', 'flow', 'decent', 'clip', 'need', 'say', 'sequence', 'comedy', 'gold', 'like', 'scene', 'restaurant', 'excellent', 'costello', 'pretending', 'work', 'ball', 'gym', 'final', 'reel']
    ]

但注：

• Word2Vec在玩具大小的数据集方面做得不好。因此，虽然这种微小的设置可能有助于检查基本的语法/格式问题，但在您使用成千上万个单词进行培训之前，不要期待实际的结果。
• 如果您已经像以前一样在实例化时提供了您的语料库，则不需要调用train()。模型将自动完成所有步骤。(另一方面，如果你没有提供你的语料库，那么你必须同时打电话给build_vocab()和train()。)如果您在信息级别启用日志记录，那么在幕后进行的所有步骤都将更加清晰。