如何找到单词列表之间的语义相似性？

Question

输入：

listToStr = 'degeneration agents alpha alternative amd analysis angiogenesis anti anti vegf appears associated based best bevacizumab blindness blood'

我正在使用的代码：

simi = []
tokens = nlp(listToStr) 
length = len(tokens)

for i in range(length):
    #print(i)
    sim = tokens[i].similarity(tokens[i+1])
    simi.append(sim)
print(simi)

错误：

[E040] Attempt to access token at 17, max length 17.

如何删除此错误？

我用的是spacy。下面是指向它的链接：https://www.geeksforgeeks.org/python-word-similarity-using-spacy/#:~:text=Python%20%7C%20Word%20Similarity%20using%20spaCy,simple%20method%20for%20this%20task.

Answer 1

在for循环中，由于tokens[i + 1]操作的结果，创建了一个超出令牌列表范围的索引。相反，你可以做这样的事情：

import spacy

nlp = spacy.load("en_core_web_sm")

listToStr = 'degeneration agents alpha alternative amd analysis angiogenesis anti anti vegf appears associated based best bevacizumab blindness blood'

simi = []
tokens = nlp(listToStr) 

for idx, tok in enumerate(tokens):
    sim = []
    for nextok in tokens[idx:]:
        sim.append(tok.similarity(nextok))
    simi.append(sim)

这测试了每个单词和句子中下一个单词的相似性，所以结果是一个列表。