TfidfVectorizer背后的数学是什么？

Question

我正在努力理解TfidfVectorizer背后的数学。我使用了这教程，但是我的代码有一点改变：

最后还说了什么，The values differ slightly because sklearn uses a smoothed version idf and various other little optimizations.

我想能够使用TfidfVectorizer，但也用我的手计算相同的简单样本。

以下是我的完整代码:从sklearn.feature_extraction.text导入熊猫作为pd从sklearn.feature_extraction.text导入CountVectorizer从sklearn.feature_extraction.text导入TfidfTransformer从sklearn.feature_extraction.text导入TfidfVectorizer

def main():
    documentA = 'the man went out for a walk'
    documentB = 'the children sat around the fire'
    corpus = [documentA, documentB]
    bagOfWordsA = documentA.split(' ')
    bagOfWordsB = documentB.split(' ')

    uniqueWords = set(bagOfWordsA).union(set(bagOfWordsB))

    print('----------- compare word count -------------------')
    numOfWordsA = dict.fromkeys(uniqueWords, 0)
    for word in bagOfWordsA:
        numOfWordsA[word] += 1
    numOfWordsB = dict.fromkeys(uniqueWords, 0)
    for word in bagOfWordsB:
        numOfWordsB[word] += 1

    tfA = computeTF(numOfWordsA, bagOfWordsA)
    tfB = computeTF(numOfWordsB, bagOfWordsB)
    print(pd.DataFrame([tfA, tfB]))

    CV = CountVectorizer(stop_words=None, token_pattern='(?u)\\b\\w\\w*\\b')
    cv_ft = CV.fit_transform(corpus)

    tt = TfidfTransformer(use_idf=False, norm='l1')
    t = tt.fit_transform(cv_ft)
    print(pd.DataFrame(t.todense().tolist(), columns=CV.get_feature_names()))

    print('----------- compare idf -------------------')
    idfs = computeIDF([numOfWordsA, numOfWordsB])
    print(pd.DataFrame([idfs]))

    tfidfA = computeTFIDF(tfA, idfs)
    tfidfB = computeTFIDF(tfB, idfs)
    print(pd.DataFrame([tfidfA, tfidfB]))

    ttf = TfidfTransformer(use_idf=True, smooth_idf=False, norm=None)
    f = ttf.fit_transform(cv_ft)
    print(pd.DataFrame(f.todense().tolist(), columns=CV.get_feature_names()))

    print('----------- TfidfVectorizer -------------------')
    vectorizer = TfidfVectorizer(smooth_idf=False, use_idf=True, stop_words=None, token_pattern='(?u)\\b\\w\\w*\\b', norm=None)
    vectors = vectorizer.fit_transform([documentA, documentB])
    feature_names = vectorizer.get_feature_names()
    print(pd.DataFrame(vectors.todense().tolist(), columns=feature_names))


def computeTF(wordDict, bagOfWords):
    tfDict = {}
    bagOfWordsCount = len(bagOfWords)
    for word, count in wordDict.items():
        tfDict[word] = count / float(bagOfWordsCount)
    return tfDict


def computeIDF(documents):
    import math
    N = len(documents)

    idfDict = dict.fromkeys(documents[0].keys(), 0)
    for document in documents:
        for word, val in document.items():
            if val > 0:
                idfDict[word] += 1

    for word, val in idfDict.items():
        idfDict[word] = math.log(N / float(val))
    return idfDict


def computeTFIDF(tfBagOfWords, idfs):
    tfidf = {}
    for word, val in tfBagOfWords.items():
        tfidf[word] = val * idfs[word]
    return tfidf


if __name__ == "__main__":
    main()

我可以比较术语频率的计算。两种结果看起来都一样。但是，当我计算以色列国防军和TF-以色列国防军时，网站上的代码和TfidfVectorizer之间存在差异(我还尝试将CountVectorizer和TfidfTransformer结合起来，以确保它返回与TfidfVectorizer相同的结果)。

编码Tf-以色列国防军结果：

​

​

TfidfVectorizer Tf-以色列国防军结果：

​

​

有人能帮我编写一个代码，它将返回与TfidfVectorizer相同的返回值，或者设置TfidfVectorizer，返回与上面的代码相同的结果吗？

Answer 1

下面是我对您的代码的即兴创作，以便为您的数据再现TfidfVectorizer输出。

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, TfidfTransformer
from IPython.display import display

documentA = 'the man went out for a walk'
documentB = 'the children sat around the fire'
corpus = [documentA, documentB]
bagOfWordsA = documentA.split(' ')
bagOfWordsB = documentB.split(' ')

uniqueWords = set(bagOfWordsA).union(set(bagOfWordsB))

print('----------- compare word count -------------------')
numOfWordsA = dict.fromkeys(uniqueWords, 0)
for word in bagOfWordsA:
    numOfWordsA[word] += 1
numOfWordsB = dict.fromkeys(uniqueWords, 0)
for word in bagOfWordsB:
    numOfWordsB[word] += 1

series_A = pd.Series(numOfWordsA)
series_B = pd.Series(numOfWordsB)
df = pd.concat([series_A, series_B], axis=1).T
df = df.reindex(sorted(df.columns), axis=1)
display(df)

tf_df = df.divide(df.sum(1),axis='index')

n_d = 1+ tf_df.shape[0]
df_d_t = 1 + (tf_df.values>0).sum(0)
idf = np.log(n_d/df_d_t) + 1

pd.DataFrame(df.values * idf,
                  columns=df.columns )

​

​

tfidf = TfidfVectorizer(token_pattern='(?u)\\b\\w\\w*\\b', norm=None)
pd.DataFrame(tfidf.fit_transform(corpus).todense(),
                  columns=tfidf.get_feature_names() )

​

​

有关实现的更多详细信息请参阅文档这里。