用word2vec算法实现意想不到的结果

Question

我在word2vec中实现了c++。我发现原来的语法不太清楚，所以我想我会重新实现它，使用c++的所有优点(std::map，std：：载体，等等)

这是每次训练样本时实际调用的方法(l1表示第一个单词的索引，l2表示第二个单词的索引，标签表示它是正的还是负的，neu1e作为梯度的累加器)。

void train(int l1, int l2, double label, std::vector<double>& neu1e)
{
        // Calculate the dot-product between the input words weights (in 
        // syn0) and the output word's weights (in syn1neg).
        auto f = 0.0;

        for (int c = 0; c < m__numberOfFeatures; c++) 
            f += syn0[l1][c] * syn1neg[l2][c];

      // This block does two things:
      //   1. Calculates the output of the network for this training
      //      pair, using the expTable to evaluate the output layer
      //      activation function.
      //   2. Calculate the error at the output, stored in 'g', by
      //      subtracting the network output from the desired output, 
      //      and finally multiply this by the learning rate.
      auto z = 1.0 / (1.0 + exp(-f));
      auto g = m_learningRate * (label - z);

      // Multiply the error by the output layer weights.
      // (I think this is the gradient calculation?)
      // Accumulate these gradients over all of the negative samples.
      for (int c = 0; c < m__numberOfFeatures; c++) 
        neu1e[c] += (g * syn1neg[l2][c]);    

      // Update the output layer weights by multiplying the output error
      // by the hidden layer weights.
      for (int c = 0; c < m__numberOfFeatures; c++) 
        syn1neg[l2][c] += g * syn0[l1][c];         
}

此方法由

void train(const std::string& s0, const std::string& s1, bool isPositive, std::vector<double>& neu1e)
    {
        auto l1 = m_wordIDs.find(s0) != m_wordIDs.end() ? m_wordIDs[s0] : -1;
        auto l2 = m_wordIDs.find(s1) != m_wordIDs.end() ? m_wordIDs[s1] : -1;
        if(l1 == -1 || l2 == -1)
            return;

        train(l1, l2, isPositive ? 1 : 0, neu1e);
    }

然后由主要的训练方法调用。

完整的代码可在

https://github.com/jorisschellekens/ml/tree/master/word2vec

具有完整示例的

8.hpp

当我运行这个算法时，最接近father的前10个单词是：

父亲 可汗 沙阿 健忘 迈阿密 皮疹 病征 丧葬 印第安纳波利斯 印象深刻

这是计算最近单词的方法：

std::vector<std::string> nearest(const std::string& s, int k) const
    {
        // calculate distance
        std::vector<std::tuple<std::string, double>> tmp;
        for(auto &t : m_unigramFrequency)
        {
            tmp.push_back(std::make_tuple(t.first, distance(t.first, s)));
        }

        // sort
        std::sort(tmp.begin(), tmp.end(), [](const std::tuple<std::string, double>& t0, const std::tuple<std::string, double>& t1)
        {
            return std::get<1>(t0) < std::get<1>(t1);
        });

        // take top k
        std::vector<std::string> out;
        for(int i=0; i<k; i++)
        {
            out.push_back(std::get<0>(tmp[tmp.size() - 1 - i]));
        }

        // return
        return out;
    }

这似乎很奇怪。我的算法有问题吗？

Answer 1

你确定，你得到了“最近”的单词(不是最直接的)吗？

        ...
        // take top k
        std::vector<std::string> out;
        for(int i=0; i<k; i++)
        {
            out.push_back(std::get<0>(tmp[tmp.size() - 1 - i]));
        }
        ...