Python矩阵分解

Question

我用python编写了以下代码

###############################################################################

"""
@INPUT:
    R     : a matrix to be factorized, dimension N x M
    P     : an initial matrix of dimension N x K
    Q     : an initial matrix of dimension M x K
    K     : the number of latent features
    steps : the maximum number of steps to perform the optimisation
    alpha : the learning rate
    beta  : the regularization parameter
@OUTPUT:
    the final matrices P and Q
"""
def matrix_factorization(R, P, Q, K, steps=5000, alpha=0.0002, beta=0.02):
    Q = Q.T
    for step in xrange(steps):
        for i in xrange(len(R)):
            for j in xrange(len(R[i])):
                if R[i][j] > 0:
                    eij = R[i][j] - numpy.dot(P[i,:],Q[:,j])
                    for k in xrange(K):
                        P[i][k] = P[i][k] + alpha * (2 * eij * Q[k][j] - beta * P[i][k])
                        Q[k][j] = Q[k][j] + alpha * (2 * eij * P[i][k] - beta * Q[k][j])
        eR = numpy.dot(P,Q)
        e = 0
        for i in xrange(len(R)):
            for j in xrange(len(R[i])):
                if R[i][j] > 0:
                    e = e + pow(R[i][j] - numpy.dot(P[i,:],Q[:,j]), 2)
                    for k in xrange(K):
                        e = e + (beta/2) * ( pow(P[i][k],2) + pow(Q[k][j],2) )
        if e < 0.001:
            break
    return P, Q.T

###############################################################################

指向此代码的链接如下：http://www.quuxlabs.com/blog/2010/09/matrix-factorization-a-simple-tutorial-and-implementation-in-python/

代码对于小矩阵工作正常，但我有两个大矩阵P(15715 ,203)和Q(203,16384)，当我尝试在P和Q上执行这段代码时，它给出了以下错误

K=203

matrix_factorization(R, P, Q, K)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-00b8211f2507> in <module>()
----> 1 matrix_factorization(R, P, Q, K)

/Users/ajinkyachandrakantbobade/Desktop/random_choicefile/trial.py in matrix_factorization(R, P, Q, K, steps, alpha, beta)
     52             for j in xrange(len(R[i])):
     53                 if R[i][j] > 0:
---> 54                     eij = R[i][j] - numpy.dot(P[i,:],Q[:,j])
     55                     for k in xrange(K):
     56                         P[i][k] = P[i][k] + alpha * (2 * eij * Q[k][j] - beta * P[i][k])

/Users/ajinkyachandrakantbobade/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/frame.pyc in __getitem__(self, key)
   1967             return self._getitem_multilevel(key)
   1968         else:
-> 1969             return self._getitem_column(key)
   1970 
   1971     def _getitem_column(self, key):

/Users/ajinkyachandrakantbobade/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/frame.pyc in _getitem_column(self, key)
   1974         # get column
   1975         if self.columns.is_unique:
-> 1976             return self._get_item_cache(key)
   1977 
   1978         # duplicate columns & possible reduce dimensionality

/Users/ajinkyachandrakantbobade/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/generic.pyc in _get_item_cache(self, item)
   1087         """ return the cached item, item represents a label indexer """
   1088         cache = self._item_cache
-> 1089         res = cache.get(item)
   1090         if res is None:
   1091             values = self._data.get(item)

TypeError: unhashable type 

有没有人能帮我解决这个错误？

Answer 1

您尝试相乘的矩阵太大，并且没有足够的内存来完成计算。下面是一些可能会有帮助的事情：

• get more RAM
• 如果你的矩阵包含很多0，你可以尝试使用稀疏矩阵。它们是常规矩阵，只存储显示不为零的值的元素。
• 您似乎已经在使用Python64位，但如果没有，这将比Python32位

工作得更好

Answer 2

您可以使用scipy包(例如scipy.sparse.coo_matrix(arg1，shape，dtype，copy))将矩阵转换为稀疏矩阵。这将允许在更大的数据集上使用MF，而不会遇到计算问题。

为了实现该方法，我使用了Jason Feriante在https://github.com/jferiante/Collaborative-Filtering-Machine-Learning/blob/master/python/nmf-learn.py上解释的非负矩阵分解