持久化sklearn.feature_extraction.text.TfidfVectorizer出错

Question

我使用模块joblib持久化了一个TfidfVectorizer。我通过方法fit_transform运行的对象是一个字符串列表。得到的矩阵具有263744列的维度。

我通过transform方法运行一个字符串列表，并得到以下错误。

有什么线索吗？

File "/usr/local/lib/python2.7/dist-      packages/sklearn/feature_extraction/text.py", 
line 1334, in transform
return self._tfidf.transform(X, copy=False)
File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/text.py", 
line 1037, in transform
X = X * self._idf_diag

File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", line    
318, in __mul__
return self._mul_sparse_matrix(other)
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py",
line 487, in _mul_sparse_matrix
other = self.__class__(other)  # convert to this format
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py",
line 31, in __init__
arg1 = arg1.asformat(self.format)
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", 
line 219, in asformat
return getattr(self,'to' + format)()
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/dia.py", 
line 241, in tocsr
return self.tocoo().tocsr()
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/dia.py", 
line 249, in tocoo

num_offsets, offset_len = self.data.shape
AttributeError: 'NDArrayWrapper' object has no attribute 'shape'

Answer 1

假设您正在将经过训练的转换器或管道持久化到磁盘，然后在看到错误之前重新加载它，您可以：

1. 尝试使用compress关键字参数将原始(工作)对象保存到joblib.dump，并使用大于0的整数值：

_= joblib.dump(python_object，persisted_file_name，compress=3)

• 如果要将持久化文件移动到新位置，请确保复制所有文件片段。如果它很大，joblib会将其拆分，例如：

persisted_model.joblib.pkl persisted_model.joblib.pkl_01.npy persisted_model.joblib.pkl_02.npy

joblib docs