AttributeError:格式未找到- pyodide + joblib.dump +scikit-学习(TfidfVectorizer)

Question

我已经腌制了一个短信垃圾邮件预测模型使用泡菜。现在，我想使用Pyodide在浏览器中加载模型。

我已经在浏览器中使用pickle.loads加载了这个被腌制的文件：

console.log("Pyodide loaded, downloading pretrained ML model...")
const model = (await blobToBase64(await (await fetch("/model.pkl")).blob())).replace("data:application/octet-stream;base64,", "")
console.log("Loading model into Pyodide...")
await pyodide.loadPackage("scikit-learn")
await pyodide.loadPackage("joblib")
pyodide.runPython(`
    import base64
    import pickle
    from io import BytesIO
    classifier, vectorizer = pickle.loads(base64.b64decode('${model}'))
`)

这个很管用。

但是，当我试图打电话：

const prediction = pyodide.runPython(`
    vectorized_message = vectorizer.transform(["Call +172949 if you want to get $1000 immediately!!!!"])
    classifier.predict(vectorized_message)[0]
`)

它给出了一个错误(在vectorizer.transform中)：AttributeError: format not found

完全错误转储如下。

Uncaught (in promise) Error: Traceback (most recent call last):
  File "/lib/python3.8/site-packages/pyodide/_base.py", line 70, in eval_code
    eval(compile(mod, "<exec>", mode="exec"), ns, ns)
  File "<exec>", line 2, in <module>
  File "/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 1899, in transform
    return self._tfidf.transform(X, copy=False)
  File "/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 1513, in transform
    X = X * self._idf_diag
  File "/lib/python3.8/site-packages/scipy/sparse/base.py", line 319, in __mul__
    return self._mul_sparse_matrix(other)
  File "/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 478, in _mul_sparse_matrix
    other = self.__class__(other)  # convert to this format
  File "/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 28, in __init__
    if arg1.format == self.format and copy:
  File "/lib/python3.8/site-packages/scipy/sparse/base.py", line 525, in __getattr__
    raise AttributeError(attr + " not found")
AttributeError: format not found

    _hiwire_throw_error https://cdn.jsdelivr.net/pyodide/v0.16.1/full/pyodide.asm.js:8
    __runPython https://cdn.jsdelivr.net/pyodide/v0.16.1/full/pyodide.asm.js:8
    _runPythonInternal https://cdn.jsdelivr.net/pyodide/v0.16.1/full/pyodide.asm.js:8
    runPython https://cdn.jsdelivr.net/pyodide/v0.16.1/full/pyodide.asm.js:8
    <anonymous> http://localhost/:41
    async* http://localhost/:46
pyodide.asm.js:8:39788

但是，在Python中，它工作得很好。

我可能做错了什么？

Answer 1

这可能是一个便携性问题。Pickles在体系结构之间应该是可移植的，这里是amd64和wasm32，但是是它们不能跨包版本进行移植。。这意味着包版本应该在您训练模型的环境和进行推断的环境(pyodide)之间是相同的。

pyodide 0.16.1包括Python3.8.2、ciply0.17.1和scikit-Learch0.22.2。不幸的是，这意味着您必须从源代码构建那个版本的3.8 (可能还有numpy)来训练模型，因为Python3.8二进制轮并不存在于这样一个过时版本的3.8中。将来，这个问题应该用pyodide#1293来解决。

您所得到的特殊错误可能是由于scipy.sparse版本的模拟(请参阅scipy#6533 )。

尽管如此，基于树的模型在scikit学习中目前还不能跨架构移植，因此在pyodide中也不会出现问题。这是应该修复的已知错误(科学知识-学习#19602)