码头集装箱装载精调简化变压器模型时的误差

Question

我使用torch.save()和torch.load()命令保存和加载模型。

在Docker容器中加载一个微调的简单变压器模型时，我面临着无法解决的错误：

 Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/xlm_roberta/tokenization_xlm_roberta.py", line 161, in __setstate__
    self.sp_model.Load(self.vocab_file)
  File "/usr/local/lib/python3.7/dist-packages/sentencepiece.py", line 367, in Load
    return self.LoadFromFile(model_file)
  File "/usr/local/lib/python3.7/dist-packages/sentencepiece.py", line 177, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
OSError: Not found: "/home/jupyter/.cache/huggingface/transformers/9df9ae4442348b73950203b63d1b8ed2d18eba68921872aee0c3a9d05b9673c6.00628a9eeb8baf4080d44a0abe9fe8057893de20c7cb6e6423cddbf452f7d4d8": No such file or directory Error #2

如果有人知道这件事，请告诉我。

我正在使用：

==1.7.1+cu101

• sentence-transformers 0.3.9

• simpletransformers 0.51.15

• transformers 4.4.2

• tensorflow 2.2.0

Answer 1

我建议使用state_dict对象-- Python，因为它们可以很容易地保存、更新和恢复，从而为以后恢复模型提供了一种灵活性。下面是使用state_dict保存模型的推荐保存/加载方法

Save

torch.save(model.state_dict(), PATH)

负载

model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()