huggingface的ReformerForMaskedLM配置问题

Question

我正在尝试将huggingface的所有...ForMaskedLM传递给FitBert模型，用于填空任务，看看哪种预训练对我准备的数据产生了最好的结果。但是在Reformer模块中，这个错误告诉我需要执行'config.is_decoder=False‘，但我并不真正理解这是什么意思(这是我第一次使用huggingface)。我试图向模型传递一个ReformerConfig(is_decoder=False)，但仍然得到相同的错误。我该如何解决这个问题呢？

我的代码：

pretrained_weights = ['google/reformer-crime-and-punishment', 
                      'google/reformer-enwik8']
configurations = ReformerConfig(is_decoder=False)
for weight in pretrained_weights:
    print(weight)
    model = ReformerForMaskedLM(configurations).from_pretrained(weight)
    tokenizer = ReformerTokenizer.from_pretrained(weight)
    fb = FitBert(model=model, tokenizer=tokenizer)
    predicts = []
    for _, row in df.iterrows():
        predicts.append(fb.rank(row['question'], options=[row['1'], row['2'], row['3'], row['4']])[0])
    print(weight,':', np.sum(df.anwser==predicts) / df.shape[0])

错误：

AssertionError                            Traceback (most recent call last)
<ipython-input-5-a6016e0015ba> in <module>()
      4 for weight in pretrained_weights:
      5     print(weight)
----> 6     model = ReformerForMaskedLM(configurations).from_pretrained(weight)
      7     tokenizer = ReformerTokenizer.from_pretrained(weight)
      8     fb = FitBert(model=model, tokenizer=tokenizer)
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1032 
   1033         # Instantiate model.
-> 1034         model = cls(config, *model_args, **model_kwargs)
   1035 
   1036         if state_dict is None and not from_tf:

/usr/local/lib/python3.7/dist-packages/transformers/models/reformer/modeling_reformer.py in __init__(self, config)
   2304         assert (
   2305             not config.is_decoder
-> 2306         ), "If you want to use `ReformerForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention."
   2307         self.reformer = ReformerModel(config)
   2308         self.lm_head = ReformerOnlyLMHead(config)

AssertionError: If you want to use `ReformerForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.

Answer 1

您可以通过单独加载模型配置并将其作为参数提供给from_pretrained()方法来覆盖某些模型配置。这将确保您通过所做的更改使用正确的模型配置：

from transformers import ReformerConfig, ReformerForMaskedLM

config = ReformerConfig.from_pretrained('google/reformer-crime-and-punishment')
print(config.is_decoder)
config.is_decoder=False
print(config.is_decoder)
model = ReformerForMaskedLM.from_pretrained('google/reformer-crime-and-punishment', config=config)

输出：

True
False