如何在RoBERTa中使用strategy.scope()？

Question

我用BERT做了一个模型，对于一个NLI问题，算法运行没有问题，但是，当我想让它适应RoBERTa时，我使用strategy.scope ()，它产生了一个我不知道如何解决的错误，我感谢任何指示。

´´´
max_len1 = 515 # 128*4 de premisa mas 128*4 de hipotesis
def build_model1():
    input_word_ids = tf.keras.Input(shape=(max_len1,), dtype=tf.int32,name="input_word_ids")
    input_mask = tf.keras.Input(shape = (max_len1,),dtype=tf.int32,name = "input_mask")
    input_type_ids = tf.keras.Input(shape = (max_len1,),dtype=tf.int32,name="input_type_ids")
    
    embedding = model([input_word_ids,input_mask,input_type_ids])[0]
    output = tf.keras.layers.Dense(3,activation='softmax')(embedding[:,0,:])
    
    model3 = tf.keras.Model(inputs=[input_word_ids, input_mask, input_type_ids], outputs=output)
    model3.compile(tf.keras.optimizers.Adam(lr=1e-5),
                 loss = 'sparse_categorical_crossentropy', metrics= ['accuracy'])
    return model3
with strategy.scope():
    model3 = build_model1()
    model3.summary()
WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f2425631d00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f2425631d00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f2425631d00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:AutoGraph could not transform <function wrap at 0x7f243c214d40> and will run it as-is.
Cause: while/else statement not yet supported
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <function wrap at 0x7f243c214d40> and will run it as-is.
Cause: while/else statement not yet supported
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
WARNING: AutoGraph could not transform <function wrap at 0x7f243c214d40> and will run it as-is.
Cause: while/else statement not yet supported
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-24-e91a2e7e4b41> in <module>()
      1 with strategy.scope():
----> 2     model3 = build_model1()
      3     model3.summary()

2 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py in _validate_compile(self, optimizer, metrics, **kwargs)
   2533               'with strategy.scope():\n'
   2534               '  model=_create_model()\n'
-> 2535               '  model.compile(...)' % (v, strategy))
   2536 
   2537     # Model metrics must be created in the same distribution strategy scope

ValueError: Variable (<tf.Variable 'tfxlm_roberta_model/roberta/encoder/layer_._0/attention/self/query/kernel:0' shape=(1024, 1024) dtype=float32, numpy=
array([[-0.00294119, -0.00129846,  0.00517603, ...,  0.03835522,
         0.0218797 ,  0.02100084],
       [-0.00933813, -0.05062149,  0.01634834, ..., -0.02387142,
         0.0113477 , -0.02262339],
       [-0.02023344, -0.04181184, -0.00581416, ..., -0.00609464,
         0.00801133,  0.00512151],
       ...,
       [-0.02129102, -0.03157991, -0.04071935, ...,  0.04682101,
         0.01948426,  0.00312433],
       [-0.04902648, -0.01055507,  0.01377375, ...,  0.00845209,
         0.01616496, -0.01041171],
       [ 0.00759454, -0.00162496, -0.00215843, ..., -0.03199947,
        -0.03871808,  0.04949447]], dtype=float32)>) was not created in the distribution strategy scope 
    of (<tensorflow.python.distribute.tpu_strategy.TPUStrategy object at 0x7f21fcbbb210>). It is most 
    likely due to not all layers or the model or optimizer being created outside the distribution 
    strategy scope. Try to make sure your code looks similar to the following.
    with strategy.scope():
  model=_create_model()
  model.compile(...)
´´´

正如我上面所说的，相同的代码对于BERT非常有效，显然，对于RoBERTa，我对记号赋予器和模型的加载进行了更改

Answer 1

我设法解决了这个问题，进行了调查，我发现roberta的实现不仅仅是调用模型