附加培训NLP

Question

我开始使用Camembert深度学习模型，这是罗伯塔在法语中的类比，我有一个问题，我如何为特定的任务重新训练这样的模型？具体地说，模型的任务是学习如何评估输入句子的正确性

class newModel(nn.Module):
    def __init__(self, numFeatures=768):
        super(newModel, self).__init__()
        self.camembert = CamembertModel.from_pretrained('camembert-base')
        self.GAP = nn.AdaptiveAvgPool2d((1, numFeatures))
        self.predictionLayer = nn.Linear(numFeatures, 2)
        self.softmax = nn.Softmax()

    def forward(self, x):
        camembertFeatures = self.camembert(**tokenized_sentence) # [BS, inputShape] -> [BS, numToken, numFeatures]
        camembertFeatures = camembertFeatures[0]
        GAPvalues = self.GAP(camembertFeatures) # [BS, numToken, numFeatures] -> [BS, 1, numFeatures]
        GAPshape = GAPvalues.shape
        sentenceFeatures = GAPvalues.view(GAPshape[0], GAPshape[2]) # [BS, 1, numFeatures] -> [BS, numFeatures]
        predictions = self.predictionLayer(sentenceFeatures) # [BS, numFeatures] -> [BS, 2]
        predictions = self.softmax(predictions) # [BS, 2] -> [BS, 2]
        return predictions

我已经构建了一个层，需要训练，怎么做才是正确的？(了解使用哪个优化器和损失函数尤其重要)

Answer 1

如果您已经在camembert上构建了一个层，即您正在尝试对其进行微调，您可以遵循微调的标准实践，即冻结预先训练的大型模型，并仅优化在其上构建的层。我相信对于初学者来说，在它上面安装一个MLP就可以了。

在正确的情况下(我相信这是一项分类任务)，您可以使用Crossentropy loss函数和adam优化器