是否实现了SOP(句序预测)？

Question

我正在复习huggingface的艾伯特版本。

但是，我找不到任何关于SOP的代码或注释。

我可以从modeling_from src/transformers/modeling_bert.py找到NSP(下一句预测)的实现。

if masked_lm_labels is not None and next_sentence_label is not None:
    loss_fct = CrossEntropyLoss()
    masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
    next_sentence_loss = loss_fct(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
    total_loss = masked_lm_loss + next_sentence_loss
    outputs = (total_loss,) + outputs

SOP是继承自这里的SOP风格标签吗？还是说我漏掉了什么？

Answer 1

句子顺序丢失是here

sentence_order_loss = loss_fn(y_true=sentence_order_label, y_pred=sentence_order_reduced_logits)

这只是一个交叉熵损失。