Pytorch:理解内联替换发生的困难

Question

这似乎是人们经常会犯的错误，但我不能真正理解真正的原因。我很难弄清楚内联替换是在哪里发生的。我的前进职能：

def forward(self, input, hidden=None):
    if hidden is None :
        hidden = self.init_hidden(input.size(0))
    out, hidden = self.lstm(input, hidden)
    out = self.linear(out)
    return out, hidden

训练回路

def training(dataloader, iterations,  device):
    torch.autograd.set_detect_anomaly(True)
    model = NModel(662, 322, 2, 1)
    hidden = None
    model.train()
    loss_fn = F.MSELoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
    running_loss =  []
    last_loss = 0
    for i, (feature, label) in tqdm(enumerate(dataloader)):
        optimizer.zero_grad()
        outputs, hidden = model(feature, hidden)
        loss =  loss_fn(outputs, label)
        print("loss item" , loss.item())
        running_loss.append(loss.item())
        loss.backward(retain_graph=True)
        optimizer.step()
        if i%1000 == 0:
            last_loss = len(running_loss) /1000
    return last_loss

错误的堆栈跟踪

Traceback (most recent call last):
  File "main.py", line 18, in <module>
    main()
  File "main.py", line 14, in main
    training(dataloader=training_loader, iterations=3, device=0)
  File "/home//gitclones/feature-extraction/training.py", line 30, in training
    loss.backward(retain_graph=True)
  File "/home/miniconda3/envs/pytorch-openpose/lib/python3.7/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home//miniconda3/envs/pytorch-openpose/lib/python3.7/site-packages/torch/autograd/__init__.py", line 156, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [322, 1288]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

当我删除optimizer.step()时，代码会运行，但我认为不会发生反向传播。

编辑奇怪，现在它是如何工作的，当我不把隐藏状态作为输入传递给前向传递时

def forward(self, input, hidden=None):
    if hidden is None :
        hidden = self.init_hidden(input.size(0))
    out, hidden = self.lstm(input)
    out = self.linear(out)
    return out, hidden

Answer 1

在hidden = tuple([each.data for each in hidden])修复错误后添加optimizer.step()，但将渐变设置为隐藏值。您可以使用hidden = tuple([each.detach() for each in hidden])实现相同的效果。