火把钩有问题吗？激活映射所有位置

Question

我在看vgg19的激活图。在我应用ReLU之前，我发现所有映射的值都是正的。

我觉得很奇怪..。如果这是正确的(可能是我没有正确地使用register_forward_hook方法？)那么为什么要应用ReLu呢？

这是我的代码，用于生成以下内容：

import torch
import torchvision

import torchvision.models as models
import torchvision.transforms as transforms

from torchsummary import summary


import os, glob
import matplotlib.pyplot as plt
import numpy as np


# settings:
batch_size = 4


# load the model
model = models.vgg19(pretrained=True)
summary(model.cuda(), (3, 32, 32))
model.cpu()

# how to preprocess??? See here:
# https://discuss.pytorch.org/t/how-to-preprocess-input-for-pre-trained-networks/683/2
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])

transform = transforms.Compose(
    [transforms.ToTensor(),
     normalize])




# build data loader
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)



# show one image
dataiter = iter(trainloader)
images, labels = dataiter.next()


# set a hook
activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

# hook at the first conv layer
hook = model.features[0].register_forward_hook(get_activation("firstConv"))

model(images)

hook.remove()



# show results:

flatted_feat_maps = activation["firstConv"].detach().numpy().flatten()

print("All positiv??? --> ",np.all(flatted_feat_maps >= 0))

plt.hist(flatted_feat_maps)
plt.show()

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1           [-1, 64, 32, 32]           1,792
              ReLU-2           [-1, 64, 32, 32]               0
            Conv2d-3           [-1, 64, 32, 32]          36,928
              ReLU-4           [-1, 64, 32, 32]               0
         MaxPool2d-5           [-1, 64, 16, 16]               0
            Conv2d-6          [-1, 128, 16, 16]          73,856
              ReLU-7          [-1, 128, 16, 16]               0
            Conv2d-8          [-1, 128, 16, 16]         147,584
              ReLU-9          [-1, 128, 16, 16]               0
        MaxPool2d-10            [-1, 128, 8, 8]               0
           Conv2d-11            [-1, 256, 8, 8]         295,168
             ReLU-12            [-1, 256, 8, 8]               0
           Conv2d-13            [-1, 256, 8, 8]         590,080
             ReLU-14            [-1, 256, 8, 8]               0
           Conv2d-15            [-1, 256, 8, 8]         590,080
             ReLU-16            [-1, 256, 8, 8]               0
           Conv2d-17            [-1, 256, 8, 8]         590,080
             ReLU-18            [-1, 256, 8, 8]               0
        MaxPool2d-19            [-1, 256, 4, 4]               0
           Conv2d-20            [-1, 512, 4, 4]       1,180,160
             ReLU-21            [-1, 512, 4, 4]               0
           Conv2d-22            [-1, 512, 4, 4]       2,359,808
             ReLU-23            [-1, 512, 4, 4]               0
           Conv2d-24            [-1, 512, 4, 4]       2,359,808
             ReLU-25            [-1, 512, 4, 4]               0
           Conv2d-26            [-1, 512, 4, 4]       2,359,808
             ReLU-27            [-1, 512, 4, 4]               0
        MaxPool2d-28            [-1, 512, 2, 2]               0
           Conv2d-29            [-1, 512, 2, 2]       2,359,808
             ReLU-30            [-1, 512, 2, 2]               0
           Conv2d-31            [-1, 512, 2, 2]       2,359,808
             ReLU-32            [-1, 512, 2, 2]               0
           Conv2d-33            [-1, 512, 2, 2]       2,359,808
             ReLU-34            [-1, 512, 2, 2]               0
           Conv2d-35            [-1, 512, 2, 2]       2,359,808
             ReLU-36            [-1, 512, 2, 2]               0
        MaxPool2d-37            [-1, 512, 1, 1]               0
AdaptiveAvgPool2d-38            [-1, 512, 7, 7]               0
           Linear-39                 [-1, 4096]     102,764,544
             ReLU-40                 [-1, 4096]               0
          Dropout-41                 [-1, 4096]               0
           Linear-42                 [-1, 4096]      16,781,312
             ReLU-43                 [-1, 4096]               0
          Dropout-44                 [-1, 4096]               0
           Linear-45                 [-1, 1000]       4,097,000
================================================================
Total params: 143,667,240
Trainable params: 143,667,240
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.01
Forward/backward pass size (MB): 5.25
Params size (MB): 548.05
Estimated Total Size (MB): 553.31
----------------------------------------------------------------

​

​

可能是因为我没有正确地使用register_forward_hook吗？

Answer 1

您应该将输出clone在

def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach().clone() #
    return hook

请注意，Tensor.detach只从图中分离张量，但两个张量仍然共享相同的底层存储。

返回的张量与原始存储共享相同的存储空间。它们中任何一个的就地修改都会被看到，并可能引发正确性检查中的错误。重要注意事项:以前，对返回的张量的就地大小/ set_ /存储更改(如resize_ / resize_as_ /set_/ transpose_)也会更新原始张量。现在，这些就地更改将不再更新原来的张量，而是会触发一个错误。对于稀疏张量:对返回的张量的原位索引/值更改(如zero_ / copy_ / add_)将不再更新原始张量，而将触发一个错误。