TensorFlow量化感知训练量化节点附加参数的研究

Question

目前，我正试图理解量化感知的TensorFlow培训。据我所知，伪量化节点需要收集动态范围信息作为量化操作的校准。当我将同一模型与“普通”Keras模型和一次量化感知模型进行比较时，后者具有更多的参数，这是有意义的，因为我们需要在量化感知训练中存储激活的最小值和最大值。

请考虑以下示例：

import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras.models import Model
def get_model(in_shape):
  inpt = layers.Input(shape=in_shape)
  dense1 = layers.Dense(256, activation="relu")(inpt)
  dense2 = layers.Dense(128, activation="relu")(dense1)
  out = layers.Dense(10, activation="softmax")(dense2)

  model = Model(inpt, out)

  return model

该模型有以下概述：

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 784)]             0         
_________________________________________________________________
dense_3 (Dense)              (None, 256)               200960    
_________________________________________________________________
dense_4 (Dense)              (None, 128)               32896     
_________________________________________________________________
dense_5 (Dense)              (None, 10)                1290      
=================================================================
Total params: 235,146
Trainable params: 235,146
Non-trainable params: 0
_________________________________________________________________

但是，如果我使我的模型优化意识到，它会打印以下摘要：

import tensorflow_model_optimization as tfmot

quantize_model = tfmot.quantization.keras.quantize_model

# q_aware stands for for quantization aware.
q_aware_model = quantize_model(standard)

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 784)]             0         
_________________________________________________________________
quantize_layer (QuantizeLaye (None, 784)               3         
_________________________________________________________________
quant_dense_3 (QuantizeWrapp (None, 256)               200965    
_________________________________________________________________
quant_dense_4 (QuantizeWrapp (None, 128)               32901     
_________________________________________________________________
quant_dense_5 (QuantizeWrapp (None, 10)                1295      
=================================================================
Total params: 235,164
Trainable params: 235,146
Non-trainable params: 18
_________________________________________________________________

我特别有两个问题：

1. ，在输入层之后有3个参数的quantize_layer的目的是什么？
2. ，为什么每个层有5个额外的不可训练参数，它们到底用于什么？

。

我很感激任何帮助我(和其他偶然发现这个问题的人)理解量化感知训练的提示或进一步的材料。

Answer 1

1. 量化层用于将浮点输入转换为int8。量化参数用于输出min/max和零点calculations.
2. Quantized稠密层，需要一些附加参数:内核为min/max，输出激活为min/max/零点。