如何从TensorFlow的量化感知训练中获得量化权重

Question

我正在使用TensorFlow的量化感知训练API，并希望部署一个具有任意位宽的模型。由于tflite部署只支持8位量化，因此我将使用自定义推理算法进行部署，但我仍然需要访问正确大小的模型权重。

目前，在使用量化感知训练之后，我的模型仍然是浮点型的，就我所见，访问量化权重的唯一方法是将模型转换为tflite格式。但是，当使用实验函数时，这是不可能的。

下面是我的量化配置类：

    class Quantizer(tfmot.quantization.keras.QuantizeConfig):
    # Configure how to quantize weights.
    def get_weights_and_quantizers(self, layer):
        return [(layer.kernel, tfmot.quantization.keras.quantizers.LastValueQuantizer(num_bits=8, symmetric=True, narrow_range=False, per_axis=False))]

    # Configure how to quantize activations.
    def get_activations_and_quantizers(self, layer):
        return [(layer.activation, tfmot.quantization.keras.quantizers.MovingAverageQuantizer(num_bits=8, symmetric=False, narrow_range=False, per_axis=False))]

    def set_quantize_weights(self, layer, quantize_weights):
        # Add this line for each item returned in `get_weights_and_quantizers`
        # , in the same order
            layer.kernel = quantize_weights[0]

    def set_quantize_activations(self, layer, quantize_activations):
        # Add this line for each item returned in `get_activations_and_quantizers`
        # , in the same order.
        layer.activation = quantize_activations[0]

    # Configure how to quantize outputs (may be equivalent to activations).
    def get_output_quantizers(self, layer):
        return []

    def get_config(self):
        return {}
    
class ModifiedQuantizer(Quantizer):
    # Configure weights to quantize with 4-bit instead of 8-bits.
    def get_weights_and_quantizers(self, layer):
        return [(layer.kernel, quantizer(num_bits=bits, symmetric=symmetric, narrow_range=narrow_range, per_axis=per_axis))]

下面是我量化模型的方法：

    supported_layers = [
    tf.keras.layers.Conv2D,
    tf.keras.layers.DepthwiseConv2D
]

class Quantizer(tfmot.quantization.keras.QuantizeConfig):
    # Configure how to quantize weights.
    def get_weights_and_quantizers(self, layer):
        return [(layer.kernel, tfmot.quantization.keras.quantizers.LastValueQuantizer(num_bits=8, symmetric=True, narrow_range=False, per_axis=False))]

    # Configure how to quantize activations.
    def get_activations_and_quantizers(self, layer):
        return [(layer.activation, tfmot.quantization.keras.quantizers.MovingAverageQuantizer(num_bits=8, symmetric=False, narrow_range=False, per_axis=False))]

    def set_quantize_weights(self, layer, quantize_weights):
        # Add this line for each item returned in `get_weights_and_quantizers`
        # , in the same order
            layer.kernel = quantize_weights[0]

    def set_quantize_activations(self, layer, quantize_activations):
        # Add this line for each item returned in `get_activations_and_quantizers`
        # , in the same order.
        layer.activation = quantize_activations[0]

    # Configure how to quantize outputs (may be equivalent to activations).
    def get_output_quantizers(self, layer):
        return []

    def get_config(self):
        return {}
    
class ModifiedQuantizer(Quantizer):
    # Configure weights to quantize with 4-bit instead of 8-bits.
    def get_weights_and_quantizers(self, layer):
        return [(layer.kernel, quantizer(num_bits=bits, symmetric=symmetric, narrow_range=narrow_range, per_axis=per_axis))]
    
    # Configure how to quantize activations.
    def get_activations_and_quantizers(self, layer):
        return [(layer.activation, tfmot.quantization.keras.quantizers.MovingAverageQuantizer(num_bits=bits, symmetric=False, narrow_range=False, per_axis=False))]

    def quantize_all_layers(layer):
        for supported_layer in supported_layers:
            if isinstance(layer, supported_layer):
                return quantize_annotate_layer(layer, quantize_config=ModifiedQuantizer())
        # print(layer.name)
        return layer
    annotated_model = clone_model(
        model,
        clone_function=quantize_all_layers
    )

with quantize_scope(
    {'Quantizer': Quantizer},
    {'ModifiedQuantizer': ModifiedQuantizer},
    {'_relu6': models._relu6}):
    q_aware_model = quantize_apply(annotated_model)

optimizer = keras.optimizers.Adam(
    learning_rate=0.001)
q_aware_model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=True),
    optimizer=optimizer, metrics=['sparse_categorical_accuracy'])

train_images, train_labels, val_images, val_labels, _, _ = cifar10.load()

q_aware_model.fit(train_images, train_labels, batch_size=64, epochs=1, verbose=1,
                  validation_data=(val_images, val_labels))

以前说过，当在ModifiedQuantizer中使用bits=4时，模型仍然是以浮点形式保存的，我不知道如何访问量化的权重。

谢谢!

Answer 1

我怀疑您可以通过对给定层的权重张量调用LastValueQuantizer.__call__来获得量化的权重。如何调用该方法是一个问题。

当前签名为：

    LastValueQuantizer.__call__(inputs, training, weights, **kwargs)

我假设inputs是层的权重，weights是LastValueQuantizer.build返回的值。如果您可以获得build返回的weights的引用，我希望直接使用LastValueQuantizer.__call__量化该层的权重。

[nav] In [1]: from tensorflow_model_optimization.quantization.keras.quantizers import LastValueQuantizer
INFO:tensorflow:Enabling eager execution
INFO:tensorflow:Enabling v2 tensorshape
INFO:tensorflow:Enabling resource variables
INFO:tensorflow:Enabling tensor equality
INFO:tensorflow:Enabling control flow v2

[nav] In [2]: q = LastValueQuantizer(num_bits=3, per_axis=True, symmetric=True, narrow_range=True)

[ins] In [3]: ??q.__call__
Signature: q.__call__(inputs, training, weights, **kwargs)
Source:   
  def __call__(self, inputs, training, weights, **kwargs):
    """Quantize tensor.

    Args:
      inputs: Input tensor to be quantized.
      training: Whether the graph is currently training.
      weights: Dictionary of weights the quantizer can use to quantize the
        tensor. This contains the weights created in the `build` function.
      **kwargs: Additional variables which may be passed to the quantizer.

    Returns:
      Quantized tensor.
    """
    return quant_ops.LastValueQuantize(
        inputs,
        weights['min_var'],
        weights['max_var'],
        is_training=training,
        num_bits=self.num_bits,
        per_channel=self.per_axis,
        symmetric=self.symmetric,
        narrow_range=self.narrow_range
    )