不能在训练中改变learning_rate --流动优化器

Question

有人能解释我为什么在训练中不能改变learningrate吗?在老的优化器中，我可以用self.updates.append(K.update(self.learning_rate, new_learning_rate))来改变它，但是不能再做了，self._set_hyper("learning_rate", new_learning_rate)不能工作了，它告诉我：TypeError:__array__() takes 1 positional argument but 2 were given

class SGD(keras.optimizers.Optimizer):
    def __init__(self, learning_rate=0.01, name="SGD", **kwargs):
        """Call super().__init__() and use _set_hyper() to store hyperparameters"""
        super().__init__(name, **kwargs)
        self._set_hyper("learning_rate", kwargs.get("lr", learning_rate)) # handle lr=learning_rate
        
    @tf.function
    def _resource_apply_dense(self, grad, var):
        learning_rate = self._get_hyper("learning_rate")
        new_learning_rate = learning_rate * 0.001
        new_var_m = var - grad * new_learning_rate
        self._set_hyper("learning_rate", new_learning_rate) #dont work
        var.assign(new_var)

    def _resource_apply_sparse(self, grad, var):
        raise NotImplementedError

    def get_config(self):
        base_config = super().get_config()
        return {
            **base_config,
            "learning_rate": self._serialize_hyperparameter("learning_rate"),
        }

Answer 1

作为一种解决方法，您可以使用_set_hyper直接访问属性集，如文档中提到的那样：

可以通过用户代码覆盖超级参数

由于它是一个tf.Variable，因此可以使用assign来使用tf.Tensor设置一个新值：

class SGD(keras.optimizers.Optimizer):
    def __init__(self, learning_rate=0.01, name="SGD", **kwargs):
        """Call super().__init__() and use _set_hyper() to store hyperparameters"""
        super().__init__(name, **kwargs)
        self._set_hyper("learning_rate", kwargs.get("lr", learning_rate)) # handle lr=learning_rate
        
    @tf.function
    def _resource_apply_dense(self, grad, var):
        learning_rate = self._get_hyper("learning_rate")
        new_learning_rate = learning_rate * 0.001
        new_var_m = var - grad * new_learning_rate
        self.learning_rate.assign(new_learning_rate)
        var.assign(new_var_m)