在Tensorflow中使用bucketing时，如何在Adam优化器中共享梯度和变量？

Question

全,

我在seq2seq任务中使用了类似于bucketing的技术：

# For different length in encoder and decoder
model_map = {}
for i in encoder_shape:
    for j in decoder_shape:
        with variable_scope.variable_scope(variable_scope.get_variable_scope(),
                                 reuse=True if tt > 0 else None):
            model = Seq2SeqModel()
            model.build(encoder[:i], decoder[:j])
            model_map[i*100+j] = model

并获取共享模型的参数：

for t in tf.all_variables():
    print t.name, t.get_shape() 

Print: 
embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding:0 (50000, 256)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias:0 (1600,)

Model的优化器如下所示：

#every model have an optimizer
params = tf.trainable_variables()
opt = tf.train.AdamOptimizer(1e-3)
gradients = tf.gradients(self.loss, params)
self.optimizer = opt.apply_gradients(zip(gradients, params))

但是我发现优化器并不共享变量：

embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding/Adam:0 (50000, 256)
embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding/Adam_1:0 (50000, 256)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix/Adam:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix/Adam_1:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias/Adam:0 (1600,)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias/Adam_1:0 (1600,)
embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding/Adam_2:0 (50000, 256)
embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding/Adam_3:0 (50000, 256)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix/Adam_2:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix/Adam_3:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias/Adam_2:0 (1600,)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias/Adam_3:0 (1600,)

随着存储桶数量的增长，GPU内存也会随之增长。同时，我在tf.train.Saver.save()中得到了一个更大的模型。

那么，在tensorflow中共享bucketing中的梯度是可能的吗？

Answer 1

我相信在你所有的模型中共享一个优化器实例将会做你想要的事情。