gcloud与本地培训的效果越来越差

Question

我试图理解为什么我的本地结果要比gcloud结果好。

在当地，我的工作是这样的：

gcloud ml-engine local train --module-name trainer.task --package-path trainer -- --vocabulary-file trainer/data/vocab.txt --class-files $CLASS_FILES --job-dir trainer/lr0001 --num-epochs 5000 --learning-rate 0.0001 --train-batch-size 4 --eval-batch-size 64 --export-format CSV

对于gcloud，我运行

gcloud ml-engine jobs submit training $JOBNAME --job-dir gs://.../lr0001 --module-name trainer.task --package-path trainer --region us-west1 --runtime-version 1.10 -- --vocabulary-file gs://.../vocab.txt --class-files $GS_CLASS_FILES --num-epochs 5000 --learning-rate 0.0001 --train-batch-size 4 --eval-batch-size 64 --export-format CSV

我已经修复了种子，多次运行它，检查python 2和python 3，但是gcloud的结果仍然比我的本地运行更糟糕。

我发现的最后一点线索是本地日志如下所示：

INFO:tensorflow:loss = 0.63639945, step = 401 (0.170 sec)
INFO:tensorflow:global_step/sec: 485.821
INFO:tensorflow:loss = 0.61793035, step = 501 (0.206 sec)
INFO:tensorflow:global_step/sec: 490.795
INFO:tensorflow:loss = 0.5869169, step = 601 (0.204 sec)
INFO:tensorflow:global_step/sec: 619.825
INFO:tensorflow:loss = 0.5738391, step = 701 (0.161 sec)
INFO:tensorflow:global_step/sec: 605.698
INFO:tensorflow:loss = 0.51589084, step = 801 (0.165 sec)

而gcloud日志看起来像是在翻倍什么的

I  master-replica-0 loss = 0.40115586, step = 2202 (0.367 sec) master-replica-0 
I  master-replica-0 global_step/sec: 555.434 master-replica-0 
I  master-replica-0 global_step/sec: 498.601 master-replica-0 
I  master-replica-0 loss = 0.4367655, step = 2402 (0.470 sec) master-replica-0 
I  master-replica-0 global_step/sec: 366.906 master-replica-0 
I  master-replica-0 global_step/sec: 408.556 master-replica-0 
I  master-replica-0 loss = 0.41198668, step = 2602 (0.492 sec) master-replica-0 
I  master-replica-0 global_step/sec: 388.73 master-replica-0 
I  master-replica-0 global_step/sec: 380.982 master-replica-0 
I  master-replica-0 loss = 0.35386887, step = 2802 (0.522 sec) master-replica-0 
I  master-replica-0 global_step/sec: 401.002 master-replica-0 
I  master-replica-0 global_step/sec: 465.647 master-replica-0 
I  master-replica-0 loss = 0.4420835, step = 3002 (0.417 sec) master-replica-0 

任何指点都将不胜感激！

到目前为止，我在互联网上发现的唯一问题是这些问题没有答案：

Results of training a Keras model different on Google Cloud

Differents outputs from predictions using Tensorflow from same data?

Answer 1

经过深入研究，我认为这是由于tensorflow 1.10 (当前的gcloud版本)中的Keras和Estimator之间的一些错误交互，而不是在>=1.11中(我在本地使用的)。

我在这里提交了一份错误报告：https://github.com/tensorflow/tensorflow/issues/24299