初始v3再培训错误(flower示例)

Question

我最近遇到了一个奇怪的bug，比如花再训练示例(https://www.tensorflow.org/versions/r0.9/how_tos/image_retraining/index.html)。

Tensorflow 0.9版是从源代码安装的，我尝试运行Tensorflow python脚本(它确实会启动并创建一些瓶颈，但随后会出现以下错误消息)。

有人知道问题出在哪里吗？我没有找到任何类似的帖子。

E tensorflow/core/kernels/check_numerics_op.cc:157] abnormal_detected_host @0x10007200300 = {1, 0} activation input is not finite.
Traceback (most recent call last):
  File "examples/image_retraining/retrain.py", line 888, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "examples/image_retraining/retrain.py", line 798, in main
    jpeg_data_tensor, bottleneck_tensor)
  File "examples/image_retraining/retrain.py", line 456, in cache_bottlenecks
    jpeg_data_tensor, bottleneck_tensor)
  File "examples/image_retraining/retrain.py", line 414, in get_or_create_bottleneck
    bottleneck_tensor)
  File "examples/image_retraining/retrain.py", line 331, in run_bottleneck_on_image
    {image_data_tensor: image_data})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: activation input is not finite. : Tensor had NaN values
         [[Node: conv_1/CheckNumerics = CheckNumerics[T=DT_FLOAT, message="activation input is not finite.", _device="/job:localhost/replica:0/task:0/gpu:0"](conv_1/batchnorm)]]
Caused by op u'conv_1/CheckNumerics', defined at:
  File "examples/image_retraining/retrain.py", line 888, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "examples/image_retraining/retrain.py", line 769, in main
    create_inception_graph())
  File "examples/image_retraining/retrain.py", line 312, in create_inception_graph
    RESIZED_INPUT_TENSOR_NAME]))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 274, in import_graph_def
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2297, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1231, in __init__
    self._traceback = _extract_stack()

Answer 1

更新:只是为了跟进，建议使用Tensorflow 1.6，因为许多操作都要快得多。如果您运行的是Nvidia GPU，请确保您安装的是CUDA 9.0而不是9.1，9.1会破坏一切。

对于cuDNN，它需要同时匹配CUDA9.0和构建Tensorflow的版本。对于TensorFlow1.6，请确保安装版本7.0.4，而不是7.1，以及构建1.6时使用的特定版本(否则，它也会崩溃)：确切的版本是CUDA9.0的cuDNN v7.0.4.31-1 (而不是9.1)。最新版本(此时为7.1.2)将抛出错误，因为Tensorflow 1.6是使用7.0.4构建的

这是我在TensorFlow中遇到的一个bug (我在Ubuntu14.04中使用2xGTX1080)

一种选择是安装Cuda 8.0。但是，Cuda 8.0并不完全受支持，您可能会遇到其他问题。

如果您只是在实验，另一种解决方法是构建它，并仅在CPU上运行它，至少在瓶颈生成阶段是这样。

bazel build -c opt --copt=-mavx tensorflow/examples/image_retraining:retrain
bazel-bin/tensorflow/examples/image_retraining/retrain --image_dir ~/flower_photos

正如您可能知道的，如果您已经构建了支持GPU的TensorFlow，然后运行以下命令：

python tensorflow/examples/image_retraining/retrain.py --image_dir ~/flower_photos

它将在GPU支持下运行，然后您可能会遇到相同的错误。

我在这里打开了一个问题：https://github.com/tensorflow/tensorflow/issues/3560

在他们解决这个问题之前，只要你没有大量的类别需要分类，这个变通方法就会起作用。