GCS路径的Kaggle

Question

在kaggle内核中，我试图在自己的数据集中训练一个神经网络，如下所示：

%%time
history = model.fit(train_dataset, 
                   steps_per_epoch=train_labels.shape[0] // BATCH_SIZE,
                   callbacks=[lr_callback],
                   epochs=EPOCHS,
                   validation_data=valid_dataset)

在启用TPU并设置如下路径之前：

GCS_DS_PATH = KaggleDatasets().get_gcs_path('my-first-data') # you can list the bucket with "!gsutil ls $GCS_DS_PATH"
!gsutil ls $GCS_DS_PATH
clear_output()

我用：

tensorflow版本2.1.0 tensorflow.keras版本2.2.4-tf

然而，这是我收到的错误。之前，我使用kaggle竞赛的数据集运行了完全相同的代码，它运行得非常完美。现在我正在尝试在我自己的数据集中运行它，我得到了这个问题。我的数据具有与竞争数据完全相同的结构，并且错误消息中的Train_17.jpg文件没有丢失(我检查过)。

我想知道，这是否与TPU有关，因为数据是从云桶和我的个人(但公开的！)读取的。kaggle数据集可能不允许这样做？

你有什么建议吗？

Train for 28 steps, validate for 5 steps

Epoch 00001: LearningRateScheduler reducing learning rate to 1e-05.
Epoch 1/40
 1/28 [>.............................] - ETA: 59:20
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<timed exec> in <module>

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    817         max_queue_size=max_queue_size,
    818         workers=workers,
--> 819         use_multiprocessing=use_multiprocessing)
    820 
    821   def evaluate(self,

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    340                 mode=ModeKeys.TRAIN,
    341                 training_context=training_context,
--> 342                 total_epochs=epochs)
    343             cbks.make_logs(model, epoch_logs, training_result, ModeKeys.TRAIN)
    344 

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py in run_one_epoch(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)
    126         step=step, mode=mode, size=current_batch_size) as batch_logs:
    127       try:
--> 128         batch_outs = execution_function(iterator)
    129       except (StopIteration, errors.OutOfRangeError):
    130         # TODO(kaftan): File bug about tf function and errors.OutOfRangeError?

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in execution_function(input_fn)
     96     # `numpy` translates Tensors to values in Eager mode.
     97     return nest.map_structure(_non_none_constant_value,
---> 98                               distributed_function(input_fn))
     99 
    100   return execution_function

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/util/nest.py in map_structure(func, *structure, **kwargs)
    566 
    567   return pack_sequence_as(
--> 568       structure[0], [func(*x) for x in entries],
    569       expand_composites=expand_composites)
    570 

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/util/nest.py in <listcomp>(.0)
    566 
    567   return pack_sequence_as(
--> 568       structure[0], [func(*x) for x in entries],
    569       expand_composites=expand_composites)
    570 

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in _non_none_constant_value(v)
    128 
    129 def _non_none_constant_value(v):
--> 130   constant_value = tensor_util.constant_value(v)
    131   return constant_value if constant_value is not None else v
    132 

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/tensor_util.py in constant_value(tensor, partial)
    820   """
    821   if isinstance(tensor, ops.EagerTensor):
--> 822     return tensor.numpy()
    823   if not is_tensor(tensor):
    824     return tensor

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py in numpy(self)
    940     """
    941     # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.
--> 942     maybe_arr = self._numpy()  # pylint: disable=protected-access
    943     return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr
    944 

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py in _numpy(self)
    908       return self._numpy_internal()
    909     except core._NotOkStatusException as e:
--> 910       six.raise_from(core._status_to_exception(e.code, e.message), None)
    911 
    912   @property

/opt/conda/lib/python3.6/site-packages/six.py in raise_from(value, from_value)

NotFoundError: {{function_node __inference_distributed_function_519795}} Error executing an HTTP request: HTTP response code 404 with body '{
  "error": {
    "code": 404,
    "message": "No such object: kds-f683341923266d33718e6f3ab31b298eb2f954595ee701388c328ce7/images/Train_17.jpg",
    "errors": [
      {
        "message": "No such object: kds-f683341923266d33718e6f3ab31b298eb2f954595ee701388c328ce7/images/Train_17.jpg",
        "domain": "global",
        "reason": "notFound"
      }
    ]
  }
}
'
     when reading metadata of gs://kds-f683341923266d33718e6f3ab31b298eb2f954595ee701388c328ce7/images/Train_17.jpg
     [[{{node ReadFile}}]]
     [[MultiDeviceIteratorGetNextFromShard]]
     [[RemoteCall]]
     [[IteratorGetNextAsOptional]]

Answer 1

我使用了一个不正确的路径来访问数据。我的照片在"my-first-data/plant-pathology-2020-cropped-images/images“里

GCS_DS_PATH = KaggleDatasets().get_gcs_path('my-first-data')
!gsutil ls $GCS_DS_PATH

显示桶包含“植物病理学-2020-裁剪-图像”文件夹。这是如何定义图像的路径，以便将它们传递给模型。

def format_path(st):
    return GCS_DS_PATH + '/plant-pathology-2020-cropped-images/images/' + st + '.jpg'
train_paths = df_train.image_id.apply(format_path).values