向dataset.interleave迁移TF.data

Question

TensorFlow nightly：1.15.0-dev20190730

filenames = tf.gfile.Glob(data_files_pattern)
dataset = tf.data.Dataset.from_tensor_slices(filenames).repeat()

def _read_fn(f):
  return tf.data.TFRecordDataset(f)

dataset = dataset.apply(tf.data.experimental.parallel_interleave(
    map_func=_read_fn,
    cycle_length=CYCLE_LENGTH,
    block_length=BLOCK_LENGTH,
    sloppy=True,
    buffer_output_elements=BUFFER_OUTPUT_ELEMENTS,
    prefetch_input_elements=BUFFER_INPUT_ELEMENTS))
dataset = dataset.batch(BATCH_SIZE, drop_remainder=False)
dataset = dataset.prefetch(PREFETCH)
return dataset

我得到以下警告：

WARNING:tensorflow:From sample.py:35: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
W0909 06:50:51.144233 140600866592512 deprecation.py:323] From sample.py:35: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.

当我为了避免警告而进行迁移时，我的读取速度会变慢，CPU使用率也会下降：

filenames = tf.gfile.Glob(data_files_pattern)
dataset = tf.data.Dataset.from_tensor_slices(filenames).repeat()

def _read_fn(f):
   return tf.data.TFRecordDataset(f)

options = tf.data.Options()
options.experimental_deterministic = True
dataset = dataset.interleave(
    map_func=_read_fn,
    cycle_length=CYCLE_LENGTH,
    block_length=BLOCK_LENGTH,      
    num_parallel_calls=tf.data.experimental.AUTOTUNE).with_options(options)
dataset = dataset.batch(BATCH_SIZE, drop_remainder=False)
dataset = dataset.prefetch(PREFETCH)
return dataset

我的迁移是否正确？

Answer 1

问题是，您正在将草率(非确定性) parallel_interleave与确定性interleave进行比较。您为parallel_interleave设置了sloppy=True，因此为了正确迁移，您需要设置

options.experimental_deterministic = False

为了interleave。