使用Keras序列的keras DataGenerator

Question

我试着提高我的模型的训练速度。我做了一些预处理和增强(它运行在CPU)，使我的训练缓慢。因此，我尝试在keras Sequence中实现数据的加载和预处理。为此，我跟踪了角蛋白文档和这个斯坦福出版社。到目前为止，这使得我的训练速度慢了很多，我很肯定我在某个地方犯了错误。因为使用4 workers和use_multiprocessing=True运行我的培训脚本，所以我得到了以下日志：

Epoch 8/10
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
8/9 [=========================>....] - ETA: 2s - loss: 444.2380Using TensorFlow backend.
9/9 [==============================] - 26s 3s/step - loss: 447.4939 - val_loss: 308.3012
Using TensorFlow backend.
Epoch 9/10
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
8/9 [=========================>....] - ETA: 2s - loss: 421.9372Using TensorFlow backend.
9/9 [==============================] - 26s 3s/step - loss: 418.9702 - val_loss: 263.9197

在我的代码中，TensorFlow似乎是为每个工作人员加载和加载的(8是因为验证集的缘故？)在每一个时代。我不认为正常情况下序列应该是这样工作的？

DataGenerator：

class DataGenerator(Sequence):
    def __init__(self, annotation_lines, batch_size, input_shape, anchors, num_classes, max_boxes=80):
        self.annotations_lines = annotation_lines
        self.batch_size = batch_size
        self.input_shape = input_shape
        self.anchors = anchors
        self.num_classes = num_classes
        self.max_boxes = max_boxes

    def __len__(self):
        return int(np.ceil(len(self.annotations_lines) / float(self.batch_size)))

    def __getitem__(self, idx):
        annotation_lines = self.annotations_lines[idx * self.batch_size:(idx + 1) * self.batch_size]

        image_data = []
        box_data = []
        for annotation_line in annotation_lines:
            image, box = get_random_data(annotation_line, self.input_shape, random=True, max_boxes=self.max_boxes)
            image_data.append(image)
            box_data.append(box)
        image_data = np.array(image_data)
        box_data = np.array(box_data)
        y_true = preprocess_true_boxes(box_data, self.input_shape, self.anchors, self.num_classes)
        return [image_data, *y_true], np.zeros(self.batch_size)

这是我训练剧本的一部分：

batch_size = batch_size_complete  # note that more GPU memory is required after unfreezing the body

data_gen_train = DataGenerator(lines, batch_size, input_shape, anchors, num_classes)
data_gen_validation = DataGenerator(validation_lines, batch_size, input_shape, anchors, num_classes)

print('Train on {} samples, val on {} samples, with batch size {}.'.format(num_train, num_val, batch_size))
r = model.fit_generator(data_gen_train,
                        steps_per_epoch=max(1, num_train // batch_size),
                        validation_data=data_gen_validation,
                        validation_steps=max(1, num_val // batch_size),
                        epochs=epochs,
                        initial_epoch=initial_epoch,
                        callbacks=[logging, checkpoint, reduce_lr, early_stopping],
                        workers=workers,
                        use_multiprocessing=True)
model.save_weights(log_dir + 'trained_weights_final.h5')

Answer 1

我看到您多次使用Tensorflow后端来获取，这似乎就像Keras在每个线程中一次又一次地初始化一样。

也许你应该试一试use_multiprocessing=False (你仍然可以有很多工人)

Answer 2

训练的速度取决于批量大小、输入图像的大小、学习速度、划时代步骤和步骤验证等因素。然后开始调查其中一个原因，并将use_multiprocessing=False放在那里，因为在培训期间编写的各种tensorflow后端不应该在那里。