无分割车牌识别模型问题

Question

我们团队的目标是创建一个无分割的车牌识别模型，其体系结构如下：

​

​

我们成功地实现了大部分的体系结构，但是我们正在为连接8个完全连接的层分支而挣扎(每个分支都有一个车牌字符)。

是否有方法对8个分支使用相同的输入( 25x5x128张量)，让它们独立工作，并将它们的输出与相应的车牌地面真伪字母进行独立比较，并根据哪个字母不正确来惩罚(损失函数)？

我们尝试了多种方法，使用keras模型(序列和模型类API)，但没有任何成功。下面是我们目前版本的模型。我们将非常感谢任何帮助。

model = models.Sequential()

model.add(layers.Conv2D(32, (3, 3), use_bias=False, input_shape=(32, 32, 3),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.Conv2D(32, (3, 3), use_bias=False, input_shape=(32, 32, 3),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.Conv2D(32, (3, 3), use_bias=False, input_shape=(32, 32, 3),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.MaxPooling2D((2, 2)))

model.add(layers.Conv2D(64, (3, 3), use_bias=False, input_shape=(100, 20, 32),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.Conv2D(64, (3, 3), use_bias=False, input_shape=(100, 20, 32),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.Conv2D(64, (3, 3), use_bias=False, input_shape=(100, 20, 32),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.MaxPooling2D((2, 2)))

model.add(layers.Conv2D(128, (3, 3), use_bias=False, input_shape=(50, 10, 64),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.Conv2D(128, (3, 3), use_bias=False, input_shape=(50, 10, 64),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.Conv2D(128, (3, 3), use_bias=False, input_shape=(50, 10, 64),padding='same'))
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
model.add(layers.MaxPooling2D((2, 2)))


model.add(layers.Flatten())


branch1 = models.Sequential()

branch1 .add(layers.Dense(128, input_shape=(16000,)))
branch1.add(layers.Dense(36, input_shape=(128,)))
branch1.add(layers.Activation("softmax"))

# Another 7 branches follows with exact same definition

final_model = keras.Model(inputs=[model, model, model, model, model, model, model, model],
                          outputs=[branch1, branch2, branch3, branch4, branch5, branch6, branch7, branch8])


final_model.compile(tf.keras.optimizers.Adam(learning_rate=0.001),
        loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=['accuracy'])

history = final_model.fit(train_images, train_labels, epochs=80,
                          validation_data=(test_images, test_labels))

Answer 1

这是一个虚拟的例子

inp_dim = (32, 32, 3)
x_inp = Input(shape=inp_dim)
x = Conv2D(32, (3, 3), use_bias=False, padding='same')(x_inp)
x = BatchNormalization()(x)
x = Activation("relu")(x)
x = MaxPooling2D((2, 2))(x)
x_out = Flatten()(x)
model = Model(x_inp, x_out)

inp = Input(shape=inp_dim)

branches = []

for _ in range(8):

    init_branch = model(inp)
    x = Dense(128)(init_branch)
    x = Dense(36)(x)
    x = Activation("softmax")(x)

    branches.append(x)

final_model = Model(inputs = inp, outputs = branches)


final_model.compile(tf.keras.optimizers.Adam(learning_rate=0.001),
                loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                metrics=['accuracy'])

对于所有8个最终分支，输入都是一个单一模型(我使用了一个简化版本)。检查输入维度((32，32，3)，我不知道它是否正确)

这是我的解决方案，正确的适合(这是我的意见，我不测试)

import string

# dummy target in the format I expected (7 digits)
train_labels = np.array([['7C24698'], ['7F84698']])
test_label = np.array([['8C24698'], ['8F84698']])

# create dict to encode digits to numbers
y_map = {}
for i,d in enumerate(string.digits+string.ascii_uppercase):
    y_map[d] = i

# utility function for split string in char and encode them
def split_encode(x):
    x = list(x[0])
    x = [y_map[d] for d in x]
    return x

# transform target
train_labels = np.apply_along_axis(split_encode, 1, train_labels)
test_label = np.apply_along_axis(split_encode, 1, test_label)

......

# fit model (this works with 7 output branches, 36 digits output prob, and sparse_catcrossent
final_model.fit(train_images, [train_labels[:,i] for i in range(7)], epochs=80,
            validation_data=(test_images, [test_label[:,i] for i in range(7)]))