用于CNN图像分类的数值/物理数据集成

Question

我正在尝试使用CNN对巨蟒中的医学图像进行分类，使用keras。这些医学图像还包括文本信息，如年龄和性别，可以影响模型的决定。我如何训练CNN既能利用图像又能利用现实世界的信息进行训练，使之成为两者的分类基础。

Answer 1

有几种可能性，我可以想出我的头部类型，但最简单的是从医学图像中提取一些特征与CNN，然后扁平的CNN的结果，并连接非图像数据。下面是一个假设您有512x512图像和10个类的想法。这是functional，允许您有多个输入。

import tensorflow as tf
import numpy as np

num_classes = 10

H,W = 512, 512
# Define inputs with their shapes
imgs = tf.keras.Input((H,W,3), dtype = tf.float32)
genders = tf.keras.Input(1, dtype = tf.float32)
ages = tf.keras.Input(1, dtype = tf.float32)

# Extract image features
features = tf.keras.layers.Conv2D(64, 4, strides = 4, activation = 'relu')(imgs)
features = tf.keras.layers.MaxPooling2D()(features)
features = tf.keras.layers.Conv2D(128,3, strides = 2, activation = 'relu')(features)
features = tf.keras.layers.MaxPooling2D()(features)
features = tf.keras.layers.Conv2D(256, 3, strides = 2, activation = 'relu')(features)
features = tf.keras.layers.Conv2D(512, 3, strides = 2, activation = 'relu')(features)

# #Flatten output
flat_features = tf.keras.layers.Flatten()(features)

#Concatenate gender and age
flat_features = tf.concat([flat_features, genders, ages], -1)

# Downsample
xx = tf.keras.layers.Dense(2048, activation = 'relu')(flat_features)
xx = tf.keras.layers.Dense(1024, activation = 'relu')(xx)
xx = tf.keras.layers.Dense(512, activation = 'relu')(xx)

#Calculate probabilities for each class
logits = tf.keras.layers.Dense(num_classes)(xx)
probs = tf.keras.layers.Softmax()(logits)

model = tf.keras.Model(inputs = [imgs, genders, ages], outputs = probs)

model.summary()

这种架构并不是特别标准的，您可能希望使解码器更深和/或减少CNN编码器中的参数数。