如何在Keras制作一个端到端的3D-2D CNN？

Question

我有两个CAE模型，一个是3D的，另一个是2D的。这个2D CAE以第一个所生成的新表示作为输入。我的目标是弄清楚如何将它们结合起来，这样我就可以拥有一个端到端的全三维二维CAE模型，我如何训练它呢？

以下是每个模型的代码：

#3D CAE (I have just implemented the first encoding part since my aim is to generate the new representation z)

in_3D = Input((100,100, 288, 1))
model_3D = Conv3D(8, (5, 5, 5), activation='relu', padding='same')(in_3D)
model_3D = MaxPooling3D((2, 2, 2), strides=(1, 1, 4), padding='same')(model_3D)
model_3D = Reshape((10000,72*8))(model_3D)
model_3D = Dense(350, activation="relu")(model_3D)
model_3D = Dense(250, activation="relu")(model_3D)
model_3D = Dense(198, activation="relu")(model_3D)
model_3D = Reshape((100,100, 198))(model_3D)
z = Permute((3,2, 1))(model_3D)

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 100, 100, 288, 1  0         
                             )]                                  
                                                                 
 conv3d_1 (Conv3D)           (None, 100, 100, 288, 8)  1008      
                                                                 
 max_pooling3d_1 (MaxPooling  (None, 100, 100, 72, 8)  0         
 3D)                                                             
                                                                 
 reshape (Reshape)         (None, 10000, 576)        0         
                                                                 
 dense (Dense)            (None, 10000, 350)        201950    
                                                                 
 dense_1 (Dense)            (None, 10000, 250)        87750     
                                                                 
 dense_2 (Dense)            (None, 10000, 198)        49698     
                                                                 
 reshape_1 (Reshape)         (None, 100, 100, 198)     0         
                                                                 
 permute (Permute)           (None, 198, 100, 100)     0         

第二个2D CAE模型作为输入，由第一个模型生成新的z (198,100,100)。这里198被通过为无。

#2D CAE  

in_2D = Input((100,100, 1))
model_2D= Conv2D(16, (3, 3), activation='relu', padding='same', name='Conv1')(in_2D)
model_2D = MaxPooling2D((2, 2), padding='same')(model_2D)
model_2D = Flatten()(model_2D)
model_2D = Dense(48, activation='relu')(model_2D)
model_2D = Dense(36, activation='relu')(model_2D)
model_2D = Dense(12)(model_2D)
model_2D= Dense(100*100, activation='linear')(model_2D)

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 100, 100, 1)]     0         
                                                                 
 Conv1 (Conv2D)              (None, 100, 100, 16)      160       
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 50, 50, 16)       0         
 2D)                                                             
                                                                 
 flatten (Flatten)          (None, 40000)             0         
                                                                 
 dense (Dense)              (None, 48)                1920048   
                                                                 
 dense_1 (Dense)             (None, 36)                1764      
                                                                 
 dense_2 (Dense)             (None, 12)                444       
                                                                 
 dense_3 (Dense)            (None, 10000)             130000    

任何帮助都将不胜感激。

Answer 1

要将这两种模型组合起来，首先可以按如下方式分别声明它们：

import keras
import tensorflow as tf
from tensorflow.keras.layers import *

in_3D = Input((100,100, 288, 1))
model_3D = Conv3D(8, (5, 5, 5), activation='relu', padding='same')(in_3D)
model_3D = MaxPooling3D((2, 2, 2), strides=(1, 1, 4), padding='same')(model_3D)
model_3D = Reshape((10000,72*8))(model_3D)
model_3D = Dense(350, activation="relu")(model_3D)
model_3D = Dense(250, activation="relu")(model_3D)
model_3D = Dense(198, activation="relu")(model_3D)
model_3D = Reshape((100,100, 198))(model_3D)
z = Permute((3,2, 1))(model_3D)
cae_model_3D = keras.Model(in_3D, z)

in_2D = Input((100,100, 1))
model_2D= Conv2D(16, (3, 3), activation='relu', padding='same', name='Conv1')(in_2D)
model_2D = MaxPooling2D((2, 2), padding='same')(model_2D)
model_2D = Flatten()(model_2D)
model_2D = Dense(48, activation='relu')(model_2D)
model_2D = Dense(36, activation='relu')(model_2D)
model_2D = Dense(12)(model_2D)
model_2D= Dense(100*100, activation='linear')(model_2D)
cae_model_2D = keras.Model(in_2D, model_2D)

然后声明将第一个模型的输出作为输入传递给第二个模型的组合模型：

combined_model_input = Input((100, 100, 288, 1))
cae_model_3D_output = cae_model_3D(combined_model_input)
cae_model_3D_output_reshaped = tf.reshape(cae_model_3D_output, (-1, 100, 100, 1))
combined_model_output = cae_model_2D(cae_model_3D_output_reshaped)
combined_model = keras.Model(combined_model_input, combined_model_output)

注意，我们必须重新构造第一个模型的输出，以便与您希望将198作为批处理维度(None)传递的想法保持一致。

最简单的训练模型的方法是调用compile和fit方法。传递给这些方法的确切参数将取决于您要解决的问题和您自己的首选项。这里有一个指向官方文件的帮助链接。