Coremltools:获得最简单卷积模型工作的错误

Question

假设我在Keras中创建了最简单的模型：

from keras.layers import *
from keras import Input, Model

import coremltools


def MyModel(inputs_shape=(None,None,3), channels=64):

        inpt = Input(shape=inputs_shape)

        # channels
        skip = Conv2D(channels, (3, 3), strides=1, activation=None, padding='same', name='conv_in')(inpt)
        out = Conv2D(3, (3, 3), strides=1, padding='same', activation='tanh',name='out')(skip)

        return Model(inputs=inpt, outputs=out)


model = MyModel()


coreml_model = coremltools.converters.keras.convert(model,
    input_names=["inp1"],
    output_names=["out1"],
    image_scale=1.0,
    model_precision='float32',
    use_float_arraytype=True,
    input_name_shape_dict={'inp1': [None, 384, 384, 3]}
    )


spec = coreml_model._spec


print(spec.description.input[0])

print(spec.description.input[0].type.multiArrayType.shape)

print(spec.description.output[0])


coremltools.utils.save_spec(spec, "test.mlmodel")

输出为：

2 : out, <keras.layers.convolutional.Conv2D object at 0x7f08ca491470>
3 : out__activation__, <keras.layers.core.Activation object at 0x7f08ca4b0b70>
name: "inp1"
type {
  multiArrayType {
    shape: 3
    shape: 384
    shape: 384
    dataType: FLOAT32
  }
}

[3, 384, 384]
name: "out1"
type {
  multiArrayType {
    shape: 3
    dataType: FLOAT32
  }
}

所以输出的形状是3，这是不正确的。当我试图摆脱input_name_shape_dict时，我得到了：

Please provide a finite height (H), width (W) & channel value (C) using input_name_shape_dict arg with key = 'inp1' and value = [None, H, W, C]
Converted .mlmodel can be modified to have flexible input shape using coremltools.models.neural_network.flexible_shape_utils

所以它想要NHWC。

尝试推理会产生以下结果：

Layer 'conv_in' of type 'Convolution' has input rank 3 but expects rank at least 4

当我尝试向输入添加额外维度时：

spec.description.input[0].type.multiArrayType.shape.extend([1, 3, 384, 384])
del spec.description.input[0].type.multiArrayType.shape[0]
del spec.description.input[0].type.multiArrayType.shape[0]
del spec.description.input[0].type.multiArrayType.shape[0]

[name: "inp1"
type {
  multiArrayType {
    shape: 1
    shape: 3
    shape: 384
    shape: 384
    dataType: FLOAT32
  }
}
]

我得到一个推论：

Shape (1 x 384 x 384 x 3) was not in enumerated set of allowed shapes

跟随this advice并使输入形状(1,1,384,384,3)无济于事。

我怎样才能让它工作并产生正确的输出呢？

推理：

From PIL import Image

model_cml = coremltools.models.MLModel('my.mlmodel')

# load image
img = np.array(Image.open('patch4.png').convert('RGB'))[np.newaxis,...]/127.5 - 1

# Make predictions
predictions = model_cml.predict({'inp1':img})

# save result
res = predictions['out1']
res = np.clip((res[0]+1)*127.5,0,255).astype(np.uint8)

Image.fromarray(res).save('out32.png')

更新：

我可以运行这个模型的输入(3,1,384,384)，结果产生的是(1,3,3,384,384)，这对我来说没有任何意义。

更新2：

在Keras中设置固定形状

def MyModel(inputs_shape=(384,384,3), channels=64):

        inpt = Input(shape=inputs_shape)

修复了输出形状问题，但我仍然无法运行模型(Layer 'conv_in' of type 'Convolution' has input rank 3 but expects rank at least 4)

更新：

下面的方法可以消除输入形状和conv_in形状不匹配的情况。

1)。降级到coremltools==3.0。版本3.3 (型号版本4)似乎坏了。

2.)keras模型使用固定形状，没有input_shape_dist，coreml模型使用可变形状

from keras.layers import *
from keras import Input, Model

import coremltools


def MyModel(inputs_shape=(384,384,3), channels=64):

        inpt = Input(shape=inputs_shape)

        # channels
        skip = Conv2D(channels, (3, 3), strides=1, activation=None, padding='same', name='conv_in')(inpt)
        out = Conv2D(3, (3, 3), strides=1, padding='same', activation='tanh',name='out')(skip)

        return Model(inputs=inpt, outputs=out)


model = MyModel()

model.save('test.model')

print(model.summary())

'''
# v.3.3
coreml_model = coremltools.converters.keras.convert(model,
    input_names=["image"],
    output_names="out1",
    image_scale=1.0,
    model_precision='float32',
    use_float_arraytype=True,
    input_name_shape_dict={'inp1': [None, 384, 384, 3]}
    )
'''

coreml_model = coremltools.converters.keras.convert(model,
    input_names=["image"],
    output_names="out1",
    image_scale=1.0,
    model_precision='float32',

    )


spec = coreml_model._spec


from coremltools.models.neural_network import flexible_shape_utils
shape_range = flexible_shape_utils.NeuralNetworkMultiArrayShapeRange()
shape_range.add_channel_range((3,3))
shape_range.add_height_range((64, 384))
shape_range.add_width_range((64, 384))
flexible_shape_utils.update_multiarray_shape_range(spec, feature_name='image', shape_range=shape_range)


print(spec.description.input)

print(spec.description.input[0].type.multiArrayType.shape)

print(spec.description.output)


coremltools.utils.save_spec(spec, "my.mlmodel")

在推理脚本中，馈送形状为(1,1,3,384,384)的数组

img = np.zeros((1,1,3,384,384))

# Make predictions
predictions = model_cml.predict({'inp1':img})
res = predictions['out1'] # (3, 384,384)

Answer 1

如果mlmodel文件不正确，可以忽略其在输出形状中的内容。这更多的是一个元数据问题，即模型仍然可以很好地工作，并做正确的事情。转换器并不总是能够计算出正确的输出形状(不确定原因)。