如何为vImage_CGImageFormat图像创建YCbCr？

Question

我想要创建一个库，它可以将使用AVCaptureDevice捕获的任何视频帧转换为目标像素格式。因此，它必须支持kCVPixelFormatType_32BGRA、kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange、kCVPixelFormatType_420YpCbCr8BiPlanarFullRange和可选的kCVPixelFormatType_420YpCbCr8Planar。

为了提高性能，我想使用加速框架，因为它包含丰富的转换函数集。由于目标像素格式可能不同，并且是由库用户设置的，所以使用通用的AnyToAny函数是很好的：

您可以使用vImage的vImageConvert_AnyToAny(_:)函数在任意颜色空间和位深度的核心视频或核心图形图像数据之间进行转换。源图像和目标图像由一个或多个缓冲区描述。例如，Y‘’CbCr图像可以由一个包含亮度信息的缓冲器和一个包含色度信息的缓冲器组成。

要使用这个函数，我必须创建vImageConverter，它定义了图像之间的转换。此类的构造函数要求以CGImageFormat的形式描述源和目标图像格式。

vImage_CGImageFormat in_format = ...;
vImage_CGImageFormat out_format = ...; 
vImageConverterRef converter = vImageConverter_CreateWithCGImageFormat(&in_format, &out_format, NULL, kvImagePrintDiagnosticsToConsole, &err);

if( err == kvImageNoError )
{
    vImage_Buffer *src_planes = ...;
    vImage_Buffer *dst_planes = ...;

    err = vImageConvert_AnyToAny(converter, src_planes, dst_planes, NULL,  kvImagePrintDiagnosticsToConsole);
}

代码是基于苹果的这篇文章：构建基本转换工作流

对于kCVPixelFormatType_32BGRA，这样的vImage_CGImageFormat很简单，并且在kCVPixelFormatType_32BGRA中进行了描述。

/*!
 * @struct vImage_CGImageFormat
 * @abstract A pixel format
 * @discussion A vImage_CGImageFormat describes the ordering of the color channels, how many there are,
 * the size and type of the data in the color channels and whether the data is premultiplied by alpha or not.
 * This format mirrors the image format descriptors used by CoreGraphics to create things like CGImageRef and
 * CGBitmapContextRef.
 *
 * This vImage_CGImageFormat:
 *
 *  <pre>@textblock
 *      vImage_CGImageFormat format = {
 *          .bitsPerComponent = 8,
 *          .bitsPerPixel = 32,
 *          .colorSpace = CGColorSpaceCreateDeviceRGB(),                                    // don't forget to release this!
 *          .bitmapInfo = kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Little,
 *          .version = 0,                                                                   // must be 0
 *          .decode = NULL,
 *          .renderingIntent = kCGRenderingIntentDefault
 *      };
 *  @/textblock</pre>
 *
 * codes for a little endian ARGB8888 pixel, or what is called in the rest of vImage, BGRA8888. Note: for 16-
 * and 32-bits per component formats (int16_t, uint16_t, half-float, float) most vImage image filters assume
 * the data is in host-endian format. (The APIs in this header do not.) Host-endian is little endian for Intel
 * and ARM, big endian for PowerPC. If the data is not in host-endian format, then you may use
 * vImagePermuteChannels_ARGB8888 or vImageByteSwap_Planar16U to swap the image data byte ordering.
 *
 * Some examples:
 *  <pre>@textblock
 *      ARGB8888     ->  {8, 32, NULL, alpha first, 0, NULL, kCGRenderingIntentDefault}     alpha first = { kCGImageAlphaFirst, kCGImageAlphaPremultipliedFirst, kCGImageAlphaNoneSkipFirst }
 *      RGBA8888     ->  {8, 32, NULL, alpha last,  0, NULL, kCGRenderingIntentDefault}     alpha last  = { kCGImageAlphaLast,  kCGImageAlphaPremultipliedLast,  kCGImageAlphaNoneSkipLast }
 *      BGRA8888     ->  {8, 32, NULL, alpha first | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault}
 *      RGB888       ->  {8, 24, NULL, kCGImageAlphaNone | kCGBitmapByteOrderDefault, 0, NULL, kCGRenderingIntentDefault}
 *      RGB565       ->  {5, 16, NULL, kCGImageAlphaNone | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
 *      ARGB1555     ->  {5, 16, NULL, alpha first | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
 *      RGBA16F      ->  {16, 64, NULL, alpha last | kCGBitmapFloatComponents | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault }
 *      CMYK8888     ->  {8, 32, CGColorSpaceCreateDeviceCMYK(), kCGImageAlphaNone, 0, NULL, kCGRenderingIntentDefault  }
 *      ARGBFFFF premultiplied    ->  {32, 128, NULL, kCGImageAlphaPremultipliedFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *      ARGBFFFF not-premultiplied -> {32, 128, NULL, kCGImageAlphaFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *      ARGBFFFF, alpha = 1 ->        {32, 128, NULL, kCGImageAlphaNoneSkipFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *  @/textblock</pre>
 *
 *  Note that some of these formats, particularly RGB565 and 16F formats are supported by vImage but
 *  not necessarily CoreGraphics. They will be converted to a higher precision format as necessary by
 *  vImage in vImageCreateCGImageFromBuffer().
 *
 *  By C rules, uninitialized struct parameters are set to zero. The last three parameters are usually zero, so can usually be omitted.
 *
 *  <pre>@textblock
 *      vImage_CGImageFormat srgb888 = (vImage_CGImageFormat){
 *          .bitsPerComponent = 8,
 *          .bitsPerPixel = 24,
 *          .colorSpace = NULL,
 *          .bitmapInfo = kCGImageAlphaNone | kCGBitmapByteOrderDefault };
 *  @/textblock</pre>
 *
 * To understand how these various parameters relate to one another, we can look at the process of converting from
 * one vImage_CGImageFormat format to another:
 *
 *  1) transform endianness of src format given by bitmapInfo to host endian  (except 8 bitPerComponent content)
 *  2) remove decode array transformation, and up convert to a higher range format as necessary to preserve precision / range
 *  3) convert src colorspace to reference XYZ colorspace (may cause upconvert to preserve range / precision)
 *  4) convert XYZ to destination colorspace + rendering intent
 *  5) convert to destination precision (given by bitsPerComponent)
 *  6) deal with any alpha changes (given by bitmapInfo) or flattening that needs to occur
 *  7) Apply any channel reordering requested, if it didn't happen at an earlier step. (As indicated by src and dest bitmapInfo)
 *  8) Apply destination decode array
 *  9) Apply endianness transform given by dest bitmapInfo
 *
 * Clearly, for most common transformations not all steps need to occur and multiple steps can be collapsed into a compound operation.
 *
 *  @field  bitsPerComponent    The number of bits needed to represent one channel of data in one pixel. For ARGB8888, this would be 8. Expected values: {1, 2, 4, 5, 8, 10, 12, 16, 32}
 *  @field  bitsPerPixel        The number of bits needed to represent one pixel. For ARGB8888, this would be 32.
 *                              It is possible that bitsPerPixel > bitsPerComponent * number of components, but in practice this is rare.
 *                              The number of color components is given by the colorspace and the number of alpha components (0 or 1) is given by
 *                              by the bitmapInfo.
 *  @field  colorSpace          A description of how the pixel data in the image is positioned relative to a reference XYZ color space.
 *                                  See CoreGraphics/CGColorSpace.h.  Pass NULL as a shorthand for sRGB. The vImage_CGImageFormat is not
 *                                  capable of managing the memory held by the colorSpace. If you created the colorspace, you must
 *                                  be sure to release it before all references to it disappear from scope.
 *  @field  bitmapInfo          The CGBitmapInfo describing the color channels. See CoreGraphics/CGImage.h.
 *                                  ARGB8888 is kCGImageAlphaFirst | kCGBitmapByteOrderDefault
 *                                  BGRA8888 is kCGImageAlphaFirst | kCGBitmapByteOrder32Little
 *  @field  version             The struct is versioned for future expansion.  Pass 0 here.
 *  @field  decode              Prior to transformations caused by the colorspace, color channels are subject to a linear transformation.
 *                              This allows for a different range than the typical [0,1.0]. NULL indicates default behavior of [0,1.0]
 *                              range, and is what you should use if you don't understand this parameter. See description of CGImageCreate()
 *                              for a discussion of decode arrays. See also Decode Arrays section of Chapter 4.8 of the PDF specification.
 *                              The vImage_CGImageFormat is not capable of managing the memory held by the decode array. If you created a
 *                              decode array on the heap, you must be sure to release it before all references to it disappear from scope.
 *
 *  @field renderingIntent      See CGColorSpace.h. kCGRenderingIntentDefault is typical here. By convention, rendering intent changes that
 *                              are not accompanied by a colorspace change are ignored.
 */

我不明白如何为vImage_CGImageFormat像素格式创建YCbCr。

首先，我认为它根本不受支持，但是这种格式有一些特殊的功能：理解YpCbCr图像格式和vImageConverter对多平面图像(如GetNumberOfSourceBuffers和GetSourceBufferOrder )都有特殊的功能。在最后一个函数中，有许多vImage缓冲型码，甚至kvImageBufferTypeCode_CVPixelBuffer_YCbCr。

因此，看起来为vImageConverter创建YCbCr是可能的，我需要帮助来了解如何做到这一点。

Answer 1

vImageConverter_CreateWithCGImageFormat创建一个转换器，用于在两种核心图形格式之间进行转换。

在这种情况下，不应该使用vImageConverter_CreateForCVToCGImageFormat将CVPixelBuffer (可以使用vImageCVImageFormat_CreateWithCVPixelBuffer生成的vImageCVImageFormat )转换为CGImage吗？vImage_CGImageFormat是您的目标格式，因此属性由您来定义。

Answer 2

有函数vImageCreateRGBColorSpaceWithPrimariesAndTransferFunction为YCbCr像素格式创建适当的颜色空间，但只用于转换RGB格式或转换为RGB格式：

const vImageTransferFunction f709 =
{
    .c0 = 1.099,
    .c1 = 1.0,
    .c2 = 0.0,
    .c3 = -0.099,
    .gamma = 0.45,
    .cutoff = 0.018,
    .c4 = 4.5,
    .c5 = 0
};

const vImageRGBPrimaries p709 =
{
    .red_x = .64,               // 1.3 red
    .green_x = .30,             // 1.3 green
    .blue_x = .15,              // 1.3 blue
    .white_x = 0.3127,          // 1.4 white

    .red_y = .33,               // 1.3 red
    .green_y = .60,             // 1.3 green
    .blue_y = .06,              // 1.3 blue
    .white_y = 0.3290           // 1.4 white
};

vImage_Error err = kvImageNoError;
CGColorSpaceRef colorSpace = vImageCreateRGBColorSpaceWithPrimariesAndTransferFunction( &p709, &f709, kvImageNoFlags, &err );

vImage_CGImageFormat format = {
    .bitsPerComponent = 8,
    .bitsPerPixel = 32,
    .bitmapInfo = (CGBitmapInfo)kCGImageAlphaNoneSkipFirst | kCGBitmapByteOrderDefault,
    .colorSpace = colorSpace
};