使用metalkit导出拼贴视频

Question

如何使用不同分辨率的视频导出拼贴视频？我试图实现像显示下面的第一个图像，我正在使用AVCustomEdit演示，到目前为止，我创建了AVMutableVideoComposition传递所有的视频轨道i到customVideoCompositorClass，并获得所有的视频CVPixelBuffer，然后在MTLTexture中转换然后渲染所有的纹理，但问题是我的视频输出大小是方形(DestinationTexture)和视频大小是纵向或横向，这就是为什么每个视频都被挤压，我如何才能旋转缩放位置和掩模形状每个视频？另外，我怎样才能应用cifilter呢？我是否应该将每个CVPixelBuffer转换为ciImage，并将ciImage转换回CVPixelBuffer

​

​

override func renderPixelBuffer(backgroundTexture: MTLTexture,
                                firstPixelBuffer: CVPixelBuffer,
                                secondPixelBuffer: CVPixelBuffer,
                                thirdPixelBuffer: CVPixelBuffer,
                                fourthPixelBuffer: CVPixelBuffer,
                                destinationPixelBuffer: CVPixelBuffer) {

    // Create a MTLTexture from the CVPixelBuffer.
    guard let firstTexture = buildTextureForPixelBuffer(firstPixelBuffer) else { return }
    guard let secondTexture = buildTextureForPixelBuffer(secondPixelBuffer) else { return }
    guard let thirdTexture = buildTextureForPixelBuffer(thirdPixelBuffer) else { return }
    guard let fourthTexture = buildTextureForPixelBuffer(fourthPixelBuffer) else { return }
    guard let destinationTexture = buildTextureForPixelBuffer(destinationPixelBuffer) else { return }

    /*
     We must maintain a reference to the pixel buffer until the Metal rendering is complete. This is because the
     'buildTextureForPixelBuffer' function above uses CVMetalTextureCacheCreateTextureFromImage to create a
     Metal texture (CVMetalTexture) from the IOSurface that backs the CVPixelBuffer, but
     CVMetalTextureCacheCreateTextureFromImage doesn't increment the use count of the IOSurface; only the
     CVPixelBuffer, and the CVMTLTexture own this IOSurface. Therefore we must maintain a reference to either
     the pixel buffer or Metal texture until the Metal rendering is done. The MTLCommandBuffer completion
     handler below is then used to release these references.
     */

    pixelBuffers = RenderPixelBuffers(firstBuffer: firstPixelBuffer,
                                      secondBuffer: secondPixelBuffer,
                                      thirdBuffer: thirdPixelBuffer,
                                      fourthBuffer: fourthPixelBuffer,
                                      destinationBuffer: destinationPixelBuffer)

    // Create a new command buffer for each renderpass to the current drawable.
    let commandBuffer = commandQueue.makeCommandBuffer()!
    commandBuffer.label = "MyCommand"

    /*
     Obtain a drawable texture for this render pass and set up the renderpass
     descriptor for the command encoder to render into.
     */
    let renderPassDescriptor = setupRenderPassDescriptorForTexture(destinationTexture)

    // Create a render command encoder so we can render into something.
    let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)!
    renderEncoder.label = "MyRenderEncoder"

    guard let renderPipelineState = renderPipelineState else { return }

    modelConstants.modelViewMatrix = matrix_identity_float4x4

    // Render background texture.
    renderTexture(renderEncoder, texture: backgroundTexture, pipelineState: renderPipelineState)

    var translationMatrix = matrix_float4x4(translation: simd_float3(-0.5, 0.5, 0))
    // var rotationMatrix = matrix_float4x4(rotationZ: radians(fromDegrees: -90))
    var scaleMatrix = matrix_float4x4(scaling: 0.25)
    var modelMatrix = translationMatrix * scaleMatrix
    modelConstants.modelViewMatrix = modelMatrix

    // Render first texture.
    renderTexture(renderEncoder, texture: firstTexture, pipelineState: renderPipelineState)

    //        translationMatrix = matrix_float4x4(translation: simd_float3(0.5, -0.5, 0))
    //        rotationMatrix = matrix_float4x4(rotationZ: radians(fromDegrees: -45))
    //        scaleMatrix = matrix_float4x4(scaling: 0.5)
    //        modelMatrix = translationMatrix * scaleMatrix * rotationMatrix
    //        modelConstants.modelViewMatrix = modelMatrix

    //        // Render second texture.
    //        renderTexture(renderEncoder, texture: secondTexture, pipelineState: renderPipelineState)
    //
    //        // Render third texture.
    //        renderTexture(renderEncoder, texture: thirdTexture, pipelineState: renderPipelineState)
    //
    //        // Render fourth texture.
    //        renderTexture(renderEncoder, texture: fourthTexture, pipelineState: renderPipelineState)

    // We're done encoding commands.
    renderEncoder.endEncoding()

    // Use the command buffer completion block to release the reference to the pixel buffers.
    commandBuffer.addCompletedHandler({ _ in
        self.pixelBuffers = nil // Release the reference to the pixel buffers.
    })

    // Finalize rendering here & push the command buffer to the GPU.
    commandBuffer.commit()
}

Answer 1

我建议使用一个名为MetalPetal的库。它是一个基于Metal.You的图像处理框架，必须将CVPixelBuffer转换成MetalImage即MTIImage。然后你可以在图像中做任何事情，比如有预先制作的滤镜，你可以应用它，或者你甚至可以使用CIFilter或你的自定义滤镜，你可以变换，旋转，裁剪每一帧，以便拼贴帧是准确的。然后，您必须再次将MTIimage转换为cvpixelbuffer。在这里你也可以使用CIImage，但是我想它会很慢。你得到的可能是渲染大小的长方体图像。请查看渲染大小。