在AVAssetExportSession中使用MTAudioProcessingTap

Question

我正在尝试用AVAssetExportSession导出一个可变的组合。我希望这是实时发生的，因为动画依赖于音频数据。如果我使用AVPlayer，这可以很好地工作，但在AVAssetExportSession中就不是很好了。在MTAudioProcessingTapProcessCallback中接收的音频数据似乎与AVAssetExportSession中的视频不同步。所以这里的问题是，有没有人知道它们是否应该在AVAssetExportSession期间同步？

这是我的MTAudioProcessingTapProcessCallback

var tapProcess: MTAudioProcessingTapProcessCallback = {
    (tap, numberFrames, flags, bufferListInOut, numberFramesOut, flagsOut) in
            let status = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut)

            let viewController = MTAudioProcessingTapGetStorage(tap)

            let viewController = Unmanaged<CanvasViewController>.fromOpaque(viewController).takeUnretainedValue()

            DispatchQueue.main.sync {
                // update my CALayer in viewController based on the audio data
            }
        }

下面是设置audiomix的代码：

func setupAudioMix(audioTrack: AVAssetTrack) -> AVAudioMix {
        var callbacks = MTAudioProcessingTapCallbacks(
            version: kMTAudioProcessingTapCallbacksVersion_0,
            clientInfo: UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque()),
            init: tapInit,
            finalize: tapFinalize,
            prepare: tapPrepare,
            unprepare: tapUnprepare,
            process: tapProcess)

        var tap: Unmanaged<MTAudioProcessingTap>?
        let err = MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks, kMTAudioProcessingTapCreationFlag_PreEffects, &tap)

        print("err: \(err)\n")
        if err == noErr {
        }

        let inputParams = AVMutableAudioMixInputParameters(track: audioTrack)
        inputParams.audioTapProcessor = tap?.takeUnretainedValue()

        let audioMix = AVMutableAudioMix()
        audioMix.inputParameters = [inputParams]

        return audioMix
    }

并将audiomix附加到AVAssetExportSession：

exporter = AVAssetExportSession(asset: mutableComposition!, presetName: AVAssetExportPresetMediumQuality)!
        exporter!.outputURL = exportUrl as URL
        exporter!.videoComposition = videoComposition!
        exporter!.audioMix = audioMix!

        exporter!.outputFileType = AVFileTypeMPEG4
        exporter!.shouldOptimizeForNetworkUse = true

Answer 1

不，AVAssetExportSession不承诺在处理类似定时的视频帧之前、之后或期间会调用音频tap。

想想看，AVAudioPlayer (AVPlayer?)也不是，所以也许你只是在这方面走运。