在swift3.2/4中正确释放MTAudioProcessingTapRef

Question

我一直在我的项目中使用MTAudioProcessingTapRef，以便在播放流媒体音频的同时实时分析缓冲区数据。问题是我不能让tap处理器在我需要它的时候正确地释放它。

我有一个引用AudioTapProcessor objective-C类的AudioViewController swift类，swift类负责告诉处理器开始和停止AVPlayerItem的处理。处理器还具有一个代理(在本例中为视图控制器)，用于在处理过程中通知缓冲区更改。

我的问题是，如果我声明处理器委托为弱(它应该是弱的)，处理器将随机崩溃，试图通知一个已经释放的委托，因为tap处理器的process方法在stop processing调用之后执行了几次。我发现解决这个问题的唯一方法是将tap处理器委托声明为一个强属性，这显然会导致保留周期，并且我的AudioViewControllers永远不会被释放。

下面是一些代码，可以帮助你从de环境中获得相关信息：

AudioTapProcessor.h

@interface AudioTapProcessor : NSObject

@property (nonatomic, strong) AVPlayerItem *item;
@property (nonatomic, strong) id<AudioProcessorDelegate> delegate;

- (instancetype)initWithDelegate:(id<AudioProcessorDelegate>)delegate 
    item:(AVPlayerItem *)item;
- (void)startProcessing;
- (void)stopProcessing;

@end

AudioTapProcessor.m

void init(MTAudioProcessingTapRef tap, void *clientInfo, void 
**tapStorageOut) {
    *tapStorageOut = clientInfo;
}

void finalize(MTAudioProcessingTapRef tap) {}

void prepare(
         MTAudioProcessingTapRef tap,
         CMItemCount maxFrames,
         const AudioStreamBasicDescription *processingFormat
         ) {}

void unprepare(MTAudioProcessingTapRef tap) {}

void process(
         MTAudioProcessingTapRef tap,
         CMItemCount numberFrames,
         MTAudioProcessingTapFlags flags,
         AudioBufferList *bufferListInOut,
         CMItemCount *numberFramesOut,
         MTAudioProcessingTapFlags *flagsOut
         ) {
 //Random crashes here if I declare the delegate weak
 //Something like AUDeferredRenderer-0x7ff8f448ef (364): EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  AudioTapProcessor *processor = (__bridge AudioTapProcessor *)MTAudioProcessingTapGetStorage(tap);

  OSStatus err = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, NULL, numberFramesOut);

  AudioBuffer *pBuffer = &bufferListInOut->mBuffers[0];
  UInt32 frameLength = pBuffer->mDataByteSize / sizeof(float);
  float *pData = (float *)pBuffer->mData;

  if (err == noErr && processor) {
    if ([processor.delegate 
      respondsToSelector:@selector(updateWith:withSize:)]) {
      [processor.delegate updateWith:pData withSize:frameLength];
    }
  }
 }

- (void)stopProcessing
{
  [self.item removeObserver:self forKeyPath:@"status"];
AVMutableAudioMixInputParameters *params =
(AVMutableAudioMixInputParameters *) _item.audioMix.inputParameters[0];
  MTAudioProcessingTapRef tap = params.audioTapProcessor;
  self.item.audioMix = nil;
  CFRelease(tap);
  //By doing this the tap processor does call its unprepare and finalize methods, so it is being deallocated fine.
}

然后在我的AudioViewController.swift中，我有：

var processor: AudioTapProcessor!

override func prepareForPlayback() {
  super.prepareForPlayback()
  if processor == nil {
    processor = AudioTapProcessor(delegate: self, item: item)
    processor.startProcessing()
  }
}

override func viewWillDisappear(_ animated: Bool) {
   super.viewWillDisappear(animated)
   player.pause()
}

deinit {
  //I tried to do this early in the lifecycle(viewWillDissapear) and it is the same thing.
   processor.stopProcessing()
}

任何提示都将不胜感激，我为此而疯狂。谢谢

Answer 1

适用于所有iOS版本

了解根本原因的⇨

1.AudioTapProcessor.m初始化

callbacks.clientInfo持有指向self的指针。这不是弱引用，也不是强引用，只是一个C指针。因此，如果self被取消分配，我们context->self指向一个坏内存地址

- (AVAudioMix *)audioMix {
    if (!_audioMix) {
        AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];
        if (audioMix) {
            AVMutableAudioMixInputParameters *audioMixInputParameters = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:self.audioAssetTrack];
            if (audioMixInputParameters) {
                MTAudioProcessingTapCallbacks callbacks;

                ...
                callbacks.clientInfo = (__bridge void *)self;
                ...
            }
        }
     }
}

2.AudioTapProcessor.m processCallback

每次调用processCallback时，都会进行安全检查，以查看是否取消分配self，但请记住，在上面的步骤1中，即使取消分配self，context->self也不是nil，而是指向导致EXC_BAD_ACCESS.的错误内存地址

static void tap_ProcessCallback(MTAudioProcessingTapRef tap, CMItemCount numberFrames, MTAudioProcessingTapFlags flags, AudioBufferList *bufferListInOut, CMItemCount *numberFramesOut, MTAudioProcessingTapFlags *flagsOut) {
    ...
    MYAudioTapProcessor *self = ((__bridge MYAudioTapProcessor *)context->self);
    if (!self) {
      NSLog(@"AudioTapProcessor - processCallback CANCELLED (self is nil)");
      return;
    }
    NSLog(@"AudioTapProcessor - processCallback PROCESSING");
}

：⇨现在，如何解决这个问题？

1.ViewController.swift或audioTapProcessor的所有者

deinit {
  print("ViewController - Dealloc")
  audioTapProcessor.stopProcessing()
}

2.AudioTapProcessor.m

我们需要一种方法来告诉我们的audioTapProcessor停止processCallback。最简单、最自然的方法是，如果(!self)返回；，则使用上面processCallback中已有的检查

所以停止上下文就是正确地将 audioTapProcessor ->设置为空。

- (void)stopProcessing {
    NSLog(@"AudioTapProcessor - stopProcessing");
    AVMutableAudioMixInputParameters *params = (AVMutableAudioMixInputParameters *)_audioMix.inputParameters[0];
    MTAudioProcessingTapRef audioProcessingTap = params.audioTapProcessor;
    AVAudioTapProcessorContext *context = (AVAudioTapProcessorContext *)MTAudioProcessingTapGetStorage(audioProcessingTap);

    // nils out the pointer so that we know in tapProcessorCallbacks that self will be dealloc'ed
    context->self = NULL;
}

作为结果，生命周期被更正为⇨

而不是这个

1. ViewController - Dealloc
2. AudioTapProcessor - stopProcessing
3. AudioTapProcessor - PROCESSING
4. EXC_BAD_ADDRESS

- processCallback

我们得到了这个

1. ViewController - Dealloc
2. AudioTapProcessor - stopProcessing
3. AudioTapProcessor -Dealloc已取消(self is nil)
4. AudioTapProcessor - unprepareCallback
5. AudioTapProcessor - finalizeCallback
6. AudioTapProcessor - processCallback

Answer 2

我最终解决了这个问题，使MTAudioProcessingTapRef保留了它的AudioTapProcessor父元素。这样，它们就不会在生命周期的不同时刻被释放。

对原始代码的更改：

1.首先，我们将委托设置为弱变量：

@property (nonatomic, weak) id<AudioProcessorDelegate> delegate;

2.然后，我们将self(我们的AudioTapProcessor)的保留引用传递给创建的MTAudioProcessingTapRef

 callbacks.clientInfo = CFRetain((__bridge void *)(self));

3.还创建了一个自定义上下文，用于在tap上传递数据：

typedef struct TapProcessorContext {
  void *self;
} TapProcessorContext;

void init(MTAudioProcessingTapRef tap, void *clientInfo, void **tapStorageOut) {
  TapProcessorContext *context = calloc(1, sizeof(TapProcessorContext));
  //Initialize TapProcessorContext context.
  context->self = clientInfo;

  *tapStorageOut = context;
}

void finalize(MTAudioProcessingTapRef tap) {
  TapProcessorContext *context = (TapProcessorContext 
*)MTAudioProcessingTapGetStorage(tap);
  // Clearing the context. THIS IS KEY TO DEALLOCATE THE AUDIOTAPPROCESSOR
  CFRelease(context->self);
  context->self = NULL;

  free(context);
}

4.最后，我对我们的stopProcessing方法的iOS中的一个已知错误应用了一个变通方法：

- (void)stopProcessing
{
   if ( @available(iOS 11.0, *) ) {
    // Starting with iOS 11, it is not required to manually nil audioTapProcessor,
    // but we need to retain the audioMix for a bit to make sure the processing callback
    // will not be called after we release (this is due to a bug in iOS 11 which calls the release
    // callback but still calls the processing callback afterwards - it also releases internal data
    // on release, so simply checking for release in the processing block is not enough)
    // rdar://34977000
    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(1 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
      [self releaseTap];
    });
  } else {
    // Prior to iOS 11 we need to manually nil the audioTapProcessor
    [self releaseTap];
  }
}

-(void)releaseTap {
  AVMutableAudioMixInputParameters *params = (AVMutableAudioMixInputParameters *) _item.audioMix.inputParameters[0];
  params.audioTapProcessor = nil;
  _item.audioMix = nil;
}