Echo cancellation not working on iOS

[jaween]

Package version
5.1.1

Environment

• OS: iOS 15.7.9

Describe the bug
Enabling RecordConfig.echoCancel doesn't successfully remove the speaker audio from being picked up by the mic recording.

My record config used for streaming is:
RecordConfig( encoder: AudioEncoder.pcm16bits, numChannels: 1, echoCancel: true, noiseSupress: true, );

To Reproduce

Steps to reproduce the behavior:

1. Begin recording

final micStream = await _record.startStream(
  const RecordConfig(
    encoder: AudioEncoder.pcm16bits,
    numChannels: 1,
    echoCancel: true,
    noiseSuppress: true,
  ),
);

2. Have the app play audio through the speaker

3. Stop recording

await _record.stop()

4. Listen to the recorded audio and hear the audio data that shouldn't have been picked up by the mic

Expected behavior

The mic should pick up no audio from the speaker, or at least significantly reduced audio from the speaker.

Additional context
Though the docs for this plugin show echo cancellation as unsupported on iOS, I believe this commit from two weeks ago enables it using Voice-Processing I/O Unit as per the Apple Docs.

However according to What's New In AVAudioEngine at 0:45 there is an audioEngine.inputNode.setVoiceProcessingEnabled(true) method now which enables echo cancellation.

I have tried this method is a separate plugin manually and it does seem to successfully stop the mic from hearing the speaker. (Note I had to also use audioSession.overrideOutputAudioPort(.speaker) to make playback come out of the speaker while recording, but that may be a separate issue).

import AVFoundation
import Foundation

class Microphone {
    private var audioEngine = AVAudioEngine()
    private var converter: AVAudioConverter?
    
    func start(onData: @escaping (Data) -> Void) {
        switch AVAudioSession.sharedInstance().recordPermission {
        case .granted:
            setupAndStartEngine(onData: onData)
        case .denied:
            print("Microphone access denied.")
        case .undetermined:
            AVAudioSession.sharedInstance().requestRecordPermission { granted in
                if granted {
                    self.setupAndStartEngine(onData: onData)
                } else {
                    print("Microphone access denied.")
                }
            }
        @unknown default:
            fatalError("Unknown record permission state.")
        }
    }

    private func setupAndStartEngine(onData: @escaping (Data) -> Void) {
        do {
            let inputNode = audioEngine.inputNode
            try inputNode.setVoiceProcessingEnabled(true)
            let inputFormat = inputNode.inputFormat(forBus: 0)
            
            guard let outputFormat = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 44100.0, channels: 1, interleaved: true) else {
                print("Failed to create output format")
                return
            }
            
            let converter = AVAudioConverter(from: inputFormat, to: outputFormat)

            inputNode.installTap(onBus: 0, bufferSize: 1024, format: inputFormat) { (buffer, when) in
                guard let converter = converter else { return }
                let pcmBuffer = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: AVAudioFrameCount(outputFormat.sampleRate) * buffer.frameLength / AVAudioFrameCount(inputFormat.sampleRate))!

                var error: NSError? = nil
                let inputBlock: AVAudioConverterInputBlock = { inNumPackets, outStatus in
                    outStatus.pointee = .haveData
                    return buffer
                }

                converter.convert(to: pcmBuffer, error: &error, withInputFrom: inputBlock)

                if let error = error {
                    print("Error during conversion: \(error)")
                    return
                }

                guard let data = self.bufferToData(buffer: pcmBuffer, audioFormat: outputFormat) else {
                    return
                }

                DispatchQueue.main.async {
                    onData(data)
                }
            }

            try audioEngine.start()
        } catch {
            print("Failed to start audio engine: \(error)")
        }
    }
    
    func stop() {
        audioEngine.inputNode.removeTap(onBus: 0)
        audioEngine.stop()
    }
    
    func playFromSpeaker() {
        let audioSession = AVAudioSession.sharedInstance()
        do {
            try audioSession.overrideOutputAudioPort(.speaker)
        } catch {
            print("Failed to force AVAudioSession to play from speaker")
        }
    }
    
    private func bufferToData(buffer: AVAudioPCMBuffer, audioFormat: AVAudioFormat) -> Data? {
        let channelCount = Int(audioFormat.channelCount)
        let audioBuffer = buffer.audioBufferList.pointee.mBuffers
        guard let bytes = audioBuffer.mData else {
            return nil
        }
        return Data(bytes: bytes, count: Int(audioBuffer.mDataByteSize) * channelCount)
    }
}

[jaween]

Thanks, the fix from that commit works well. The only issue is now the constant print statements that are emitted while the microphone is on:

throwing -1
  from , render err: -1
throwing -1
  from , render err: -1
throwing -1
  from , render err: -1

[llfbandit]

Thanks for the feedback. I did not see it when testing this.
I'll have another look before releasing new version. Could it be a conversion error? There's not much print statement around here...

[jaween]

It appears to only happen when using the startStream() with echoCancel set to true, here is the method I'm using:

final micStream = await _recorder.startStream(
   RecordConfig(
     encoder: AudioEncoder.pcm16bits,
     echoCancel: true,
   ),
);

When echoCancel is false, or when using start() (with or without echoCancel), there are no print statements.

[llfbandit]

Echo cancel and auto gain are only available when streaming for now.

[callmephil]

Echo cancel and auto gain are only available when streaming for now.

You meant it's not available?

I have the same issue with the following config:

return recorder.startStream(
      const RecordConfig(
        echoCancel: true,
        noiseSuppress: true,
        encoder: AudioEncoder.pcm16bits,
        numChannels: 1,
        sampleRate: 16000,
        bitRate: 16000,
      ),
    );

[llfbandit]

Available with iOS 13+ only

[callmephil]

Available with iOS 13+ only

my pod version is set to 15. both in podfile and xcode and my lock file is updated.

[jaween]

Not sure but the version on pub.dev may not include the fix yet, but try setting your dependency to the latest commit:

  record:
    git:
      url: [email protected]:llfbandit/record.git
      path: record
      ref: 809a5e47c473f8c7296efd9fa16627a30f8344fd