.mp3 files not supported for SAD

[dannima]

Looks like pyannote-audio is using soundfile to load audio files, and soundfile does not support mp3 files. Is it possible to switch to audioread/ffmpeg?

[hbredin]

pyannote.audio relies on SoundFile because

• it is faster than most alternatives
• it supports seeking (which is used heavily during training to extract audio chunk at random position)

The counterpart is, indeed, the limited support for other audio formats.

For more insights, see this useful benchmark by @faroit: https://github.com/faroit/python_audio_loading_benchmark

That being said, once the model is trained, seeking is indeed not a mandatory features.
I’d gladly merge a PR that add support for other formats at inference time.

An alternative option is to use your own pyannote.database preprocessor to pre-load the waveform using any library you see fit.

https://github.com/pyannote/pyannote-database#preprocessors

pyannote-audio/pyannote/audio/applications/config.py

Lines 119 to 124 in cd2f2b5

	# preprocessors:
	# key:
	# name: package.module.ClassName
	# params:
	# param1: value1
	# param2: value2

[nryant]

Hi @hbredin , those audio loading benchmarks were originally run with a now 2 year old version of librosa. Since then, librosa switched to using soundfile by default, then falling back to an alternate backend if soundfile could not handle the format. When rerun with modern versions of librosa, the performance is MUCH BETTER.

For WAV and FLAC, the pure soundfile solution and librosa are essentially identical. If you'd like, I could make a pull request replacing the soundfile references with wrappers around librosa.

[faroit]

@nryant thanks for reminding me to finish this PR ;-)

[hbredin]

Thanks for the offer, @nryant.

I'd definitely merge such a PR adding support for more file formats.

However, I also want to make sure this does not slow things down. In particular (but this is true for all methods and functions in pyannote.audio.features.utils), we need to make sure that RawAudio.crop is at least as fast as it currently is for WAV files.

I am also concerned about the lack of unit tests for pyannote.audio.features.utils.
These tests should probably be added first, to avoid any regression when switching to librosa (and also catch any future regression with future releases of librosa).

[nryant]

Hi @hbredin ,

Yes, if there are not currently unit tests, those should definitely be added first. Since this code is relevant for performance, we should probably also add some basic benchmarks so that we can check for regressions. Perhaps using airspeed velocity?

As to changes to the underlying audio io in order to support additional formats, it seems that ideally the following four audio file formats would minimally be supported:

• wav
• flac
• mp3 (since it shows often in very large corpora such as Common Voice or the raw chapters from which LibriSpeech was sliced)
• sphere files NOT compressed using SHORTEN (since they appear so frequently in older LDC and NIST packages)

librosa supports the last two formats, though random reads from MP3 files are SLOOOW. Torchaudio also supports all four formats and is quite a bit faster with MP3, so that might also be an option. Direct comparison of librosa and tochaudio for loading an entire 8 minute long, 16 kHz recording into a Tensor (mean times in seconds as reported by iPython's timeit command):

format	librosa	torchaudio
wav	0.017831	0.058931
flac	0.115104	0.135125
mp3	0.580566	0.449611
sph	0.017851	0.058338

And the same table for reading a random 500 ms chunk of the same recording :

format	librosa	torchaudio
wav	0.000174	0.000125
flac	0.000405	0.000253
mp3	0.256025	0.003628
sph	0.000165	0.000142

Do you want to open an issue and/or pull request to organize this work?

[nryant]

After looking at the librosa and audioread code, there are other reasons to not rely on librosa for MP3 other than speed. In particular, audioread will use one of the supported backends available on the system, which may end up being ffmpeg. In which case the estimated (and potentially inaccurate) MP3 duration will be parsed from the output logged by ffmpeg to STDOUT, but only to the nearest 100 ms:

https://github.com/beetbox/audioread/blob/8d02710f5cd9f8db1290e4a3a08115e222b0790a/audioread/ffdec.py#L268

Maybe it would be safest to either depend on torchaudio or a combination of soundfile and torchaudio.

[hbredin]

Thanks @nryant for this analysis -- that is very helpful!
torchaudio looks like a very good candidate.

I just created a new issue to continue the discussion and keep track of the switch.

Closing this one (will try to import your analysis in there).

[Infinitay]

I was having issues with this lately myself. My issue went away for mp3s when I manually specified to install the latest available soundfile package, albeit I was forced to use v0.12.0 since that's the maximum version allowed for the dependency whilst 0.12.1 has been recently released

[nryant]

I'm not 100%, but I believe torchaudio uses soundfile, which in turn depends on libsndfile for reading formats. As of version 1.10, libsndfile does handle MP3:

https://github.com/libsndfile/libsndfile/issues/258

However, which version of libsndfile the soundfile package uses is more complicated. For Linux, soundfile always uses the system version, though they are working toward building and distributing wheels for Linux systems:

https://github.com/bastibe/python-soundfile/issues/353

For OS X/WIndows, they provide wheels which contain a pre-compiled shared library, though in my experience on OS X, if you already have libsndfile installed, soundfile often picks up the wrong library.

[Infinitay]

Last I checked torchaudio makes using soundfile optional but I believe we use it primarily. At this rate the more I learn about more dependencies the more lost I get on what is the main culprit. Although, I will look more into libsndfile personally. As I said, I updated my soundfile package to v0.12.0 which allowed me to use mp3s. However, I have a lot more m4as and unless I convert them into mp3s or another format that soundfile supports, I can't directly process them. I'll see if updating my libsndfile is possible and could help with that.

though in my experience on OS X, if you already have libsndfile installed, soundfile often picks up the wrong library.
I wonder if that's where my problems first started on Windows, it having chosen the wrong version. I guess I'll find out if I manage to attempt this later.

Thanks for the additional information

EDIT: That was short lived. soundfile v0.12.0 updated to use libsndfile v1.2.0 which is the current latest. Looking at their issues, doesn't seem like m4a will be supported yet due to patents or something? At this rate spending a few hours converting my files is much simpler.