There are not many speech recognition engines/models for the Catalan language. This benchmark aims to collect and compare the available models and serve as reference for projects willing to include catalan speech recognition.
Two datasets have been initially used to benchmark the different models:
- The test set of the Common Voice dataset from June 2020 (ca_579h_2020-06-22)
- The crowdsourced high-quality Catalan speech data set (https://www.openslr.org/69/)
As it is not possible to assure that neither of the test datasets has been used when training the evaluated models, a new dataset is in preparation for future tests.
This benchmark considers three metrics: word error rate, real-time factor and model size.
Defined as the ratio of the Levenshtein distance between the reference transcript and the prediction, to the number of words in the reference transcript.
The ratio of the recognition response time to the audio length. The smaller the ratio, the faster it takes to infer speech. This is not considered for cloud-based recognition engines.
As this metric depends on the hardware the model runs, all tests have been executed sequentially on the same machine, an AWS g4dn.xlarge instance:
- 4 vCPU (2nd Generation Intel Xeon Scalable (Cascade Lake) processors)
- 16 GiB Memory
- NVIDIA T4 Tensor Core GPU (16 GiB GPU Mem)
GPU was used when the model supported it.
The uncompressed size of the model as needed by the benchmarking scripts.
The following models have been tested, if you publish or know of any other speech recognition model for the Catalan language please open an issue!
A CMUSphinx based model released by Col·lectivaT in November 2018 with AGPL-3.0 license. The model was trained on the ParlamentParla and TV3Parla datasets. More information can be found in their website or github repository.
A Mozilla DeepSpeech based speech recognition model, trained on the Common Voice June dataset, ParlamentParla and FestCat.
A Vosk-API compatible lightweight wideband model for Android and RPi for Catalan by Alpha Cephei
The speech recognition service offered by Google
The speech to text service offered by Azure
| Model/Engine | WER | RTF | Model size |
|---|---|---|---|
| Col·lectivaT CMUSphinx v0.4.0 | 45,89% | 0,69 | 106 MB |
| deepspeech-català v0.7.0 | 21,69% | 0,18 | 233 MB |
| vosk-model-small-ca-0.4 | 11,46% | 0,04 | 91 MB |
| Google Speech-To-Text | 10,97% | N/A | N/A |
| Azure Cognitive Services | 11,72% | N/A | N/A |
| Model/Engine | WER | RTF | Model size |
|---|---|---|---|
| Col·lectivaT CMUSphinx v0.4.0 | TBD | TBD | 106 MB |
| deepspeech-català v0.7.0 | 18,41% | 0,18 | 233 MB |
| vosk-model-small-ca-0.4 | 11,71% | 0,04 | 91 MB |
| Google Speech-To-Text | TBD | N/A | N/A |
| Azure Cognitive Services | TBD | N/A | N/A |
- Both Google and Azure convert numbers and time to their numerical representation (quite accurately) but the original test data represents them textually. These were converted back in a semi-automatic post-processing step in order to correct the WER when they were semantically correct.
- Azure also adds punctuation, which was removed in the post-processing step.
- Azure fails to add middle dots and hyphens, which worsens its WER notably. These were not corrected as it didn't seem right to do so (the previous two corrections did not change the semantics of the sentences whilst failing to add hyphens or middle dots renders different words)
- Azure's profanity check appears not to be language specific as it marked "munt" (mount, hill) and "retard" (delay) as offensive.