How to combine one large dataset and one good-quality smaller dataset by giving more weights to a smaller dataset?

[npovey]

Hi,
I have 15K hours of a good-quality dataset and 50K hours OK dataset.

Question: Does Lhotse offer a way to combine two manifests in a smart way? I want big dataset to be weighted less in my new combined model. I am new to combining datasets, so any feedback would be appreciated. I believe Lhotse combine function only combines two JSON files without weighting them.

Thanks,
Nadira

[csukuangfj]

I don't think you.need to.combine them materially.

You can have a look at how we handle multiple datasets in icefall.

Hint: Just grep "mux". You can control how often we sample a dataset.

[pzelasko]

Yes, I believe that mux is the best dataset combination method, especially when combined with natural weights (and possibly some rebalancing on top of that) and infinite iterables (cuts.repeat()), as per Canary paper https://arxiv.org/abs/2406.19674

We have a not too outdated tutorial here https://github.com/lhotse-speech/lhotse/blob/master/examples/03-combining-datasets.ipynb

[pzelasko]

I also recommend splitting your manifests into smaller pieces for each dataset and using CutSet.from_files() API to build a per-dataset CutSet - if you change the random seeds across different training runs, it will reshuffle the order of shards improving your sampling randomness.