About samplers for contrastive learning

[t13m]

Hi everyone, I'm trying to implement such a scenario for TTS that every utterance is provided along with some other utterance randomly sampled on the same speaker.

For example, I have utterances A, B and C of speaker 1, utterances D, E, F of speaker 2. I would like to feed the system like this:

{
  "feature": "A",
  "feature-sampled-from-same-speaker": "part of C", # for C is too long, it is truncated.
},
{
  "feature": "F",
  "feature-sampled-from-same-speaker": "D,E", # for D and E are short utterance, they are concatenated.
},

I am aware that the CutPairSampler can be used, however the pairs are fixed before the training start but not dynamic sampled. Any help would be appreciated!

[t13m]

In case there is no direct way to do this now, can I just use a bucket sampler in which every speaker has a bucket, and then the main sampler also draw samples from this sampler according to the consistent speaker id?

[pzelasko]

Hmmm if you want to sample it dynamically I'd suggest sth like this: inside your Dataset object, create per-speaker samplers or cutsets (kind of like buckets), and fetch an item from the one that corresponds to speaker, e.g.:

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, ..., speaker_emb_cuts: CutSet):
        ...
        self.spk2cuts = {spk: speaker_emb_cuts.filter(lambda c: c.supervisions[0].speaker == spk).to_eager() for spk in speaker_emb_cuts.speakers}

    def __getitem__(self, cuts):
        ...
        spk_emb_cuts = CutSet([self.spk2cuts[c.supervisions[0].speaker].sample() for c in cuts])
        ...

(if you had very very large data and/or used webdataset/lhotse shar you'd need to adjust the code a bit, otherwise it'll work in most cases)