Training multiple speakers - what is the point?

[Pentalimbed]

I have attempted to train multiple speakers at the same time, but the resulting voices are less distinct from each other. Is this what people call voice leaking / 音色泄漏? Is it because I need to train for more epochs, even though all losses seem to converge at the time?

With such a disadvantage, what is the point of training multiple speakers together? Does it improves pitches or generalization when there are less training materials for each speakers, compared to training them separately?

[nikich340]

I think it's a result of averaging some weights (speakers share some layers). In original VITS repo it was an interesting feature which allowed to do voice conversion between model speakers. Here, it seems to be a rudimentary option (as model itself is doing any-to-one conversion).
How many actors did you try?
I am interested, what is a maximum amount to still get good results.

[skol101]

Yep, after training for 36 epochs on VCTK dataset it looks like it's no better than QuickVC.