zipformer and SSL feature adaptation

[alanshaoTT]

I was wondering if zipformer has any specific modules or functions designed for fbank features? I’m using pretrained wav2vec2.0 representations as input for zipformer training, but I’m having trouble with the model’s loss not converging well. I’m following the librispeech/zipformer recipe, but when I used the same representations with librispeech/pruned_transducer_stateless7, it converged just fine. I noticed the main difference between these recipes is the zipformer encoder. Is there something specifically designed for fbank features in librispeech/zipformer that could be causing this?

[marcoyang1998]

What do you mean by not converging well? Is it having poor WERs?

[alanshaoTT]

My model's loss is quite high, fluctuating around 2.0, and it hasn’t decreased much. The WER is also high.

[alanshaoTT]

this is my models loss