Wav2Vec-U 2.0: could not training with fp16

[xiabingquan]

❓ Questions and Help

Before asking:

1. search the issues.
2. search the docs.

What is your question?

When training Wav2Vec-U 2.0 models following the official configuration, I tried training with fp16 but leads to errors. The losses will be NaN.

fairseq/examples/wav2vec/unsupervised/config/gan/w2vu2.yaml

Lines 3 to 10 in 3f6ba43

	common:
	fp16: false
	fp16_no_flatten_grads: true
	log_format: json
	log_interval: 100
	tensorboard_logdir: tb
	reset_logging: false
	suppress_crashes: false

Code

No code needed.

What have you tried?

Kind of stuck on debugging. Have no idea.

What's your environment?

• fairseq Version: '0.12.2'
• PyTorch Version: '1.13.0+cu117'
• OS: Linux avsu-ESC8000-G4 5.15.0-69-generic
• How you installed fairseq (pip, source): pip -e install
• Build command you used (if compiling from source):
• Python version: Python 3.8.13
• CUDA/cuDNN version: 11.7
• GPU models and configuration: 4 GeForce RTX 3090
• Any other relevant information:

[XR1988]

Thanks for your work. What's the current status? I'm not getting good results; my UER is stuck around 90.
I've cloned this repo (it might have environment problems new): https://github.com/oneapi-src/ai-transcribe
Others cloned it with a virtual environment: https://github.com/voidful/wav2vec-u-exp
I'm having trouble with this: #5572
@xiabingquan