Dimension Mismatch Error in parseq Model

[ep0p]

I have been fine-tuning recognition models and encountered an error when using the parseq model. I am using the same dataset that worked with other models, but I encounter an error when running the following command:

python references/recognition/train_pytorch.py parseq --train_path /home/epop/DATASET/ds_3p5_rec_v1/train --val_path /home/epop/DATASET/ds_3p5_rec_v1/val --name parseq_v1 --wb --epochs 10 --batch_size 32 --pretrained --vocab="frenchplus" --device 1

Error Traceback:

  File "/home/epop/dataset_prep/doctr/references/recognition/train_pytorch.py", line 489, in <module>
    main(args)
  File "/home/epop/dataset_prep/doctr/references/recognition/train_pytorch.py", line 394, in main
    fit_one_epoch(model, train_loader, batch_transforms, optimizer, scheduler, amp=args.amp)
  File "/home/epop/dataset_prep/doctr/references/recognition/train_pytorch.py", line 124, in fit_one_epoch
    train_loss = model(images, targets)["loss"]
  File "/home/epop/anaconda3/envs/ds3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/epop/anaconda3/envs/ds3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/epop/dataset_prep/doctr/doctr/models/recognition/parseq/pytorch.py", line 345, in forward
    mask = (target_mask.bool() & padding_mask.bool()).int()  # (N, 1, seq_len, seq_len)
RuntimeError: The size of tensor a (32) must match the size of tensor b (31) at non-singleton dimension 3

[felixdittrich92]

Hi @ep0p 👋,

Excuse the late response, that's a problem with the max_length could you filter your reco dataset please and keep only samples with max 30 chars and try it again ?

Best regards,
Felix

[felixdittrich92]

Hi @whoami02 Unfortunately no.
The detection and recognition works only on word boundaries (not lines or paragraphs) which is also the reason that the vocabs doesn't contains whitespaces.

Example datasets can be found here: #1654

Best regards,
Felix :)

[ep0p]

Hi @felixdittrich92 ,

You were right, thank you! After running the training without --amp, the NaN loss issue disappeared. Parseq is my last hope :)

I’ve been fine-tuning several recognition models and testing them, but I’ve noticed that their performance during testing is much poorer than expected. I understand that these models are pre-trained on French documents, and I’m also fine-tuning them on French legal documents, so I’m not sure where I might be going wrong (ie)

Could you provide some guidance on what common pitfalls or mistakes I should watch out for when fine-tuning these models?

Any advice would be greatly appreciated!

[felixdittrich92]

Hi @ep0p 👋,

How much samples are in your dataset actually ?

It's a common issue in many libraries because the models "forget" over time .. So in an optimal case you would merge the dataset which was used for pretraining with your new data partial .. which is unfortunately not possible.

Have you already tried to freeze the backbone with --freeze-backbone ? This makes only the decoder trainable..so feature extraction will be "untouched".

[ep0p]

My dataset is currently 1.5 million words (I read in a previous post that good results are usually seen with datasets starting from 1 million words and above). I haven’t tried the --freeze-backbone option yet, I didn’t fully grasp how it works at first, but I’ll give it a go now. :)

Thanks again for the suggestion, I’ll keep you posted on how it goes!

[felixdittrich92]

Maybe with >1 million you could also try to train from scratch so without --pretrained.
But i agree this needs a bit testing :)