Fine-tuning and word generation for new font

[hanshupe007]

I want to fine-tune a model. As for my use case only arial is used and no other fonts, i want to try:

python references/recognition/train_pytorch.py crnn_vgg16_bn --epochs 5 --font "arial.ttf" --pretrained

This should execute the word generator. I was wondering if I can pass any additional parameters to the word generator, e.g. to specify how many training and validation samples are generated, which words it generates, introducing any scan artifacts, etc?

[felixdittrich92]

Hi @hanshupe007 👋,

The training with the internal generator doesn't works well enough atm.
We use it more for debugging, etc.

Take a look at:
https://github.com/felixdittrich92/synthtiger/tree/doctr-modified
I used this to generate ~1M samples (80/20 split) to fine tune: https://huggingface.co/Felix92/doctr-torch-parseq-multilingual-v1

[felixdittrich92]

There is a branch named: doctr-modified you need to chekout :)
It's not something official yet i used this on my own to create a synth multilingual dataset :)

[felixdittrich92]

Depends a bit what you want to do:

Still training on french vocab only adding a few new charaters / training on a complete new vocabulary (latin ? or something like cyrillic ?)

[hanshupe007]

the goal is to tune the model for Arial font, as the used documents are only in this font (latin, including numbers), currently there are smaller issues with Os and 0s etc. Any recommendations regarding training from scratch / fine-tuning / training last layers and training samples are welcome :)

[felixdittrich92]

reference datasets can be found here: #1654

• Take a look here i think it's a really helpful discussion: Unexpected Poor Inference Despite High Evaluation Scores: Validation Loss 0.0031 (Exact: 99.70% | Partial: 99.74%) #1677 and Dimension Mismatch Error in parseq Model #1679
• In general if you can provide real examples for the validation dataset it's super helpful and makes it easier to validate the performance / for training i would generate some data
• training from scratch would make only sense if all the data it will "see" while inference are really close to the training samples (complete fine tuning and decoder fine tuning (backbone freeze) seems to be a better fit for your case)
• depending on your hardware i suggest parseq as recognition model it's the best model (compared quality and inference latency) we provide atm.
• option outside of docTR: use an light LM for grammar correction

[hanshupe007]

thanks for that, inference latency is less a priority, the default model gave best performance so far but will give parseq a chance :⁠-⁠)