QUESTION ABOUT TRAINING TEXT DETECTION MODEL

[anhalu]

when I train a text detection model using pretrained db_resnet50, I find it quite slow to calculate. it only use 1 thread in CPU, ( I have 24 thread in my CPU and 24GB in CUDA ). How do I speed up the calculation faster ?
I use pytorch and use this command for training scrip :
python references/detection/train_pytorch.py path/to/your/train_set path/to/your/val_set db_resnet50 --epochs 5 --device 0 --pretrained
I see at base.py in DB of doctr model, it has a lot of numpy, so it is using too much CPU to calculate, how can I increase the thread CPU for training or use more GPU for training ? I need help. Thank a lots

[anhalu]

@felixdittrich92 can u help me ? thank u so much

[anhalu]

@odulcy-mindee plz, can you take a look ? thank a lot :((

[felixT2K]

Hi @anhalu 👋🏼,

Excuse the late response 😅
1 thread in CPU this sounds weird ... could you share some more information about this behaviour (maybe htop log)

Try to specify the workers by yourself with -j <NUMBER_WORKERS> (e.g.: -j 16) and with 24 GB VRAM you can also increase the batch size (try 12 / 14 / 16)

This has nothing to do with the models postprocessing it's more we apply by default a lot of augmentations which takes some time.
Feel free to modify this to your needs :)

doctr/references/detection/train_pytorch.py

Line 269 in 8e0609d

img_transforms=Compose(

[anhalu]

[anhalu]

@felixT2K when training, i see in htop, it only use 1 thread cpu to calculator, and it takes up 5gb gpu, I don't think it's a data loading problem, I think it has a problem with the loss calculation of base.py in detection forder. I see a lot of places using numpy in base.py, does that affect the calculation when training the model? and how do I speed up the training process

[anhalu]

My data is only 27 sample to test training, but It takes a lot of time to process calculations on the CPU side. Did I do anything wrong at any stage? Thanks a lot for response

[anhalu]

@felixT2K I don't do anything in lib, I just git clone and run the command " python references/detection/train_pytorch.py /home/anhm/data/word_det/data_doctr /home/anhm/data/word_det/data_doctr db_resnet50 --epochs 50 --device 0 --pretrained -j 16
". and it run slowly. it took 16 minute to finish one epochs.

[anhalu]

@felixT2K oh, I have question, Does it matter if I leave img_hash as empty in labels.json when I prepare the data for training ?

[felixT2K]

No that's fine

[felixT2K]

but .. your polygon annotations looks not correct

-> 'polygons': [[[x1, y1], [x2, y2], [x3, y3], [x4, y4]], ...]

[anhalu]

Oh, It need 4 point ?, I just use top left and bottom right for rectange box. let's me try prepare data again. Does it need to be calculated in any order?

[felixT2K]

The order of the points does not matter inside a polygon. Points are (x, y) absolutes coordinates.

:)

[anhalu]

Thank a lot, it worked. I didn't carefully check the number of waypoints. <3

[anhalu]

I missed 2 point, so it cannot run property. I just fix that, So it worked now.