pos loss cannot be reduced

[FengheTan9]

Hello author, I used your original voco_head file for pre-training, but found that the pos loss could not decrease normally. What could be the problem?

[Luffy03]

Hi, many thanks for your kind attention to our work!
Would you please provide more details about your pre-training datasets, settings, and environment? And would you please print the labels and logits for debugging?

VoCo/models/voco_head.py

Line 203 in f70606b

print('labels and logits:', label[0].data, logits[0].data)

We also provide our training log here https://www.dropbox.com/scl/fi/rmqy9n2gio5tptbhlt239/20240115_232208.txt?rlkey=0jmnpz3n77bb1b9r9wt9aqkrv&dl=0, for comparisons.

[FengheTan9]

bs=4 ir=2e-4 10k ct datasets:

loss function:

[Luffy03]

Weird, seems the loss did not converge as our provided log. I cannot find the problems now.
The batch_size is 4 and sw_batch_size is still 2, right? And what are your GPU, versions of torch and Cuda? For some environments, we cannot use amp. And would you please also try lr=1e-4?

[FengheTan9]

Thanks for your reply. The batch_size is 4 and sw_batch_size is 1. Because I used voco_head_old, which does not include student-teacher. In addition, the reason for setting it to 1 is that I saw that there was an operation that modified the random cropping to 1 before.

[Luffy03]

Oh I see. You are using the old version. The link here is some researcher modified by himself, which may be due to the limitation of GPU memory. For the old version, I recommend you to try sw_batch_size=4 if you have enough GPU resources.
By the way, our current version with student-teacher is more stable and we will release more powerful models soon.