After finishing the training, continuing the training revealed that the curve of learning rate had reversed。

[Mr47121836]

This is my log and tensorboard log.

[p0p4k]

Did you stop and continue training?

[Mr47121836]

这是来自QQ邮箱的假期自动回复邮件。你好，我最近正在休假中，无法亲自回复你的邮件。我将在假期结束后，尽快给你回复。

[Mr47121836]

Did you stop and continue training?

yes，when i stop and continue training，i discover learnning rate is not the last learnning rate.And I also discover orignal vits also have this situation.
This is vits train

[p0p4k]

Yea its normal, the checkpoint saved at different step and continue is from that step, while model was trained a little bit more steps in the first run, the board shows everything. You can sort using Wall button in the tensorboard in the left side.

[Mr47121836]

Yea its normal, the checkpoint saved at different step and continue is from that step, while model was trained a little bit more steps in the first run, the board shows everything. You can sort using Wall button in the tensorboard in the left side.

but,When I continued training, I found a huge change in the curve.



I thing it can't load the checkpoint ?

[Mr47121836]

Yea its normal, the checkpoint saved at different step and continue is from that step, while model was trained a little bit more steps in the first run, the board shows everything. You can sort using Wall button in the tensorboard in the left side.

but,When I continued training, I found a huge change in the curve. I thing it can't load the checkpoint ?

The total loss have a huge change