Combining CLI and COLAB for using pre-trained models

[auesro]

Dear all,

We (@isabelperezf and me) are using Sleap in the CLI for labeling frames and Colab for training but have some doubts:

We have already trained a top-down model in Colab, downloaded the "models" folder, opened the project locally, ran predict using the downloaded .json files, corrected the worst instances (according to the score). Now our doubts start:
-How do we proceed so that we can train a model starting from the previous one but taking into account the newly added frames? What would be the step by step process?
-We have tried exporting the training package as the first time but that doesnt export any model.h5 files so we are guessing COLAB will start training from scratch, right?

Thanks!

A

[talmo]

Hi @auesro and @isabelperezf,

You are correct that training does not pick up from the pre-trained model. This is intentional as we find that randomly initialized models trained from scratch every time are more stable and can train just as well. This is partially because we avoid "reinforcing" errors that would be learned early on when there is little labeled data available and the model is heavily overfitting to it.

That said, if you want to try it yourself, we have a notebook on resumable training you're welcome to run through. It's a bit more advanced than the basic training notebook, but if you follow it step-by-step it'll do what you're looking for.

Let us know if you this works for you or if you run into any issues.

[auesro]

Dear @talmo
Thanks a lot for the informative reply. In such case, we will continue adding the frames where the model predicts with lower scores.
Is there an accessible way to evaluate if a project is improving, as in getting better at predicting the positions of the bodyparts? Is there a measure of how well (in average of bodyparts I guess) the model is doing?

Thanks!

A

[talmo]

Hi @auesro,

Yep! There's an option to view evaluation metrics for trained models from within the GUI menu (below Run Training), but here's a notebook on evaluating model accuracy programmatically.

Cheers,

Talmo