Two Mouse Topdown 4000 + labeled frames help

[MweinbergUmass]

Hello Sleap Team,

I have been working on a project for a few months now. I have 35 videos and am now at around 4000+ labeled frames. I have been having a lot of trouble in parameter selection and wanted the input of some experts. Below, I have added screenshots from the evaluation process, the model parameters (of the models that have so far performed the best), a few labeled frames, and a few typical errors.

Evals:

Examples:
(My label)

(No labels)

Params:

I am worried about a few things:

1. The variation in noise due to variation in focus of the camera.
2. There is some (very little) change in the zoom of the camera.
3. The animals are of varying sizes and ages.
4. The labels are heavily loaded towards the first few videos. (Each video has at least 50 labels.)
5. Lighting conditions and motion blur (not totally sure what to do about this.)
6. Overlap between animals. Not totally sure how to deal with this... When I decrease the sigma, I find a decrease in model performance overall.
7. I'm worried about changing the structure of my skeleton, although I'm aware that decreasing the number of nodes should increase performance. There are certainly some nodes I can do without, perhaps I should try this? How can I do it such that I don't have to relabel all the frames I've already labeled?
8. Varying posture of the mice. I'm not labeling the tail, and that makes having a super consistent midpoint for the centroid difficult. What key point do you recommend for the centroid? Should I just use the metrics to guide which point I choose?

For any changes I would make, I am, of course, concerned that I've already labeled 4000 frames, but I would be willing to make significant changes if, after trying some little things, we get no results.

A few things I've tried:

1. Online keypoint mining
2. Increased batch size
3. Turning on and off up-interpolate
4. Lowering and increasing the sigma for both models.
5. Increasing and decreasing the input scaling.

For everything I've tried, I'm worried that some of it may actually be helpful, but just not in the exact combination of parameters I used. Therefore, some guidance about parameters would be extraordinarily useful. Currently, I'm thinking of trying to up the filters to 32 and max stride to 32 (for the centered instance model). Does that make sense?

Any help/guidance for parameter selection and further fine-tuning would be greatly appreciated.

[Marielamend]

Hello @MweinbergUmass,

Thank you for your detailed inquiry!

Currently, I'm thinking of trying to up the filters to 32 and max stride to 32 (for the centered instance model). Does that make sense?

For centered instance:

• You can try this as increasing your max stride to 32 will increase your receptive field size which should be around the size of the animal that you want to track. (Receptive field represented by blue box on the left of the model configuration window

In regards to some of your worries,

3. Make sure that you have enough examples in your labels that are representative of all ages/sizes. You'll want to make sure that the model has a large enough receptive field size for the largest animals, this can be done as stated before by increasing the max stride.

4. You definitely can change the structure of your skeleton (all previously annotated instances will automatically be updated to the new skeleton) :
   Open the project
   File --> Save as to a new .slp file.
   Delete the nodes you want to delete from the Skeleton panel.
   Continue labeling as normal, add new videos, etc.

5. When using the top-down approach, it’s a good idea to choose an anchor part that has a relatively stable position and that can be consistently found. In your case, you could choose “Tail_start”.
   

Regarding your concerns about your camera issues,
The zoom should not be much of an issue and the rest is hard to tackle but hopefully, with these suggestions, you can get a better performance regardless of these variations.

Additional note:
If you can modify your skeleton, you can try a bottom-up approach! This approach first predicts on the location of body parts and then groups them into distinct animals. This approach sometimes works better in mouse datasets. Considering your skeleton as is, this approach is not recommended but if you simplify your skeleton this could be an approach you can try and experiment with! You can take a look at an example template under Skeleton (excluding the tail).

Let us know how it goes!

Best,
Mariela

[MweinbergUmass]

This is super helpful, thank you so much!

Besides trying what you mentioned, do you think the parameters set as they are make sense?

Also, another, unrelated question:

If I want to export my labels (just the labels I've done - not predictions), but I don't want to export the image data (just the xy coordinates for the points for each instance, how can I do that? I've tried deleting all predictions and exporting, but the returned h5 has only half the labels; it's missing the other instance (maybe because neither is associated with a track for the labels?).

Thanks so much!

[Marielamend]

Hello @MweinbergUmass!

You are right! If there are no tracks in the project, then the export function assumes that the project is a single animal project hence the missing instance. I hope you were able to get were able to get better predictions with our suggestions, if there's anything else we can help with let us know!

Best,
Mariela

[MweinbergUmass]

Thanks for your response Mariela.

Yes, my predictions improved significantly!

Thank you!