Guideline to configure both bottom-up model and top down

[ttunja]

When trying to configure the bottom-up model and top-down I have read this page https://sleap.ai/guides/choosing-models.html#choosing-models

What I am concretely trying to figure out is the values Sigma values for Nodes and Sigma values for Edges for both of these types of models. Because these values are configurable for both of the model (top down and bottom up) for the top down anchor point is also available for which I know what to set.

And i have hovered over the input and saw the discription however I am not sure practically what does this mean to me.

Image size is 920x1200

And the relative size of the animal is shown underneath:

Additionally, should I change Output stride and Loss weight for the above sigmas in "Bottom-up model configuration"?

My 2 final question is regarding the augmentation and model selection.

1. You only suggest changing the rotation and brightness. Could you point me to other important things one can configure, there are a lot of parameters, and theoretically they make sense, however practically (depending on the animals, resolution etc) it is hard to guess what is a good starting point to configuring them. I doubt its the default ones? (since again image dependent)
2. Model selection, default is Unet, what are your suggestions for other (resnet etc) just try out and see or are there guidelines based on the size of the images, number of labels, accuracy ones want to achieve etc?

[talmo]

Hi @ttunja,

Do you mind trying out the steps described in my reply here?

If you're still having issues with accuracy, here are a couple more resources to check out:

• Instances were predicted on 0 frames #854 (comment)
• https://sleap.ai/guides/troubleshooting-workflows.html

Let me take a quick stab at the specific questions you had though:

What I am concretely trying to figure out is the values Sigma values for Nodes and Sigma values for Edges for both of these types of models. Because these values are configurable for both of the model (top down and bottom up) for the top down anchor point is also available for which I know what to set.

The sigma trades off between precision and ease-of-training. If sigma is <2, SLEAP will have a hard time training the model to predict confidence maps since they'll be too sparse. If sigma is >5, SLEAP will probably have an easy time training the model, but your localization precision will be pretty bad (i.e., the points will jump around a lot).

For the part affinity fields in bottom-up models, you'll likely want to leave that at the default (75 I think). This controls how wide the PAFs are around the edges -- here you pretty much always do better with having them larger since it provides more "support" for the model. If you're seeing a lot of part grouping errors, e.g., skeletons merged across instances when animals are interacting, then you might consider lowering it slightly to 50.

You only suggest changing the rotation and brightness. Could you point me to other important things one can configure, there are a lot of parameters, and theoretically they make sense, however practically (depending on the animals, resolution etc) it is hard to guess what is a good starting point to configuring them. I doubt its the default ones? (since again image dependent)

We're in the process of investigating the effect of more complex augmentation configurations, but I'll say for now that the single most effective one we've tried has been rotation. Since you have a camera view from above, I'd also recommend changing the rotation angles to -180 to 180. Looking at your data, I don't believe you'll see much improvement from using most of the other augmentations.

Model selection, default is Unet, what are your suggestions for other (resnet etc) just try out and see or are there guidelines based on the size of the images, number of labels, accuracy ones want to achieve etc?

Yep, the links above have some helpful info. In our experience, we've had a hard time beating UNet pretty much across the board. See, for example, the section on architectures in our paper and this supplementary figure where we evaluate a ton of the other architectures.

The key thing to consider is the receptive field size, and the built-in profiles span a good range that should work with nearly all types of data. My personal recommendation would be to give the bottom-up large RF profile a go.

Let me know how it goes!

Cheers,

Talmo