Pose estimation slightly off

[justinecorsilia]

Hi! I have a training set of 6743 instances and am getting tracking errors in odd places (see below). I was wondering if there are any troubleshooting ideas or parameter suggestion you have other than those listed on the SLEAP webpage? (The first image below is the expected tracks and the second image is the tracks SLEAP produced)

[roomrys]

Hi @justinecorsilia,

Thanks for the question! Looking at the user-labeled Instance/annotation and the PredictedInstance, I definitely see that quite a few body part locations were predicted a few pixels off.

This could be due to using too low of an input scaling. Input scaling is nice because it allows us to save on computational cost, but when predicted locations are scaled back to full resolution on the original image, we lose information as to precisely which pixel the body part is located. Also, since the legs of your fly are only about 1pixel wide, any decrease in resolution might accidentally wipe-out the legs entirely!

My suggestion would be to use a top-down pipeline which actually uses two models: centroid and centered instance. Both models should have the same augmentation settings of +-180 rotation just to provide more examples at different orientation.

Centroid

The centroid model will find just a single point and take a centered crop around that point, passing the cropped image to the centered instance model.

Anchor Point

For the centroid model, we will need to specify an anchor point. The anchor point should be a body part location that is consistently visible. We will be training the centroid model to find and predict just the single anchor point body part - so it is important that the model is able to find it. Usually the anchor point is also somewhere in the center of our animal, but this is not imperative. I would suggest using the thorax (which might cause trouble for us if the animal flips over often, but we can always try different anchor points later).

Input Scaling

The input scaling should always be determined by how large the smallest body part is in terms of pixels. After multiplying by the input scaling, we still want to have the smallest body part be represented by at least 2 pixels (we only need 1 pixel, 2 is for safety). Since the centroid model only finds a single body part, this calculation should be simple. If we are using the thorax, which is a relatively large body part compared to your 1 pixel legs, then we can safely set the input scaling to 0.5 (or even to 0.25 if you're feeling frisky).

Centered Instance

The centered instance model will take in the cropped image and find all body parts at FULL RESOLUTION. The present input scaling is 1.0 (and the model does not support any other input scaling). We already save on memory with the top-down pipeline from finding centered crops at a low resolution and then using just the crops for the rest of the pipeline.

If you are still having trouble, then would you mind also attaching a screenshot of your Training Pipeline GUI with all the settings filled in (for both models please if you decide to top-down). Even if you aren't having troubles, we'd be happy to hear if things went smoothly.

Thanks,
Liezl

[justinecorsilia]

Hi Liezl, Thank you so much for your response, I really appreciate it. I believe my current input scaling settings are as you suggested, would you be willing to take a look at my current Training pipeline and let me know if there is anything obvious that I might want to change? Thank you again for your help! Sincerely, Justine [image: Image 6-6-24 at 4.43 PM.jpeg][image: Image 6-6-24 at 4.44 PM.jpeg][image: Image 6-6-24 at 4.44 PM (1).jpeg]

…

On Thu, 6 Jun 2024 at 15:46, Liezl Maree ***@***.***> wrote: Hi @justinecorsilia <https://github.com/justinecorsilia>, Thanks for the question! Looking at the user-labeled Instance/annotation and the PredictedInstance, I definitely see that quite a few body part locations were predicted a few pixels off. This could be due to using too low of an input scaling. Input scaling is nice because it allows us to save on computational cost, but when predicted locations are scaled back to full resolution on the original image, we lose information as to precisely which pixel the body part is located. Also, since the legs of your fly are only about 1pixel wide, any decrease in resolution might accidentally wipe-out the legs entirely! My suggestion would be to use a top-down pipeline which actually uses two models: centroid and centered instance. Both models should have the same augmentation settings of +-180 rotation just to provide more examples at different orientation. Centroid The centroid model will find just a single point and take a centered crop around that point, passing the cropped image to the centered instance model. Anchor Point For the centroid model, we will need to specify an anchor point. The anchor point should be a body part location that is consistently visible. We will be training the centroid model to find and predict just the single anchor point body part - so it is important that the model is able to find it. Usually the anchor point is also somewhere in the center of our animal, but this is not imperative. *I would suggest using the thorax* (which might cause trouble for us if the animal flips over often, but we can always try different anchor points later). Input Scaling The input scaling should always be determined by how large the smallest body part is in terms of pixels. After multiplying by the input scaling, we still want to have the smallest body part be represented by at least 2 pixels (we only need 1 pixel, 2 is for safety). Since the centroid model only finds a single body part, this calculation should be simple. If we are using the thorax, which is a relatively large body part compared to your 1 pixel legs, then we can safely set the *input scaling to 0.5* (or even to 0.25 if you're feeling frisky). Centered Instance The centered instance model will take in the cropped image and find all body parts at FULL RESOLUTION. The present input scaling is 1.0 (and the model does not support any other input scaling). We already save on memory with the top-down pipeline from finding centered crops at a low resolution and then using just the crops for the rest of the pipeline. If you are still having trouble, then would you mind also attaching a screenshot of your Training Pipeline GUI with all the settings filled in (for both models please if you decide to top-down). Even if you aren't having troubles, we'd be happy to hear if things went smoothly. Thanks, Liezl — Reply to this email directly, view it on GitHub <#1795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BI6ETHDMR4YPBO4GAWK4IX3ZGC4B3AVCNFSM6AAAAABIY6ZVD6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMOJTGQ3TC> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[justinecorsilia]

Hi again, I'm only just realizing that the photos attached in an odd way. These are the images I meant to attach yesterday:

[eberrigan]

@justinecorsilia you mentioned tracks in your first message. Just to clarify, you will get the tracks after you get pose predictions on each frame.

Please try adjusting the Rotation Min and Max Angle to -180, + 180 respectively for both models.

Also, was there a reason your Batch Size is 1 for the centered instance model?

Lastly, you should go through your labels and make sure they are correct. I see there that your Proboscis is marked as invisible. Is that intentional?

Thanks,

Elizabeth

[justinecorsilia]

Thank you, I will try the -180 +180 min and max angles. I have the Batch size as 1 because I had heard that lowering the batch size would help with prediction accuracy, is that wrong to assume?

[roomrys]

Hi @justinecorsilia,

batch size and training time

The batch size determines how many examples the networks sees together at a time. The network computes a gradient each pass, and, with a smaller batch size, the gradient is more likely to point in a sub-optimal direction (taking a less efficient path toward the desired minimum/solution). Thus, this would lead to slower convergence and therefore slower training.

batch size and prediction accuracy

However, since the smaller batch size explores different parts of the parameter space (by taking the path determined by the "sub-optimal" gradient), the eventually converged upon solution may actually be a better minima than if we had larger batch sizes (which would have used more "optimal" gradients and thus explored less of the parameter space).

So yes, it is possible (although not guaranteed) that the smaller batch size may lead to a better predictions, but this will make training take longer.

Thanks,
Liezl

[justinecorsilia]

Hi Liezel,

Thank you so much for your help! The model is actually working a lot better now!