New to SLEAP - predicting tracks creates new instances?

[alexandra-clem]

Hello! I'm an undergrad fairly new to SLEAP and programming, and I was wondering if I might be doing something wrong along the inference pipeline to generate extra instances when I run inference. I have videos with two mice, and when I run inference, tracks are not "assigned" to previously labeled instances but rather generated as new instances - so I have four instances per frame instead of two. The previously labeled instances are then tagged as "not in training data set." Is this normal, or is there something I'm missing? From what I understood reading the documentation and some discussions here, tracks should categorize user-labeled instances into separate groups.

To work around this I've been going through each frame and deleting the instances I previously labeled so the only remaining instances are the ones assigned to tracks, but this isn't very efficient.

Attaching screenshots of my inference pipeline and "instances" tab. Sometimes the nodes of predicted tracks also deviate pretty significantly from the ones I labeled, as shown in the third screenshot. Thanks so much for any guidance and clarification!

[roomrys]

Hi @alexandra-clem,

Oh boy - yeah that sounds pretty like a pretty cumbersome workflow.

Some jargon and background

Alright, so... there are two distinct pieces here: pose estimation and tracking. SLEAP uses user-labeled data to train models to output pose estimations. But, then we also need a way to keep track of which animal is which, so we decided to use the term "Track"s. Tracking was traditionally done separate from the pose estimations as a second step. Whereas the pose estimations use machine learning to train a model to output predictions, tracking was just implemented as a temporal, frame-by-frame comparison of the distances between poses of all animal in adjacent frames - no model required, just a cost matrix.

However, we (not so) recently now added "supervised ID" models that also allow users to label the "Track"s of visually distinct animals which just bundles in the tracking step into the model and foregoes the additional second pass previously required for tracking.

Action items

Your mice look very similar, so I would NOT advise trying to annotate the "Track" yourself and using an supervised ID model. Instead, I would forget about tracking until the very end of your project - until after the pose estimations look good. Then, you can run tracking on SEQUENTIAL frames (i.e. the entire video) and fix/proofread any track switches at the end of your project.

Anytime there is a predicted instance/animal/pose-estimation on a frame that is not linked to a user-labeled instance, it will be outlined in a yellow box to highlight that it will not be used in training. In fact, predicted instances will never be used in training as they cannot be taken as ground truth. Only user-labeled instances are used in training. The yellow box is just a warning in case the user corrected some predictions in a frame but accidentally forgot to correct all predictions. By "correct", I mean "double-click to create a user-labeled instance from the underlying prediction and then make any adjustments to the expected pose". The yellow box is supposed to remind you to always fully label a frame - never leave some instances labeled and others unlabeled.

Hope this helps clear some things up,
Liezl