ID Animal Tracking

[lenacochrane]

Hello!

I hope all is well.

For context, we are establishing a model which can effectively predict and track larvae in a Petri dish (between 2 and 10). I had found sleap fairly accurate at predicting the number and orientation of the larvae in a video, thus I decided to begin tracking; however, the tracking predictions weren't great, particularly in the videos containing 10 larvae.

I decided to begin with a new training topdown model which identifies track IDs simultaneously (multi-animal-topdown-id). For simplicity, I used videos containing 2 larvae. However, when running inferences post training, I found that all my predictions are in identical places across the video. For example, in the video below I ran inferences twice and each time two different tracks were allocated but predicted instances remained in an identical location throughout the video- nowhere near the larvae themselves!

Further, I retrained the model with data across two videos and now it doesnt predict inferences at all!

How do I fix such issue and am I training the model wrong? - I presumed through training with ID tracking that when running inferences it would simultaneously predict instances and tracks, however it has done neither?

Also, is there an issue with tracking across multiple videos in the same project, i.e each video with a distinct set of larvae? In the second photo attached I have multiple track 1’s due to multiple videos. Ideally each video should start from 1 but it seems strange to assign number 1’s identity and colour to a new video. Unsure if this will will affect the model also.

Further, can anyone recommend a more effective strategy to track multiple animals across videos?

Many thanks,

Lena

[Marielamend]

Hello @lenacochrane!

Some follow-up questions to better understand your workflow:

I had found sleap fairly accurate at predicting the number and orientation of the larvae in a video, thus I decided to begin tracking; however, the tracking predictions weren't great, particularly in the videos containing 10 larvae.

1. Which model type did you already attempt to use?
2. Did you set a limit to the # of tracks?
3. Can you send a screenshot of your settings when doing inference tracking?

For supervised ID models the animals have to be visually distinct allowing users to label the tracks associated with an instance. Based on your dataset, we recommend using a regular top-down approach + tracking.

We can continue working with you on making tweaks to your settings and other troubleshooting but also the team is currently working on an improved ID tracking model, so that is something to look forward to!!

Best,
Mariela

[lenacochrane]

Hello Mariela,

Sorry for my delayed reply.

1. We are using the multi animal top-down model.

2. We have been limiting the max number of instances (10) (see screenshot) but not the max number of tracks. Should we be doing this the other way, and if so, why?

3. Screenshot attached. The combination of flow, centroid and hungarian has been best so far (although this is video dependant?).

In regards to the kalman filter-based tracking, is there an ideal way to get this to work efficiently? I have tried adding a few frames with track IDs at the beginning of my video but it appears to make tracking worse. Also, I had a question regarding post-tracker data cleaning: when the option is ticked, it doesn't seem to be making any difference at all- is this to be expected? I am unsure of the benefits of allowing this data cleaning when it doesn't seem to improve the tracking?

That is something to look forward to- thank you for continually improving SLEAP. :)

Best,

Lena

[Marielamend]

Hello @lenacochrane!

Here are some answers for you from the SLEAP team:)

We are using the multi-type animal top-down model.

Great! In regards to the type of model, the regular top-down model will work best with the size of the animals relative to the frame.

We have been limiting the max number of instances (10) (see screenshot) but not the max number of tracks. Should we be doing this the other way, and if so, why?

You can do the max number of tracks + 1 for the instances. Limiting the number of instances can help reject extraneous detections, but sometimes you may get a duplicate of one instance and thus reject a correct one entirely, which will throw off the tracking. Having a bit of wiggle room can get you the best of both worlds.

In regards to the kalman filter-based tracking, is there an ideal way to get this to work efficiently?

In the team’s experience, the kalman filter based tracking has been a bit finicky, in part because of the sensitivity to missing values in our current implementation. We would not recommend it in its current state, but you're welcome to experiment with it. We've found the most success when you choose a subset of nodes that are really reliable for use with the kalman filter.

Also, I had a question regarding post-tracker data cleaning: when the option is ticked, it doesn't seem to be making any difference at all- is this to be expected?

Yes, with the newest update where we use track-local buffers for limiting the max number of tracks, it does the job of the 'Connect Single Track Breaks' option. It was left it in because it's still useful for correcting spurious track switches when new tracks are created when they are not limited to a max number.

Have a great rest of your day,
Mariela

[lenacochrane]

Hello Mariela,

I hope all is well! Thank you for answering these questions, very useful!

I was hoping you could help answer a few more questions (sorry)?

1. We have been noticing that even if predictions in isolation are near perfect, the tracking output is not accurate. Is there a way to predict every frame and then from this join up the dots in a form of track? Unsure how the current tracking framework works and why there is such a distinction between the predictions and tracking accuracy.

2. We are using our model to predict and track various size larvae (images 1 and 2). Is it okay to use larvae of such different sizes, and if so, will it take far more training to allow the model to compensate for this?

3. Further, we have larvae on two different types of petri dishes: agarose and food-filled (images 3 and 4). It is slightly confusing because the ideal tracking parameters for the agarose filled dishes rely on flow, but for the food filled dishes, it's the simple algorithm. Is this discrepancy to be expected? Further, the tracking is significantly worse on the food filled petri dishes despite equal agarose and food petri dish training?

Many thanks and look forward to your reply!,

Lena

[Marielamend]

Hello @lenacochrane!

Unsure how the current tracking framework works and why there is such a distinction between the predictions and tracking accuracy.

No worries! Hopefully, I can provide a bit of clarification:
The current workflow consists of iterations of reviewing and fixing predictions until you have accurate frame-by-frame predictions. After manually labeling animal pose(s) in selected frames, then training a machine-learning model using the labeled frames would proceed → the output of this step are pose estimations.

Then once you have accurate predictions (just like you have now!) You can now predict for entire clips by going to Predict > Run Inference… > Predict On: entire current video.

Tracking is done through a frame-by-frame comparison of the distances between poses of all animals across frames - no model/learning required, just a cost matrix. (which actually→ The team is working on a new tracking model that involves a learning framework, the model I had mentioned before:) )

Here is a great explanation of how the different accuracy scores are derived for more insight!

We have been noticing that even if predictions in isolation are near perfect, the tracking output is not accurate. Is there a way to predict every frame and then from this join up the dots in a form of track?

• A follow-up question, in what way is your tracking output not accurate, do you mean the identities are not being tracked correctly or the poses are not correct after you track?

Poses are not accurate → This should not be occuring
Identities are not accurate→ This is a common mistake made by the tracking code, the next step would be to do some quality control such as proofreading

We are using our model to predict and track various size larvae (images 1 and 2). Is it okay to use larvae of such different sizes, and if so, will it take far more training to allow the model to compensate for this?

Yes, you can have these varying sizes, it will just require labeling enough examples that are representative of all the sizes across. If it’s feasible to do so based on your datasets, you can also train a model for each size group such as the larger larvae/ smaller larvae. We find that specialist models perform better than training a model that can generalize across different sizes.

Further, we have larvae on two different types of petri dishes: agarose and food-filled (images 3 and 4). It is slightly confusing because the ideal tracking parameters for the agarose filled dishes rely on flow, but for the food filled dishes, it's the simple algorithm. Is this discrepancy to be expected? Further, the tracking is significantly worse on the food filled petri dishes despite equal agarose and food petri dish training?

These qualitative differences could be happening by chance or not measured systematically. If one works better relative to the dish, you can use the specific parameter (flow vs simple) that works best for you! We recommend starting with simple because it is faster overall whereas using flow can be slower but can compensate when there is fast movement or overlapping of animals.

Best,
Mariela:)