Influence of the ML model performanace on tracking

[ttunja]

What is the exact extent of influence of the model on the tracking (either top down or the bottom up versions).

Here is what I mean by that, taking for example the top down model. For every frame a network will first finds each animal (object detection task) and then a separate network estimates the pose of each found animal (pose estimation task).

1. object detection
2. pose estimation

Now tracking works (depending on the exact configuration) on some basic computer vision manipulations, not really machine learning. But how much does failing on the two tasks above influence the results of the tracking.

So my concrete questions are:

1. Is it true that object detection and pose estimation is done on every frame? Wouldnt this be inefficient, and we ought to do it sporadically mostly in the initial frames? Hopefully this is configurable too?
2. What are the ways that object detection and pose estimation influences the tracking. Its obvious that if the networks initially (first frames) are bad in detecting the sperate objects and estimating the pose then the tracking will fail. But are there any other situations where this is detrimental?

[roomrys]

Hi @ttunja,

Great question!

1. Yes, object detection and pose estimation is run on every frame (that you specify during in the inference pipeline) to find the locations of the nodes/body parts.
2. We currently need the locations of the nodes/body parts across sequential frames to perform tracking as we have a temporal tracking method implemented (no classifiers are used). The tracking is implemented as a cost function either summed across all frames (hungarian) or constrained to the current frame (greedy) with three different options for cost - instance: the distance between all node locations of multiple instances across frames, centroid: the distance between only the centroid, and iou: the intersection over union of the bounding boxes. There is a bit of documentation about the tracking methods here.

Hope this helps clear things up - please let us know if you have any more questions.

Thanks,
Liezl

[ttunja]

Hi @roomrys,

thank you for this. Just to clear up

1. I cant make this configurable (i.e. run it not on every frame, but with that loose accuracy) maybe a good feature for future releases?
2. That means that if my nodes/body parts were wrongly predicted (i.e. pose estimation task of ML failed) we will also have poor tracking, correct?

[roomrys]

Hi @ttunja,

1. You will need to run object detection and pose estimation on any frame that you want a prediction on. We have a few options for which frames to predict on as seen in the sleap-track documentation.
2. If your nodes/body parts were wrongly predicted, then the tracking will be affected since we use a distance-based cost function.

Thanks,
Liezl