Multiple Instances for the Same Animal, Even Though Max Instances is Restricted in Training Pipeline

[majaneubauer]

I am currently using Google Colab and SLEAP (Version 1.3.3) to train a bottom-up multi-animal model. I have videos of two mice (one black and one white) in an open field arena, filmed from below. I am training my model so that it is able to identify the two animals in my video. So far I have trained on 515 frames. However, I have run into some issues:

When predicting on a new video after training, I often get up to four predicted instances for the same animal (see attached screenshot). What I have been doing so far is double-clicking one of the instances and correcting the predictions, while ignoring the other instances (see screenshot). I assumed that the yellow "not in training data" means that these predictions won't be used in future training.

After looking into this issue I realised that I can restrict the max instances when exporting my training package. I set it to 2 and predicted again on my user labelled frames (see Training pipeline screenshot). However, I still get up to four predicted instances for the same animal.

I now have three questions:

1. Is there another way to restrict how many animals to get predicitions on, i.e. tell the model that there are only two mice in my videos?

2. Am I correct in my assumption that the other ignored instances (labelled "not in training data") will in fact not get used for future training?

3. Do you have any suggestions as to what I should be doing to make my model realise that there are only two animals, and therefore there should only be two instances per frame?

Thank you!

[roomrys]

Hi @majaneubauer,

Update

The user was running tracking through the CLI! There was a missing argument to specify which type of tracker to use (flowmaxtracks) which uses/implements the max tracking. See designated answer from user.

---

After looking at your discussion - we did a little test case over here. Similar to you, I trained a bottom-up (non-supervised-id) model on a dataset with two animals. I then ran inference on "user-labeled frames" two different times. The first time, I ran inference with max instances set to "No max" and 2 instances were predicted on my frame - as expected. The second time, I ran inference on the same "user-labeled frames" with max instances set to 1 and only 1 instance was predicted on my frame - also as expected. But not great for recreating the issue or figuring out what is goin on.

Max of 1 predicted instance (yellow-boxed instance) per user-labeled frame

I am fairly sure that I checked that max instances was working before releasing... but perhaps we did not identify an edge case that you are running into.

Inference configuration

I call inference as part of the Inference Pipeline (which uses an already trained model) whereas you have run inference as the next step after training a new model in the Training Pipeline - I should do that to see if anything is amiss there.

After running inference another time on the same frames, we see that the previous predictions were replaced with new predictions

I though maybe there were predictions left over from another inference run and that might be why there were more predictions than we had specified. Although I was pretty sure we remove predictions before adding new ones, I double checked the code to verify and ran a test which show that: yes, predictions are indeed removed before adding new predicitons.

A few differences between our runs

1. You are running inference as part of the training process whereas I ran inference on it's own
2. You are on the official 1.3.3 release, I am on develop (but we don't seem to have any track related commits since 1.3.3 so this should not be the cause)

I will now re-run training + inference to see if there is a difference and update below. In the meantime, I would encourage running just inference (not as part of the training) with the max instances set.

... updates still loading ...

Thanks,
Liezl

[majaneubauer]

Hi @roomrys,

Thank you for your quick reply!

I have been doing some testing on my own the past two days and just now realised where I went wrong. Exporting the training package specifying the max instances to 2 and predicting on user-labeled frames was not the problem. Rather, I made a mistake while running inference on Colab using sleap-track. I reported my issue in #1679, which I have now closed. I had to specify my tracking.tracker to flowmaxtracks in order for tracking.max_tracking to work.

I will go ahead and close this discussion as well now, but thank you for your efforts, I appreciate it!

Best,
Maja

[roomrys]

Hi @majaneubauer,

Awesome news, thanks for following up on that. The max tracks is a somewhat new feature which I have not had much interaction with yet - totally forgot about the flowmaxtracks! But, I will be remembering/double-checking that from now on.

Not so quick, but always trying to be better - appreciate the positive reinforcement!

Thanks,
Liezl