Multiple Tracks in merging with multi-animal tracks

[djg9324]

I have been having trouble keeping consistent tracks/instances in my projects. I am trying to track 2 animals but keep getting more than 2 after merging. I had recently received help from someone who provided me with this code:

import sleap
import sys

def merge(old_labels, new_labels):
    labels = sleap.load_file(old_labels)
    labels.remove_predictions()
    preds = sleap.load_file(new_labels, match_to=labels)

    # I don't know why doing this twice fixes it...
    merged, old_conflicts, new_conflicts = sleap.Labels.complex_merge_between(labels, preds, unify=False)
    merged, old_conflicts, new_conflicts = sleap.Labels.complex_merge_between(labels, preds, unify=False)
    print("Merged videos:", len(merged))
    print("Old labels conflicts:", len(old_conflicts))
    print("New labels conflicts:", len(new_conflicts))

    # Merge
    for video, video_preds in merged.items():
        for fidx, insts in video_preds.items():
            new_frame = False
            if (video, fidx) not in labels:
                lf = sleap.LabeledFrame(video=video, frame_idx=fidx)
                new_frame = True
            else:
                lf = labels[(video, fidx)]

            lf.instances.extend(insts)

            if new_frame:
                labels.append(lf)

    print("Tracks after merging:")
    print(labels.tracks)
   
    merged_filename = old_labels + ".merged.slp"
    labels.save(merged_filename)
    print("Saved:", merged_filename)
   

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Call this script with syntax: 'python merge.py labels.slp predictions.slp'")
    merge(sys.argv[1], sys.argv[2])

Strangely, the code does merge the frames into one project but still leaves 4 tracks instead of just two (as shown in the attached image). I am wondering if there is a step that I am missing to ensure that the tracks merge together with their respective track to only define two animals.

May I also provide the information that command line provided me, if that helps

:

(sleap) C:\Users\schneideradmin>python merge.py "C:\Users\schneideradmin\Desktop\Deanna\sleap\LEAVES_social.slp" "C:\Users\schneideradmin\Desktop\Deanna\sleap\1_run_inference.slp"
Merged videos: 0
Old labels conflicts: 0
New labels conflicts: 0
Tracks after merging:
[Track(spawned_on=10669, name='animal1'), Track(spawned_on=10669, name='animal2'), Track(spawned_on=4286, name='animal1'), Track(spawned_on=4286, name='animal2')]
Saved: C:\Users\schneideradmin\Desktop\Deanna\sleap\LEAVES_social.slp.merged.slp

[talmo]

Hi @djg9324,

A couple of things:

1. If you update to SLEAP v1.3.3, we have a new feature in the tracking CLI that strictly enforces the max number of tracks, which would bypass the need for this script entirely. (You still should proofread manually in case there are switches.) Here's an example of the CLI syntax for this feature.
2. In the Tracks menu, there is an option you can use to delete extraneous, unused tracks, which would also get rid of the extra tracks (assuming there are no instances with those tracks assigned):

Let us know if that works for you.

Cheers,

Talmo

[djg9324]

Hello,

I did what you suggest of updating the version of sleap as well as running inference through the CLI. The prediction files being created are still creating multiple instance tracks across the videos despite my model only have two of them defined. This is what my model shows:

This is what my code looks like:

And this is what the predictions from the code produced:

[djg9324]

Also, I have the ears labeled in the model and the predictions are not including the ears. How may I fix that issue?

[ralphpeterson]

Thanks for the suggestions @talmo! I am helping Deanna troubleshoot this issue. We are using a bottom-up (no id) model with "animal1" and "animal2" tracks in our training set. If we predict on new frames, they are assigned new track labels, e.g. "track_0", "track_1", etc.

Is this expected behavior for bottom-up models? Would it assign our labels if we used an ID model?

[talmo]

Hello,

I did what you suggest of updating the version of sleap as well as running inference through the CLI. The prediction files being created are still creating multiple instance tracks across the videos despite my model only have two of them defined. This is what my model shows: ...

This is what my code looks like: ...

And this is what the predictions from the code produced: ...

Hi @djg9324,

Judging from the printed outputs in the screenshot, I suspect you might be missing a flag. It's a bit annoying but (for historical reasons) you basically need all of these flags to enable track-local queues to globally constrain the maximum number of tracks:

sleap-track [...] --tracking.tracker simplemaxtracks --tracking.max_tracking 1 --tracking.max_tracks N [...]

where N is the max number of animals. Also note that the tracker is a different and that you need to enable --tracking.max_tracking as well.

[talmo]

Hi @djg9324,

Also, I have the ears labeled in the model and the predictions are not including the ears. How may I fix that issue?

This one is a bit less obvious from what we see so far.

It could be an issue with how the skeleton is configured for bottom-up models as the edges need to form a tree in those models. SLEAP should detect when this happens, but if you trained this on an older version it might have been missing that check. Additionally, there was a bug a few versions ago that made it dependent on the ordering of the edges (but this is fixed in v1.3.3). It'll help if you can provide a screenshot of the skeleton edges table in case there's something funky going on there.

Honestly though, for your data I'd actually recommend trying a top-down model with spine2 as the anchor node. Give it a go and let us know how it goes!

[talmo]

Thanks for the suggestions @talmo! I am helping Deanna troubleshoot this issue. We are using a bottom-up (no id) model with "animal1" and "animal2" tracks in our training set. If we predict on new frames, they are assigned new track labels, e.g. "track_0", "track_1", etc.

Is this expected behavior for bottom-up models? Would it assign our labels if we used an ID model?

Hi @ralphpeterson,

Right, the track labels don't matter at all for non-ID models, so no need to bother with those.

If you do want to assign them tracks to use the ID models, you'll need to make sure that the animals have distinguishing visual appearances and assign them to tracks corresponding to identity classes (e.g., "bleached" vs "non_bleached") and then train an ID model variant.

I'd start with the above suggestions for sorting out the tracking though and see where you get.

Cheers,

Talmo