Skeleton Compatibility Issues with SLEAP Inference Between Mac and NVIDIA GPU Cluster

[pelikanGIS]

I’m encountering a skeleton-related issue when transferring and running inference models between different environments.

Model Training:
I trained a model on my Mac Studio using SLEAP version 1.3.3.
The training data consisted of frames extracted from a video.
Inference works on the [Mac] (although for some movie when I apply inference I get a message about a Semaphore leak #1430) including applying inference to the entire video.

Inference on NVIDIA GPU Cluster:
I copied the trained models to an NVIDIA GPU cluster running SLEAP version 1.4.1a2.
When applying inference to the same video, I get the following error:
ValueError: Found multiple nodes named (name).

Despite this error (expection), the program continues to process approximately 12% of the movie before the process gets killed.

Testing with SLEAP 1.3.3 on the Cluster:
I attempted to run inference on the same NVIDIA GPU cluster using SLEAP version 1.3.3.
This also resulted in a skeleton-related exception, but the program does not process any frames at all.

Questions:
What steps can I take to debug or resolve skeleton-related errors like Found multiple nodes named (name)?

Thanks
Ehud Sussman

[talmo]

Hi @pelikanGIS,

This looks related to #1918. In v1.3.4, there's a fix for this, but depending on what version created your labels vs model, it may need to be resaved or corrected using some of the recipes in that thread.

Do you mind giving those a go and letting us know what kind of errors you're getting?

Talmo

[pelikanGIS]

FAQ: Resolving Issues with SLEAP Skeleton and Model Compatibility

If you’re encountering issues with SLEAP models or .slp files due to differences between versions, such as errors when trying to use files prepared in SLEAP 1.3.3 with SLEAP 1.3.4, here’s a guide to address these concerns.

---

Issue: Incompatibility Between Versions

Some users may encounter issues with reading or using skeletons or models created in one version of SLEAP (e.g., 1.3.3) when transitioning to a newer version (e.g., 1.3.4). Here's how to address this:

---

Steps to Resolve the Issue

1. Ensure Compatibility Between .slp Files

• Action: Open your .slp file (created in SLEAP 1.3.3) in SLEAP 1.3.4 and resave it.
• Outcome: Files prepared in version 1.3.3 can be read seamlessly in 1.3.4 once they have been resaved. The updated .slp format ensures compatibility with new features and updates in version 1.3.4. Version 1.3.4 is available on a MAC and stores the json skeleton in a format that is readable in version 1.4.1a2

2. Verify Skeleton Model Changes (optional)

• Action: Note that skeletons saved in SLEAP 1.3.4 may differ from those saved in SLEAP 1.3.3. Check the skeletons and adjust them if necessary.
• Outcome: Skeleton models in 1.3.4 are now saved in a format designed to enhance compatibility with the new features of SLEAP. Models trained in 1.3.4 will automatically adopt this updated skeleton structure.

3. Train New Models in SLEAP 1.3.4

• Action: If you’re transitioning to 1.3.4, retrain your models in the updated version to ensure the skeleton model is in the appropriate format.
• Outcome: Once retrained, models created in SLEAP 1.3.4 will work seamlessly with the updated skeleton format.

4. Cross-Version Compatibility

• Scenario: You wish to run a model built using SLEAP 1.3.4 on a DLC cluster running SLEAP 1.4.1a2.
• Action: Ensure the skeleton and models are correctly formatted and compatible by using the recipes from Error during Inference. ValueError: Found Multiple nodes named (name). #1918 to resolve any version-related issues before deploying the model. Or better yet try to open the .slp file created in 1.3.3, train the model in 1.3.4 and save the model.
• Outcome: The model can run smoothly across platforms, including an NVIDIA cluster using SLEAP 1.4.1a2, provided compatibility steps are followed.

---

FAQ Summary

• SLEAP 1.3.4 introduces fixes for issues encountered in earlier versions, such as skeleton format inconsistencies.
• Data stored in .slp files from SLEAP 1.3.3 can be seamlessly read and used in 1.3.4 once resaved in the new version. Or one can attempt to solve the issue based on Error during Inference. ValueError: Found Multiple nodes named (name). #1918.
• Skeletons in 1.3.4 are saved in a format also work in version 1.4.1a2 , so retraining models in this version ensures compatibility.
• Models built on a Mac using SLEAP 1.3.4 can be deployed (for inference) on an NVIDIA cluster running SLEAP 1.4.1a2.