How to extract original videos from the dataset?

[tctco]

Hi, thanks for this exciting project!

Is there a way to get the original video from the dataset (namely the slp files)? I would like to compare the performance between SLEAP and my own algorithm.

Thanks in advance!

[roomrys]

Hi @tctco,

Super cool that you are developing your own algorithm!

If you imported a mp4/avi into SLEAP, then SLEAP will just reference/point-to that mp4/avi. Unfortunately, if you imported a dataset as a set of images (i.e. import from DLC), then SLEAP does not currently support outputting an mp4/avi.

Thanks,
Liezl

[tctco]

I see that you have made your datasets publicly available, but they are created for sleap specifically and I don't know where I can find the original videos that generate these datasets. My algorithm depends on the spatial-temporal info and couldn't infer the identity of animals directly from images :(

[tctco]

Thanks a lot!!!!

BTW, is there an API that I can export the labeled frames to images and files containing all labeled points (just like the DeepLabCut datasets)?

[roomrys]

Hi @tctco,

There is not an official API to export labeled frames to images and files containing all labeled points. SLEAP exports a SLEAP Analysis file that exports all the labeled/predicted points (preferring user-labeled points). SLEAP also exports a SLEAP Package file for a compact version of the project.

However, if you need a specific format, you can create a script to export through the command line (in the SLEAP conda environment)

import sleap
labels = sleap.Labels.load_file("path_to_labels/slp_project.slp")
labeled_frames = labels.labeled_frames

# Extract all the images for every labeled frame
for lf in labeled_frames:
     # Get the image for each LabeledFrame
     img = lf.image  # Returns np.ndarray

     # Add some code to save the image here...
     # Your code here

     # Add some code to add the image filename to a csv here...
     # Your code here

# Construct a numpy array from instance points
tracks = labels.numpy()  # Returns shape `(n_frames, n_tracks, n_nodes, 2)`

# Add some code to write the points to a csv here...
# Your code here

The SLEAP Classes that will be useful here are LabeledFrame and Labels. With the template above, you should just need to add your own code. Some third-party libraries (pandas.DataFrame.to_csv might be helpful along with PIL.Image.fromarray).

Let us know if you need any help/clarification.

Thanks,
Liezl

[tctco]

Thanks!

[STAR-811]

Hi @roomrys! According to the code you offer, the labeled images are exported from the .slp file. However, the exported neighboring frames weren't taken at an adjacent time point. So, I can't finish pose tracking through this discontinuous video. The example images are shown below. And I also want to ask @tctco how you solve this problem.
T=1:

T=2:

[roomrys]

Hi @STAR-811,

I converted you issue on this to a discussion (see #917) and will follow up there. In summary, the exported labeled frames are exported in the order they were labeled - I added a bit of code in #917 to help order those labeled frames. Note that even after ordering based on video/frame index, the labeled frames may still not be adjacent as we only export the frames that were labeled in .pkg.slp files to help condense the file.

Thanks,
Liezl