Large CLI Jobs

[jramborger78]

Hey guys,

Thank you for the continued help, I was able to improve my model from #1612 to now 1000 frames and a bit better outcomes and ready to start tracking. Tracking has gone well for the first test video and exporting the data to a csv (somehow keep having trouble with python in some way to use h5 and unfamiliar with MATLAB) to play around with in R that I am most familiar with.

I am attempting to put together a script that is purely CLI based using OSG's supercomputers to run the many many videos we have. Not 100% sure if this is going to be as easy as I hope considering the need to correct tracks, although each of our videos is a single animal and notice that as long as there isn't two tracks on the same image it seems avoidable by simply cleaning the csv.

Long story short, the hope is to have a script that takes in a folder as a path and runs inference on each video and exports the analyses as a csv. But sleap-convert I do not see a csv option. Not sure if you guys have say a base iteration I can adjust for this portion. Next would be for each csv to then run through R as I have been playing around with a script that then runs them through R, interpolates the missing, and running a few analyses and saving as such.

Any pointers or best practices are much appreciated! Of course the hope is to have it all automated with a press of "play" but even tips on points that would require human-in-the-loop would save time in trying to write an all encompassing script instead of a couple of scripts. But like what I have seen thus far. Thanks guys.

[jramborger78]

Update:

This is a python script that I am testing right now for the first portion from what I have gathered looking around through the Guides section and what not. As mentioned, not seeing a csv option as I am way more adept at R than Python graphing and analyses. Thanks guys.

SCRIPT:

import subprocess
import os

# Set paths to the centroid and centered models
centroid = "C:/Users/jramb/Documents/Sleap/models/1_centroid_model/training_config.json"
centered = "C:/Users/jramb/Documents/Sleap/models/1_centered_model/training_config.json"

# Set parameters for SLEAP tracking
parameters = "--tracking.tracker simple --tracking.similarity instance --tracking.match hungarian --tracking.track_window 5 --tracking.post_connect_single_breaks 0 --verbosity json --no-empty-frames"

# Set path to the directory containing videos
video_path = "C:/Users/jramb/Videos/Raspi_Vids/Row_B/Coc/C22_Coc_B_LgA14"

# Set path to inference result location
inf_path = "C:/Users/jramb/Documents/Sleap/results/track"

# Set path to analysis output location
out_path = "C:/Users/jramb/Documents/Sleap/results/analyses"

# Step 1: Use SLEAP to run inference on each video and convert slp to h5
for video_file in os.listdir(video_path):
    if video_file.endswith(".mp4"):
        video_path_full = os.path.join(video_path, video_file)
        video_name_no_ext = os.path.splitext(video_file)[0]
        
        # Run sleap-track
        track_command = f"sleap-track -m \"{centroid}\" -m \"{centered}\" {parameters} -o \"{inf_path}/{video_name_no_ext}.slp\" \"{video_path_full}\""
        subprocess.run(track_command, shell=True)

        # Run sleap-convert
        convert_command = f"sleap-convert \"{inf_path}/{video_name_no_ext}.slp\" --format analysis \"{out_path}/{video_name_no_ext}.analysis.h5\""
        subprocess.run(convert_command, shell=True)

[jramborger78]

Worked smoothly! Notice that the output shows a number of tracks as opposed to so many duplicates across frames. I added "--tracking.max_tracking True --tracking.max_tracks 1" that hopefully curbs this. Any further suggestions are always welcome, figured I'd keep you in the loop. Thank you, sir.

[jramborger78]

Also, I now continue to get this error. I have been trying to find solutions online with memory allocation and what not. It has happened in the past without issue but now at about the 50% completion point the inference terminates after running out of memory.

Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ETA: 0:54:14 41.5 FPS2023-12-08 08:26:26.337851: W tensorflow/core/data/root_dataset.cc:167] Optimization loop failed: Cancelled: Operation was cancelled
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1% ETA: 0:54:11 41.2 FPS2023-12-08 08:26:49.955429: W tensorflow/core/data/root_dataset.cc:167] Optimization loop failed: Cancelled: Operation was cancelled
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1% ETA: 0:52:57 42.1 FPS2023-12-08 08:26:51.185286: W tensorflow/core/data/root_dataset.cc:167] Optimization loop failed: Cancelled: Operation was cancelled
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1% ETA: 0:53:24 41.7 FPS

[jramborger78]

Sorry for all the updates but, again, wanted to keep you in the loop in case someone got to this thread. I added this portion to try and fix the OOM error:

import tensorflow as tf

# Limit GPU memory growth
gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)

This did not help. After the last adjustments I decided to remove and reestablish sleap in case there was some update needed given he warning of dependencies. My GPU was then no longer being noticed and used the suggestion to downgrade Tensorflow=2.6.3 in the installation section. This also did not seem to help. So I decided to just "mamba update tensorflow" and now I no longer am getting the optimization loop fail. Currently at 26% with GPU support of my inference. Hopeful to report desired results with the csv output and one track.

But of course right when I wanted to submit this this occured:

Predicting... ---------- -----------------------------  26% ETA: 0:36:46 45.1 FPS2023-12-08 09:53:28.914038: E tensorflow/stream_executor/cuda/cuda_driver.cc:794] failed to alloc 17179869184 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-12-08 09:53:28.915245: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 17179869184
Predicting... -------------------- -------------------  50% ETA: 0:22:54 48.9 FPS2023-12-08 10:05:36.408962: E tensorflow/stream_executor/cuda/cuda_driver.cc:794] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-12-08 10:05:36.409126: W .\tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 34359738368

Appears the optimization loop is no longer failing but I can't breach 50% without an OOM suddenly.

[talmo]

Hey @jramborger78,

Thanks for the updates!! This is actually super helpful.

Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ETA: 0:54:14 41.5 FPS2023-12-08 08:26:26.337851: W tensorflow/core/data/root_dataset.cc:167] Optimization loop failed: Cancelled: Operation was cancelled

This is actually just a warning and totally safe to ignore. It just happens because TensorFlow is trying to optimize the data pipeline graph for optimal performance, but sometimes it fails (typically when the data flow changes). Nothing to worry about :)

My GPU was then no longer being noticed and used the suggestion to downgrade Tensorflow=2.6.3 in the installation section. This also did not seem to help. So I decided to just "mamba update tensorflow" and now I no longer am getting the optimization loop fail.

Out of curiosity: what version of tensorflow did you land on?

I added this portion to try and fix the OOM error: ...

The limiting GPU memory growth option is actually enabled by default in SLEAP, and manually via sleap.disable_preallocation().

It's weird that you're getting OOM errors well into the inference process -- memory usage on the GPU should be constant after the first batch 😕

Is this maybe a shared computing environment where other people might be using the GPU as well?

[jramborger78]

Hey @talmo,

Not sure how much I have done since this thread but this is where i currently am:

(sleap) C:\Users\jramb\Documents>python -c "import sleap; sleap.versions()"
SLEAP: 1.3.3
TensorFlow: 2.6.3
Numpy: 1.19.5
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0

Have not gotten those errors since. For the OOM I am on my own pc, but eventually closed out of everything, even reinstalled sleap and that appears to have helped some things. So far so good! The more annoying issue is on the other thread I just posted this morning #1632