Optimization loop failed

[IsabelleSajonia]

How can I resolve this message? I get this when running inference, but my results look good for the most part, maybe just on the slower side (130 FPS).

[Marielamend]

Hi @IsabelleSajonia,

Can you provide us with your full logs to see if there is anything before or after that can help us further diagnose the issue?

Best,
Mariela

[IsabelleSajonia]

what is in the terminal before:
Started inference at: 2024-05-08 14:27:09.152876
Args:
{
'data_path': 'C:/Users/irs3th/Desktop/toggled/flow_in_hungarian_no_tail.slp',
'models': [
'C:/Users/irs3th/Desktop/toggled\models\240508_135057.single_instance.n=10531\training_config.json'
],
'frames': '0,-180042',
'only_labeled_frames': False,
'only_suggested_frames': False,
'output': 'C:/Users/irs3th/Desktop/toggled\predictions\flow_in_hungarian_no_tail.slp.240508_142706.predictions.slp',
'no_empty_frames': True,
'verbosity': 'json',
'video.dataset': None,
'video.input_format': 'channels_last',
'video.index': '16',
'cpu': False,
'first_gpu': False,
'last_gpu': False,
'gpu': 'auto',
2024-05-08 14:27:10.379471: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
'max_edge_length_ratio': 0.25,
'dist_penalty_weight': 1.0,
'batch_size': 4,
'open_in_gui': False,
'peak_threshold': 0.2,
'max_instances': None,
2024-05-08 14:27:10.753973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21346 MB memory: -> device: 0, name: NVIDIA GeForce RTX 4090, pci bus id: 0000:01:00.0, compute capability: 8.9
'tracking.tracker': 'flow',
'tracking.max_tracking': None,
'tracking.max_tracks': None,
'tracking.target_instance_count': None,
'tracking.pre_cull_to_target': None,
'tracking.pre_cull_iou_threshold': None,
'tracking.post_connect_single_breaks': 0,
'tracking.clean_instance_count': None,
'tracking.clean_iou_threshold': None,
'tracking.similarity': 'instance',
'tracking.match': 'greedy',
'tracking.robust': None,
'tracking.track_window': 5,
'tracking.min_new_track_points': None,
'tracking.min_match_points': None,
'tracking.img_scale': None,
'tracking.of_window_size': None,
'tracking.of_max_levels': None,
'tracking.save_shifted_instances': None,
'tracking.kf_node_indices': None,
'tracking.kf_init_frame_count': None
}

INFO:sleap.nn.inference:Auto-selected GPU 0 with 23637 MiB of free memory.
2024-05-08 14:27:13.373525: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2024-05-08 14:27:14.039084: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] INTERNAL: ptxas exited with non-zero error code -1, output:
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
Versions:
SLEAP: 1.3.3
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0

System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True

2024-05-08 14:27:19.453876: W tensorflow/core/data/root_dataset.cc:163] Optimization loop failed: CANCELLED: Operation was cancelled

[IsabelleSajonia]

and this is my GPU performance during so should be accessing it

[eberrigan]

Hi @IsabelleSajonia,

I believe the error is starting at this line:

Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.

How did you install SLEAP?

Thanks,

Elizabeth

[IsabelleSajonia]

I might be misremembering because this was 7 months ago, but I think conda from source (using git). Either that or from conda package since we run Windows. Our version is 1.3.3.

[eberrigan]

You should find the ptxas application that is being used when you issue the command. Are you using a conda environment?

I have a conda environment named "sleap_v1.3.3" and my ptxas compiler is located at "\miniforge3\envs\sleap_v1.3.3\bin\ptxas.exe". Please add the parent folder of the application to the PATH environment variables for your system so that ptxas compiler can be accessed.

[IsabelleSajonia]

Yes I have a conda env. I can add the application to the PATH environment!

[talmo]

Hey folks, just to chime in here: this is a warning from TensorFlow and something which can be safely ignored. It tries to automatically parallelize several steps of the data loading pipeline, but if it can't, there's no real issue.

It also shouldn't matter much for inference/tracking since those are bottlenecked mostly by I/O and other steps not affected by this data loading pipeline.

[IsabelleSajonia]

Ah ok, good to know! Thank you both!