Error with centroid and centered instance models

[sciencemassa]

I have been training a model for the past few weeks, and we are finally on the last batch of videos... But unfortunately I keep getting an error message when I try to train on the frames I just labelled.
The error goes something like this:
The "preparing for training" phase lasts a few minutes with no progress. Then I get two popups saying that an error occurred with the centroid model and then the centered instance model. I have tried troubleshooting by changing computers, predicting on nothing as opposed to a new video, and removing and re-adding the new videos.
The error says to check the terminal, but I can't quite decipher what the issue is. I was wondering if there is a maximum number of videos you can upload? Is there a maximum "n"?

[roomrys]

Hi @sciencemassa,

We do not have an upper limit on the number of videos that you can upload in SLEAP. Do you mind copy+pasting the error message from your terminal?

Thanks,
Liezl

[sciencemassa]

Polling: /nadata/snlkt/spb_psilicon/Pilot_BehParameters/SLEAP/SLEAP_phase4/models/220804_153045.centered_instance.n=2332/viz/validation.*.png
Start training centered_instance...
['sleap-train', '/tmp/tmpay57pgmc/220804_153045_training_job.json', '/nadata/snlkt/spb_psilicon/Pilot_BehParameters/SLEAP/SLEAP_phase4/fsm_mb_abs_MedPC_p4_v39.slp', '--zmq', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.2.4
TensorFlow: 2.6.3
Numpy: 1.19.5
Python: 3.7.12
OS: Linux-4.19.0-21-amd64-x86_64-with-debian-10.12
INFO:sleap.nn.training:Training labels file: /nadata/snlkt/spb_psilicon/Pilot_BehParameters/SLEAP/SLEAP_phase4/fsm_mb_abs_MedPC_p4_v39.slp
INFO:sleap.nn.training:Training profile: /tmp/tmpay57pgmc/220804_153045_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/tmp/tmpay57pgmc/220804_153045_training_job.json",
    "labels_path": "/nadata/snlkt/spb_psilicon/Pilot_BehParameters/SLEAP/SLEAP_phase4/fsm_mb_abs_MedPC_p4_v39.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "tensorboard": false,
    "save_viz": true,
    "zmq": true,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": 0
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": null,
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": null,
            "validation_inds": null,
            "test_inds": null,
            "search_path_hints": [],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": null,
            "resize_and_pad_to_target": true,
            "target_height": null,
            "target_width": null
        },
        "instance_cropping": {
            "center_on_part": "haunch",
            "crop_size": 112,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 4,
                "filters": 24,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": null,
            "centered_instance": {
                "anchor_part": "haunch",
                "part_names": null,
                "sigma": 2.5,
                "output_stride": 4,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        }
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -15.0,
            "rotation_max_angle": 15.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": true
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": null,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": null,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "220804_153045.centered_instance.n=2332",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "/nadata/snlkt/spb_psilicon/Pilot_BehParameters/SLEAP/SLEAP_phase4/models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.2.4",
    "filename": "/tmp/tmpay57pgmc/220804_153045_training_job.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/5 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
  Device: /physical_device:GPU:1
         Available: False
        Initalized: False
     Memory growth: None
  Device: /physical_device:GPU:2
         Available: False
        Initalized: False
     Memory growth: None
  Device: /physical_device:GPU:3
         Available: False
        Initalized: False
     Memory growth: None
  Device: /physical_device:GPU:4
         Available: False
        Initalized: False
     Memory growth: None
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /nadata/snlkt/spb_psilicon/Pilot_BehParameters/SLEAP/SLEAP_phase4/fsm_mb_abs_MedPC_p4_v39.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 2099 / Validation = 233.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
2022-08-04 15:31:18.385708: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-08-04 15:31:18.974614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 15405 MB memory:  -> device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0
2022-08-04 15:31:19.652940: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2022-08-04 15:31:21.486798: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 360 } dim { size: 640 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "111" frequency: 2599 num_cores: 40 environment { key: "cpu_instruction_set" value: "AVX SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2" } environment { key: "eigen" value: "3.3.90" } l1_cache_size: 32768 l2_cache_size: 262144 l3_cache_size: 26214400 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 112 } dim { size: 112 } dim { size: 1 } } }
INFO:sleap.nn.training:Loaded test example. [9.535s]
INFO:sleap.nn.training:  Input shape: (112, 112, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=24, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=2, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 4,311,542
INFO:sleap.nn.training:  Heads:
INFO:sleap.nn.training:    [0] = CenteredInstanceConfmapsHead(part_names=['nose', 'left_ear_tip', 'left_ear_base', 'right_ear_base', 'right_ear_tip', 'skull_base', 'shoulders', 'haunch', 'tail_base', 'tail_seg', 'left_arm', 'right_arm', 'right_leg', 'left_leg'], anchor_part='haunch', sigma=2.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs:
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 28, 28, 14), dtype=tf.float32, name=None), name='CenteredInstanceConfmapsHead/BiasAdd:0', description="created by layer 'CenteredInstanceConfmapsHead'")
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 2099
INFO:sleap.nn.training:Validation set: n = 233
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: /nadata/snlkt/spb_psilicon/Pilot_BehParameters/SLEAP/SLEAP_phase4/models/220804_153045.centered_instance.n=2332
INFO:sleap.nn.training:Setting up visualization...
2022-08-04 15:31:23.953847: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 360 } dim { size: 640 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "111" frequency: 2599 num_cores: 40 environment { key: "cpu_instruction_set" value: "AVX SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2" } environment { key: "eigen" value: "3.3.90" } l1_cache_size: 32768 l2_cache_size: 262144 l3_cache_size: 26214400 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 112 } dim { size: 112 } dim { size: 1 } } }
2022-08-04 15:31:25.234398: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 360 } dim { size: 640 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "111" frequency: 2599 num_cores: 40 environment { key: "cpu_instruction_set" value: "AVX SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2" } environment { key: "eigen" value: "3.3.90" } l1_cache_size: 32768 l2_cache_size: 262144 l3_cache_size: 26214400 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 112 } dim { size: 112 } dim { size: 1 } } }
INFO:sleap.nn.training:Finished trainer set up. [13.4s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
2022-08-04 15:32:56.302232: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 360 } dim { size: 640 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "111" frequency: 2599 num_cores: 40 environment { key: "cpu_instruction_set" value: "AVX SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2" } environment { key: "eigen" value: "3.3.90" } l1_cache_size: 32768 l2_cache_size: 262144 l3_cache_size: 26214400 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 112 } dim { size: 112 } dim { size: 1 } } }
2022-08-04 15:33:08.209209: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 360 } dim { size: 640 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "111" frequency: 2599 num_cores: 40 environment { key: "cpu_instruction_set" value: "AVX SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2" } environment { key: "eigen" value: "3.3.90" } l1_cache_size: 32768 l2_cache_size: 262144 l3_cache_size: 26214400 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 112 } dim { size: 112 } dim { size: 1 } } }
INFO:sleap.nn.training:Finished creating training datasets. [103.1s]
INFO:sleap.nn.training:Starting training loop...
2022-08-04 15:33:08.970134: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 360 } dim { size: 640 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "111" frequency: 2599 num_cores: 40 environment { key: "cpu_instruction_set" value: "AVX SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2" } environment { key: "eigen" value: "3.3.90" } l1_cache_size: 32768 l2_cache_size: 262144 l3_cache_size: 26214400 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 112 } dim { size: 112 } dim { size: 1 } } }
Epoch 1/200
2022-08-04 15:33:10.872754: F tensorflow/core/platform/default/env.cc:73] Check failed: ret == 0 (11 vs. 0)Thread tf_data_iterator_resource creation via pthread_create() failed.

Let me know if this is comprehensive enough or if you need any more info!
Thanks for the help.

[talmo]

Hi @sciencemassa,

It looks like that error is mentioned in this issue, which describes it as a problem with your system limit on the max number of threads. Tensorflow/SLEAP parallelize a lot of the data loading, so could be happening due to other things running on the server that take up too many threads and/or RAM.

The max thread limit is something defined on your server's environment but there are some potential solutions here.

I'd recommend trying the ulimit -u first (though you might need to ask someone if you don't have sudo).

[jmdelahanty]

Not sure if relevant, but doing:

ulimit

On one of the machines people are commonly using (tesla) gives this as an answer:

unlimited

Also this:

ulimit -u
1031140

From linked issues above/people's suggestion:

cat /proc/sys/kernel/threads-max
2062281

Not sure what these numbers actually mean yet...

It looks like it literally means that a given user could theoretically spawn 1,031,140 threads if they wanted which is a whole bunch of threads! Interesting!

[sciencemassa]

We are still having issues getting training to run, despite checking our max limit.
It doesn't help that we also keep getting completely different error messages.
Here is the most recent one:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  indices[0] = 1 is not in [0, 1)
         [[node GatherV2 (defined at /lib/python3.7/site-packages/sleap/nn/losses.py:142) ]]
         [[IteratorGetNext]]
  (1) Invalid argument:  indices[0] = 1 is not in [0, 1)
         [[node GatherV2 (defined at /lib/python3.7/site-packages/sleap/nn/losses.py:142) ]]
         [[IteratorGetNext]]
         [[ReadVariableOp_16/_64]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14407]

Errors may have originated from an input operation.
Input Source operations connected to node GatherV2:
 IteratorGetNext (defined at /lib/python3.7/site-packages/sleap/nn/training.py:923)

Input Source operations connected to node GatherV2:
 IteratorGetNext (defined at /lib/python3.7/site-packages/sleap/nn/training.py:923)

Function call stack:
train_function -> train_function

Another error we sometimes get when we try to run training is a data type error.
So far, nothing has worked and we still haven't been able to run training on any projects.

[jmdelahanty]

Just ran a training on a machine from my account and things are working well it looks like. They're going to try running on the same computer when the training finishes. Can report back. Also can help them open an Issue as well.

[jmdelahanty]

Somehow Anousheh's run worked fine on this other machine. The computers that she's having trouble with are computers that others are not encountering problems. There must be something off about how things are configured in the cluster or something...

[roomrys]

Hi @sciencemassa,

Can you open an issue on this? Filling out the relevant info on issue form would be useful.

A few questions/clarifications (which you can include in the issue form if you'd like):

1. To clarify, you were able to run training, then you added a new video, and are now unable to run training?
2. Does removing the video allow you to run training again? (Can you train on older versions of the project that does not include the new video?)
3. Have you changed anything to get the new error messages?
4. How are you running training (local, remote, on Colab?)

If feasible, can you share your project with [email protected]?

Sorry you are experiencing so much trouble, but thanks for helping us figure it out!

Thanks,
Liezl

[jmdelahanty]

At the moment, I believe their team is using a working machine I haven't been able to reproduce the problem elsewhere yet...

EDIT:
"Wasn’t able to run SLEAP on Poseidon on a new sleap file I have, but it worked on this big one again. So the plot thickens" - Anousheh

Poseidon is a machine on the network that was working without this threading issue. Now with a new SLEAP file she's running into the threading issue again it seems. Very odd...

[sciencemassa]

Okay so here's an update:
The problem seems to be local to our servers. It likely originated when our data center got an upgrade and some things got moved around. The computers we were running training on basically just stopped working for us, and we don't have an answer yet for why/what happened. We are able to run training on a different machine in the meantime.
Thanks for all the help. We will keep trouble shooting over here and figure out what got switched around.

[jmdelahanty]

See here for potential solution to the problem! Long story short, it seems that the incorrect drivers were being used accidentally.

[jmdelahanty]

Turns out this is unlikely to have been a solution because it is still happening intermittently. An Issue will be spawned soon @roomrys. If we can't spawn threads we can at least spawn issues.