Error in PredictCost() for the op: op: "CropAndResize"

[cosimogonnelli]

Hi,

I am training on Google Colab. I am working on frames from RGB videos. I am training with the bottom-up approach.
I haven't been able to find good resources on this issue so far to be able to fix it. This is why I am posting here.

Thanks for your help!

I am encountering the following error. See output screenshot below:

INFO:numexpr.utils:NumExpr defaulting to 4 threads.
INFO:sleap.nn.training:Versions:
SLEAP: 1.2.3
TensorFlow: 2.8.2
Numpy: 1.21.5
Python: 3.7.13
OS: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
INFO:sleap.nn.training:Training labels file: /content/drive/MyDrive/48HoursDegus/SLEAP/Colab/labels.v000.pkg.slp
INFO:sleap.nn.training:Training profile: /content/drive/MyDrive/48HoursDegus/SLEAP/Colab/multi_instance.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/content/drive/MyDrive/48HoursDegus/SLEAP/Colab/multi_instance.json",
    "labels_path": "/content/drive/MyDrive/48HoursDegus/SLEAP/Colab/labels.v000.pkg.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "tensorboard": false,
    "save_viz": false,
    "zmq": false,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": 0
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": null,
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": null,
            "validation_inds": null,
            "test_inds": null,
            "search_path_hints": [],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": true,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": null,
            "resize_and_pad_to_target": true,
            "target_height": null,
            "target_width": null
        },
        "instance_cropping": {
            "center_on_part": null,
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 32,
                "output_stride": 4,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": null,
            "centered_instance": null,
            "multi_instance": {
                "confmaps": {
                    "part_names": null,
                    "sigma": 2.5,
                    "output_stride": 4,
                    "loss_weight": 1.0,
                    "offset_refinement": false
                },
                "pafs": {
                    "edges": null,
                    "sigma": 75.0,
                    "output_stride": 8,
                    "loss_weight": 1.0
                }
            },
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        }
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -15.0,
            "rotation_max_angle": 15.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": true,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": true,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": true
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": null,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": null,
        "min_val_batches_per_epoch": 10,
        "epochs": 20,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "220620_150149",
        "run_name_prefix": "",
        "run_name_suffix": ".multi_instance",
        "runs_folder": "models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": false,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": false,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.2.2",
    "filename": "/content/drive/MyDrive/48HoursDegus/SLEAP/Colab/multi_instance.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /content/drive/MyDrive/48HoursDegus/SLEAP/Colab/labels.v000.pkg.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 9 / Validation = 1.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
INFO:sleap.nn.training:Loaded test example. [2.479s]
INFO:sleap.nn.training:  Input shape: (1088, 1920, 3)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=5, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 32
INFO:sleap.nn.training:  Parameters: 7,818,534
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = MultiInstanceConfmapsHead(part_names=['snout', 'left_ear', 'right_ear', 'shoulder', 'back', 'tail_base', 'tail_center', 'tail_end'], sigma=2.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training:    [1] = PartAffinityFieldsHead(edges=[('shoulder', 'back'), ('back', 'tail_base'), ('tail_base', 'tail_center'), ('tail_center', 'tail_end'), ('shoulder', 'snout'), ('shoulder', 'left_ear'), ('shoulder', 'right_ear')], sigma=75.0, output_stride=8, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 272, 480, 8), dtype=tf.float32, name=None), name='MultiInstanceConfmapsHead/BiasAdd:0', description="created by layer 'MultiInstanceConfmapsHead'")
INFO:sleap.nn.training:    [1] = KerasTensor(type_spec=TensorSpec(shape=(None, 136, 240, 14), dtype=tf.float32, name=None), name='PartAffinityFieldsHead/BiasAdd:0', description="created by layer 'PartAffinityFieldsHead'")
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 9
INFO:sleap.nn.training:Validation set: n = 1
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.training:Created run path: models/220620_150149.multi_instance
INFO:sleap.nn.training:Setting up visualization...
2022-06-20 22:05:01.770710: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -33 } dim { size: -34 } dim { size: -35 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla P100-PCIE-16GB" frequency: 1328 num_cores: 56 environment { key: "architecture" value: "6.0" } environment { key: "cuda" value: "11010" } environment { key: "cudnn" value: "8005" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 16154099712 bandwidth: 732160000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -36 } dim { size: -37 } dim { size: 1 } } }
2022-06-20 22:05:03.289734: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -33 } dim { size: -34 } dim { size: -35 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla P100-PCIE-16GB" frequency: 1328 num_cores: 56 environment { key: "architecture" value: "6.0" } environment { key: "cuda" value: "11010" } environment { key: "cudnn" value: "8005" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 16154099712 bandwidth: 732160000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -36 } dim { size: -37 } dim { size: 1 } } }
Unable to use Qt backend for matplotlib. This probably means Qt is running headless.
INFO:sleap.nn.training:Finished trainer set up. [7.1s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [5.4s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/20
200/200 - 175s - loss: 9.3878e-04 - MultiInstanceConfmapsHead_loss: 2.7198e-04 - PartAffinityFieldsHead_loss: 6.6681e-04 - val_loss: 8.0499e-04 - val_MultiInstanceConfmapsHead_loss: 2.4420e-04 - val_PartAffinityFieldsHead_loss: 5.6079e-04 - lr: 1.0000e-04 - 175s/epoch - 877ms/step
Epoch 2/20
200/200 - 164s - loss: 9.2697e-04 - MultiInstanceConfmapsHead_loss: 2.6815e-04 - PartAffinityFieldsHead_loss: 6.5882e-04 - val_loss: 8.0783e-04 - val_MultiInstanceConfmapsHead_loss: 2.4231e-04 - val_PartAffinityFieldsHead_loss: 5.6552e-04 - lr: 1.0000e-04 - 164s/epoch - 822ms/step
Epoch 3/20
200/200 - 173s - loss: 9.1590e-04 - MultiInstanceConfmapsHead_loss: 2.6419e-04 - PartAffinityFieldsHead_loss: 6.5171e-04 - val_loss: 8.3987e-04 - val_MultiInstanceConfmapsHead_loss: 2.3928e-04 - val_PartAffinityFieldsHead_loss: 6.0059e-04 - lr: 1.0000e-04 - 173s/epoch - 864ms/step
Epoch 4/20
200/200 - 169s - loss: 8.5822e-04 - MultiInstanceConfmapsHead_loss: 2.5189e-04 - PartAffinityFieldsHead_loss: 6.0633e-04 - val_loss: 9.3843e-04 - val_MultiInstanceConfmapsHead_loss: 2.3601e-04 - val_PartAffinityFieldsHead_loss: 7.0242e-04 - lr: 1.0000e-04 - 169s/epoch - 846ms/step
Epoch 5/20
200/200 - 154s - loss: 6.7421e-04 - MultiInstanceConfmapsHead_loss: 2.0300e-04 - PartAffinityFieldsHead_loss: 4.7121e-04 - val_loss: 8.5504e-04 - val_MultiInstanceConfmapsHead_loss: 2.0548e-04 - val_PartAffinityFieldsHead_loss: 6.4956e-04 - lr: 1.0000e-04 - 154s/epoch - 769ms/step
Epoch 6/20
200/200 - 161s - loss: 4.0680e-04 - MultiInstanceConfmapsHead_loss: 1.2040e-04 - PartAffinityFieldsHead_loss: 2.8640e-04 - val_loss: 7.2331e-04 - val_MultiInstanceConfmapsHead_loss: 1.6678e-04 - val_PartAffinityFieldsHead_loss: 5.5653e-04 - lr: 1.0000e-04 - 161s/epoch - 803ms/step
Epoch 7/20
200/200 - 169s - loss: 2.1749e-04 - MultiInstanceConfmapsHead_loss: 5.4675e-05 - PartAffinityFieldsHead_loss: 1.6281e-04 - val_loss: 6.8497e-04 - val_MultiInstanceConfmapsHead_loss: 1.6092e-04 - val_PartAffinityFieldsHead_loss: 5.2406e-04 - lr: 1.0000e-04 - 169s/epoch - 845ms/step
Epoch 8/20
200/200 - 149s - loss: 1.3127e-04 - MultiInstanceConfmapsHead_loss: 2.8738e-05 - PartAffinityFieldsHead_loss: 1.0253e-04 - val_loss: 7.2444e-04 - val_MultiInstanceConfmapsHead_loss: 1.7155e-04 - val_PartAffinityFieldsHead_loss: 5.5289e-04 - lr: 1.0000e-04 - 149s/epoch - 744ms/step
Epoch 9/20
200/200 - 159s - loss: 7.1462e-05 - MultiInstanceConfmapsHead_loss: 1.3239e-05 - PartAffinityFieldsHead_loss: 5.8223e-05 - val_loss: 7.4206e-04 - val_MultiInstanceConfmapsHead_loss: 1.6581e-04 - val_PartAffinityFieldsHead_loss: 5.7624e-04 - lr: 1.0000e-04 - 159s/epoch - 793ms/step
Epoch 10/20
200/200 - 159s - loss: 5.5629e-05 - MultiInstanceConfmapsHead_loss: 1.0163e-05 - PartAffinityFieldsHead_loss: 4.5466e-05 - val_loss: 7.1255e-04 - val_MultiInstanceConfmapsHead_loss: 1.6881e-04 - val_PartAffinityFieldsHead_loss: 5.4374e-04 - lr: 1.0000e-04 - 159s/epoch - 796ms/step
Epoch 11/20
200/200 - 160s - loss: 4.4206e-05 - MultiInstanceConfmapsHead_loss: 7.8443e-06 - PartAffinityFieldsHead_loss: 3.6362e-05 - val_loss: 6.8842e-04 - val_MultiInstanceConfmapsHead_loss: 1.6070e-04 - val_PartAffinityFieldsHead_loss: 5.2772e-04 - lr: 1.0000e-04 - 160s/epoch - 800ms/step
Epoch 12/20

Epoch 12: ReduceLROnPlateau reducing learning rate to 4.999999873689376e-05.
200/200 - 159s - loss: 3.7331e-05 - MultiInstanceConfmapsHead_loss: 6.5763e-06 - PartAffinityFieldsHead_loss: 3.0755e-05 - val_loss: 6.9770e-04 - val_MultiInstanceConfmapsHead_loss: 1.6466e-04 - val_PartAffinityFieldsHead_loss: 5.3305e-04 - lr: 1.0000e-04 - 159s/epoch - 793ms/step
Epoch 13/20
200/200 - 159s - loss: 2.7997e-05 - MultiInstanceConfmapsHead_loss: 4.3432e-06 - PartAffinityFieldsHead_loss: 2.3654e-05 - val_loss: 6.9984e-04 - val_MultiInstanceConfmapsHead_loss: 1.6470e-04 - val_PartAffinityFieldsHead_loss: 5.3514e-04 - lr: 5.0000e-05 - 159s/epoch - 797ms/step
Epoch 14/20
200/200 - 158s - loss: 2.5521e-05 - MultiInstanceConfmapsHead_loss: 3.8989e-06 - PartAffinityFieldsHead_loss: 2.1622e-05 - val_loss: 7.1469e-04 - val_MultiInstanceConfmapsHead_loss: 1.6838e-04 - val_PartAffinityFieldsHead_loss: 5.4631e-04 - lr: 5.0000e-05 - 158s/epoch - 791ms/step
Epoch 15/20
200/200 - 168s - loss: 2.3930e-05 - MultiInstanceConfmapsHead_loss: 3.6252e-06 - PartAffinityFieldsHead_loss: 2.0305e-05 - val_loss: 6.9776e-04 - val_MultiInstanceConfmapsHead_loss: 1.6717e-04 - val_PartAffinityFieldsHead_loss: 5.3059e-04 - lr: 5.0000e-05 - 168s/epoch - 841ms/step
Epoch 16/20
200/200 - 150s - loss: 2.2972e-05 - MultiInstanceConfmapsHead_loss: 3.5370e-06 - PartAffinityFieldsHead_loss: 1.9435e-05 - val_loss: 6.9937e-04 - val_MultiInstanceConfmapsHead_loss: 1.6550e-04 - val_PartAffinityFieldsHead_loss: 5.3387e-04 - lr: 5.0000e-05 - 150s/epoch - 748ms/step
Epoch 17/20
200/200 - 158s - loss: 2.1731e-05 - MultiInstanceConfmapsHead_loss: 3.3550e-06 - PartAffinityFieldsHead_loss: 1.8376e-05 - val_loss: 7.0372e-04 - val_MultiInstanceConfmapsHead_loss: 1.6824e-04 - val_PartAffinityFieldsHead_loss: 5.3548e-04 - lr: 5.0000e-05 - 158s/epoch - 789ms/step
Epoch 17: early stopping
INFO:sleap.nn.training:Finished training loop. [45.7 min]
INFO:sleap.nn.training:Deleting visualization directory: models/220620_150149.multi_instance/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2022-06-20 22:50:59.797019: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -65 } dim { size: -66 } dim { size: -67 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla P100-PCIE-16GB" frequency: 1328 num_cores: 56 environment { key: "architecture" value: "6.0" } environment { key: "cuda" value: "11010" } environment { key: "cudnn" value: "8005" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 16154099712 bandwidth: 732160000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -68 } dim { size: -69 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━  89% ETA: 0:00:01 20.0 FPS2022-06-20 22:51:04.886307: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -65 } dim { size: -66 } dim { size: -67 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla P100-PCIE-16GB" frequency: 1328 num_cores: 56 environment { key: "architecture" value: "6.0" } environment { key: "cuda" value: "11010" } environment { key: "cudnn" value: "8005" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 16154099712 bandwidth: 732160000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -68 } dim { size: -69 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 1.1 FPS
INFO:sleap.nn.evals:Saved predictions: models/220620_150149.multi_instance/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: models/220620_150149.multi_instance/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.835520
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2022-06-20 22:51:09.983386: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -65 } dim { size: -66 } dim { size: -67 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla P100-PCIE-16GB" frequency: 1328 num_cores: 56 environment { key: "architecture" value: "6.0" } environment { key: "cuda" value: "11010" } environment { key: "cudnn" value: "8005" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 16154099712 bandwidth: 732160000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -68 } dim { size: -69 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 ?
INFO:sleap.nn.evals:Saved predictions: models/220620_150149.multi_instance/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: models/220620_150149.multi_instance/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.100990

[roomrys]

Hi @cosimogonnelli,

The CropAndResize error should only only affect the training visualizations (which allows users to visually monitor training - when training locally). Training should still complete successfully despite the CropAndResize error. Have you noticed any abnormal predictions?

Thanks,
Liezl

[cosimogonnelli]

Hi @roomrys,

if by predictions you mean looking at the predictions.slp file, I am not able to merge that file into my project (as i think you are aware by seen your answer to the other message I posted). Otherwise what should I check to be sure this error doesn't affect my data?
Do I have another way to see the predictions on the GUI?

thanks for your help!

[roomrys]

Hi @cosimogonnelli,

This error should not affect your data (the visuals are made using Qt which can only be run locally - not on colab). By predictions, I meant the predictions on the validation/training sets that are updated at the end of each training epoch - not the ones stored in predictions.slp. The predictions stored in predictions.slp will be output from the best model returned after training completes. I will follow up on the merging issue in #713.

Thanks,
Liezl

[cosimogonnelli]

Hi @roomrys,

This is an example of my predictions that I get using tensorBoard. This is the best case with 20 labeled frames. It seems it is training even if results are not that great. it seems like that the tail of the mouse on left is missing predictions (I don't see any red dots). Can this have to do with the issue we are talking about here? thanks for your help!

[roomrys]

Hi @cosimogonnelli,

This should just be taken as a metric of how your model is performing - not a result of the CropAndResize error. Either changing the hyperparameters of your model, increasing the quality (precise and accurate labeling of nodes) of your labels, or adding more high quality labels should help.

Thanks,
Liezl