trained in Colab, but when opening predictions.slp file, it seems like it didn't label anything

[delaroob]

Hi everyone,

I am trying to train a network to recognize certain parts of a single fish in videos. Since I defined a skeleton with 27 labels and 27 edges, I thought it would be a good idea to traing using a GPU in Colab. However, something seems to have gone wrong because when I open the predictions.slp file, it shows that there are 0 labeled frames.
Here I attach everything I've done so far, any help would be appreciated!
Thanks in advance.

in conda: creating the project, exporting training job package

Added a video (male.mp4)
Defined a skeleton (dorsal_skeleton.json) with 27 edges and 27 labels, saved it
Labeled 20 images (single instance), saved it (Labels.v001.slp)

Predict > run training

save config files

Export training job package

in drive

created a folder named "sleap", and one named "colab" in sleap
uploaded labels.v001.slp.training_job.zip in folder named "colab"

in colab

Change runtime > GPU (V100: GPU RAM: 16.0 GB

installing packages

!pip uninstall -qqq -y opencv-python opencv-contrib-python
!pip install -qqq "sleap[pypi]>=1.3.3"

mounting drive

from google.colab import drive
drive.mount('/content/drive/')
%cd /content/drive/MyDrive/sleap/colab
!ls

output:

/content/drive/MyDrive/sleap/colab
labels.v001.slp.training_job.zip

unzip

import os
%cd /content/drive/MyDrive/sleap/colab
!unzip labels.v001.slp.training_job.zip
!ls

output:

/content/drive/MyDrive/sleap/colab
Archive:  labels.v001.slp.training_job.zip
  inflating: inference-script.sh     
  inflating: jobs.yaml               
  inflating: labels.v001.pkg.slp     
  inflating: single_instance.json    
  inflating: train-script.sh         
inference-script.sh  labels.v001.pkg.slp	       single_instance.json
jobs.yaml	     labels.v001.slp.training_job.zip  train-script.sh

train

!sleap-train single_instance.json labels.v001.pkg.slp
output:

INFO:numexpr.utils:NumExpr defaulting to 8 threads.
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.3
TensorFlow: 2.8.4
Numpy: 1.22.4
Python: 3.10.12
OS: Linux-5.15.120+-x86_64-with-glibc2.35
INFO:sleap.nn.training:Training labels file: labels.v001.pkg.slp
INFO:sleap.nn.training:Training profile: single_instance.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "single_instance.json",
    "labels_path": "labels.v001.pkg.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "base_checkpoint": null,
    "tensorboard": false,
    "save_viz": false,
    "zmq": false,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": null,
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": null,
            "validation_inds": null,
            "test_inds": null,
            "search_path_hints": [],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": true,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": null,
            "resize_and_pad_to_target": true,
            "target_height": null,
            "target_width": null
        },
        "instance_cropping": {
            "center_on_part": null,
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 2,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": {
                "part_names": null,
                "sigma": 2.5,
                "output_stride": 2,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "centroid": null,
            "centered_instance": null,
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -15.0,
            "rotation_max_angle": 15.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": true,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": null,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": null,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "231210_122851",
        "run_name_prefix": "makrop",
        "run_name_suffix": ".single_instance",
        "runs_folder": "models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": false,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": false,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.3.3",
    "filename": "single_instance.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Auto-selected GPU 0 with 16148 MiB of free memory.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: labels.v001.pkg.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 18 / Validation = 2.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
INFO:sleap.nn.training:Loaded test example. [2.384s]
INFO:sleap.nn.training:  Input shape: (1088, 1920, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 1,953,963
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = SingleInstanceConfmapsHead(part_names=['orr', 'hat1', 'hat2', 'hat3', 'hat4', 'hat5', 'hat6', 'hat7', 'hat8', 'hat9', 'hat10', 'hat11', 'hatfarfel', 'fartofel', 'farfel1', 'farfel2', 'farfel3', 'fartokozep', 'farle3', 'farle2', 'farle1', 'fatole', 'hatfarle', 'has4', 'has3', 'has2', 'has1'], sigma=2.5, output_stride=2, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 544, 960, 27), dtype=tf.float32, name=None), name='SingleInstanceConfmapsHead/BiasAdd:0', description="created by layer 'SingleInstanceConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 18
INFO:sleap.nn.training:Validation set: n = 2
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.training:Created run path: models/makrop231210_122851.single_instance
INFO:sleap.nn.training:Setting up visualization...
Unable to use Qt backend for matplotlib. This probably means Qt is running headless.
Unable to use Qt backend for matplotlib. This probably means Qt is running headless.
INFO:sleap.nn.training:Finished trainer set up. [3.5s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [3.4s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/200
2023-12-10 11:38:23.897054: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 27 } dim { size: 544 } dim { size: 960 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla V100-SXM2-16GB" frequency: 1530 num_cores: 80 environment { key: "architecture" value: "7.0" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 98304 memory_size: 15315632128 bandwidth: 898048000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -5 } dim { size: -6 } dim { size: 1 } } }
200/200 - 163s - loss: 3.8168e-05 - val_loss: 3.7605e-05 - lr: 1.0000e-04 - 163s/epoch - 817ms/step
Epoch 2/200
200/200 - 157s - loss: 3.7597e-05 - val_loss: 3.7597e-05 - lr: 1.0000e-04 - 157s/epoch - 785ms/step
Epoch 3/200
200/200 - 163s - loss: 3.7592e-05 - val_loss: 3.7595e-05 - lr: 1.0000e-04 - 163s/epoch - 817ms/step
Epoch 4/200
Epoch 6: ReduceLROnPlateau reducing learning rate to 4.999999873689376e-05.
200/200 - 158s - loss: 3.7586e-05 - val_loss: 3.7592e-05 - lr: 1.0000e-04 - 158s/epoch - 792ms/step
Epoch 7/200
200/200 - 144s - loss: 3.7585e-05 - val_loss: 3.7591e-05 - lr: 5.0000e-05 - 144s/epoch - 719ms/step
Epoch 8/200
200/200 - 151s - loss: 3.7584e-05 - val_loss: 3.7591e-05 - lr: 5.0000e-05 - 151s/epoch - 755ms/step
Epoch 9/200
200/200 - 160s - loss: 3.7583e-05 - val_loss: 3.7590e-05 - lr: 5.0000e-05 - 160s/epoch - 798ms/step
Epoch 10/200
200/200 - 140s - loss: 3.7582e-05 - val_loss: 3.7589e-05 - lr: 5.0000e-05 - 140s/epoch - 702ms/step
Epoch 11/200
200/200 - 148s - loss: 3.7581e-05 - val_loss: 3.7591e-05 - lr: 5.0000e-05 - 148s/epoch - 741ms/step
Epoch 12/200
200/200 - 147s - loss: 3.7581e-05 - val_loss: 3.7590e-05 - lr: 5.0000e-05 - 147s/epoch - 737ms/step
Epoch 13/200
Epoch 13: ReduceLROnPlateau reducing learning rate to 2.499999936844688e-05.
200/200 - 151s - loss: 3.7579e-05 - val_loss: 3.7588e-05 - lr: 5.0000e-05 - 151s/epoch - 755ms/step
Epoch 14/200
200/200 - 157s - loss: 3.7579e-05 - val_loss: 3.7588e-05 - lr: 2.5000e-05 - 157s/epoch - 784ms/step
Epoch 14: early stopping
INFO:sleap.nn.training:Finished training loop. [35.7 min]
INFO:sleap.nn.training:Deleting visualization directory: models/makrop231210_122851.single_instance/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2023-12-10 12:11:29.268810: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla V100-SXM2-16GB" frequency: 1530 num_cores: 80 environment { key: "architecture" value: "7.0" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 98304 memory_size: 15315632128 bandwidth: 898048000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━  89% ETA: 0:00:01 46.5 FPS2023-12-10 12:11:29.916067: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla V100-SXM2-16GB" frequency: 1530 num_cores: 80 environment { key: "architecture" value: "7.0" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 98304 memory_size: 15315632128 bandwidth: 898048000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 9.4 FPS
/usr/local/lib/python3.10/dist-packages/sleap/nn/evals.py:539: RuntimeWarning: Mean of empty slice
  "dist.avg": np.nanmean(dists),
/usr/local/lib/python3.10/dist-packages/sleap/nn/evals.py:572: RuntimeWarning: Mean of empty slice.
  mPCK = mPCK_parts.mean()
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/usr/local/lib/python3.10/dist-packages/sleap/nn/evals.py:666: RuntimeWarning: Mean of empty slice.
  pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1)
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:181: RuntimeWarning: invalid value encountered in true_divide
  ret = um.true_divide(
/usr/local/lib/python3.10/dist-packages/sleap/nn/evals.py:668: RuntimeWarning: Mean of empty slice.
  metrics["oks.mOKS"] = pair_oks.mean()
WARNING:sleap.nn.evals:Failed to compute metrics.
INFO:sleap.nn.evals:Saved predictions: models/makrop231210_122851.single_instance/labels_pr.train.slp
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2023-12-10 12:11:31.913007: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla V100-SXM2-16GB" frequency: 1530 num_cores: 80 environment { key: "architecture" value: "7.0" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 98304 memory_size: 15315632128 bandwidth: 898048000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 ?
/usr/local/lib/python3.10/dist-packages/sleap/nn/evals.py:539: RuntimeWarning: Mean of empty slice
  "dist.avg": np.nanmean(dists),
/usr/local/lib/python3.10/dist-packages/sleap/nn/evals.py:572: RuntimeWarning: Mean of empty slice.
  mPCK = mPCK_parts.mean()
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/usr/local/lib/python3.10/dist-packages/sleap/nn/evals.py:666: RuntimeWarning: Mean of empty slice.
  pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1)
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:181: RuntimeWarning: invalid value encountered in true_divide
  ret = um.true_divide(
/usr/local/lib/python3.10/dist-packages/sleap/nn/evals.py:668: RuntimeWarning: Mean of empty slice.
  metrics["oks.mOKS"] = pair_oks.mean()
WARNING:sleap.nn.evals:Failed to compute metrics.
INFO:sleap.nn.evals:Saved predictions: models/makrop231210_122851.single_instance/labels_pr.val.slp

track

!sleap-track "/content/drive/MyDrive/sleap/colab/male.mp4" -m "/content/drive/MyDrive/sleap/colab/models/makrop231210_122851.single_instance"

output:

INFO:numexpr.utils:NumExpr defaulting to 8 threads. Started inference at: 2023-12-10 12:14:48.930744 Args: {
│   'data_path': '/content/drive/MyDrive/sleap/colab/male.mp4',
│   'models': ['/content/drive/MyDrive/sleap/colab/models/makrop231210_122851.single_instance'],
│   'frames': '',
│   'only_labeled_frames': False,
│   'only_suggested_frames': False,
│   'output': None,
│   'no_empty_frames': False,
│   'verbosity': 'rich',
│   'video.dataset': None,
│   'video.input_format': 'channels_last',
│   'video.index': '',
│   'cpu': False,
│   'first_gpu': False,
│   'last_gpu': False,
│   'gpu': 'auto',
│   'max_edge_length_ratio': 0.25,
│   'dist_penalty_weight': 1.0,
│   'batch_size': 4,
│   'open_in_gui': False,
│   'peak_threshold': 0.2,
│   'max_instances': None,
│   'tracking.tracker': None,
│   'tracking.max_tracking': None,
│   'tracking.max_tracks': None,
│   'tracking.target_instance_count': None,
│   'tracking.pre_cull_to_target': None,
│   'tracking.pre_cull_iou_threshold': None,
│   'tracking.post_connect_single_breaks': None,
│   'tracking.clean_instance_count': None,
│   'tracking.clean_iou_threshold': None,
│   'tracking.similarity': None,
│   'tracking.match': None,
│   'tracking.robust': None,
│   'tracking.track_window': None,
│   'tracking.min_new_track_points': None,
│   'tracking.min_match_points': None,
│   'tracking.img_scale': None,
│   'tracking.of_window_size': None,
│   'tracking.of_max_levels': None,
│   'tracking.save_shifted_instances': None,
│   'tracking.kf_node_indices': None,
│   'tracking.kf_init_frame_count': None
}
INFO:sleap.nn.inference:Auto-selected GPU 0 with 16148 MiB of free memory.
Versions:
SLEAP: 1.3.3
TensorFlow: 2.8.4
Numpy: 1.22.4
Python: 3.10.12
OS: Linux-5.15.120+-x86_64-with-glibc2.35
System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
Video: /content/drive/MyDrive/sleap/colab/male.mp4
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2023-12-10 12:14:54.314324: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -47 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla V100-SXM2-16GB" frequency: 1530 num_cores: 80 environment { key: "architecture" value: "7.0" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 98304 memory_size: 15315632128 bandwidth: 898048000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -50 } dim { size: -51 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 100% ETA: 0:00:01 24.1 FPS2023-12-10 12:23:43.554564: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -47 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "Tesla V100-SXM2-16GB" frequency: 1530 num_cores: 80 environment { key: "architecture" value: "7.0" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 98304 memory_size: 15315632128 bandwidth: 898048000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -50 } dim { size: -51 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 17.8 FPS
Finished inference at: 2023-12-10 12:23:44.714343 Total runtime: 535.7836267948151 secs Predicted frames: 12519/12519 Provenance: {
│   'model_paths': [
│   │   '/content/drive/MyDrive/sleap/colab/models/makrop231210_122851.single_instance/training_config.json'
│   ],
│   'predictor': 'SingleInstancePredictor',
│   'sleap_version': '1.3.3',
│   'platform': 'Linux-5.15.120+-x86_64-with-glibc2.35',
│   'command': '/usr/local/bin/sleap-track /content/drive/MyDrive/sleap/colab/male.mp4 -m /content/drive/MyDrive/sleap/colab/models/makrop231210_122851.single_instance',
│   'data_path': '/content/drive/MyDrive/sleap/colab/male.mp4',
│   'output_path': '/content/drive/MyDrive/sleap/colab/male.mp4.predictions.slp',
│   'total_elapsed': 535.7836267948151,
│   'start_timestamp': '2023-12-10 12:14:48.930744',
│   'finish_timestamp': '2023-12-10 12:23:44.714343'
}
Saved output: /content/drive/MyDrive/sleap/colab/male.mp4.predictions.slp

inspect

!sleap-inspect /content/drive/MyDrive/sleap/colab/male.mp4.predictions.slp

output:

INFO:numexpr.utils:NumExpr defaulting to 8 threads.
Labeled frames: 12519
Tracks: 0
Video files:
  /content/drive/MyDrive/sleap/colab/male.mp4
    labeled frames: 12519
    labeled frames from 0 to 12518
    user labeled frames: 0
    tracks: 0
    max instances in frame: 0
Total user labeled frames: 0

Provenance:
  model_paths: ['/content/drive/MyDrive/sleap/colab/models/makrop231210_122851.single_instance/training_config.json']
  predictor: SingleInstancePredictor
  sleap_version: 1.3.3
  platform: Linux-5.15.120+-x86_64-with-glibc2.35
  command: /usr/local/bin/sleap-track /content/drive/MyDrive/sleap/colab/male.mp4 -m /content/drive/MyDrive/sleap/colab/models/makrop231210_122851.single_instance
  data_path: /content/drive/MyDrive/sleap/colab/male.mp4
  output_path: /content/drive/MyDrive/sleap/colab/male.mp4.predictions.slp
  total_elapsed: 535.7836267948151
  start_timestamp: 2023-12-10 12:14:48.930744
  finish_timestamp: 2023-12-10 12:23:44.714343
  args: {'data_path': '/content/drive/MyDrive/sleap/colab/male.mp4', 'models': ['/content/drive/MyDrive/sleap/colab/models/makrop231210_122851.single_instance'], 'frames': '', 'only_labeled_frames': False, 'only_suggested_frames': False, 'output': None, 'no_empty_frames': False, 'verbosity': 'rich', 'video.dataset': None, 'video.input_format': 'channels_last', 'video.index': '', 'cpu': False, 'first_gpu': False, 'last_gpu': False, 'gpu': 'auto', 'max_edge_length_ratio': 0.25, 'dist_penalty_weight': 1.0, 'batch_size': 4, 'open_in_gui': False, 'peak_threshold': 0.2, 'max_instances': None, 'tracking.tracker': None, 'tracking.max_tracking': None, 'tracking.max_tracks': None, 'tracking.target_instance_count': None, 'tracking.pre_cull_to_target': None, 'tracking.pre_cull_iou_threshold': None, 'tracking.post_connect_single_breaks': None, 'tracking.clean_instance_count': None, 'tracking.clean_iou_threshold': None, 'tracking.similarity': None, 'tracking.match': None, 'tracking.robust': None, 'tracking.track_window': None, 'tracking.min_new_track_points': None, 'tracking.min_match_points': None, 'tracking.img_scale': None, 'tracking.of_window_size': None, 'tracking.of_max_levels': None, 'tracking.save_shifted_instances': None, 'tracking.kf_node_indices': None, 'tracking.kf_init_frame_count': None}

download

from google.colab import files
files.download('/content/drive/MyDrive/sleap/colab/male.mp4.predictions.slp')

in conda

Conda activate sleap
Sleap-label
File > open project (male.mp4.predictions.slp)

located file

here it is:

it seems like a blank, new project, even though I thought that I have done everything according to the user guide/tutorial

so, what am I doing wrong?

[delaroob]

sooo update, I've had a look at the single_instance.json that is created after "Predict/Run training/Save configuration files", and it is pretty much empty (or at least with null values everywhere):

{
    "data": {
        "labels": {
            "training_labels": null,
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": null,
            "validation_inds": null,
            "test_inds": null,
            "search_path_hints": [],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": true,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": null,
            "resize_and_pad_to_target": true,
            "target_height": null,
            "target_width": null
        },
        "instance_cropping": {
            "center_on_part": null,
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 2,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": {
                "part_names": null,
                "sigma": 2.5,
                "output_stride": 2,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "centroid": null,
            "centered_instance": null,
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -15.0,
            "rotation_max_angle": 15.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": true,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": null,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": null,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "231210_122851",
        "run_name_prefix": "makrop",
        "run_name_suffix": ".single_instance",
        "runs_folder": "models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": false,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": false,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.3.3",
    "filename": "single_instance.json"
}

And even after training, doesn't change at all, looks the same, still null values. I'm beginning to think that I've missed a pretty important step but I can't seem to figure out what.
Any ideas?

@talmo

[Marielamend]

Hello, @delaroob!

It looks like your issue stems from your model configuration. The receptive field based on your screenshot, (illustrated by the blue box above) appears to be too small relative to the animal you are trying to track. Based on your training logs we can see your loss is not decreasing which is detected by SLEAP, and will consequently reduce the learning rate to prevent overfitting. Since the training was not working and was not improving in its validation, SLEAP terminated this whole process earlier. In terms of how we can fix this issue, you can attempt a few things.

1. You can change the configuration under your single instance model, by selecting in the dropdown (baseline_large_rf.single(baseline_large_rf.single.json)). This will increase your model receptive field size!

2. If this does not work, you can also change your input scaling in your model configuration (under data) to 0.50 this is another way to increase your receptive field. This will reduce the resolution of the image and will lose some information, but can overall help with your issue.

3. Lastly, you can try a different model approach, (in the training pipeline) using the drop-down menu to select multi-animal top-down. You can try this method to see if it produces better results since your data set has an animal that is small relative to its frame.

If you have any other questions let us know!

Best,
Mariela

[delaroob]

hi @Marielamend,

in the dropdown menu, I selected baseline_large_rf.single(baseline_large_rf.single.json), and scaled the input to 0.5.
It worked, thank you so much! :D

Though the single_instance.json is still basically a bunch of null values, but I guess thats not a problem then.