Input scale

[WeheliyeHashi]

Hi

Many thanks for your code. I just had a small question regarding the training and validation of images. If my training images are of size 100 by 100 pixel see attached image, can I validate it on Images that are 3000 by 3000 pixels (also see the attached image). Note here that the worm size in the large image in pixels are like half of the training images. Just to clarify the large image with all the worms are the one I am using for validation. The smaller one with the single worm is for training.

[talmo]

Hi @WeheliyeHashi,

Yes, you technically can do this, but you might need to tweak some of the model configuration parameters so we don't automatically scale it.

What model type are you training (top-down, bottom-up)?

Do you mind sharing the model training_config.json file here so we can have a look?

Cheers,

Talmo

[WeheliyeHashi]

Also it seems as thought all my points are invisible (red colour text) even thought I have them visible in my coco dataset (i.e. keypoints = [x,y, 2]). It seems sleap its not detecting this input. Should I make the points visible manually by clicking on them and setting it visible. It may take a while for large image dataset.

[roomrys]

Piping in here that the red color text does not denote point visibility as stated in this bit of documentation on prediction-assisted labeling:

All node labels on the regular instance will be colored red. After moving nodes, the node labels will colored green. This is just a visual indicator to keep track of which nodes have been moved from the original prediction.

Nodes are denoted as not visible when text is grayed-out (and highlighted in gray).

[roomrys]

As you can see in the training image I connect the tail>tail_base_1>neck_1>neck_1_base>hips_1>midpoint_1.....etc do you reckon its better if I connect midpoint_1>tail, midpoint_1>tail_base_1, midpoint_1>neck_1 etc....I was thinking this might work better because the points are better separated and the PAF's will be much smoother and better linkage.

Yes, this is a good idea. When labeling the tail on a mouse we often recommend connecting the tail base (which is less likely to be occluded) as a source to all other sections of the tail (same idea here).

[WeheliyeHashi]

Thanks @roomrys for your reply!

I am still struggling with the input scale of the images. I can't somehow get even a prediction for the larger image (3036 by 3036 pixel) when I do training on the small image (100 pixel by 100 pixel). Any useful tips to solving this issue? Thanks !

[roomrys]

Hi @WeheliyeHashi,

I am running some tests on my end using two videos with different ratios between animal and image size. The second video is a scaled and cropped version of the first video. These are the dimensions:

I labeled just a single frame in the second video and had many frames labeled in the first video. Running training (I tested on both top-down and bottom-up) then inference on the "all videos", SLEAP was able to predict on the both videos nicely.

---

The top-down training configs I used:

Centroid training config

{
    "data": {
        "labels": {
            "training_labels": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": true,
            "training_inds": [ ],
            "validation_inds": [ ],
            "test_inds": null,
            "search_path_hints": [
                "",
                "",
                "",
                "",
                ""
            ],
            "skeletons": [ ]
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 0.5,
            "pad_to_stride": 16,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": "thorax",
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 2,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": {
                "anchor_part": "thorax",
                "sigma": 2.5,
                "output_stride": 2,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "centered_instance": null,
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        }
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -15.0,
            "rotation_max_angle": 15.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": true
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 20
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "221118_200659.centroid.n=149",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": false,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.2.9",
    "filename": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\models\\221118_200659.centroid.n=149\\training_config.json"
}

Centered Instance training config

{
    "data": {
        "labels": {
            "training_labels": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [ ],
            "validation_inds": [ ],
            "test_inds": null,
            "search_path_hints": [
                "",
                "",
                "",
                "",
                "",
                ""
            ],
            "skeletons": [ ]
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": 1,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": "thorax",
            "crop_size": 144,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 4,
                "filters": 24,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": null,
            "centered_instance": {
                "anchor_part": "thorax",
                "part_names": [
                    "head",
                    "thorax",
                    "abdomen",
                    "wingL",
                    "wingR",
                    "forelegL4",
                    "forelegR4",
                    "midlegL4",
                    "midlegR4",
                    "hindlegL4",
                    "hindlegR4",
                    "eyeL",
                    "eyeR"
                ],
                "sigma": 2.5,
                "output_stride": 4,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        }
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": true
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "221123_092731.centered_instance.n=150",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.2.9",
    "filename": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\models\\221123_092731.centered_instance.n=150\\training_config.json"
}

---

The bottom-up training config I used:

{
    "data": {
        "labels": {
            "training_labels": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [ ],
            "validation_inds": [ ],
            "test_inds": null,
            "search_path_hints": [
                ""
            ],
            "skeletons": [ ]
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": 32,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": null,
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 32,
                "output_stride": 4,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": null,
            "centered_instance": null,
            "multi_instance": {
                "confmaps": {
                    "part_names": [
                        "head",
                        "thorax",
                        "abdomen",
                        "wingL",
                        "wingR",
                        "forelegL4",
                        "forelegR4",
                        "midlegL4",
                        "midlegR4",
                        "hindlegL4",
                        "hindlegR4",
                        "eyeL",
                        "eyeR"
                    ],
                    "sigma": 2.5,
                    "output_stride": 4,
                    "loss_weight": 1.0,
                    "offset_refinement": false
                },
                "pafs": {
                    "edges": [
                        [
                            "thorax",
                            "head"
                        ],
                        [
                            "thorax",
                            "abdomen"
                        ],
                        [
                            "thorax",
                            "wingL"
                        ],
                        [
                            "thorax",
                            "wingR"
                        ],
                        [
                            "thorax",
                            "forelegL4"
                        ],
                        [
                            "thorax",
                            "forelegR4"
                        ],
                        [
                            "thorax",
                            "midlegL4"
                        ],
                        [
                            "thorax",
                            "midlegR4"
                        ],
                        [
                            "thorax",
                            "hindlegL4"
                        ],
                        [
                            "thorax",
                            "hindlegR4"
                        ],
                        [
                            "head",
                            "eyeL"
                        ],
                        [
                            "head",
                            "eyeR"
                        ]
                    ],
                    "sigma": 75.0,
                    "output_stride": 8,
                    "loss_weight": 1.0
                }
            },
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        }
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": true
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 8,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "221123_101230.multi_instance.n=150",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.2.9",
    "filename": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\models\\221123_101230.multi_instance.n=150\\training_config.json"
}

---

I notice you are using a collection of SingleImageVideos - it is possible this might be an issue with the SingleImageVideo backend. Are you able to share your dataset (the .slp, the model(s) used, and the videos/images) with [email protected] for troubleshooting purposes? In the meantime, I'll keep looking into the problem and update you if I find what the issue might be.

Thanks and sorry you are experiencing so much trouble!
Liezl

[WeheliyeHashi]

If you have any findings, please share with us!! @talmo

Oh I fixed the issue with the invisible points....

[roomrys]

For posterity, the images being predicted on were differently sized than the training images. Prior to SLEAP v1.3.0a0, models expected the same input size as the images they were trained on. However, this is not necessarily needed for models that are fully convolutional. Thus, we opened PR #1084 to allow flexible resizing of the input layer to allow inference on any size input regardless of the training image sizes. (This was followed up via email, but the loop was never closed on Github!)