ValueError: Invalid segmentation mask

[eppane]

Hello,

Got the following error when using mmdet model (rtmdet-ins):

 res = get_prediction(
          ^^^^^^^^^^^^^^^
  File "/mmdet/lib/python3.11/site-packages/sahi/predict.py", line 100, in get_prediction
    detection_model.convert_original_predictions(
  File "/mmdet/lib/python3.11/site-packages/sahi/models/base.py", line 168, in convert_original_predictions
    self._create_object_prediction_list_from_original_predictions(
  File "/mmdet/lib/python3.11/site-packages/sahi/models/mmdet.py", line 305, in _create_object_prediction_list_from_original_predictions
    object_prediction = ObjectPrediction(
                        ^^^^^^^^^^^^^^^^^
  File "/mmdet/lib/python3.11/site-packages/sahi/prediction.py", line 79, in __init__
    super().__init__(
  File "/mmdet/lib/python3.11/site-packages/sahi/annotation.py", line 526, in __init__
    raise ValueError("Invalid segmentation mask.")

ValueError: Invalid segmentation mask.

The validity of (boolean) mask is already verified here by checking if bbox can be extracted from it, but when it gets converted to coco-formatted polygon, the bbox cannot be extracted anymore (i.e. this fails).

[fcakyon]

Can you please check again with the latest release? @eppane

[ygswlgldmy]

Hello, I also encountered this question when using a model trained based on YOLOv11m-seg for prediction in combination with the sahi (version: 0.11.19). The following error occurs during execution:

There are 50 listed files in folder: Testset_50/
Performing prediction on 80 slices.
Performing inference on images:   0%|                                                               | 0/50 [00:10<?, ?it/s]
Traceback (most recent call last):
  File "/yolov11/tools/SAHI_predict.py", line 4, in <module>
    predict(
  File "anaconda3/envs/yolov11/lib/python3.10/site-packages/sahi/predict.py", line 559, in predict
    prediction_result = get_sliced_prediction(
  File "anaconda3/envs/yolov11/lib/python3.10/site-packages/sahi/predict.py", line 283, in get_sliced_prediction
    object_prediction_list = postprocess(object_prediction_list)
  File "anaconda3/envs/yolov11/lib/python3.10/site-packages/sahi/postprocess/combine.py", line 555, in __call__
    object_prediction_list[keep_ind] = merge_object_prediction_pair(
  File "anaconda3/envs/yolov11/lib/python3.10/site-packages/sahi/postprocess/utils.py", line 220, in merge_object_prediction_pair
    return ObjectPrediction(
  File "anaconda3/envs/yolov11/lib/python3.10/site-packages/sahi/prediction.py", line 79, in __init__
    super().__init__(
  File "anaconda3/envs/yolov11/lib/python3.10/site-packages/sahi/annotation.py", line 526, in __init__
    raise ValueError("Invalid segmentation mask.")
ValueError: Invalid segmentation mask.

when I use the Ultralytics official pre-trained YOLOv11m-seg model for predictions with sahi slicing, it works correctly. Are there specific configurations or preprocessing steps required for sahi to work correctly with custom-trained models?

Any guidance or insights on troubleshooting this issue would be greatly appreciated. Thank you for your support!

[calebdeejohnson]

For me the problem lies here: https://github.com/obss/sahi/blob/main/sahi/utils/shapely.py#L41

If I have a geometry collection that has multipolygons and line strings, but no polygons, then this will return an empty polygon. If both tiles return an empty polygon then the merge is empty and leads to this problem. Just a slight refactoring of this to allow for multipolygons fixed all my problems. Here is a visualization of the geometry collection that gets returned as empty and helps lead to this problem.

@fcakyon is this something that easily change in a new release? Happy to help in any way from my end

[Jquinto64]

@calebdeejohnson Could you share what you changed in your code to get this working?

[fcakyon]

@calebdeejohnson If you could open a PR with your fix, I would gladly help you and merge it!

[mookghg]

Hi! I wanted to add my 2 cents here. I recently encountered the ValueError: Invalid segmentation mask. error with a Detectron2 instance segmentation implementation running from the main repo. After doing some digging I discovered that this piece of code within the Detectron2 model file:

object_prediction_list = [
                ObjectPrediction(
                    bbox=box.tolist() if mask is None else None,
                    segmentation=(
                        get_coco_segmentation_from_bool_mask(mask.detach().cpu().numpy()) if mask is not None else None
                    ),
                    category_id=category_id.item(),
                    category_name=self.category_mapping[str(category_id.item())],
                    shift_amount=shift_amount,
                    score=score.item(),
                    full_shape=full_shape,
                )
                for box, score, category_id, mask in zip(boxes, scores, category_ids, masks)
                if mask is None or get_bbox_from_bool_mask(mask.detach().cpu().numpy()) is not None
            ]

would fail where get_coco_segmentation_from_bool_mask would return an empty list this would then cause an error within the annotation.py file raising the error.

It seems that the get_bbx_frombool_mask code occasionally returns not None values when bool masks might look like this:

This occurs with small pixel detections which in my case could be common as the data im working with is low res and a single pixel could actually be a significant area.

I do not yet have a fix for this. I am intending to filter these empty lists however I think the get_bbox_from_bool function probably needs to act per object not per mask as the width x height dimensions are incorrectly returned.

further comment, filtering like this may not be desired for low-res images, sub pixel approach might be needed.

finally, im not sure if others have had a similar issue and if a fork exists which fixes this