the results of coco evaluate of sahi and test.py of mmdetection are different?

[SleepyFanz]

mmdetection:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.617
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.900
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.733
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.581
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.642
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.631
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.689
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.689
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.689
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.656
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.711
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.698

coco evaluate:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=500 ] = 0.569
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=500 ] = 0.861
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=500 ] = 0.658
Average Precision (AP) @[ IoU=0.50 | area= small | maxDets=500 ] = 0.823
Average Precision (AP) @[ IoU=0.50 | area=medium | maxDets=500 ] = 0.880
Average Precision (AP) @[ IoU=0.50 | area= large | maxDets=500 ] = 0.871
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=500 ] = 0.489
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=500 ] = 0.606
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=500 ] = 0.611
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=500 ] = 0.537
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=500 ] = 0.665
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=500 ] = 0.669

parameters used in coco evaluate are the same as mmdetetion command, by the way , I add --no_sliced_prediction command, so the results should be same, I can't figure out why, even change --postprocess_match_metric and --postprocess_type, it doesn't work.

[fcakyon]

maxDets param is different for both eval, you should take eval at same maxDets value (100, 500 or 1000)

[imadgohar]

@fcakyon How can we set maxDets parameter, I am using this parameter --proposal_nums "[10 100 500]" in --> sahi evaluation but have no effect.

[fcakyon]

you can set it with max_detections param:

sahi/sahi/scripts/coco_evaluation.py

Line 348 in 72b426d

max_detections (int): Maximum number of detections to consider for AP alculation. Default: 500

[SleepyFanz]

thanks a lot, but still,
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.569
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.861
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.658
Average Precision (AP) @[ IoU=0.50 | area= small | maxDets=1000 ] = 0.823
Average Precision (AP) @[ IoU=0.50 | area=medium | maxDets=1000 ] = 0.880
Average Precision (AP) @[ IoU=0.50 | area= large | maxDets=1000 ] = 0.871
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.489
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.606
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.611
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.537
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.665
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.669
COCO evaluation results are successfully exported to runs/predict/exp19/eval.json

[fcakyon]

What do you use as the input? Do you use coco formatted results.json as input or you perform prediction with sahi?

[SleepyFanz]

yes, as for sahi coco evaluate,i use results.json got by sahi coco prediction as input, on the other hand, i use test.py to get results.pkl, but they turn out to be different.

[fcakyon]

That is normal as test.py and sahi predict outputs are different (bounding box integer casting, postprocessing functions and parameters etc)

Can you tell me what is your postprocessing parameters in test.py from mmdet? (iou and conf thresholds)

You should use NMS as postprocess_type in SAHI by adding --postprocess_type NMS into the sahi predict command.

You should also lower the confidence threshold. Its 0.01 by default in mmdet but 0.25 in sahi . Add --model_confidence_threshold 0.01 into the sahi predict command.

[SleepyFanz]

After setting the --model_confidence_threshold 0.01, and --postprocess_type NMS, the results,
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.587
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.891
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.675
Average Precision (AP) @[ IoU=0.50 | area= small | maxDets=1000 ] = 0.871
Average Precision (AP) @[ IoU=0.50 | area=medium | maxDets=1000 ] = 0.911
Average Precision (AP) @[ IoU=0.50 | area= large | maxDets=1000 ] = 0.886
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.509
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.626
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.625
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.586
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.704
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.694
still smaller than mmdet's

[fcakyon]

There is only 0.009 AP difference in AP@50 which is more acceptable considering sahi casts prediction boxes into integers.

[SleepyFanz]

Thanks, reasonable answer.