Validation of YOLO detection results

[cspino]

This issue is to discuss the validation method for the YOLO detection model.

After each training, a slice-wise validation is automatically completed by the YOLO library using the built-in val mode.
Using this mode, you can set different parameters including the confidence threshold.

But when playing with the confidence threshold, although the metrics changed, the saved images (with boxes overlapping the images) did not change (they seem to be for a fixed conf value). So with this mode, you don't have access to an accurate visual representation of the model's performance, and there is no easy way to validate the metrics.

For that reason I decided to implement my own validation script. I also thought it made more sense to use the actual predict mode and validate those results. And since the results will be used to train a 3d segmentation model, I chose to implement a volume-wise validation method, hoping it would be more representative of the model's performance.

Volume-wise merging

For every volume, I get a single set of prediction boxes (and I repeat the same steps to get one set of ground truth boxes):

1- Get a list of all slice-wise bounding boxes for given volume
2- Merge bounding boxes that overlap within a given threshold:

• Compare boxes two at a time:
• Calculate what percentage of the smallest box overlaps with the bigger box (intersection over smallest box area) and merge them if this value is over the threshold (I've been using 20% as a threshold):
  

Confusion matrix

To decide whether a prediction box matches with a ground truth box, the intersection over union (IoU) must be over a given threshold.

• A prediction that matches with a ground truth is a TP
• A prediction that has no match is a FP
• A label that has no match is a FN

Here's an example with a 40% IoU threshold:

The IoU is calculated between every ground truth - prediction box combination, and TP, FP, FN values are calculated for every volume.

Metrics

Recall ($\frac{TP}{TP+FN}$) and precision ($\frac{TP}{TP+FP}$) are calculated for the whole batch and saved in a csv file along with the TP, FP, FN values for every volume.

Nifti files from bounding boxes

Nifti files containing the prediction and ground truth boxes are saved in the validation script. They can then be opened in fsleyes along with the image and segmentation files:

[plbenveniste]

Great work !

2- Merge bounding boxes that overlap within a given threshold:

For the merging of the 2D boxes to compute the 3D box, are you making sure that the boxes are on consecutive slices ? I am thinking of a problematic case when a subject has two lesions at the same (inferior-superior) level, but one is on the right side of the SC and the other one on the left side. Maybe this should be taken into account ?

(I've been using 20% as a threshold):

This is worth justifying to avoid tricky questions on this.

Nifti files containing the prediction and ground truth boxes are saved in the validation script. They can then be opened in fsleyes along with the image and segmentation files:

I think that the bounding boxes should also be cropped on the left-right axis (using the location of the most right and left boxes as limits).

I am happy to discuss all the above if you need.

[cspino]

For now, I've kept it very simple so I'm basically just gathering all the bounding boxes for a volume, performing 2d validation and applying the resulting boxes to every slice. So I'm not considering whether two boxes are from consecutive slices or not when merging them.
But taking an actual 3D approach as you suggest @plbenveniste would definitely make more sense, I can start looking into that! And I could then display the boxes on their respective slices.

As for the 20% threshold to merge boxes, I just tested out a few different thresholds and 20% seemed reasonable (boxes within a similar area were able to be merged).. And because of the slice thickness, I've noticed that for a given lesion, the bounding box can shift quite a bit from one slice to the next, so the threshold can't be too high.

[cspino]

IoU calculation is now done in 3D and boxes are only displayed on their appropriate slices!

And merging is now only done for boxes on consecutive slices. I found an image (sub-tor097_ses-M12_PSIR) where two different lesions were being merged together before:

Before	Now