How evaluation metrics are calculated?

[KennyTC]

After running training models, I see a lot of evaluation metrics, for example

vis.tp
vis.fp
vis.tn
vis.fn
vis.precision
vis.recall
dist.dists
dist.avg
dist.p50
dist.p75
dist.p90
dist.p95
dist.p99
pck.thresholds
pck.pcks
pck.mPCK_parts
pck.mPCK
oks.mOKS
oks_voc.match_score_thresholds
oks_voc.recall_thresholds
oks_voc.match_scores
oks_voc.precisions
oks_voc.recalls
oks_voc.AP
oks_voc.AR
oks_voc.mAP
oks_voc.mAR
pck_voc.match_score_thresholds
pck_voc.recall_thresholds
pck_voc.match_scores
pck_voc.precisions
pck_voc.recalls
pck_voc.AP
pck_voc.AR
pck_voc.mAP

For distance related metrics such as (dist.dists, dist.avg, dist.p50, dist.p75.....), I can understand.
But for other metrics, I could not find how these metrics are calculated. How can I get the overview of how these metrics calculated?

Any recommendation of which metrics should I focus when comparing these models?

[roomrys]

Hi @KennyTC,

I'll start by listing the names of the metrics below:
- "vis.tp": Visibility - True Positives
- "vis.fp": Visibility - False Positives
- "vis.tn": Visibility - True Negatives
- "vis.fn": Visibility - False Negatives
- "vis.precision": Visibility - Precision
- "vis.recall": Visibility - Recall
- "dist.avg": Average Distance (ground truth vs prediction)
- "dist.p50": Distance for 50th percentile
- "dist.p75": Distance for 75th percentile
- "dist.p90": Distance for 90th percentile
- "dist.p95": Distance for 95th percentile
- "dist.p99": Distance for 99th percentile
- "dist.dists": All distances
- "pck.mPCK": Mean Percentage of Correct Keypoints (PCK)
- "oks.mOKS": Mean Object Keypoint Similarity (OKS)
- "oks_voc.mAP": VOC with OKS scores - mean Average Precision (mAP)
- "oks_voc.mAR": VOC with OKS scores - mean Average Recall (mAR)
- "pck_voc.mAP": VOC with PCK scores - mean Average Precision (mAP)
- "pck_voc.mAR": VOC with PCK scores - mean Average Recall (mAR)

Calculations are tied together through the evaluate function.

All visibility (vis):

sleap/sleap/nn/evals.py

Lines 549 to 582 in 5093f69

	def compute_visibility_conf(
	positive_pairs: List[Tuple[Instance, Instance, Any]]
	) -> Dict[Text, float]:
	"""Compute node visibility metrics.
	Args:
	positive_pairs: A list of tuples of the form `(instance_gt, instance_pr, _)`
	containing the matched pair of instances.
	Returns:
	A dictionary of visibility metrics, including the confusion matrix.
	"""
	vis_tp = 0
	vis_fn = 0
	vis_fp = 0
	vis_tn = 0
	for instance_gt, instance_pr, _ in positive_pairs:
	missing_nodes_gt = np.isnan(instance_gt.points_array).any(axis=-1)
	missing_nodes_pr = np.isnan(instance_pr.points_array).any(axis=-1)
	vis_tn += ((missing_nodes_gt) & (missing_nodes_pr)).sum()
	vis_fn += ((~missing_nodes_gt) & (missing_nodes_pr)).sum()
	vis_fp += ((missing_nodes_gt) & (~missing_nodes_pr)).sum()
	vis_tp += ((~missing_nodes_gt) & (~missing_nodes_pr)).sum()
	return {
	"vis.tp": vis_tp,
	"vis.fp": vis_fp,
	"vis.tn": vis_tn,
	"vis.fn": vis_fn,
	"vis.precision": vis_tp / (vis_tp + vis_fp) if (vis_tp + vis_fp) else np.nan,
	"vis.recall": vis_tp / (vis_tp + vis_fn) if (vis_tp + vis_fn) else np.nan,
	}

The distances (dist.dists):

sleap/sleap/nn/evals.py

Lines 472 to 492 in 5093f69

	def compute_dists(
	positive_pairs: List[Tuple[Instance, PredictedInstance, Any]]
	) -> np.ndarray:
	"""Compute Euclidean distances between matched pairs of instances.
	Args:
	positive_pairs: A list of tuples of the form `(instance_gt, instance_pr, _)`
	containing the matched pair of instances.
	Returns:
	An array of pairwise distances of shape `(n_positive_pairs, n_nodes)`.
	"""
	dists = []
	for instance_gt, instance_pr, _ in positive_pairs:
	points_gt = instance_gt.points_array
	points_pr = instance_pr.points_array
	dists.append(np.linalg.norm(points_pr - points_gt, axis=-1))
	dists = np.array(dists)
	return dists

The distance errors (dist.avg, dist.pXX):

sleap/sleap/nn/evals.py

Lines 495 to 520 in 5093f69

	def compute_dist_metrics(dists: np.ndarray) -> Dict[Text, np.ndarray]:
	"""Compute the Euclidean distance error at different percentiles.
	Args:
	dists: An array of pairwise distances of shape `(n_positive_pairs, n_nodes)`.
	Returns:
	A dictionary of distance metrics.
	"""
	results = {
	"dist.dists": dists,
	"dist.avg": np.nanmean(dists),
	"dist.p50": np.nan,
	"dist.p75": np.nan,
	"dist.p90": np.nan,
	"dist.p95": np.nan,
	"dist.p99": np.nan,
	}
	is_non_nan = ~np.isnan(dists)
	if np.any(is_non_nan):
	non_nans = dists[is_non_nan]
	for ptile in (50, 75, 90, 95, 99):
	results[f"dist.p{ptile}"] = np.percentile(non_nans, ptile)
	return results

All Percentage of Correct Points (pck):

sleap/sleap/nn/evals.py

Lines 523 to 546 in 5093f69

	def compute_pck_metrics(
	dists: np.ndarray, thresholds: np.ndarray = np.linspace(1, 10, 10)
	) -> Dict[Text, np.ndarray]:
	"""Compute PCK across a range of thresholds.
	Args:
	dists: An array of pairwise distances of shape `(n_positive_pairs, n_nodes)`.
	thresholds: A list of distance thresholds in pixels.
	Returns:
	A dictionary of PCK metrics evaluated at each threshold.
	"""
	dists = np.copy(dists)
	dists[np.isnan(dists)] = np.inf
	pcks = np.expand_dims(dists, -1) < np.reshape(thresholds, (1, 1, -1))
	mPCK_parts = pcks.mean(axis=0).mean(axis=-1)
	mPCK = mPCK_parts.mean()
	return {
	"pck.thresholds": thresholds,
	"pck.pcks": pcks,
	"pck.mPCK_parts": mPCK_parts,
	"pck.mPCK": mPCK,
	}

Mean Object Keypoint Similarity (oks.mOKS)

sleap/sleap/nn/evals.py

Lines 632 to 635 in 5093f69

	pair_oks = np.array([oks for _, _, oks in positive_pairs])
	pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1)
	metrics["oks.mOKS"] = pair_oks.mean()

where positive_pairs is calculated here

sleap/sleap/nn/evals.py

Lines 340 to 386 in 5093f69

	def match_frame_pairs(
	frame_pairs: List[Tuple[LabeledFrame, LabeledFrame]],
	stddev: float = 0.025,
	scale: Optional[float] = None,
	threshold: float = 0,
	user_labels_only: bool = True,
	) -> Tuple[List[Tuple[Instance, PredictedInstance, float]], List[Instance]]:
	"""Match all ground truth and predicted instances within each pair of frames.
	This is a wrapper for `match_instances()` but operates on lists of frames.
	Args:
	frame_pairs: A list of pairs of `sleap.LabeledFrame`s in the form
	`(frame_gt, frame_pr)`. These can be obtained with `find_frame_pairs()`.
	stddev: The expected spread of coordinates for OKS computation.
	scale: The scale for normalizing the OKS. If not set, the bounding box area will
	be used.
	threshold: The minimum OKS between a candidate pair of instances to be
	considered a match.
	user_labels_only: If False, predicted instances in the ground truth frame may be
	considered for matching.
	Returns:
	A tuple of (`positive_pairs`, `false_negatives`).
	`positive_pairs` is a list of 3-tuples of the form
	`(instance_gt, instance_pr, oks)` containing the matched pair of instances and
	their OKS.
	`false_negatives` is a list of ground truth `sleap.Instance`s that could not be
	matched.
	"""
	positive_pairs = []
	false_negatives = []
	for frame_gt, frame_pr in frame_pairs:
	positive_pairs_frame, false_negatives_frame = match_instances(
	frame_gt,
	frame_pr,
	stddev=stddev,
	scale=scale,
	threshold=threshold,
	user_labels_only=user_labels_only,
	)
	positive_pairs.extend(positive_pairs_frame)
	false_negatives.extend(false_negatives_frame)
	return positive_pairs, false_negatives

All Visual Object Classes (_voc):

sleap/sleap/nn/evals.py

Lines 389 to 469 in 5093f69

	def compute_generalized_voc_metrics(
	positive_pairs: List[Tuple[Instance, PredictedInstance, Any]],
	false_negatives: List[Instance],
	match_scores: List[float],
	match_score_thresholds: np.ndarray = np.linspace(0.5, 0.95, 10), # 0.5:0.05:0.95
	recall_thresholds: np.ndarray = np.linspace(0, 1, 101), # 0.0:0.01:1.00
	name: Text = "gvoc",
	) -> Dict[Text, Any]:
	"""Compute VOC metrics given matched pairs of instances.
	Args:
	positive_pairs: A list of tuples of the form `(instance_gt, instance_pr, _)`
	containing the matched pair of instances.
	false_negatives: A list of unmatched instances.
	match_scores: The score obtained in the matching procedure for each matched pair
	(e.g., OKS).
	match_score_thresholds: Score thresholds at which to consider matches as a true
	positive match.
	recall_thresholds: Recall thresholds at which to evaluate Average Precision.
	name: Name to use to prefix returned metric keys.
	Returns:
	A dictionary of VOC metrics.
	"""
	detection_scores = np.array([pp[1].score for pp in positive_pairs])
	inds = np.argsort(-detection_scores, kind="mergesort")
	detection_scores = detection_scores[inds]
	match_scores = match_scores[inds]
	precisions = []
	recalls = []
	npig = len(positive_pairs) + len(false_negatives) # total number of GT instances
	for match_score_threshold in match_score_thresholds:
	tp = np.cumsum(match_scores >= match_score_threshold)
	fp = np.cumsum(match_scores < match_score_threshold)
	rc = tp / npig
	pr = tp / (fp + tp + np.spacing(1))
	recall = rc[-1] # best recall at this OKS threshold
	# Ensure strictly decreasing precisions.
	for i in range(len(pr) - 1, 0, -1):
	if pr[i] > pr[i - 1]:
	pr[i - 1] = pr[i]
	# Find best precision at each recall threshold.
	rc_inds = np.searchsorted(rc, recall_thresholds, side="left")
	precision = np.zeros(rc_inds.shape)
	is_valid_rc_ind = rc_inds < len(pr)
	precision[is_valid_rc_ind] = pr[rc_inds[is_valid_rc_ind]]
	precisions.append(precision)
	recalls.append(recall)
	precisions = np.array(precisions)
	recalls = np.array(recalls)
	AP = precisions.mean(
	axis=1
	) # AP = average precision over fixed set of recall thresholds
	AR = recalls # AR = max recall given a fixed number of detections per image
	mAP = precisions.mean() # mAP = mean over all OKS thresholds
	mAR = recalls.mean() # mAR = mean over all OKS thresholds
	return {
	name + ".match_score_thresholds": match_score_thresholds,
	name + ".recall_thresholds": recall_thresholds,
	name + ".match_scores": match_scores,
	name + ".precisions": precisions,
	name + ".recalls": recalls,
	name + ".AP": AP,
	name + ".AR": AR,
	name + ".mAP": mAP,
	name + ".mAR": mAR,
	}

The model evaluation notebook provides further info on what metrics might be helpful to look at.

Hope this helps!

Thanks,
Liezl