Question about likelihood (mean score？)

[nova8761]

When I was using DLC before, a likelihood (confidence) was given to each frame after training or prediction, which was a value between 0 and 1, meaning the probability of being correct was between 0% and 100%.
Now I'm going to try to use SLEAP instead. I find that SLEAP has a similar parameter called "mean score" for training and "score" for inference, but it seems to be a value between 0 and 2.
May I ask: How is this value defined and how is it calculated? It does not seem to be a probability value?

[roomrys]

Hi @nova8761,

The mean score is the mean of all the instance scores in the frame. The calculation of SLEAP's instance scores differ from type of model to type of model (see Predictors below). In general, the instance score aims to measure the grouping of nodes (predicted points) into an instance. For most predictors, SLEAP uses the likelihood as a score which is proportional to the probability of a point being correct, but not normalized. Thus the higher the score the more "confident" SLEAP is in that prediction/grouping, but this confidence is not constrained between 0 to 1. That being said, one downfall of having a nonnormalized score means that it is difficult to compare predictions across different types of models.

Thanks,
Liezl

---

Related #1268 (and #1219)

Hi @druz, @mollywestbrook,

Point

The score comes directly from the output of our keras model (heatmaps that indicate the likelihood of each pixel containing the center of a particular keypoint). Each node will get its own heatmap (aka confidence map) and within each heatmap there will be multiple peaks for where SLEAP thinks there is some non-zero likelihood that a pixel contains the center of a node given the annotated frames. Note that this likelihood is not normalized, but is proportional to the joint probability of a "pixel containing the center of a particular keypoint" given the annotated data. This proportionality (and comparison to other likelihoods) is all we need to determine which peak in our heatmap to use as the predicted point for that node. This is done for all nodes and the likelihood is then used as the score for that point.

Instances

Per the doc strings, the instance score is supposed to measure the grouping of nodes.

sleap/sleap/instance.py

Lines 1166 to 1167 in b5aab0e

	instance_score: Scalar float representing the overall instance score, e.g.,
	the PAF grouping score.

SingleInstancePredictor

The score is a sum of all the point scores (not a grouping score because there is not grouping to be done - all predicted nodes must belong to this single instance). This score would not even fall within the range of the other Predictors since it would be scaled by the number of nodes.

TopDownPredictor

Much like the score for a point, this score comes from the heatmap of the first model in top-down (the centroid model) which finds the centroid/"anchor point" (the location of the node closest to the centroid of the object being tracked). Thus the instance score is the likelihood of a pixel containing the center of the "anchor point".

BottomUpPredictor

The instance score for a BottomUpPredictor is the sum of all the edge scores in an instance. The edge score comes from the PAFs and is explained in the doc strings for score_paf_lines as:

sleap/sleap/nn/paf_grouping.py

Lines 354 to 361 in b5aab0e

	Returns:
	The line scores as a `tf.Tensor` of shape `(n_candidates,)` and dtype
	`tf.float32`. Each score value is the average dot product between the PAFs and
	the normalized displacement vector between source and destination peaks.
	Scores range from roughly -1.5 to 1.0, where larger values indicate a better
	connectivity score for the candidate. Values can be larger or smaller due to
	prediction error.

TopDownMulticlassPredictor

The multiclass predictors have tracking baked in as another network that uses annotated Tracks to predict instance tracks. So, now the instance score is the classification probabilities that an instance is in the assigned Track. Similar to the point scores, these classification probabilities are directly output from the keras model used to run inference. Note that these are referred to as probabilities not likelihoods; thus we expect them to fall within 0 to 1.

BottomUpMulticlassPredictor

The same as the TopDownMulticlassPredictor

MoveNetPredictor

The same as SingleInstancePredictor.

Tracking

So the tracking score is derived using both an instance similarity score to form a cost matrix and then using a matching method to choose which instances to assign to which track (essentially selecting an element in the cost matrix to assign to the instance as a tracking score). The similarity functions to choose from are either instance similarity, centroid distance, or instance Intersection Over Union (IOU). The output from each of these will form the cost matrix. The method chosen for matching will then select which element of the cost matrix to assign each to each Instance.tracking_score.

sleap/sleap/nn/tracker/components.py

Lines 29 to 78 in b5aab0e

	def instance_similarity(
	ref_instance: InstanceType, query_instance: InstanceType
	) -> float:
	"""Computes similarity between instances."""
	ref_visible = ~(np.isnan(ref_instance.points_array).any(axis=1))
	dists = np.sum(
	(query_instance.points_array - ref_instance.points_array) ** 2, axis=1
	)
	similarity = np.nansum(np.exp(-dists)) / np.sum(ref_visible)
	return similarity
	def centroid_distance(
	ref_instance: InstanceType, query_instance: InstanceType, cache: dict = dict()
	) -> float:
	"""Returns the negative distance between the centroids of two instances.
	Uses `cache` dictionary (created with function so it persists between calls)
	since without cache this method is significantly slower than others.
	"""
	if ref_instance not in cache:
	cache[ref_instance] = ref_instance.centroid
	if query_instance not in cache:
	cache[query_instance] = query_instance.centroid
	a = cache[ref_instance]
	b = cache[query_instance]
	return -np.linalg.norm(a - b)
	def instance_iou(
	ref_instance: InstanceType, query_instance: InstanceType, cache: dict = dict()
	) -> float:
	"""Computes IOU between bounding boxes of instances."""
	if ref_instance not in cache:
	cache[ref_instance] = ref_instance.bounding_box
	if query_instance not in cache:
	cache[query_instance] = query_instance.bounding_box
	a = cache[ref_instance]
	b = cache[query_instance]
	return utils.compute_iou(a, b)