Accuracy scores?

[druoz]

Hello,
I've been trying Sleap and so far I've been very happy with how easy it is to tag with the GUI. This is great software!

After exporting data, I see the file using Google colab.
Regarding accuracy data:
What is the unit of measurement of accuracy for point, instances and tracking accuracy?
In my case, accuracy score for tracking ranges 0-1, instance scores above 5 (does this depend on number of body parts, is this a sum?) and point score above 1. Can you explain the range?
Thanks in advance.
Ugur

[mollywestbrook]

I have this same question. Thanks!

[roomrys]

Hi @druz, @mollywestbrook,

Point

The score comes directly from the output of our keras model (heatmaps that indicate the likelihood of each pixel containing the center of a particular keypoint). Each node will get its own heatmap (aka confidence map) and within each heatmap there will be multiple peaks for where SLEAP thinks there is some non-zero likelihood that a pixel contains the center of a node given the annotated frames. Note that this likelihood is not normalized, but is proportional to the joint probability of a "pixel containing the center of a particular keypoint" given the annotated data. This proportionality (and comparison to other likelihoods) is all we need to determine which peak in our heatmap to use as the predicted point for that node. This is done for all nodes and the likelihood is then used as the score for that point.

Instances

Per the doc strings, the instance score is supposed to measure the grouping of nodes.

sleap/sleap/instance.py

Lines 1166 to 1167 in b5aab0e

	instance_score: Scalar float representing the overall instance score, e.g.,
	the PAF grouping score.

SingleInstancePredictor

The score is a sum of all the point scores (not a grouping score because there is not grouping to be done - all predicted nodes must belong to this single instance). This score would not even fall within the range of the other Predictors since it would be scaled by the number of nodes.

TopDownPredictor

Much like the score for a point, this score comes from the heatmap of the first model in top-down (the centroid model) which finds the centroid/"anchor point" (the location of the node closest to the centroid of the object being tracked). Thus the instance score is the likelihood of a pixel containing the center of the "anchor point".

BottomUpPredictor

The instance score for a BottomUpPredictor is the sum of all the edge scores in an instance. The edge score comes from the PAFs and is explained in the doc strings for score_paf_lines as:

sleap/sleap/nn/paf_grouping.py

Lines 354 to 361 in b5aab0e

	Returns:
	The line scores as a `tf.Tensor` of shape `(n_candidates,)` and dtype
	`tf.float32`. Each score value is the average dot product between the PAFs and
	the normalized displacement vector between source and destination peaks.
	Scores range from roughly -1.5 to 1.0, where larger values indicate a better
	connectivity score for the candidate. Values can be larger or smaller due to
	prediction error.

TopDownMulticlassPredictor

The multiclass predictors have tracking baked in as another network that uses annotated Tracks to predict instance tracks. So, now the instance score is the classification probabilities that an instance is in the assigned Track. Similar to the point scores, these classification probabilities are directly output from the keras model used to run inference. Note that these are referred to as probabilities not likelihoods; thus we expect them to fall within 0 to 1.

BottomUpMulticlassPredictor

The same as the TopDownMulticlassPredictor

MoveNetPredictor

The same as SingleInstancePredictor.

Tracking

So the tracking score is derived using both an instance similarity score to form a cost matrix and then using a matching method to choose which instances to assign to which track (essentially selecting an element in the cost matrix to assign to the instance as a tracking score). The similarity functions to choose from are either instance similarity, centroid distance, or instance Intersection Over Union (IOU). The output from each of these will form the cost matrix. The method chosen for matching will then select which element of the cost matrix to assign each to each Instance.tracking_score.

sleap/sleap/nn/tracker/components.py

Lines 29 to 78 in b5aab0e

	def instance_similarity(
	ref_instance: InstanceType, query_instance: InstanceType
	) -> float:
	"""Computes similarity between instances."""
	ref_visible = ~(np.isnan(ref_instance.points_array).any(axis=1))
	dists = np.sum(
	(query_instance.points_array - ref_instance.points_array) ** 2, axis=1
	)
	similarity = np.nansum(np.exp(-dists)) / np.sum(ref_visible)
	return similarity
	def centroid_distance(
	ref_instance: InstanceType, query_instance: InstanceType, cache: dict = dict()
	) -> float:
	"""Returns the negative distance between the centroids of two instances.
	Uses `cache` dictionary (created with function so it persists between calls)
	since without cache this method is significantly slower than others.
	"""
	if ref_instance not in cache:
	cache[ref_instance] = ref_instance.centroid
	if query_instance not in cache:
	cache[query_instance] = query_instance.centroid
	a = cache[ref_instance]
	b = cache[query_instance]
	return -np.linalg.norm(a - b)
	def instance_iou(
	ref_instance: InstanceType, query_instance: InstanceType, cache: dict = dict()
	) -> float:
	"""Computes IOU between bounding boxes of instances."""
	if ref_instance not in cache:
	cache[ref_instance] = ref_instance.bounding_box
	if query_instance not in cache:
	cache[query_instance] = query_instance.bounding_box
	a = cache[ref_instance]
	b = cache[query_instance]
	return utils.compute_iou(a, b)

[mfkeles]

I am a bit confused about point scores as they do not seem to be a direct match to "likelihood" value other models provide (i.e. DLC). I noticed that point scores can be greater than 1. What does this mean? Is there a way to use this information to determine an occluded body part or pick a keypoint that has a mirror keypoint.

Thanks,
Mehmet

[msjindal]

I also have the same question. Our instance score is also above 1 occasionally and how do we interpret that score?

[Marielamend]

Hello @mfkeles @msjindal !

Hopefully, this clarifies things, if not let us know!

Point Score
This is the score for each body point, which you can use to determine the types of examples your model would like more annotations of. Given that this score is

likelihood that a pixel contains the center of a node given the annotated frames

This score can technically be above 1 but you can interpret it as a probability.

Instance scores
In general, this measures the grouping of the nodes into an instance/ “animal”. The higher the score the more confident SLEAP is in that grouping. This score is not normalized meaning it is not constrained between 0 to 1.

So, the instance score should measure how confident SLEAP is that the nodes grouped together in an Instance should be grouped together in that Instance. Instance scores are used to solve the grouping problem: "should Node X belong to Instance A or Instance B?".
(#1680 (comment))

Keep in mind, that the calculation of the instance scores varies relative to the model that was chosen by the user which is described above by the different predictor subcategories.

A bit more insight -> From the sleap team:
The underlying reason for this is that the output of our neural networks has a linear activation function, which doesn't constrain it to be between 0 and 1 (even though that's what we train them on). We empirically find that this gives us better performance over applying a sigmoid activation which would strictly constrain the range.

Best,
Mariela