hidden nodes

[chrisch12138]

Hi,

I recently realized that all the nodes that are hidden (marked as invisible) would eventually be output as NaNs. But we would still like to have even an estimated position of each node that is invisible for our analysis. In the prediction assisted labeling, the hidden nodes are greyed out, and so is it possible to output that greyed out hidden nodes?

Thanks!

[roomrys]

Hi @chrisch12138,

It does not seem like there is an easy way to do this at the moment. Logging what needs to happen to make this possible here:

When writing the analysis file, SLEAP populates the locations matrix with points from the instance here:

sleap/sleap/info/write_tracking_h5.py

Line 178 in 1a0404c

locations_matrix[frame_i, ..., track_i] = inst.numpy()

Instance.numpy() just calls Instance.points_array

sleap/sleap/instance.py

Lines 833 to 843 in 1a0404c

	def numpy(self) -> np.ndarray:
	"""Return the instance node coordinates as a numpy array.
	Alias for `points_array`.
	Returns:
	Array of shape `(n_nodes, 2)` of dtype `float32` containing the coordinates
	of the instance's nodes. Missing/not visible nodes will be replaced with
	`NaN`.
	"""
	return self.points_array

And Instance.points_array calls Instance.get_points_array() with invisible_as_nan=True

sleap/sleap/instance.py

Lines 821 to 831 in 1a0404c

	@property
	def points_array(self) -> np.ndarray:
	"""Return array of x and y coordinates for visible points.
	Row in array corresponds to order of points in skeleton. Invisible points will
	be denoted by NaNs.
	Returns:
	A numpy array of of shape `(n_nodes, 2)` point coordinates.
	"""
	return self.get_points_array(invisible_as_nan=True)

So, in order to give users the option to export the analysis file with either nan or approximate locations for invisible nodes, we would need to

1. call self.get_points_array(invisible_as_nan=invisible_as_nan) in Instance.numpy where invisible_as_nan is a keyword argument passed into Instance.numpy
2. call Instance.numpy(invisible_as_nan=invisible_as_nan) from get_occupancy_and_points_matrix where invisible_as_nan is a keyword argument passed into get_occupancy_and_points_matrix
3. call get_occupancy_and_points_matrix(..., invisible_as_nan=invisible_as_nan) from sleap.info.write_tracking_h5.main where invisible_as_nan is a keyword argument passed into sleap.info.write_tracking_h5.main
4. Add invisible_as_nan as an argument for the sleap-convert CLI in sleap.io.convert.create_parser
5. Pass in args.invisibale_as_nan into the write_analysis function within sleap.io.convert.main

Thanks,
Liezl

[roomrys]

https://github.com/talmolab/sleap/compare/liezl/export_invisible_coor_analysis

[talmo]

Hey guys,

Just jumping in here since I think there's a more straightforward answer than what @roomrys had in mind -- I think the goal here is just to get the predicted points back that are currently being marked as not visible, correct?

Landmarks that are detected by the models are replaced with NaNs when the confidence is below a certain threshold (by default 0.2). This drastically reduces the amount of false positive and spurious predictions, but if you want to get those points back anyway, you can do so by reducing the confidence threshold.

I'm not sure if we have this exposed in the GUI yet (@roomrys?) but you can do it via the sleap-track CLI using the --peak_threshold flag. For example:

sleap-track -m path/to/model --peak_threshold 0.01 --only-suggested-frames -o predictions.slp labels.v001.slp

You can then import predictions.slp and merge it into your project from the File menu in the GUI.

Let us know if that works for you!

Talmo

[chrisch12138]

Thank you guuys for the timely response! I'll make sure to try that!

[chrisch12138]

Actually one more question,, when I manually label the instances and hide some nodes because they are blocked (either by other animals or themselves), will the nodes I marked as invisible be learned from the machine? --(Just wondering if it would be helpful for me to accurately mark the position of the invisible nodes).. Thanks!

[roomrys]

Hi @chrisch12138,

The nodes that are marked as invisible will not be learned from the machine, so no need to accurately label them unless you want to use those in your analysis.

Exporting the analysis file will prefer user labels as opposed to predictions, so if you would prefer to use the predicted nans, then you will want to remove user labels from the final copy of your project before exporting.

Thanks,
LIezl

[chrisch12138]

If this is the case, would you guys suggest I unhide the invisible nodes so that the machine can potentially learn the complete estimated poses for the purpose our analysis? Or is it just gonna mess up with the training for actually visible nodes?

[roomrys]

Hi @chrisch12138,

If you have very little idea of where the node is, then do not label it. If you are pretty sure you know where the node is, then it's fairly safe to label it.

Labeling occluded nodes will add some variability in the expected visual appearance of the node so it is more likely that you will get predictions on pixels that do not look like the average node.

Thanks,
Liezl