Deleting human labeled features by location in space.

[catubc]

Hello

We're trying to figure out a programatic way to edit a human-labelled .slp track.

In particular we'd like to delete all human labeled features in the bottom left hand of these videos as they represent huddles of gerbils together and we've decided to completely remove/occlude this part of the video during training + prediction.

https://drive.google.com/file/d/1FzXH6Vi-Pg_jOUHVOAGTzb_8WNMgKQ3o/view?usp=share_link

We have an .slp file here with 1000 human labeled frames. And the goal would be to be able to edit the .slp file programatically and delete any feature within N pixels of location (x,y).

https://drive.google.com/file/d/1folo1zAyePzL84GVSJzr_ipnlCgneDCv/view?usp=share_link

Can someone help / show us how to do this? I can do it easily after converting .slp->h5->.npy file, but I don't know how to pack the arrays back into the .slp file.

Thanks so much,
catubc

[roomrys]

Hi @catubc,

Update: Just use the API as suggested below. The below suggestion is too complicated.

You do not need to convert back to a .slp as we have a feature to import SLEAP analysis files (caveat being that you will need to redefine the edges of the skeleton).
Since you already have a method for editing the labels using numpy, you can load and edit the "tracks" dataset (as you likely are already doing) and then overwrite that dataset in the existing hdf5 file. Alternatively, you can rewrite the entire analysis file using our sleap.info.write_tracking_h5.write_occupancy_file function (given that you have pulled ALL the datasets from the original h5py).
This is the function SLEAP uses to create the analysis file (sleap.info.write_tracking_h5.write_occupancy_file):

sleap/sleap/info/write_tracking_h5.py

Lines 242 to 284 in 72a4ae2

	def write_occupancy_file(
	output_path: str, data_dict: Dict[str, Any], transpose: bool = True
	):
	"""Write HDF5 file with data from given dictionary.
	Args:
	output_path: Path of HDF5 file.
	data_dict: Dictionary with data to save. Keys are dataset names,
	values are the data.
	transpose: If True, then any ndarray in data dictionary will be
	transposed before saving. This is useful for writing files
	that will be imported into MATLAB, which expects data in
	column-major format.
	Returns:
	None
	"""
	with h5.File(output_path, "w") as f:
	for key, val in data_dict.items():
	if isinstance(val, np.ndarray):
	print(f"{key}: {val.shape}")
	if transpose:
	# Transpose since MATLAB expects column-major
	f.create_dataset(
	key,
	data=np.transpose(val),
	compression="gzip",
	compression_opts=9,
	)
	else:
	f.create_dataset(
	key, data=val, compression="gzip", compression_opts=9
	)
	else:
	if isinstance(val, (str, int, type(None))):
	print(f"{key}: {val}")
	else:
	print(f"{key}: {len(val)}")
	f.create_dataset(key, data=val)
	print(f"Saved as {output_path}")

And these are the datasets (keys of data_dict) that are written to the analysis file:

sleap/sleap/info/write_tracking_h5.py

Lines 352 to 366 in 72a4ae2

	data_dict = dict(
	track_names=track_names,
	node_names=get_nodes_as_np_strings(labels),
	edge_names=get_edges_as_np_strings(labels),
	edge_inds=labels.skeletons[0].edge_inds,
	tracks=locations_matrix,
	track_occupancy=occupancy_matrix,
	point_scores=point_scores,
	instance_scores=instance_scores,
	tracking_scores=tracking_scores,
	labels_path=str(labels_path), # NoneType cannot be written to hdf5.
	video_path=video.backend.filename,
	video_ind=labels.videos.index(video),
	provenance=json.dumps(labels.provenance), # dict cannot be written to hdf5.
	)

Let me know if this was clear and/or if you'd prefer an example.

Thanks,
Liezl

[roomrys]

I realize that the above answer might get a bit hairy when dealing with a project with lots of videos (and therefore lots of analysis files) - if this is the case, let me know, and I can give an example for editing the .slp directly using our API.

[catubc]

Hi @roomrys, thanks for the comments.

So you're suggesting that we:

1. convert our .slp file -> .npy.
2. edit the .npy file as required.
3. resave the .npy file as an .h5 file using this wrapper above.

It is not clear to us what all these variables are? I think a working example would help via cli or api otherwise.

Maybe you can also comment on why tracks in .slp files can't be edited directly? There are at least 15-30 videos per training datasets of 1000 labeled frames, so not sure manually tracking all these details individually in each video (e.g. which frame in which video is being labeled) is that easy.

Otherwise, here's some of our questions:

track_names=track_names      # Is this the animal_name?  eg. "mouse1", "mouse2"

node_names=get_nodes_as_np_strings(labels),   # is this "nose", "tail1" etc.

edge_names=get_edges_as_np_strings(labels),  # unclear 

edge_inds=labels.skeletons[0].edge_inds,   # unclear

tracks=locations_matrix,  # unclear

track_occupancy=occupancy_matrix,   # unclear

point_scores=point_scores,   # is this even something we save for human labeled data?

instance_scores=instance_scores,   #  Also not clear

tracking_scores=tracking_scores,   # are these all 1s? 

labels_path=str(labels_path),  # NoneType cannot be written to hdf5. #  what is this the path to?  The videos?

video_path=video.backend.filename, # unclear also

video_ind=labels.videos.index(video),   # is this some video order of some sort?

provenance=json.dumps(labels.provenance),  # dict cannot be written to hdf5.

Thanks so much for the help.
catubc

[roomrys]

Hi @catubc,

This was a bad suggestion - too many moving parts. Using the API as suggested below is much simpler.

Thanks,
Liezl

[roomrys]

Hi @catubc,

Through the SLEAP API, you can access the points of instances in a certain area and remove instances meeting a certain criteria:

import sleap

ds = "path/to/slp"
labels = sleap.load_file(ds)

labels.videos  # Use in ipython to print list of videos in the labels object
video_idx = 0  # Change this select the video of interest from the labels.video list
video = labels.videos[video_idx]
labeled_frames_from_video = labels.get(video)

# Loop through all labeled frames (from a specific video)
for lf in labeled_frames_from_video:
    # Loop through all user instances in the current labeled frame
    for inst in lf.user_instances:
        remove_inst = False
        points_array = inst.points_array  # Returns a (num_nodes, 2) np.recarray

        # Add some logic here to determine whether to remove instance using points_array

        if remove_inst:
            lf.instances.remove(inst)

# Save the modified slp
new_filename = "path/to/save/modified/slp"  # Use a different path from the current slp to be safe
labels.save_file(labels, new_filename)

This is somewhat undesirable because of the nested for loop, but should do the trick. Let me know if you have any trouble.

Thanks,
Liezl

[catubc]

Thank you so much Liezl, this worked. See attached human labeled feature locations with the removed offending region :)