How to use Anipose for 3D tracking after 2D tracking with sleap

[frenchnat]

Hi,

I am currently doing pose estimation tracking with sleap on 2 camera views and I would like to do 3D tracking with Anipose but their tutorial is based on 2D tracking with DLC. Is there a tutorial for using Anipose with sleap out there?

Thanks a lot!

[talmo]

Hi there @frenchnat,

We've built an integration with Anipose here: https://github.com/talmolab/sleap-anipose

This is still a WIP, but the README has some information about how to get it set up.

[frenchnat]

Hi @talmo,

Thanks a lot for this. I used a checkerboard to present to the cameras for calibration but I can see in the documentation that sleap only supports charuco board calibration, does it mean I can't do 3D tracking based on my 2D tracking with sleap then?

[talmo]

Hi @frenchnat,

If you already calibrated then you should be able to just provide that .toml file as input and it'll work. You can calibrate using the regular Anipose app if not through sleap-anipose.

Cheers,

Talmo

[frenchnat]

Alright, I'll try! Thank you!

[frenchnat]

Sorry it took me a while to get there but it looks to me that I need to provide aruco parameters if I want to generate a calibration board parameters file through sleap-anipose to do the calibration (while the calibration board used during the experiment was a checkerboard) and, on the other hand, if I want to do the calibration through the regular Anipose it looks like I need the config.toml that is usually generated by DeepLabCut but I used sleap to do the 2D tracking and I don't see any generated config.tml file. Is there something I am missing? Many thanks in advance!

[roomrys]

Hi @frenchnat,

I looked through the sleap-anipose code and it seems they only implemented CharucoBoard calibration as of right now. I opened a pull request to support Checkerboard calibration as well. I will update here when it is merged in after it has been reviewed.

If you would rather not wait, you can use Anipose for calibration - it does not matter which software you used to annotate. In fact, calibration is usually done before starting the annotation process using just raw videos (wherein the calibration board was filmed). The Aniposelib docs on their config.yaml for calibration are a bit much in my opinion. I would personally prefer to follow along the aniposelib calibration tutorial and use a script such as the following to perform calibration:

from aniposelib.boards import Checkerboard
from aniposelib.cameras import CameraGroup

# TODO: Replace with paths to your videos
vidnames = [['calib-charuco-camA-compressed.MOV'],
            ['calib-charuco-camB-compressed.MOV'],
            ['calib-charuco-camC-compressed.MOV']]

# TODO: Replace with names you would like to use for your cameras
cam_names = ['A', 'B', 'C']

n_cams = len(vidnames)

# TODO: Fill out the parameters specific to your checkerboard
board = Checkerboard(
                     squaresX=7, 
                     squaresY=10,
                     square_length=25, # here, in mm but any unit works
                     marker_length=18.75,
                     marker_bits=4, dict_size=50)


# TODO: Change the fisheye=False if your cameras are not fisheye
# the videos provided are fisheye, so we need the fisheye option
cgroup = CameraGroup.from_names(cam_names, fisheye=True)

# this will take about 15 minutes (mostly due to detection)
# it will detect the charuco board in the videos,
# then calibrate the cameras based on the detections, using iterative bundle adjustment
cgroup.calibrate_videos(vidnames, board)

# if you need to save and load
# example saving and loading for later
cgroup.dump('calibration.toml')

Let us know if you need any help.

Thanks,
Liezl