single frame inference slow

[RubenTeunisse]

Hi!
I want to use sleap for a real-time application, but predicting a single frame is taking about 220ms.
The model and the data are already loaded, and timing is done right around the predict call:

        start = time.time()
        labeled_frame = self.sleap_model.predict(input)
        print("sleap prediction time: ", time.time() - start)

Any ideas for how I could speed it up?

Thanks!
Ruben

---

Details:
12th Gen Intel(R) Core(TM) i9-12900K
128GB RAM
GeForce RTX 3090

bottom up (~same time for top-down)
Unet
max stride 32
16 filters
input shape = (1, 512, 640, 1)
2 instances of six nodes per frame

[talmo]

Hi @RubenTeunisse,

Can you try self.sleap_model.inference_model.predict_on_batch(img)?

This lower level API doesn't do some preprocessing steps, but this might not be necessary for your data anyway. After the first call(s), the model will get traced and inference should be significantly faster as well.

We have some info on this topic in the Interactive and realtime inference notebook. This Discussion thread might also be relevant.

Let us know if any of that works for you!

Cheers,

Talmo

[RubenTeunisse]

Hi Talmo,

Thanks, just that one line of code already reduced computation time to about 30ms!

Best,
Ruben