slow inference speed

[akshitasax]

Hi,

Firstly, thank you for all the help in the past!

Currently, I want to track 2 flies over 12 hour videos using the top-down model pipeline. However, the inference time is close to ~5 hours for one 12 hour video (res 640x480, roughly 646,000 frames) and the prediction rate is about 30 FPS. I remember seeing higher speeds in the SLEAP Nature paper. Our GPU model is NVIDIA RTX A4000. What do you think I can change or do to improve inference speeds?

I reviewed previous discussion threads about similar issues (#1099 #1126 ) - but the suggested settings for max stride, output stride, and filters are already being used in my model configuration. I am also not using KF. I'm attaching screenshots of the configuration below.

Could it be an issue with my GPU? I am not entirely sure how to diagnose this, but I'm attaching a couple of screenshots from the CLI that I thought might help.

Thanks,
Akshita

[talmo]

Hi @akshitasax,

One more optimization you could try is to set the filters rate to 1.5 on the centered instance model to get a solid reduction in parameter count on that model, but there's probably a couple more sources of performance hits:

1. Tracking. The tracker speed at the moment is not ideal. Try running inference without tracking and see how you do in terms of speed.
2. I/O. Reading the video from disk and decoding it can be a significant bottleneck depending on where you're reading it from and the nature of the video compression. What's the video resolution and can you do an ffprobe video.mp4 and paste the output?

If (1) accounts for most of the slowdown, there might be some optimizations we can do by running the tracking separately -- just let us know and we can work with you on it.

If (2) accounts for most of the slowdown, then there's not much to be done if the data is just slow to read, but if it's an issue with video decoding performance, we're currently working on trying a new backend for the video reading which should hopefully be a lot faster.

Let us know what the initial investigation turns up and we can keep troubleshooting from there.

Cheers,

Talmo

[akshitasax]

Hi @talmo

I tried reducing filter rate on the centered instance model to 1.5 when training my model, then ran inference. the speed was very similar to what I've been getting (30 FPS).

1. Tracking: I tried running inference with the "tracker method" set to none, instead of simple. This made inference run slightly faster, at ~ 40 FPS.
2. I/O: The video resolution is 640 x 480. And here's the output for ffprobe

C:\Users\DonleaLab> ffprobe C:\Users\DonleaLab\Downloads\SLEAP_videos\exp1_tim_CS\2022_11_08_03_00_02.mp4
ffprobe version N-103134-gaccfd015e3-g65fdc0e589+2 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 10.3.0 (Rev5, Built by MSYS2 project)
  configuration:  --pkg-config=pkgconf --cc='ccache gcc' --cxx='ccache g++' --disable-autodetect --enable-amf --enable-bzlib --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --enable-sdl2 --enable-ffnvcodec --enable-nvdec --enable-cuda-llvm --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libdav1d --enable-libaom --disable-debug --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libopenmpt --enable-version3 --enable-libsrt --enable-libgsm --enable-libvmaf --enable-libsvtav1 --enable-librtmp --enable-mbedtls --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++ --extra-cflags=-DLIBXML_STATIC --extra-libs=-liconv --disable-w32threads
  libavutil      57.  3.100 / 57.  3.100
  libavcodec     59.  4.100 / 59.  4.100
  libavformat    59.  4.101 / 59.  4.101
  libavdevice    59.  0.100 / 59.  0.100
  libavfilter     8.  1.103 /  8.  1.103
  libswscale      6.  0.100 /  6.  0.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C:\Users\DonleaLab\Downloads\SLEAP_videos\exp1_tim_CS\2022_11_08_03_00_02.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2022-11-08T22:30:04.000000Z
  Duration: 11:30:07.93, start: 0.000000, bitrate: 654 kb/s
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480, 654 kb/s, 15 fps, 15 tbr, 15k tbn (default)
    Metadata:
      creation_time   : 2022-11-08T22:30:04.000000Z
      handler_name    : h264:[email protected]: 0.5.2-426-gc5ad4e4+dfsg5-5
      vendor_id       : `[0][0][0][0]`

Best,
Akshita

[roomrys]

Hi @akshitasax,

Apologies for the delay! We'd still like to troubleshoot this performance on your data.

Would you be able to send the SLEAP model folders (centroid+centered instance) and the video file? You can email a Google Drive/Dropbox/etc link to [email protected].

We can try to profile the individual steps within the inference pipeline to see if we can pinpoint where this slowdown is coming from.

Thanks!