Purpose of buffered video source? Guaranteed video-audio delay, may be very big.

[viric]

The buffered video source uses a queue of frames, and then the source picks the best next frame, based on their timestamps. But that makes the offset between input timestamps and output timestamps grow in case that the video source temporarily stops having frames ready (like CPU too busy, video acq lags a bit).

I have a situation where the camera returns between 30 and 60fps, and my output is at 30fps.

Then imagine that there is a lag in receiving frames (like the CPU is too busy). There will be a while without new frames. But nevertheless the last_frame_ts advances sys_offset (1/30 of second at 30fps). Here it falls into this case:

obs-studio/libobs/obs-source.c

Line 4299 in 1451554

source->last_frame_ts += sys_offset;

finishing the function with no frame:

obs-studio/libobs/obs-source.c

Line 4346 in 1451554

blog(LOG_DEBUG, "no frame!");

Despite we end with no-frame, last_frame_ts advanced a lot. With usual frames, I get a 20ms frame_offset, but when there are no frames because of temporary CPU lag, the last_frame_ts advances 33ms!

So when CPU becomes free again, the video source has last_ts_frame maybe one second into the future, making the video images go 1s behind audio (up to 2s or as much as the queue length), and it will never recover back to ilttle delay.

It's fair to say that, with buffered video source, video will be delayed by a random amount of time, always increasing every time some frame lags a bit behind, and it will never be resynched back.

This looks clearly wrong, and I don't see when anyone would desire this situation. So is the ready_async_frame algorithm wrong, or really people have just to use the video sources unbuffered, for recording live video?

One hard limit I thought was something like:

if (source->last_frame_ts > sys_time)
   source->last_frame_ts = sys_time;

That limits the delay to the acq latency at least, and that's too little to make any use of buffer. Maybe "= sys_time + 50ms"?
I don't know what would be desirable to improve this: try to keep the frame->timestamp within some boundaries? Use a smaller queue than 30 frames (that's 1s delay at 30fps input)?