Trust absolute timestamps in linux audio/video sources

[viric]

OBS is structured towards synchronising audio/video based on relative timestamps, instead of absolute timestamps.
For example, linux-v4l2 changes all frame timestamps relative to first frame:

obs-studio/plugins/linux-v4l2/v4l2-input.c

Lines 267 to 270 in 1451554

	out.timestamp = timeval2ns(buf.timestamp);
	if (!frames)
	first_ts = out.timestamp;
	out.timestamp -= first_ts;

Also, linux-pulseaudio ignores the timestamp set in pulseaudio frames, and picks the cpu timestamp at frame reception time, with a hardcoded offset:

obs-studio/plugins/linux-pulseaudio/pulse-input.c

Lines 212 to 215 in 1451554

	out.timestamp = get_sample_time(out.frames, out.samples_per_sec);
	if (!data->first_ts)
	data->first_ts = out.timestamp + STARTUP_TIMEOUT_NS;

The sequential startup of drivers makes all relative timestamps to be offset between them. For example I have to delay about 200ms my audio, to match video. Then, due to timestamps at reception time, OBS stills needs to figure out how to synchronise all them without common clock (

obs-studio/libobs/obs-source.c

Lines 1556 to 1571 in 1451554

	/* smooth audio if within threshold */
	if (diff > MAX_TS_VAR && !using_direct_ts)
	handle_ts_jump(source, source->next_audio_ts_min,
	in.timestamp, diff, os_time);
	else if (diff < TS_SMOOTHING_THRESHOLD) {
	if (source->async_unbuffered && source->async_decoupled)
	source->timing_adjust = os_time - in.timestamp;
	in.timestamp = source->next_audio_ts_min;
	} else {
	blog(LOG_DEBUG,
	"Audio timestamp for '%s' exceeded TS_SMOOTHING_THRESHOLD, diff=%" PRIu64
	" ns, expected %" PRIu64 ", input %" PRIu64,
	source->context.name, diff,
	source->next_audio_ts_min, in.timestamp);
	}
	}

etc). That's a hard problem and we often see how streams get AV-desync because of eventual busy CPU.

But the fact is that the both v4l2 and pulseaudio timestamp audio and video frames with good absolute CPU timestamps set at interrupt time, that could be perfectly used for proper av sync without any effort

I think that it would be nice if the obs_source_frame allowed a flag that said "timestamp is in absolute time" (in os_gettime_ns() clock), so drivers can provide timestamps in absolute time. That would solve many AV-desync problems and it's trivial to implement I think. At least for v4l2+pulseaudio.

Would any of that be welcome?

[kkartaltepe]

Sure but you would need to verify it is impossible to send invalid timestamps, as such all timestamps must not ever originate in devices (or untrusted code) in any case. I don't think this is actually true and that v4l2 and pulseaudio probably have options to trust devices. If that is the case then we must continue not to trust timestamps which may be garbage but just so happen not to be garbage for you.

[viric]

What do you mean about untrusted code? In v4l2 and pulseaudio I think you can ask in what clock the timestamps are. You would have to make sure it's in the same clock as the os_gettime_ns(), which is the usual monotone clock afaik. That's what the linux drivers will usually provide, additional to more timestamps if required.
Some webcams, for example a Creative webcam I have, has an internal clock. And the driver can be asked to figure out the offset between the two clocks, so you can have cpu-monotone-clock timestamps at actual exposure moment. Not all webcams/drivers support that though, or by default.
In case the timestamp comes from the interrupt code, instead of the exposure moment, that's nevertheless a lot more stable than anything you can get otherwise.
I don't see much difference into trusting the absolute timestamps or the relative timestamps. They are taken either with the same clock or not.

[kkartaltepe]

What do you mean about untrusted code?

Literally any code you cannot trust. I think we can trust that if we see kernel code adding timestamps via ktime its right. If you expect us to trust random device firmware or userspace programs its not going to happen.

I think you can ask in what clock the timestamps are.

I think this is definitely required if timestamps other than those from the v4l2 or pulseaudio system code are used. But its not sufficient. We need to know those clock source reports can be trusted. The second any poorly authored or cleverly authored code can report the wrong clock source it just introduces a new way for desync to enter, and we should just stick with the existing code.

[viric]

The only timestamps we can trust have to be in kernel monotonic clock, sure. Any other clock would require synchronisation.
The webcam driver can, if the user desires, use the webcam clock but it will care about translating it to kernel monotonic clock. That's up to driver parameters, not the application.

[viric]

Hm maybe it's not so easy to get the monotonic kernel timestamps from the pulseaudio driver. The alsa driver provides them, but not pulseaudio.

I'm checking in my system with linux-v4l2 and linux-pulseaudio, and it seems that for v4l2 I have a timestamp-to-systemtime of 32ms, and for pulseaudio a latency between 10 and 30ms.

I'm struggling to understand why I need to set a 200ms sync_offset to the audio stream, to get more or less synchronous AV in obs.

[viric]

It seems I had enabled "Low latency audio" in settings; disabling this, pulseaudio recording latency went down from the ~10-30ms to quite steady 9ms.