Add LCEVC support to improve video quality and lower bitrates

[CeeJayDK]

Select Topic Area

Feature Request

Body

LCEVC (Low Complexity Enhancement Video Coding) is not a regular video codec, rather it's an enhancement layer that can augment an existing video stream and it can do so with very small power and performance requirements.

Here is an explanation of how it works : https://www.lcevc.org/how-lcevc-works/

I believe it could add an excellent way of enhancing the video output and lowering the bandwidth requirements of a Moonlight + Sunshine session.
It also adds a way to increase the quality over what the hardware encoder of your video card is capable of producing since it can be added to H.264, HEVC and AV1 alike. Practically any encoder could work as the base encoder.

It will require implementation on both the client side and the server side though so this would also require implementation in Moonlight.

[ReenigneArcher]

This just sounds like longer encoding times, and more latency.

[CeeJayDK]

This just sounds like longer encoding times, and more latency.

That is the exact opposite takeaway that I got.

I'll try to explain it better :

With LCEVC you take the original video input and downsample it to half height and width and thus 1/4 of the original number of pixels.
This quarter size is sent to the original encoder (for example it could be HEVC) which will then encode it in 1/4 of the time because it had to do 1/4 of the work. So 4x faster.

While it's doing this (typically on the GPU hardware) the original sized video is being used by LCEVC to create an enhancement stream. LCEVC was created to be fast and efficient so it completes this faster than it takes the original encoder to encode the quarter sized stream.

Then the two streams are muxed together and sent.

The performance numbers I have seen say the combined stream is up to 40% smaller then the original encoders full res stream, while providing up to 18% higher quality and being 3x to 4x faster to encode.

On the decoding side, the combined stream is demuxed, and the quarter sized video is sent to the original decoder while the enhancement stream is sent to the LCEVC decoder. They decode both simultaneously (just like with encoding) and once both are done, the results are combined into a high res video and displayed.

The decode side should also finish faster because the original decoder has 1/4 of the work to do and because LCEVC is very fast and the two decoders can work simultaneously.

This all spells faster encoding, decoding and transmit times, so it should result in LESS latency.

[ReenigneArcher]

We have to encode and create the video first, it doesn't exist. So this is an extra step, hence slower and more latency.

Probably could be good for a video streaming server app, but I don't think it helps with game streaming, unless I'm missing something.

[CeeJayDK]

Of course you have to encode the video stream, but with this you have a 4 times reduction in workload with a little bit of overhead for the downsampling and muxing which is why it "only"ends up being between 3 to 4 times faster.

Say the stream is 4K.

Which is faster - encoding a 4K stream or encoding a 1080P stream?
The 1080P stream is faster which is why this ends up being faster.

[CeeJayDK]

Here are some graphs from the study V-NOVA did : https://docs.v-nova.com/v-nova/lcevc/lcevc-best-practices that show what kind of performance you can expect when combining LCEVC with other encoding techs.

The takeaway I got from this is that LCEVC helps with lowering the bitrate the most on the oldest codecs, and the least on the newest (AV1, VVC, EVC) but it does help all of them, and likewise it helps lowering complexity and thus increasing performance with the biggest improvements on the most complex codecs which are also the newest (AV1, VVC, EVC) and less so on the oldest codecs, but it does help all of them.

Their documentation site also contain a lot more info on LCEVC if you want to know more.