Support for stereoscopic video using the MPEG_video_texture glTF extension

[nlsdvl]

Context

Our goal is to implement stereoscopic video texture for content playback in a virtual movie theater scene. MV-HEVC could be used for encoding such video content, but note that the problem applies for any stereoscopic content.

Problem description

Stereoscopy achieves the illusion of depth by providing a pair of images each tailored either for Left or Right eye. When rendering stereoscopic content in an HMD, the presentation engine uses the appropriate view from the stereoscopic video for each eye.

I couldn’t find any way, neither in the base glTF specification, nor using the MPEG-I SD extensions, to achieve signaling of stereoscopic media that would allow implementing our scenario.

The presentation engine needs to know which image/texture/buffer should be used when rendering either for Left or Right eye.

Suggested solution

Extend MPEG_video_texture to allow the presentation engine to render only the appropriate images for each eye.

One option would be to use two textures that differ only by their accessor, with an extra property signaling which eye it is intended for.

Optionally, specifying a default a.k.a. ‘hero’ eye could be useful when not rendering the scene on an HMD.

Additional context

cc @ibouazizi - Do you have other solutions to suggest ? Did I miss something in the latest developments of MPEG extensions to glTF to support this use case ?

[nlsdvl]

The issue has been duplicated to MPEGGroup's Scene Description repository: MPEGGroup/Scene-Description#17