Give developers control over "overlay" browser

[cabanier]

Meta Quest recently introduced support to show the 2d browser while staying in VR/AR.
This mode can only entered/left by the user using system gestures.

We've gotten multiple requests since launching this features to allow for developer control over this. Basically, developers want to trigger overlay by calling an API so they can show a configuration or shopping screen. Then they want to present the user with a button to resume the WebXR session.

/facetoface give developers control over "overlay" browser

[KooIaIa]

On other platforms like PC it is common to be able to open a web browser overlay inside of an XR application without freezing it. Instead of having this new developer control halt the WebXR session - could you also have an option to let the overlay browser be accessible inside the WebXR session? I think developers would love being able to access the overlay browser from inside a WebXR session and control it.

[cabanier]

Instead of having this new developer control halt the WebXR session - could you also have an option to let the overlay browser be accessible inside the WebXR session? I think developers would love being able to access the overlay browser from inside a WebXR session and control it.

That's what this proposal is

[Utopiah]

Would it be a single overlay or multiple overlays? Would the overlay (or overlays) be fixed or positional?

I'd prefer multiple positional overlays.

[cabanier]

It's just one and it will always be centered. Basically what you have today except you can trigger it as an author

[hybridherbst]

It could also be interesting to have drag-and-drop functionality between that browser window and the XR session (and back?).

[josephrocca]

There's the potential for something really powerful here - more than just a userland API for the current system function.

It'd be very useful to be able to open multiple browser windows while still being active in the current session.

I think the browser window(s) would have to locally (i.e. in spatially-aware manner) "capture" user interaction from the parent WebXR session - kinda like an iframe does for mouse/keyboard/etc. events in HTML (or, more analogously, OS windows, I guess).

So e.g. could seamlessly tap pause button on YouTube window while remaining in game/social world, or e.g. copy text back and forth between the WebXR world and a Discord browser tab. And if it's just another browser tab then you could stream the browser window/tab to your friends in social VR using the web's Media Capture API. This sort of paradigm feels like it leans into the web's strengths.

Perhaps too architecturally ambitious for now, but I hope the spec that's produced here can leave some possibilities open for future proposals. It's a very exciting direction!

[KooIaIa]

It's just one and it will always be centered. Basically what you have today except you can trigger it as an author

Today, a VR browser user is able to size, position, and place a browser html window wherever they want (almost). Imagine opening a website in AR passthrough that opens WebXR. Could the page keep its original user set position and then just be visible in WebXR? Picture clicking the Enter XR button and the webpage never disappears and cleanly transitions into WebXR. The webpage disappearing when webXR starts might be the opposite of the ideal default behavior.

There's the potential for something really powerful here - more than just a userland API for the current system function.

The powerful feature I am most excited for is detatched 3D CSS. Once a HTML page can be loaded in VR like this, the next step could be adding VR support to the web's existing 3D functionality to make it immersive XR 3D. Then standard HTML and CSS wil work in VR and allow placing as many 3D overlays as a page could want and work in 2D too.

[coderofsalvation]

Sounds promising.
However, from a user and developer point of view I'd like the author to be able to specify features/permissions:

• enable URL-bar
• enable bookmarks
• enable back/forward buttons
• enable text-select/copy/paste
• content permissions (piggyback iframe permission policy attributes)
• unrestricted mode: just overlay the browserwindows from homescreen (worstcase enabled via browser flag)

Reason: the web requires knowledge-navigation: even triggering an payment-link will in most cases require the user to look up info elsewhere (having to exit immersive mode to open a new tab would defeat the purpose of the overlay imho).

Unrestricted mode would resonate with doing serious work in (Web)XR.

[josephrocca]

Potentially relevant viral (1.7k likes) tweet in the VRChat community from yesterday:

DXOverlay, Show your friends when you have the Overlay open. Integration with XSOverlay directly into VRChat via OSC. With easy customization of colors and screens.

I'm guessing this feature could be in MVP/v1.0 of this spec - just requires exposing position, orientation, and size of each overlay browser/panel, which is kind of a bare minimum anyway if this proposal goes beyond a simple "toggleOverlay()" at all.

Note that the above image shows sub-windows within the panels - not sure if they're cosmetic/fake, but for WebXR this would not be relevant/appropriate (can use web's Media Capture API as mentioned earlier if dev wants to allow users to share their overlays).

[josephrocca]

A security consideration: If the user's hands are near the keyboard that's associated with an overlay, hand/finger tracking information should not be exposed to WebXR, since this could leak passwords and other info. Might also need to consider lowering eye and arm tracking resolution - in the extreme case (probably not necessary), the user's body (or all except head/centroid?) could completely "freeze" from WebXR's perspective while they're interacting with an overlay.

As mentioned earlier, this is akin to the browser iframe (or OS window) capturing mouse/keyboard events - usually based on the spatial position of the input device (though there is also the concept of frame "focus" in browser/OS land - not sure if that's relevant here - maybe gaze factors into that).

Seems like it'd be enough to freeze hand joint positions when they're near overlay or keyboard, and lower the resolution of some other stuff based on conservative heuristics (e.g. gaze, when looking at an overlay, and wrist joint from body tracking when near overlay).