Feature Proposal: Support XRI and WebXR by UPM packaging

[bnco-dev]

Issue

Many Unity toolkits and plugins exist for adding VR/AR functionality. Particularly for mixed-reality apps, vendor plugins like Meta's and MRTK contain essential functionality that cannot be easily replicated in another package. Ubiq's core features (messaging, avatars, distributed logging, VOIP, etc) run parallel to these other plugins. Ideally we want to exist alongside them, adding the kind of networking functionality one might need for social VR/AR.

Currently Ubiq's player prefab, custom interaction code and samples make this difficult, as they're not compatible with any other library out of the box. We've worked to increasingly remove reliance on them from the core features above, and have now reached the point where all core features (Messaging, Avatars, VOIP) work cross-platform across Windows, Mac, Linux, Android, WebGL and UWP without them. This is great, but still requires developers to do some wrangling to connect other libraries. This isn't much fun as it's not always clear to developers what steps to take to make this happen for different libraries. Even afterwards, it will break the samples, as they're based on Ubiq interactions.

We need to think about how to organise Ubiq for the greatest possible compatibility, hopefully without losing our current 'click-play-to-start' ethos!

Proposal

Ubiq will be separated out into a ubiq package with minimal samples (networking-only) and a ubiq-xri package with the current samples updated for XRI. WebXR will be made a lot more straightforward to use with ubiq-webxr, which will build on top of ubiq-xri so that applications can be built for both by swapping the player prefab out at runtime. Here's what the result would look like (dotted packages are external dependencies):

Avatars can be separated into their own packages too: ubiq-avatars-floating, ubiq-avatars-rocketbox, ubiq-avatars-readyplayerme etc. They'll depend on ubiq, and users could import as many or as few as they like. See the current ubiq-avatars-readyplayerme package for an example.

Discussion

The really good news is that most of the vendor packages seem to be cohering around Unity's XR Interaction Toolkit for interaction. For example, both MRTKv3 and Meta's new mixed reality tools for the Quest 3 build off that base. This makes it a good base for us to build from too. Separating out core Ubiq into another package has the benefit of making it easier to integrate Ubiq into other (non-XRI, even non-XR) projects too. This comes at the cost of managing an extra repo. We'd be interested in thoughts on this; would it be worth it to you? Might you use Ubiq in non-XR projects?

This should not require any server changes, but in the future it may make sense to separate the server into its own project and encourage use through NPM or a Docker image.

Thanks for reading. Any and all comments welcome.

[sebjf]

Thanks @bnco-dev!

I agree this needs to be done, here are some purely personal thoughts on how...

I am coming at this from two main considerations:

1. The nature of our team/employment means effort on Ubiq ebbs and flows over time, so difficulty of maintaining repos and samples is a real consideration because it takes time from new features. Already we are seeing how hard it is to maintain the few samples we have.

2. By volume, our main users are students. Student teams vary a lot in their development experience & capabilities. They are also under some of the greatest time pressure, typically having to learn all the tooling (Ubiq & Unity & XR Concepts & C#) and build something with it within a single term.

In my experience, even one additional step can derail junior developers, especially when it's not common/unexpected. (It's why I never use git submodules, for example.)

For Us

I like a lot how you have set up the project so when we release we get the Unity package in a branch. I would recommend leaning into this and minimising how much we split the repo.

Personally, doing this server refactor, I already regret splitting the extended browser samples into a separate repo - there was no choice but it's still a pain! I think we should avoid this as much as possible.

The way the repo is configured at the moment is that each root folder represents a 'platform'. Perhaps we can keep this pattern, and have multiple Unity projects for different SDKs, and have the automated build tools create branches for different UPMs from this one repo?

Originally, the recommended way for users to consume Ubiq was by cloning the repo into their project, so we needed to be careful with the file structure of the repo. Now we have the UPM support, I think we can be a little more flexible with the repo itself.

I do agree that some SDKs will need their own repo though, simply due to the size. Meta's SDK for one. However, if these can be included as UPM dependencies, can we avoid committing the package contents? And rely on Unity downloading them when its first opened? Maybe we can even use this pattern for our own code...

For Users

For projects like this, I personally prefer the "path of least resistance" approach: where the default way of doing everything just works, and does everything very junior people need, and if you go off the beaten path then you are expected to know what you're doing.

To that end, I think it is OK to have 'favoured' dependencies.

XRIT

For example, I would recommend instead of splitting ubiq and ubiq-xri into two different packages, what we have is that core components (i.e. anything in the Runtime folder) never reference XRIT, however the samples fully embrace XRIT.
In theory XRIT is a library like any other, and should be treated like any other, so this isn't 'pure'. However, for one, multiple users have asked us for XRIT for a year or two now, and importantly, XRIT is maintained by Unity itself, so in practice it is special. Finally, interaction is itself a special case, being the one feature that is effectively non-optional for a working app, making it very difficult for us not to 'take a position' on.

We can have similar discussions around other features, like Avatars. For example, I think in principle we've settled on having a small number of avatars that are compatible with RPM avatars to make the samples work out of the box, and recommend users use RPM. If we could avoid a dependency on RPM's package (for various reasons: size, licensing, etc), I would personally say its OK to include a Component that brings in RPM avatars, at least in the Samples, if not Runtime, too in the future.

The main thing is simply that XRIT doesn't preclude anything else. So long as nothing in Runtime depends on it, then non XR projects can still follow the same instructions, and just use the Components in Runtime along with the traditional UX API, while never interacting with XRIT, even if its cloned.

And regarding other projects: realistically, if using Unity, the only other platform that we haven't already considered is Android - if users are going to write Peers for native C# apps or microcontrollers, they won't use Unity. So, if our samples support Desktop out of the box, which they should continue to do so, we've already covered 98%, with just a slight gap for touch screens that needs to be filled.

Other SDKs

Things are different for, e.g. MRTK or WebXR though I recognise (and as you say, other feature packages, etc), for which we will need separate UPMs.

I will think and put some more comments about this, but it maybe is something to be solved through Editor tooling. We need to consider what to do if, for example, someone has both the Meta and MRTK packages pulled in. In that case there is going to have to be some user intervention since there's no way to decide which takes priority automatically.

[bnco-dev]

Thanks for your thoughts @sebjf. I agree wholeheartedly on points 1 and 2!

We can absolutely keep Ubiq as a monorepo and build separate upms with Github actions. I think in many ways that's the better option, as we keep all our issues and git history etc in one place. My only concern is that it's not clear which project is being modified in a given release. We already have this issue with our server. We should definitely have a think about how to manage this. No worries about the size of dependencies. In theory they are pulled in by the package manager and don't end up in version control, so they should not be a blocker to a monorepo.

I'm a bit concerned about the proposed solution of using XRI but only in the samples. We cannot do samples-only dependencies, so the Ubiq package would pull in XRI regardless of whether the user needed it. Given that, we may as well use it in Runtime, as it would already be a dependency. Perhaps we can just commit to Ubiq building on XRI. This approach also seems fine to me, and may be worth it for the sake of simplicity.

Having come across it in other projects I'm very strongly against doing package management through editor-tooling. For the manual approach like ReadyPlayerMe's, dependency-resolving code is being compiled along with the dependencies it is resolving. If a dependency is missing it won't compile. All the user sees is a lot of errors, and the only solution for the user is finding out which package is missing and importing it themselves. Let's stay well away from this! We could also edit the user's manifest.json, but that too feels error-prone and a bit of a kludge.

About components to bring in other packages, like ReadyPlayerMe: We'd need to make sure we don't reference the other packages anywhere, so Ubiq will compile. The thing is, those other packages will require connecting code (see the ubiq-avatars-readyplayerme package). We probably also want samples for them, to save the user having to connect them up themselves without guidance on how. We could cram both connecting code and the samples in a Ubiq sample. It'd just be very important that the user to import the other packages using those components FIRST, before importing the sample. Otherwise they'll get a bunch of errors and nothing will work until they delete it. I think this is opening a can of worms.

To me, separate packages imported through the Unity Package Manager are better. It's obvious what each should do. There are no extra actions required by the user, or strange pitfalls full of errors to fall into. And everything comes already connected up and ready to use. It should also be straightforward for our students - they import ubiq-xri, job done!

[sebjf]

Having come across it in other projects I'm very strongly against doing package management through editor-tooling.

Yes I remember 😄! Sorry that original point wasn't clear: I don't propose to do package management via custom Editor tools, for reasons already stated.
What I wonder though (not sure yet) is if there's scope to use Editor tooling to help manage the Samples.

This example is a bit contrived but makes the point: say a user has our basic Samples with the PlayerController in the scene. Then they bring in both the Meta and MRTK packages via UPM.

Now, we don't know which interaction system they want to use (by virtue of them bringing in two - perhaps their project is going to target both Quest and Hololens?). But perhaps the PlayerController Editor has some buttons that say "Switch to Meta" or "Switch to MRTK". Then they just press one and the scene/sample updates.

This would mean we didn't need to remake multiple samples per, e.g. interaction SDK. We just need one compatible Prefab to show how the PlayerController is built for that platform, specific to that package. Then, the user can manually update the 'primary' sample scene, or, have the Editor button to do it for them. May not work for types of feature..