Proposal: Upgrade schema to be more generic

[ljoonal]

I think that it'd be nice to be able to re-use the same schema across similar modding scenes & games. The current NMG mod manifest schema though has a a few specifics that make it not properly reusable.

Changes

One of them is that the top level key is "mods", where some loaders call the things "plugins" for example. This would change that key to just a very generic "items", since it also contains things such as the actual loaders already.

Secondly, changes the "plugin" flag to a more generic "loader" flag. The Category and tags can still remain, as I feel like those would be game specific in all cases anyways.

Thirdly, changes the neosVersionCompatibility to gameVersionCompatibility.

What spurred this change?: The influx of VRC modders & the creation of https://chilloutvr-bepinex-modders.github.io/

For

• Makes tooling re-usable, more maintainers, Neos tooling can benefit from CVR modding tooling and vice versa
• Generally the "mods" top level key is already misleading anyways since it includes non-mods

Against

• Will need to update existing tooling like mod updater/installer
  • Should be a really simple fix though, and can delay this untill the pre-existing tooling have been updated.
• Need to update the schema docs
  • I'd be happy to do that
• It won't be as cool of an elite Neos-only club kinda thing anymore.
  • ...Fair I guess? But it's CC0 licensed so like, don't think that's a goal

[EIA485]

just to state my thoughts here:
seems like a positive change to me, allows for some probono development with more developers

[zkxs]

I'm fine with this as well. I just don't like how it's a breaking change. So if we do it, we shouldn't do it in manifest.json. We should instead make a new manfest-v2.json and deprecate the v1 manifest.

[XDelta]

We may want to think of a more reusable method of changing schema versions, do we somehow indicate in a current schema that a new one exists, or possibly better that the current version is changing 'soon'. I don't really like the idea of it becoming a new file that much.

[XDelta]

Just thinking of a way such that software/tools can easily grab the 'latest schema' manifest or grab the schema version it knows how to process if the manfest.json isn't always going to the latest verison. Not that I expect that the schema will change often.

[XDelta]

Top level of mods being something more generic as it includes mods, libraries, and plugins makes sense. Dunno if there might be a better word to describe it than items.

I'd lean towards keeping the plugin flag for plugins but specifically for NeosModLoader an additional flag for 'modloader' or 'loader' can be added. (atm it has a tag for modloader atleast).

Thoughts for keeping plugin as a distant tag being it is an addition to a game platform that is supported in some capacity and can be directly loaded where as mods require something else to load it.

[zkxs]

We may want to think of a more reusable method of changing schema versions, do we somehow indicate in a current schema that a new one exists, or possibly better that the current version is changing 'soon'. I don't really like the idea of it becoming a new file that much.
Just thinking of a way such that software/tools can easily grab the 'latest schema' manifest or grab the schema version it knows how to process if the manfest.json isn't always going to the latest verison. Not that I expect that the schema will change often.

The New File

Having a whole separate file is pretty typical for a breaking change.

If we don't add a new manifest-v2.json

All software that's currently parsing the manifest must add support for the v2 schema before we switch over. The software must be able to detect if it's been served the v1 or v2 manifest and handle it accordingly. v1-only software will break when we make the switchover. v2-only software will start broken, and will only stop being broken once we make the switchover.

If we do add a new manifest-v2.json

We can add the new v2 manifest immediately. Software can switch over to the v2 schema whenever they're ready, and their implementations don't need to handle both v1 and v2. If there's a pressing need we can backport mod additions to the v1 schema, but such a need probably won't exist as we can afford to serve v1-only software old data (for a little bit, at least).

Outdated Indicator

Do we need a v1.1 schema that has an indicator that it's outdated and/or that a newer major version exists?

If we don't indicate it's outdated

We just stop updating v1 and focus on v2. It doesn't matter if users don't know, as they can't do anything about it. We just directly bother the devs of manifest-consuming software to update to v2. This will be easy, as every single one of those devs is in this thread already.

If we do indicate it's outdated

As this would be a v1.1 addition, no v1.0 software currently exists that can handle it. Devs aren't going to update their software from 1.0 to 1.1. They're going straight to 2.0. Therefore while this feature could be useful, it would only be useful in v2.0+.

If we link to the new schema

v1.1 software can't use the link, as v1.1 software can't parse the v2 manifest. v2 software doesn't need the link, as it just reads the v2 manifest directly. There's also no reason to link to a minor version bumped schema because it would be the same file.

[EIA485]

this seems like a good standard for future changes.
but for this change i dont think we should move to a manifest v2 because:
currently there are vary few apps that read the manifest, the fix for those apps will be vary simple for this change.
the effort of cross pollinating the old prs to manifestv2 would be significantly greater than making old apps work with the changed schema.

[zkxs]

currently there are vary few apps that read the manifest, the fix for those apps will be vary simple for this change.

True, but delta's mod manager is a desktop app, so it doesn't matter how easy of a code change it is. What matters is how easy rolling out the update to all users is.

the effort of cross pollinating the old prs to manifestv2 would be significantly greater than making old apps work with the changed schema.

We can release v2 whenever it's convenient, namely after we process the PR backlog.

[XDelta]

It is lenient on schema changes and many it should be able to sort out and still mostly work, the only one slated that would currently break it would be the top level changing from mods to something else. The only way currently anything knows that a schema change has happened is if the schema version number changes. Mainly up for debate on how to handle breaking changes.

[ljoonal]

Okay, I'll update my proposal then based on the feedback: Let us (in addition to bumping the schema version to 2.0.0) start a v2 branch, that will become the default one. If the mod updaters are setup correctly, they should be referencing the master branch currently.

Before it comes the default one, I would like to look into making managing the manifest easier by splitting it up into per-author files, and using actions to generate the final manifest, even possibly a V1 compatible version as well, or even XML or CSV. That way we could keep compatibility. Though, I think it would require a separate repository. Maybe we could make a api NMG repo that is the fully generated one and have that be what any V2 consumers would use? Issue with that too is that we would be doing double maintenance if we wanted to keep the v1 here functional. I don't think that'd be worth it. I think that if we're going to make such a change, it would require coordination so that there's no new PRs for the old manifest format once the new one would be in place. Additionally the API repo could include pre-fetched and hash validated DLL files so that we can just directly link to them on the website for downloads for example, without having to fear that their contents are changed to something malicious later on.

[XDelta]

I don't see too much need to continue to update the v1 after we have a v2 sorted out and more stable in changes, more just allowing it to continue to exist for history/archival and so all tooling doesn't immediately break or have issue when a schema change happens.

[XDelta]

Fine with moving to a generated manifest from 'author' files, makes merge conflicts only occur within author specific files if at all. We can then update the structure of the resulting manifest without needing to manually edit entries