Registry API specification

[MythicManiac]

We should specify what the package registry API should look like and how can websites implement a gpm-compatible interface for accessing the package registry.

Use this issue for related conversation and drafts

[MythicManiac]

Let's start off with requirements we have for the API.

Guarantees

• A gpm registry should only show packages that follow the gpm-supported package format(s).
• A gpm registry should guarantee package naming and versioning scheme follows the package specification
• All packages in a registry should be immutable
• Packages should be kept available to download on the registry even after "removal", care for legal requirements, to avoid breaking ecosystem integrity (e.g. dependency references should remain valid even after a package is unlisted from a registry)

Functionality

• A gpm registry should have an endpoint for listing all available packages
• A gpm registry should have an endpoint for viewing a specific package's details and available versions
• A gpm registry should have an endpoint for viewing a specific package's specific version details
• A gpm registry should have authentication support
• A gpm registry should have an endpoint for uploading gpm packages

Other considerations and open questions

• Should we enforce HTTPS usage? It might not be a bad default but also could hinder local registry usage
• Rate limiting and how to standardize it? Should the API have an endpoint for rate limits or otherwise serve the configuration?
• Should we standardize a search endpoint, or make it optional to support? Search and filtering is likely to be implemented in mod manager UIs, which could be good to have an endpoint for to avoid requiring the client to cache all package metadata locally.
• For future proofing, we probably should track what metadata-format a package supports, so that clients can filter only the packages they know how to handle. Additional guarantees we could have is that packages of backwards-incompatible differing metadata formats shouldn't be allowed to depend on each other, meaning some form of semver would be ideal.
• The API itself should probably be versioned
• Pagination support, optional or requirement?

Any comments so far?

[MythicManiac]

Based on the above requirements, I'd propose an URL scheme as follows, in python pseudocode:

# The URL we need when adding a new registry. Can be any arbitrary URL. 
BASE_API_URL = "https://example.org/registry"

# API URLs are formed with <base_url>/<api_version>/<endpoint>

URLS = (
    # POST, Obtaining an authentication token
    f"{BASE_API_URL}/v1/token/"

    # GET, Listing all packages.
    # Possible search and filtering could be done with GET parameters here
    # Pagination support possibly?
    f"{BASE_API_URL}/v1/packages/"

    # POST, Uploading a new package, requires an authentication token
    f"{BASE_API_URL}/v1/packages/"

    # GET, Showing details of version 1.0.0 of package "SomePackage" by author "AuthorName"
    f"{BASE_API_URL}/v1/packages/AuthorName/SomePackage/1.0.0/"
    
    # GET, Download version 1.0.0. of package "SomePackage" by author "AuthorName"
    f"{BASE_API_URL}/v1/packages/AuthorName/SomePackage/1.0.0/download/"
)

Users then could add the registry to their GPM configuration (assuming it doesn't exist there yet) e.g. with

gpm registry add example https://example.org/registry

For listing packages from this registry, GPM would then query https://example.org/registry/v1/packages/

[zSoulweaver]

If searching is going to be potentially implemented, might I suggest a /authors route? Generally most searching will likely occur on the /packages route, but I'd imagine that users will want to search for info on specific authors.

I suppose you could just search within the /packages route for packages from a specific author, if the API returns any results you can then GET the /packages/:author route. I figure you could put less stress on the database and cache a /authors route easier.

Or, restructure the routes.
GET /packages

• OPTIONAL specific author
• OPTIONAL (with author query) package name returns specific package details
• OPTIONAL (with author and package name query) version number returns specific package version details
• OPTIONAL queries for searching/filtering otherwise returns all packages on the registry
• Code example below

// Returns details all packages on registry
request('/packages')

// Returns details of package 'some-mod-name' from user 'Soulweaver' with version '1.2.0'
request('/packages', {
  author: 'Soulweaver',
  package: 'some-mod-name',
  version: '1.2.0' // If missing, just return latest
})

// Returns all packages by user 'Soulweaver' ordered by amount of 'likes' with 'desc'ending value
request('/packages', {
  author: 'Soulweaver',
  orderBy: 'likes',
  sort: 'desc'
})

This could be extended to something like /packages/dependencies to get the dependencies of the package too, assuming it's using the specific queries.

POST /packages - Creates a new package on the registry
GET /authors - Accepts queries for searching/filtering otherwise returns all authors on the registry
GET /authors/packages - Returns list of packages from specific author

• REQUIRED author name
• OPTIONAL queries for searching/filtering packages otherwise returns all packages from specified author on the registry

[Schaken]

Would it help if i can provide what all my site can track, as maybe some suggestions and possibilities? For example Category, Author, File Version, Link back to File, Download ID number, Review Rating, and some other things that may be useful. In fact if one of you all would like access to personally view these things in person on my site, I am willing to grant access to someone who knows that stuff.

[metherul]

Should we standardize a search endpoint, or make it optional to support? Search and filtering is likely to be implemented in mod manager UIs, which could be good to have an endpoint for to avoid requiring the client to cache all package metadata locally.

As registries gain more and more packages it may become challenging to scale without an API-specific search function. The Nexus has over 65k mods in a single category -- an API shim will not be able to return every single package, and even if so, effectively serve them at volume.

For example Category, Author, File Version, Link back to File, Download ID number, Review Rating

In instances where we're not relying on data from a GPM registry (for example, an API shim) we cannot rely on consistent metrics for categories, file versions, ID numbers, ratings, etc. I think these values should be optional or omitted entirely.

[MythicManiac]

For example Category, Author, File Version, Link back to File, Download ID number, Review Rating

According to our package format specification, a lot of these are covered by the package metadata:

• Category: Available in the package metadata in the form of one or more tags
• Author: Required field in the package metadata, since all packages are namespaced under their author
• File Version: Required field in the package metadata
• Download ID number: We should not use ID numbers for identification, just relying on combination of author + package name + version should be enough
• Review Rating: This is something that could be supported as an optional metric from the registry.

So based on this, only Review Rating is a purely API-sourced metric, and the other metadata mentioned is already contained in the packages themselves, which should be available on the API. Another metric that could be useful to have is a download count.

So that would put registry sourced metadata to be at least:

• Download count
• Rating

These are probably metrics we shouldn't required, but could be optional for a registry to support.

[MythicManiac]

If searching is going to be potentially implemented, might I suggest a /authors route? Generally most searching will likely occur on the /packages route, but I'd imagine that users will want to search for info on specific authors.

@zSoulweaver
Did you mean that an /authors route would return all the known authors of a site, and their profile information? I do agree that's probably something we should have, just making sure I understood you correctly

[Schaken]

All looks good to me. I was just tossing information out there of what may be helpful on my end? Im really good with Papyrus, and python, and med at JSON, but what you all are doing i may as well wear my helmet and play with my crayons in the corner. lol. Ping me if there is anything i can do that would be beneficial to the project, i would love to help in any way. I use the same system as nexus, so if there is some information in the backend of the website stuff that i can provide, im willing.

[MythicManiac]

@Schaken yeah definitely, registry sourced metrics (like reviews or download count) that you mentioned are an important consideration.

Not sure if you've seen the package format spec at WolvenKit/CP77Wiki#1, but that might give you some idea on what the required metadata we want for all packages is going to look like

[zSoulweaver]

If searching is going to be potentially implemented, might I suggest a /authors route? Generally most searching will likely occur on the /packages route, but I'd imagine that users will want to search for info on specific authors.

@zSoulweaver
Did you mean that an /authors route would return all the known authors of a site, and their profile information? I do agree that's probably something we should have, just making sure I understood you correctly

Yeah, that's correct. As examples by the /packages route you could then tack queries onto it to to search and filter.

[jobobby04]

Based on what you want, I am not sure how it could be compatible with a Invision based website like Schaken-Mods, there is not much freedom on APIs with Invision Community, you pretty much are only allowed to use the REST api after OAUTH authentication(with a key and the user account). Custom APIs would have to be implemented separately on a subdomain, totally separate from the website itself, it would have to have direct access to the Invision DB and file store, this all would be very difficult with how Invision stores and keeps track of files.