Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add serviceinfo URI #232

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

AlpinDale
Copy link
Contributor

@AlpinDale AlpinDale commented Nov 4, 2024

In an effort to create a standard well-known URI for Inference servers, we, @db0 (AI Horde), @AlpinDale (Aphrodite Engine), and @LostRuins (KoboldCPP) have agreed on a set of key info to be broadcasted by every API server to identify itself. The spec contains other fields not included here, such as logo and version (for the software). As far as I know, tabby doesn't have an official logo, and since it's a rolling-release software, it doesn't have a version either. I thought about logging the commit here, but we don't have the infrastructure to log and store the commit hash yet. I can PR that in if needed.

Sample response:

{
  "version": 0.1,
  "software": {
    "name": "TabbyAPI",
    "repository": "https://github.com/theroyallab/tabbyAPI",
    "homepage": "https://github.com/theroyallab/tabbyAPI"
  },
  "api": {
    "openai": {
      "name": "OpenAI API",
      "base_url": "http://127.0.0.1:5000/v1",
      "documentation": "https://theroyallab.github.io/tabbyAPI",
      "version": 1
    },
    "koboldai": {
      "name": "KoboldAI API",
      "base_url": "http://127.0.0.1:5000/api",
      "documentation": "https://theroyallab.github.io/tabbyAPI",
      "version": 1
    }
  }
}

References:
PygmalionAI/aphrodite-engine#807
Haidra-Org/AI-Horde@eb5ec3d
LostRuins/koboldcpp@f153a14

https://github.com/Haidra-Org/AI-Horde/wiki/serviceinfo

@db0
Copy link

db0 commented Nov 4, 2024

Would be good to ensure that a version alweys exists, even if, as you said it's only a commit hash

@henk717
Copy link

henk717 commented Nov 4, 2024

I don't think it will be a big deal if its a pseudo version number that gets bumped for major things.
Like for example if there is a critical inference bug tabby could bump it by 0.1 so we can identify the bad ones.
So if integrating proper versioning isn't feasible that would be a workaround.

@@ -166,3 +168,28 @@ async def embeddings(request: Request, data: EmbeddingsRequest) -> EmbeddingsRes
)

return response

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The route should be in endpoints/core since those are propagated across all API servers.

@bdashore3
Copy link
Member

This spec looks fine by me. Added comments on the commits themselves. Also to note, semver is a common way of releasing points of software, but it's not the only option.

Rolling release is used for applications in ecosystems that are always changing (such as the exl2/AI ecosystem at large). Since TabbyAPI isn't built, it makes sense to keep it in a rolling state. Embedding the commit sha makes the most sense here, but using a semver of 0.0.1 will work as well (like the docs).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants