[Question]: How to use an OpenAI compatible API?

[decentropy]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

For example, GooseAI is AI compatible...

This works

import openai
openai.api_key = "sk-hidemykey"
openai.api_base = "https://api.goose.ai/v1"
completion = openai.Completion.create(
  engine="gpt-j-6b",
  prompt="What is a Goose?",
  max_tokens=250,
  stream=True)
for c in completion:
  print (c.choices[0].text, end = '')

but this fails

from llama_index.llms import OpenAI
resp = OpenAI(
    model="gpt-j-6b",
    api_key="sk-hidemykey", 
    api_base="https://api.goose.ai/v1").complete("What is a goose?")

ValueError: Unknown model 'gpt-j-6b'. Please provide a valid OpenAI model name in: gpt-4, gpt-4-32k, gpt-4-0613, gpt-4-32k-0613, gpt-4-0314, gpt-4-32k-0314, gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-3.5-turbo-0613, gpt-3.5-turbo-16k-0613, gpt-3.5-turbo-0301, text-davinci-003, text-davinci-002, gpt-3.5-turbo-instruct, text-ada-001, text-babbage-001, text-curie-001, ada, babbage, curie, davinci, gpt-35-turbo-16k, gpt-35-turbo

It seems that even though I change the api_base, it restricts itself to static openai models

[dosubot[bot]]

🤖

Hello,

Thank you for your question. It seems like you're trying to use a custom model with the OpenAI class in LlamaIndex, but you're encountering an error because the model name is not recognized.

The issue you're facing is due to the fact that the OpenAI class in LlamaIndex is designed to work with specific OpenAI models. When you try to use a model that is not recognized, it throws a ValueError. This is why changing the api_base does not solve the issue, as the restriction is due to the implementation of the OpenAI class, not the base API endpoint.

However, there are a couple of issues in the LlamaIndex repository that might be relevant to your situation:

1. Feature Request: Fastchat RESTful OpenAI API server drop-in for Llama 2
2. run llama_index with locally llm that have compatible OpenAI's API

These issues suggest that there might be a way to use a custom LLM model with LlamaIndex. You can refer to the documentation at https://gpt-index.readthedocs.io/en/latest/how_to/customization/custom_llms.html#example-using-a-custom-llm-model for more information on how to use a custom LLM model.

As for the stream_complete and stream_chat methods, they are not implemented in the LlamaAPI class. These methods are typically used for handling streaming responses from models that can generate responses incrementally. Since these methods are not implemented, the LlamaAPI can only work with models that generate responses all at once, i.e., static models.

If you're using the OpenAI class, you should be able to use streaming responses. If you're still unable to use streaming responses, it could be due to the _use_chat_completions method, which determines whether to use the chat completions endpoint based on the model and the use_chat_completions keyword argument. If this method returns False, the stream_chat and stream_complete methods will use the non-streaming _complete and _chat methods instead.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• [Feature Request]: Fastchat RESTful OpenAI API server drop-in for Llama 2
• run llama_index with locally llm that have compatible OpenAI's API
• llama_index/llms/llama_api.py
• llama_index/llms/openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

PR in progress: #7973

will try to get that merged soon 🙏🏻

[benbot]

Yep finishing up the last few comments on it now. Was out of commission for a few days 😅

[OM-EL]

This issue is still blocking me :(
i cant use NLSQLTableQueryEngine with my OpenIA like interface ( i have litellm ).