Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Azure OpenAI endpoint support #41

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

datlife
Copy link

@datlife datlife commented Mar 26, 2024

Added Azure OpenAI endpoint support. For Azure OpenAI, users need to specify OPENAI_API_VERSION and update the header to api-key: <key> instead of Authorization: Bearer <token>

Reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#chat-completions

Testing: I was able top use llmperf for Azure endpoint

# Azure openAI
export OPENAI_API_BASE="https://<resporce_name>.openai.azure.com/openai/deployments/gpt-3.5-turbo"
export OPENAI_API_KEY=<azure_key_from_resource_name>
export OPENAI_API_VERSION="2023-03-15-preview"

python token_benchmark_ray.py \
--model "gpt-3.5-turbo" \
--mean-input-tokens 550 \
--stddev-input-tokens 150 \
--mean-output-tokens 150 \
--stddev-output-tokens 10 \
--timeout 600 \
--results-dir "result_outputs" \
--llm-api openai%

@NicolasGrosjeanProbayes

Thanks a lot @datlife, it saved me a lot of time

@@ -59,6 +59,12 @@ def llm_request(self, request_config: RequestConfig) -> Dict[str, Any]:
if not address.endswith("/"):
address = address + "/"
address += "chat/completions"
if "azure"in address:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to replace "azure" by "openai.azure.com" because we can use other models (for example Mistral Large) in "ml.azure.com" and we don't want to enter in this bloc of code.

jmhessel added a commit to jmhessel/llmperf that referenced this pull request Aug 14, 2024
jmhessel added a commit to jmhessel/llmperf that referenced this pull request Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants