Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] Support for llm-emit-token-metric in AIProjectClient ChatCompletionsClient for Azure OpenAI Token Tracking #39385

Open
o1100 opened this issue Jan 24, 2025 · 2 comments
Open

[Feature Request] Support for llm-emit-token-metric in AIProjectClient ChatCompletionsClient for Azure OpenAI Token Tracking #39385

o1100 opened this issue Jan 24, 2025 · 2 comments
Labels
AI Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. feature-request This issue requires a new behavior in the product in order be resolved. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@o1100
Copy link

o1100 commented Jan 24, 2025

Feature Request: Enable llm-emit-token-metric or azure-openai-emit-token-metric in AIProjectClient's ChatCompletionsClient

Is your feature request related to a problem? Please describe.

I am looking to use the llm-emit-token-metric or azure-openai-emit-token-metric headers within Azure API Management (APIM) to capture token usage. My setup uses Azure Foundry alongside the new azure.ai.projects AIProjectClient and its corresponding ChatCompletionsClient.

While I understand how to enable token metric headers when performing regular POST requests to Azure OpenAI endpoints, I have been unable to find functionality to do this within the ChatCompletionsClient.

The goal is to configure an APIM instance for the endpoint https://<deployment>.openai.azure.com/ and log token metrics to Azure Log Analytics.

Describe the solution you'd like

I would like the ChatCompletionsClient (from azure.ai.projects) to include functionality for enabling and utilizing the llm-emit-token-metric or azure-openai-emit-token-metric headers when interacting with Azure OpenAI services.

This would allow token usage tracking to seamlessly integrate with APIM and Log Analytics while leveraging the new client library.

Describe alternatives you've considered

  • I have been able to use regular POST requests to Azure OpenAI endpoints and successfully capture token metrics with the llm-emit-token-metric or azure-openai-emit-token-metric headers.
  • However, I cannot find similar functionality within the ChatCompletionsClient of azure.ai.projects.

Additional context

  • I have reviewed the Journey of the Geek blog post for guidance on this matter. While their solution effectively captures token metrics, it does not utilize the new AIProjectClient or ChatCompletionsClient from the azure.ai.projects library.
@o1100 o1100 changed the title Support for llm-emit-token-metric in AIProjectClient ChatCompletionsClient for Azure OpenAI Token Tracking [Feature Request] Support for llm-emit-token-metric in AIProjectClient ChatCompletionsClient for Azure OpenAI Token Tracking Jan 24, 2025
@github-actions github-actions bot added Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team OpenAI question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jan 24, 2025
@github-actions GitHub Actions
Copy link

Thank you for your feedback. Tagging and routing to the team member best able to assist.

@kristapratico kristapratico added feature-request This issue requires a new behavior in the product in order be resolved. Service Attention Workflow: This issue is responsible by Azure service team. AI and removed question The issue doesn't require a change to the product in order to be resolved. Most issues start as that OpenAI labels Jan 24, 2025
@kristapratico kristapratico removed their assignment Jan 24, 2025
@github-actions GitHub Actions
Copy link

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @achauhan-scc @kingernupur @luigiw @needuv @paulshealy1 @singankit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
AI Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. feature-request This issue requires a new behavior in the product in order be resolved. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kristapratico @o1100