feat: Write OTel GenAI semantic convention client-side metrics for token usage #1545
Conversation
bryanatkinson
force-pushed
the
bryanatkinson-semconv-metrics
branch
2 times, most recently
from
6795c8d to
1aa5cb4
Compare
bryanatkinson
force-pushed
the
bryanatkinson-semconv-metrics
branch
from
1aa5cb4 to
0901d71
Compare
| @@ -422,7 +422,7 @@ describe('GoogleCloudMetrics', () => { | |||
| ); | |||
| }); | |||
|
|
|||
| it('writes feature label to generate and action metrics when running inside an action', async () => { | |||
| it.only('writes feature label to generate and action metrics when running inside an action', async () => { | |||
There was a problem hiding this comment.
I think this needs to be undone
There was a problem hiding this comment.
oops. Removed.
| * conditions we defer the instantiation of the metric to when it is first | ||
| * ticked. | ||
| */ | ||
| class Metric<T> { |
There was a problem hiding this comment.
Not a huge fan of having this be in core - are we positive that this can't live in the plugin?
There was a problem hiding this comment.
There's no reason it couldn't be done in the plugin, but these semantic conventions are intended so that different tools can understand the telemetry data, so only exporting them when you're using GCP somewhat defeats the purpose.
|
I've also got a follow-up that includes writing the span attributes and I think cleans things up a little bit: https://github.com/firebase/genkit/compare/bryanatkinson-semconv-metrics...bryanatkinson-semconv-spans?expand=1 I can switch this PR to using that if you're OK with the general approach. |
Adds support for writing the
gen_ai.client.token.usagemetric as described by the OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiclienttokenusageI added this directly to the framework instead of in the google-cloud plugin since they seem generally useful. So far I have only added support to the Gemini and Imagen models to see if this approach looks good. If it does, I will carry it through to the other models in the vertex model garden and add the
gen_ai.client.operation.durationmetric.Checklist (if applicable):