Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enable automatic latency recording for consumers #308

Merged
merged 1 commit into from
Nov 22, 2023

Conversation

lynnagara
Copy link
Member

The goal of this change is to record a common latency metric for all consumers that use Arroyo.

The new metric is called arroyo.consumer.latency -- it represents the time in seconds from when the message was initially produced (broker timestamp) to when the consumer successfully committed the offset for a message.

Note that if a consumer is doing either manual batching (not using Arroyo's built in Reduce/Batch) or manual committing of offsets (not using Arroyo's CommitOffsets strategy), then the metric will not be recorded automatically.

Closes #307

The goal of this change is to record a common latency
metric for all consumers that use Arroyo.

The new metric is called `arroyo.consumer.latency` -- it
represents the time in seconds from when the message was
initially produced (broker timestamp) to when the consumer
successfully committed the offset for a message.

Note that if a consumer is doing either manual batching (not using
Arroyo's built in Reduce/Batch) or manual committing of offsets
(not using Arroyo's CommitOffsets strategy), then the metric
will not be recorded automatically.

Closes #307
@lynnagara lynnagara requested a review from a team as a code owner November 22, 2023 18:33
Copy link
Member

@untitaker untitaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may want to followup at some point to record something more accurate for reduced/unfolded messages (instead of "last value wins", for example aggregate avg/max). nice work!

@lynnagara lynnagara merged commit 33d0b75 into main Nov 22, 2023
8 checks passed
@lynnagara lynnagara deleted the latency-recording branch November 22, 2023 21:18
lynnagara added a commit to getsentry/sentry that referenced this pull request Nov 30, 2023
Since getsentry/arroyo#308 Arroyo's CommitOffsets strategy
records consumer latency automatically for consumers immediately prior
to committing offsets.
lynnagara added a commit to getsentry/sentry that referenced this pull request Dec 1, 2023
Since getsentry/arroyo#308 Arroyo's
CommitOffsets strategy records consumer latency automatically for
consumers immediately prior to committing offsets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automatically record consumer latency in Arroyo
2 participants