feat: Add metric for receiver block backfill failures #912
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a metric for tracking receiver block backfill status. Since the backfill has an end state, I can't rely on constantly incrementing a counter. Instead, I use a gauge and check for non zero values. If the backfill starts and completes without issue, the value will be 0. But, if the backfill fails, it will increment the gauge and then skip to Lake backfill, leaving the gauge at a nonzero value. We can alert on this. If the stream is restarted, then a successful attempt at the backfill will reset the gauge to 0.