output: Retry document-level 429s by default
#13620
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updates the APM Server to automatically retry document-level `429`s from Elasticsearch to avoid dropping data. It can be configured/overwritten by `output.elasticsearch.max_retries`, and defaults to `3`. It uses the default backoff configuration, which could wait up to 1m if enough retries are configured, but can be overwritten as well. Signed-off-by: Marc Lopez Rubio <[email protected]>
|
This pull request does not have a backport label. Could you fix it @marclop? 🙏
NOTE: |
mergify
bot
added
the
backport-skip
carsonip
reviewed
Jul 8, 2024
There was a problem hiding this comment.
Retrying on document level 429 is the default if RetryOnDcoumentStatus is empty: https://github.com/elastic/go-docappender/blob/main/config.go#L76
|
@carsonip yes, I'm aware. I wanted to make it explicit to avoid any potential breaking changes going forward. It's also clearer when reading the code what will happen. |
carsonip
reviewed
Jul 8, 2024
Signed-off-by: Marc Lopez Rubio <[email protected]>
carsonip
reviewed
Jul 8, 2024
Co-authored-by: Carson Ip <[email protected]>
carsonip
approved these changes
Jul 8, 2024
marclop
added
backport-8.15
mergify bot
pushed a commit
that referenced
this pull request
Jul 8, 2024
Updates the APM Server to automatically retry document-level `429`s from Elasticsearch to avoid dropping data. It can be configured/overwritten by `output.elasticsearch.max_retries`, and defaults to `3`. It uses the default backoff configuration, which could wait up to 1m if enough retries are configured, but can be overwritten as well. --------- Signed-off-by: Marc Lopez Rubio <[email protected]> Co-authored-by: Carson Ip <[email protected]> (cherry picked from commit 28436dc) # Conflicts: # changelogs/head.asciidoc
1 task
mergify bot
added a commit
that referenced
this pull request
Jul 15, 2024
… (#13621) * output: Retry document-level `429`s by default (#13620) Updates the APM Server to automatically retry document-level `429`s from Elasticsearch to avoid dropping data. It can be configured/overwritten by `output.elasticsearch.max_retries`, and defaults to `3`. It uses the default backoff configuration, which could wait up to 1m if enough retries are configured, but can be overwritten as well. --------- Signed-off-by: Marc Lopez Rubio <[email protected]> Co-authored-by: Carson Ip <[email protected]> (cherry picked from commit 28436dc) # Conflicts: # changelogs/head.asciidoc * Fix conflicts, update changelog Signed-off-by: Marc Lopez Rubio <[email protected]> --------- Signed-off-by: Marc Lopez Rubio <[email protected]> Co-authored-by: Marc Lopez Rubio <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation/summary
Updates the APM Server to automatically retry document-level
429s from Elasticsearch to avoid dropping data. It can be configured/overwritten byoutput.elasticsearch.max_retries, and defaults to3.It uses the default backoff configuration, which could wait up to 1m if enough retries are configured, but can be overwritten as well.
This reduces data loss whenever Elasticsearch is overhelmed and reduces the chances of data loss due to ES push-back.
Checklist
- [ ] Documentation has been updatedHow to test these changes
This may need a bit of setup. It's thoroughly tested in the
go-docappender, but we may want to verify that retries happen when document-level 429s happen.I'd suggest writing a small "proxy" that returns 429s randomly and test that things are indexed (as duplicates).
Related issues
Closes #13619