Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DistributedTrace.generate_sampling timeouts. #385

Open
noizu opened this issue Jun 15, 2022 · 2 comments · May be fixed by #460
Open

DistributedTrace.generate_sampling timeouts. #385

noizu opened this issue Jun 15, 2022 · 2 comments · May be fixed by #460

Comments

@noizu
Copy link

noizu commented Jun 15, 2022

Describe the bug
Sampling timeouts occur under load on my IOT facing API nodes.

Need a way to increase timeouts or expose other configuration settings to avoid this exception.

image

Environment

  • Elixir & Erlang version (elixir -v):
    Erlang/OTP 22 [erts-10.7.2.16] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

  • Agent version (mix deps | grep new_relic_agent):
    new_relic_agent (Hex package)
    (mix) locked at 1.27.7 (new_relic_agent) a0aa7208
@noizu
Copy link
Author

noizu commented Dec 9, 2022

I was eventually able to resolve this by tweaking the following settings.

config :new_relic_agent,
analytic_event_per_minute: 250, # Reducing caps to avoid overwhelming api node telemetry
custom_event_per_minute: 250,
error_event_per_minute: 100,
span_event_per_minute: 100,
sampling_target: 5

Should we prepare a doc update to cover options for tweaking sampling rates and caps for high load environments to safe the next person some time?

@binaryseed
Copy link
Collaborator

Updating your sampling rates is a good call, and might be something you would want to do anyway for your app, but this particular error could be addressed by converting the BackoffSampler to use ets for storage instead of being a GenServer (which can be become overloaded if things get too hot)

I'll make a PR for that

@binaryseed binaryseed linked a pull request Dec 16, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Speed up BackoffSampler with counters newrelic/elixir_agent
2 participants
@binaryseed @noizu