Ingestion burst size not working as desired

[rishabhkumar92]

Describe the bug

We are trying to set a acceptable ingestion rate but higher burst size to reduce distributor rate limiting but received rate is capped at ingestion rate.

To Reproduce

1. Start Mimir on 2.12
2. Set ingestion rate at a certain value(say 20K) and ingestion burst size(say 100K), when writers are sending samples at higher rate then ingestion rate (40-60K), distributor is rejecting those samples and received samples rate is same as allowed ingestion rate 20K.

Expected behavior

• Burst size should take care of elevated sample rate and we shouldn't observe rate limiting from distributor's POV.

Environment

• Infrastructure: Kubernetes

Additional Context

Distributor is rate limiting with following error:

the request has been rejected because the tenant exceeded the ingestion rate limit, set to 42533 items/s with a maximum allowed burst of 800000. This limit is applied on the total number of samples, exemplars and metadata received across all distributors (err-mimir-tenant-max-ingestion-rate). To adjust the related per-tenant limits, configure -distributor.ingestion-rate-limit and -distributor.ingestion-burst-size, or contact your service administrator.
Sample incoming rate is 80K for this particular request and there are 20 distributor with fair balancing.

[56quarters]

Can you post the limits configuration you're using when you get the mentioned error?

[56quarters]

Set ingestion rate at a certain value(say 20K) and ingestion burst size(say 100K), when writers are sending samples at higher rate then ingestion rate (40-60K), distributor is rejecting those samples and received samples rate is same as allowed ingestion rate 20K.

This is expected behavior. If you set the ingestion rate to 20K samples/sec and send more than that number of samples per second, e.g. 40K samples/sec, eventually you will be rate limited regardless of the burst size. The ingestion rate is the rate at which the distributors will accept samples. The burst allows for temporarily exceeding the ingestion rate limit but eventually you will be rate limited.

[rishabhkumar92]

@56quarters can you explain a bit on "The burst allows for temporarily exceeding the ingestion rate limit but eventually you will be rate limited.", is it just for first 10 seconds, based on code I assumed that burst size will be applicable for every 10 second window?
Also is there any metric which I can look up to validate burst size exhaustion?

[56quarters]

The underlying code in question has a pretty good explanation: link. The burst is the max size of a bucket and the ingestion rate is the rate at which the bucket is refilled.

The burst allows for temporarily exceeding the ingestion rate limit but eventually you will be rate limited.", is it just for first 10 seconds, based on code I assumed that burst size will be applicable for every 10 second window?

The name of the parameter is recheckPeriod. 10 seconds in this case is how often the in-memory limits are compared against the configured limits for a particular tenant in case they've changed. The ingestion rate still controls how fast the bucket is filled and burst still controls the max size.

[rishabhkumar92]

thanks for clarification @56quarters , just a follow up clarification: what's the interval when tokens are getting reset, I am asking for case where ingestion rate <<< ingestion rate limit for a period and exceeds post that but still less than burst size(e.g if there is no reset, token consumption will be successful beyond ingestion rate limit)?

[rishabhkumar92]

@56quarters bumping this up for above question.

[56quarters]

just a follow up clarification: what's the interval when tokens are getting reset

I'm not really sure what you mean, tokens don't get reset. Every time there's another request, the rate limiter checks how long it has been since the last request and adds new tokens to the bucket based on the defined rate and max capacity (burst) before deducting them from the bucket (if there are enough).

I am asking for case where ingestion rate <<< ingestion rate limit for a period and exceeds post that but still less than burst size(e.g if there is no reset, token consumption will be successful beyond ingestion rate limit)?

You can only exceed the rate limit for a little while until the bucket is empty. As an example:

your rate limit = 20 samples / sec
your burst size = 100 samples

If you send 20 samples / sec or fewer, this will work indefinitely.

If you send 30 samples / sec , this will work for 10 seconds before you've exhausted the "burst" which is the size of the bucket. The bucket is refilled at 20 samples / sec but you're using an extra 10 samples / sec from your "burst".

If you send 40 samples / sec, this will work for 5 seconds before you've exhausted the "burst" which is the size of the bucket. The bucket is refilled at 20 samples / sec but you're using an extra 20 samples / sec from your "burst".

[rishabhkumar92]

thanks for clarification @56quarters , Closing this thread.