Adaptive Request Concurrency (ARC) should maintain separate state per (some) partition keys

[jszwedko]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Broken off from #21402

Currently, ARC maintains a global view of requests originating from a given sink; however, different request partitions can have correlated response behavior from the upstream. For example, a Datadog Logs sink where some incoming events have different API keys, one of which is invalid, can cause ARC to think it should back off all requests, and not just the partition with the invalid API key.

Ideally, ARC would maintain separate limits per partition key that can impact upstream responses so that partitions don't conflict.

Open questions:

• How should the documentation/configuration for ARC be updated to support the limits being applied per relevant partition key?
• Should we tag ARC metrics with the partition key used to segment the state? I think we'd need to, but this could result in a high cardinality depending on the partition key.

Configuration

No response

Version

v0.42.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

• Datadog sinks batch together events with different API keys #21402