Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sample ratio not align with configured SampleRate during Stress Relief #1391

Open
VinozzZ opened this issue Oct 18, 2024 · 0 comments
Open
Labels
type: bug Something isn't working

Comments

@VinozzZ
Copy link
Contributor

VinozzZ commented Oct 18, 2024

Description:

When Stress Relief mode is activated, a fraction of traffic is sampled through a deterministic sampler based on the hash of trace id calculated using wyhash. When observing the kept_from_stress and dropped_from_stress metrics, the ratio between the two does not always align with the SampleRate configured for stress relief

Potential Cause
Below test showed that with smaller iteration n, wyhash result sometimes can have less distribution. This can be a reason why more traces are kept than configured SampleRate

func TestWyhash(t *testing.T) {
	n := 10000
	const frac = 100
	var upperBound uint64 = math.MaxUint64 / frac
	for i := 0; i < 10; i++ {
		t.Run(fmt.Sprintf("frac=%d", frac), func(t *testing.T) {
			count := 0
			for i := 0; i < n; i++ {
				traceID := fmt.Sprintf("%016x%016x", rand.Int63(), rand.Int63())
				hash := wyhash.Hash([]byte(traceID), hashSeed)
				if hash <= upperBound {
					count++
				}
			}
			assert.InDelta(t, count, n/frac, 0.1*float64(n/frac))
		})
	}
}
@VinozzZ VinozzZ added the type: bug Something isn't working label Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant