Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System memory limit fallback doesn't work as expected when sizing APM Server configs #13837

Closed
1pkg opened this issue Aug 6, 2024 · 0 comments
Assignees
Labels

Comments

@1pkg
Copy link
Member

1pkg commented Aug 6, 2024

APM Server version (apm-server version): current

Description of the problem including expected versus actual behavior:

When validating cgroups memory limit invariant in fallback we compare the limit from cgroups in GB against the system meminfo limit in bytes. Therefore, on a sufficiently large machine a situation could manifest where the "invalid" memory limit value from cgroups in GB is smaller that total amount of memory in bytes. For example, with the setup

cgget -n --values-only --variable memory.limit_in_bytes /test_apm
10737418240
cat /proc/meminfo | grep 'MemTotal:'
MemTotal:        2021860 kB

memory limit of 10 GB will be assumed instead of lower limit. The issue is more pronounced on cgroups v1 because of how default memory limit is presented to APM Server.

Provide logs (if relevant):

{"log.level":"info","@timestamp":"2024-08-06T11:30:01.399-0700","log.origin":{"file.name":"beatcmd/beat.go","file.line":394},"message":"apm-server started.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-06T11:30:01.401-0700","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":220},"message":"MaxConcurrentDecoders set to 1024 based on 80 percent of 10.0gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant