Relay Server: Not Enough Memory on Health Check even though stats show otherwise

[LordSimal]

Self-Hosted Version

24.8.0

CPU Architecture

x86_64

Docker Version

27.2.1

Docker Compose Version

2.29.2

Steps to Reproduce

Can't really tell how to reproduce, since it just happens out of nowhere.

Expected Result

Sentry receives errors again

Actual Result

Sentry stops receiving errors after 1-2 days of normal usage.

Checking the docker logs there are a lot of these entries present:

relay-1                                         | 2024-09-17T06:33:00.437945Z ERROR relay_server::services::health_check: Not enough memory, 32351698944 / 33568419840 (96.38% >= 95.00%)
relay-1                                         | 2024-09-17T06:33:00.437982Z ERROR relay_server::services::health_check: Health check probe 'system memory' failed

but checking either htop we have enough RAM

as well as checking docker container stats there are no containers using > 95% RAM

CONTAINER ID   NAME                                                                CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
48244b39b406   sentry-self-hosted-nginx-1                                          0.10%     11.74MiB / 31.26GiB   0.04%     829MB / 832MB     11.8MB / 131kB    13
4188cd46c1fe   sentry-self-hosted-relay-1                                          0.54%     515.6MiB / 31.26GiB   1.61%     896MB / 2.06GB    363MB / 284MB     40
ca96eedda64d   sentry-self-hosted-generic-metrics-consumer-1                       0.56%     342.4MiB / 31.26GiB   1.07%     177MB / 290MB     17.8MB / 82MB     21
5dedcba8621c   sentry-self-hosted-monitors-clock-tick-1                            0.28%     162.4MiB / 31.26GiB   0.51%     36MB / 33.5MB     35.8MB / 29.5MB   6
8e98edce4698   sentry-self-hosted-subscription-consumer-generic-metrics-1          0.28%     323.9MiB / 31.26GiB   1.01%     37.7MB / 34.5MB   13.2MB / 68.7MB   13
ca542ffd958a   sentry-self-hosted-attachments-consumer-1                           0.48%     500.2MiB / 31.26GiB   1.56%     16.9MB / 15.2MB   26.8MB / 66.6MB   19
b300b32205a5   sentry-self-hosted-snuba-replacer-1                                 0.28%     115.6MiB / 31.26GiB   0.36%     35MB / 31.5MB     20.2MB / 68.9MB   5
1a77958a745c   sentry-self-hosted-ingest-monitors-1                                0.37%     169.8MiB / 31.26GiB   0.53%     17.2MB / 15.8MB   41.3MB / 19.8MB   11
b089c100ada2   sentry-self-hosted-worker-1                                         5.47%     1.444GiB / 31.26GiB   4.62%     9.41GB / 13.3GB   238MB / 126MB     227
8a69077b8025   sentry-self-hosted-snuba-replays-consumer-1                         0.46%     157.2MiB / 31.26GiB   0.49%     35.4MB / 31.9MB   2.92MB / 123MB    30
d77331751542   sentry-self-hosted-events-consumer-1                                0.37%     281.7MiB / 31.26GiB   0.88%     974MB / 1.02GB    18.1MB / 114MB    17
9414826f3339   sentry-self-hosted-subscription-consumer-transactions-1             0.26%     258.9MiB / 31.26GiB   0.81%     37.3MB / 34MB     27.9MB / 137MB    13
e59e3a288fbf   sentry-self-hosted-vroom-1                                          0.00%     11.75MiB / 31.26GiB   0.04%     233kB / 0B        34.8MB / 4.35MB   11
0acb817cf62d   sentry-self-hosted-snuba-subscription-consumer-events-1             0.39%     147.5MiB / 31.26GiB   0.46%     40.9MB / 34.8MB   7.13MB / 23.4MB   9
f996440e6e0d   sentry-self-hosted-post-process-forwarder-issue-platform-1          0.57%     274.9MiB / 31.26GiB   0.86%     71.8MB / 65.6MB   22.7MB / 121MB    18
c9578898e6f1   sentry-self-hosted-sentry-cleanup-1                                 0.00%     7.328MiB / 31.26GiB   0.02%     260kB / 28.5kB    136MB / 557kB     6
c148234ce38f   sentry-self-hosted-subscription-consumer-events-1                   0.27%     226.4MiB / 31.26GiB   0.71%     36.4MB / 33.1MB   16.8MB / 171MB    13
4651f280662d   sentry-self-hosted-metrics-consumer-1                               0.50%     339.4MiB / 31.26GiB   1.06%     34.8MB / 31.4MB   21.2MB / 65.4MB   21
91bc0382285e   sentry-self-hosted-ingest-profiles-1                                0.27%     155.3MiB / 31.26GiB   0.49%     33.5MB / 30MB     15.7MB / 37MB     6
82f8557caafa   sentry-self-hosted-snuba-metrics-consumer-1                         0.55%     191.5MiB / 31.26GiB   0.60%     34.6MB / 31.4MB   2.88MB / 96.5MB   34
1c7c176be51e   sentry-self-hosted-transactions-consumer-1                          0.40%     347.4MiB / 31.26GiB   1.09%     30.6MB / 29.6MB   26.3MB / 42.5MB   17
7f04ed421f85   sentry-self-hosted-post-process-forwarder-errors-1                  0.69%     350MiB / 31.26GiB     1.09%     32.8MB / 21.3MB   18.2MB / 43.4MB   23
d8ad278dcd0a   sentry-self-hosted-ingest-occurrences-1                             0.52%     146.3MiB / 31.26GiB   0.46%     36.9MB / 33.3MB   31.5MB / 58.7MB   16
a77658238b91   sentry-self-hosted-snuba-subscription-consumer-metrics-1            0.39%     132.6MiB / 31.26GiB   0.41%     36.2MB / 33.2MB   9.47MB / 41.3MB   9
8c9d85e099b8   sentry-self-hosted-web-1                                            0.12%     734.6MiB / 31.26GiB   2.29%     90.4MB / 332MB    346MB / 194MB     41
74ad5875b983   sentry-self-hosted-monitors-clock-tasks-1                           0.25%     147.3MiB / 31.26GiB   0.46%     35.3MB / 31.9MB   16.9MB / 49MB     6
12994ef232b6   sentry-self-hosted-billing-metrics-consumer-1                       0.46%     157.6MiB / 31.26GiB   0.49%     63.6MB / 37.2MB   13.5MB / 36MB     9
f7f6a96c18ec   sentry-self-hosted-ingest-replay-recordings-1                       0.43%     159.8MiB / 31.26GiB   0.50%     36.2MB / 32.7MB   21.8MB / 35.5MB   13
c0d724db862f   sentry-self-hosted-snuba-issue-occurrence-consumer-1                0.55%     333.4MiB / 31.26GiB   1.04%     34.8MB / 31.3MB   25.1MB / 61.4MB   41
37fb6a1dbf5e   sentry-self-hosted-cron-1                                           0.00%     179.1MiB / 31.26GiB   0.56%     17.4MB / 146MB    28.9MB / 45.1MB   3
6c46a69fa981   sentry-self-hosted-snuba-outcomes-billing-consumer-1                0.35%     200.9MiB / 31.26GiB   0.63%     18.3MB / 16.5MB   11MB / 49.2MB     26
564d6cedc04a   sentry-self-hosted-post-process-forwarder-transactions-1            0.64%     397.1MiB / 31.26GiB   1.24%     5.76GB / 462MB    21.2MB / 104MB    23
f135db607aec   sentry-self-hosted-subscription-consumer-metrics-1                  0.28%     298.9MiB / 31.26GiB   0.93%     36.5MB / 33.3MB   19.1MB / 94MB     13
c3ebde813802   sentry-self-hosted-snuba-transactions-consumer-1                    0.47%     322.3MiB / 31.26GiB   1.01%     21.4MB / 17MB     21.4MB / 60.4MB   37
4e4475215c07   sentry-self-hosted-ingest-feedback-events-1                         0.39%     238.4MiB / 31.26GiB   0.74%     35.7MB / 32.1MB   12.8MB / 156MB    15
d266f38dc1aa   sentry-self-hosted-snuba-spans-consumer-1                           0.35%     179MiB / 31.26GiB     0.56%     142MB / 582MB     11.3MB / 101MB    26
756587f4a091   sentry-self-hosted-snuba-generic-metrics-gauges-consumer-1          0.54%     181.1MiB / 31.26GiB   0.57%     117MB / 37.3MB    3.19MB / 120MB    34
ea958e5724d3   sentry-self-hosted-snuba-profiling-profiles-consumer-1              0.34%     128.8MiB / 31.26GiB   0.40%     35MB / 31.5MB     2.27MB / 120MB    26
cc77019cca49   sentry-self-hosted-symbolicator-cleanup-1                           0.00%     4.367MiB / 31.26GiB   0.01%     234kB / 0B        35.2MB / 0B       6
8c1ca1507b75   sentry-self-hosted-snuba-profiling-functions-consumer-1             0.34%     134.2MiB / 31.26GiB   0.42%     35MB / 31.5MB     2.53MB / 116MB    26
c479e6353a01   sentry-self-hosted-snuba-generic-metrics-counters-consumer-1        0.57%     255.2MiB / 31.26GiB   0.80%     18MB / 16.3MB     10.8MB / 23.2MB   34
a64b4393273e   sentry-self-hosted-snuba-errors-consumer-1                          0.57%     233.1MiB / 31.26GiB   0.73%     360MB / 317MB     4.6MB / 57.9MB    34
475833b0af89   sentry-self-hosted-snuba-generic-metrics-sets-consumer-1            0.67%     213.9MiB / 31.26GiB   0.67%     118MB / 41.6MB    6.73MB / 95.7MB   34
8a412f926058   sentry-self-hosted-snuba-generic-metrics-distributions-consumer-1   0.56%     270.6MiB / 31.26GiB   0.85%     124MB / 559MB     4.73MB / 60.4MB   34
4be03fe74a97   sentry-self-hosted-snuba-outcomes-consumer-1                        0.33%     162.1MiB / 31.26GiB   0.51%     33.8MB / 30.4MB   2.17MB / 92.4MB   26
f9689ba0b412   sentry-self-hosted-snuba-group-attributes-consumer-1                0.47%     322MiB / 31.26GiB     1.01%     34.8MB / 31.7MB   17.9MB / 86.7MB   37
acb9904b4aa4   sentry-self-hosted-snuba-subscription-consumer-transactions-1       0.38%     123.5MiB / 31.26GiB   0.39%     42.5MB / 36.3MB   8.68MB / 41.4MB   9
c44eb35c0248   sentry-self-hosted-vroom-cleanup-1                                  0.00%     3.602MiB / 31.26GiB   0.01%     234kB / 0B        8.87MB / 0B       6
e896238b7399   sentry-self-hosted-memcached-1                                      0.03%     23.07MiB / 31.26GiB   0.07%     673MB / 1.81GB    8.15MB / 2.45MB   10
f5170f49f238   sentry-self-hosted-snuba-api-1                                      0.05%     113.7MiB / 31.26GiB   0.36%     9.83MB / 16.3MB   66.3MB / 68.9MB   5
f7f090a153a3   sentry-self-hosted-symbolicator-1                                   0.00%     35MiB / 31.26GiB      0.11%     307kB / 59.4kB    25.5MB / 142MB    38
dec490aea24f   sentry-self-hosted-smtp-1                                           0.00%     1.371MiB / 31.26GiB   0.00%     255kB / 15.7kB    28.8MB / 4.1kB    2
3e57a3611024   sentry-self-hosted-postgres-1                                       0.01%     231.6MiB / 31.26GiB   0.72%     1.53GB / 728MB    17.6GB / 13.2MB   53
c5eaa992ea9b   sentry-self-hosted-kafka-1                                          1.97%     1.257GiB / 31.26GiB   4.02%     5.5GB / 11.8GB    1.73GB / 461MB    111
4bac36975fe0   sentry-self-hosted-clickhouse-1                                     0.39%     469MiB / 31.26GiB     1.47%     1.71GB / 83.7MB   3.21GB / 93.3MB   481
f1f35f95d7a7   sentry-self-hosted-redis-1                                          0.17%     56.23MiB / 31.26GiB   0.18%     11.9GB / 7.22GB   763MB / 2.97MB    5

docker_compose_logs.txt
latest_install_logs.txt

Event ID

No response

[LordSimal]

Duplicate of #3327

[barisyild]

I think it's a memory leak.

[LordSimal]

What I can confirm is the fact, that doing the upgrade to 24.9.0 did NOT fix the issue.
After a few hours of incoming events I repeatedly get the same

relay-1                                         | 2024-09-17T14:16:40.186772Z ERROR relay_server::services::health_check: Not enough memory, 32202633216 / 33568419840 (95.93% >= 95.00%)
relay-1                                         | 2024-09-17T14:16:40.186811Z ERROR relay_server::services::health_check: Health check probe 'system memory' failed
relay-1                                         | 2024-09-17T14:16:40.449995Z ERROR relay_server::endpoints::common: error handling request error=failed to queue envelope

error in the docker compose logs.

[hubertdeng123]

What does the event volume look like for you? Did this start happening after upgrading to 24.8.0?

[LordSimal]

We did the 24.8.0 Sentry update on the 1st of September.
This is our stats page for the last 30 days

As you can see there are sections where it works fine but then sometimes for a few hours, sometimes even for days there are no events being processed.

[hubertdeng123]

Could you track your RAM/CPU usage as well? Wondering if there is a correlation there.