Possible memory leak

[surister]

User at https://community.cratedb.com/t/disk-space-issues-on-prometheus-integration/1848/3 reports 85% memory usage, I checked out my own long running container and it was:
35%

5fd5f000593a   crate-prometheus-exporter   0.00%  5.462GiB / 15.51GiB   35.22%  4.26GB / 1.62GB   0B / 0B  12

Some other long running container was at 1.6/6GB ~25%

These last two findings are less than the users's 85% but still too much in my opinion and makes me think something is going on.

[surister]

I just checked my machine and after turning it off and on its now stable at:

5fd5f000593a   crate-prometheus-exporter  0.00%  43.88MiB / 15.51GiB   0.28%  72.1MB / 154MB  0B / 0B  11

[cyberplant]

Hello!

I had it running for 12 hours, and the memory usage right now is at 11.6% of system memory:

0227e05def58   cratedb-prometheus-adapter   215.22%   14.55GiB / 125.5GiB   11.60%    0B / 0B   5.13MB / 0B   53

I have the /metrics output from half an hour and now, is it useful? I can upload it here to see if it helps debugging the issue.

[cyberplant]

I have to shutdown everything to continue with my work, so I'm dumping all hoping it will help.

I stopped prometheus-server, and after a while memory usage decreased a bit but then stayed there:

0227e05def58   cratedb-prometheus-adapter   53.34%    11.24GiB / 125.5GiB   8.96%     0B / 0B   5.18MB / 0B   53

And 20+ minutes later:

0227e05def58   cratedb-prometheus-adapter   60.95%    11.19GiB / 125.5GiB   8.92%     0B / 0B   5.18MB / 0B   53

I'm also attaching the metrics on different moments:

• 14:39 - adapter-metrics-1.txt
• 15:08 - adapter-metrics-2.txt
• 15:24 - Stopped prometheus-server container - adapter-metrics-3-prometheus-server-just-stopped.txt
• 15:44 - prometheus-server still stopped - adapter-metrics-4-prometheus-server-stopped-20-minutes-ago.txt

--

Noticed that memory usage is going down, but very slowly.. maybe we need a more aggressive GC? each command run manually over some minutes span:

0227e05def58   cratedb-prometheus-adapter   56.56%    11.13GiB / 125.5GiB   8.87%     0B / 0B   5.18MB / 0B   53
0227e05def58   cratedb-prometheus-adapter   3092.44%   11.13GiB / 125.5GiB   8.87%     0B / 0B   5.18MB / 0B   53
0227e05def58   cratedb-prometheus-adapter   34.82%    11.1GiB / 125.5GiB   8.85%     0B / 0B   5.18MB / 0B   53
0227e05def58   cratedb-prometheus-adapter   52.53%    11.08GiB / 125.5GiB   8.83%     0B / 0B   5.18MB / 0B   53

And now ran in a loop to have more accurate stats:

[cyberplant]

Note that prometheus-server is stopped, nothing is connecting to the adapter, and the CPU usage is still a bit high (with some peaks, maybe GC?). I need to stop everything and move along, I'm sharing here the last metrics output before shutting it down. Hope it helps!!

adapter-metrics-5-before-shutdown.txt

[surister]

Updating all dependencies and go version made it so I could no longer quickly reproduce the memleak, but let's be careful, you never know with this things.

I released 0.5.2 and will deploy it on a couple of long running machines that I have and let it be for a few days, let's see if it manifests.