Reduce Ingester memory usage while uploading blocks to long-term storage

[mglaub]

hi i see very high ram usage from the ingester pods on startup.
They are regularly between 20-30GB:

NAME CPU(cores) MEMORY(bytes)
mimir-ingester-0 441m 19227Mi
mimir-ingester-1 43m 19232Mi
mimir-ingester-2 11m 7655Mi

NAME CPU(cores) MEMORY(bytes)
mimir-ingester-0 2329m 30102Mi
mimir-ingester-1 1349m 30663Mi
mimir-ingester-2 1552m 16556Mi

In the ingester logs i can see that they are uploading blocks to the remote storage. I attached one ingester logfile so you can have a look yourself.

ingester-log.txt

After the ingesters are done uploading the blocks the ram usage dropps dramatically:

NAME CPU(cores) MEMORY(bytes)
mimir-ingester-0 6m 3573Mi
mimir-ingester-1 22m 5563Mi
mimir-ingester-2 8m 1278Mi

I see that the mimir-ingester-2 tends to have lower RAM usage. I just deployed it today. So it does not have that much data in its pvc yet. so the obvious solution seems to be lower retention time -> lower ram usage on startup.

Is there any other way where i can keep my rentention period but still have lower ram usage on startup?

if you need any more information i will happily provide it.

Cheers
Marcel