How to release memory used by Group by after code execution is finished ?

[MHK107]

Hi There,

I have a project running on vaex v4.0.0, I also have it wrapped around flask to have API's running off it. I was hopping to get some help related to memory.

I do face memory leak issues while using group by here's an example.

df.groupby(['rooms_count'], agg={vx.agg.mean('price_per_meter'),vx.agg.min('price_per_meter'),vx.agg.max('price_per_meter'),vx.agg.count('price_per_meter')})

My issue is not with the amount of memory being used. But after the API call is executed the memory used is not released back to the OS. Scale it to multiple API requests and soon I am out of memory on server. I have tried using garbage collection but still the memory isn't released back to the OS.

I would really appreciate if I could get some insight on how to release that memory back once a request is finished. I would love to know if there's something I've been doing wrong which may have caused this issue ?

@maartenbreddels

[maartenbreddels]

Hi,

That is strange. You should not need to release memory. If you can isolate the problem, please open an issue with a reproducible example so we can look into this.

cheers,

Maarten

[MHK107]

Hi @maartenbreddels,

Hope you are doing well, Sorry took me some time.

With help of my data science team, I've been able to have a reproducible example ready for you. I've opened it up as an issue over here #2022

The link to my repo is mentioned in the issue itself which has the code and steps to replicate the issue I'm facing. Sharing the link to my repo here as well

https://github.com/MHK107/vaex-groupby-memory-issue