big memory usage

[bburdette]

The first thing I did with this was run it on my phone photos folder, just to see what the output looked like. That's 9673 images. The result was my machine UI froze, ha. Last update from btop showed 19G memory usage from elbum (I have 32G), but hard to say what final memory usage was as the screen was frozen, mouse pointer wouldn't move, etc. Rebooted and ran on a smaller folder with fewer images, but still impressive memory usage of like 5.4G. So it works, but better for a curated gallery than a comprehensive catalog it seems.

[jerith666]

Yes, this is a known issue (to me anyway 😉) and my current biggest frustration with this project.

I've collected a bunch of blog posts and such about space leaks in Haskell, but have not managed to make any progress beyond that on fixing this -- I have no real experience with this kind of issue. 😦

There are two scripts in the src/utils/ directory, split-large-album.sh and full-year-rerun.sh, which I've used to work around the problem in my own albums.

You could try those on your phone photos folder, though fair warning that I have suffered some data loss with both of them in the past. (I've fixed all the data loss bugs I know of, but there may be more.)

[bburdette]

Space leaks and Haskell! It Is Known. I haven't done any haskell hacking in quite a while so not much help. Sprinkling some strictness annotations around might help?

[jerith666]

Yeah, I did try that a while ago. Hadn't bothered pushing it here since it didn't pan out, but you can see what I tried at https://github.com/jerith666/elbum/compare/gen-album-mem-leak if you're curious.