'cnmf_refit.estimates.detrend_df_f' in demo_pipeline.ipynb requires tons of memory?

[brian-arnold]

Please fill in the following for any issues

Your setup:
Repeatable on both my university's computing cluster and my Mac laptop

1. Operating System (Linux, MacOS, Windows):

• Linux and MacOS (14.2)

2. Hardware type (x86, ARM..) and RAM:

• Hardware: x86 and ARM
• RAM: 150GB+ on computing cluster, 96GB on Mac

3. Python Version (e.g. 3.9):

• 3.11.7

4. Caiman version (e.g. 1.9.12):

• 1.10.2

5. Which demo exhibits the problem (if applicable):

• demo_pipeline.ipynb, ran cell by cell without making any changes, using the data provided by the tutorial: download_demo('Sue_2x_3000_40_-46.tif')

6. How you installed Caiman (pure conda, conda + compile, colab, ..):

• conda, but for my MAC I had to additionally 'pip install tensorflow-macos' to get the following command to run:
  cnmf_refit.estimates.evaluate_components

7. Details:

Thank you for such a fabulous set of well-documented tools!

I'm running demo_pipeline.ipynb, using data provided by the demo, and all but one cell in the notebook works like a charm, both when I run the jupyter notebook (via VS Code) interactively on our computing cluster with lots of memory and my Mac, which also has a decent amount of memory.

However, every time I reach the cell containing the function that computes delta_f/f (using cnmf_refit.estimates.detrend_df_f), the kernel crashes. I suspect it's a memory issue because when I monitor memory consumption on our computing cluster using 'top', over the course of ~15 min memory usage gradually increases to ~200GB before it hits 0 and the kernel crashes. Likewise, on my Mac with 96GB of memory, after 15 min it says the application ran out of memory.

Are there any options or parameters I can change to help this? Or is the only solution to just get more memory? We have some large datasets we want to analyze, so I worry that if I'm hitting memory issues with the demo data that I will certainly encounter this issue on the data we're interested in.

Thanks for any help or advice!
Brian

[EricThomson]

Yikes that's not good. This is a relatively simple calculation that shouldn't require that much memory. Let me try to reproduce soon: we've started to adjust things around that method recently (see #1260 ). I should be able to get to it tomorrow.

[kushalkolar]

What if you give use_residuals=False ? The residuals are huge.

[brian-arnold]

I had use_residuals=False since it's specified that way in the demo_pipeline.ipynb notebook (although I see the default behavior is True).

[restrepd]

I was the person that filed nel-lab/mesmerize-core#276 in mesmerize. I gave use_residuals=False, and ran get_detrend_dfof and I still get the same problem. I agree that this is a simple calculation and should not require much memory leading to a crash. Here is a screenshot with nvtop. I suppose I could work around this by running the old matlab code exxtract_DF_F.m.

Thanks!

[kushalkolar]

@restrepd That's your GPU VRAM not your system RAM. Caiman doesn't use the GPU.

[restrepd]

@kushalkolar yes, the graph in nvtop is GPU and it is not used by caiman. But at the bottom it does list "CPU HOST MEM" (memsmerize is restrepd...). At any rate it crashed.

[EricThomson]

Yep this is #1262 . Ummm....avoid use_residuals=False right now this is the infinite while loop bug 😬 . I will start the fix on this today or asap this is pretty urgent. It has to do with some deprecations from scipy that we need to accommodate. Sorry about this. We are exposing things that have been hidden for a while in the dff calculation, so bugs are getting revealed and I need to fix them. Yikes I thought I had switched that in the demo notebook let me fix that I'm basically forcing the bug to run in the demo notebook 😨

Thanks for pushing on this! I'll fix the demo notebook asap. The underlying bug will be fixed as soon as I can.

[restrepd]

Thanks a bunch!

[EricThomson]

Ok I wasn't crazy I thought I had fixed it (with a hack): the workaround in the notebook to avoid the infinite while loop bug is to set flag_auto=False. If you have flag_auto=True and use_residuals=True then you will hit the bug.

I'll work on an actual longer-term solution, but I should mention that this flag is pretty important because it's holding things together with spit and paper currently basically. 😬

[restrepd]

Thanks! It works!

[EricThomson]

Just to clarify: it isn't taking tons of CPU load because it is a computationally demanding computation, but because you are sucked into an infinite while loop. I will work on fixing this my guess is within a week, so people can once again use flag_auto. Please see programmer time translation guide. 😄

[brian-arnold]

Also confirming that setting use_residuals=True and flag_auto=False makes everything work!

[EricThomson]

Just to clarify, as I was a bit murky above: it should work with either value of use_residuals as long as flag_auto is set to False.

[EricThomson]

This is fixed as of #1288 the different permutations of dff calculation should now work (I think it is merged into main so should be part of conda install now).

[restrepd]

Thanks! Diego Diego Restrepo, PhD Professor of Cell and Developmental Biology http://www.restrepolab.org/ University of Colorado Anschutz Medical Campus Department of Cell and Developmental Biology MS 8108 Bldg RC1 South, Room L18-11119 12801 E 17th Ave Aurora, CO 80045 Tel: 303-724-3405 Fax:303-724-3420 From: Eric Thomson ***@***.***> Date: Friday, March 22, 2024 at 10:22 AM To: flatironinstitute/CaImAn ***@***.***> Cc: Restrepo, Diego ***@***.***>, Mention ***@***.***> Subject: Re: [flatironinstitute/CaImAn] 'cnmf_refit.estimates.detrend_df_f' in demo_pipeline.ipynb requires tons of memory? (Issue #1274) [External Email - Use Caution] This is fixed as of #1288<#1288> the different permutations of dff calculation should now work (I think it is merged into main so should be part of conda install now). — Reply to this email directly, view it on GitHub<#1274 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A2FV7SB4VGHIXEFMD6J2NUDYZRLFDAVCNFSM6AAAAABDHDCEIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJVGQ2DQOBZGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>