output of memory requirements

[turbulentphil]

Dear Neks,

I've been using Nek5000 (v17) and want to switch to NekRS (v23).

Background to my below question
In Nek5000, makenek provides at its end information about the memory usage per MPI task. This allows to adjust the number of tasks prior to submitting a full batch job.

In NekRS, I found the option to pre-compile a job for a given (maximum) number of MPI tasks. Unfortunately, this does not output the resulting memory usage. Instead, I can submit a batch job to run the simulation and find an output of occa memory usage: X.XXXXXX GB before the time stepping begins.
Unfortunately, there are two issues with this:

1. This output results only if the device memory is sufficient.
2. There is no information on the memory requirements on the host. This might become more important when running NekRS with --backend CPU where the OCCA memory usage is close to zero.

Question
Is there any chance to know/output the memory requirements of a simulation in advance of a successful simulation (i.e., with enough memory)?
Printing out the projected OCCA memory usage per task at the end of the pre-compilation (assuming that the final/production number of MPI tasks will coincide with the specified maximum number of tasks in the pre-compilation) would already be a first helpful step.

Relevance of the question
Knowing the memory requirements is a frequent need, for instance when (i) starting a totally new simulation or (ii) adjusting the polynomial order / number of spectral elements over different subsequent chain jobs.
The need for knowing the memory requirements came also already up in other issues such as #261, #279, #537.

I would appreciate any help or advice.

Best,
Philipp

[stgeke]

Estimating memory usage is complex and depends on various factors, such as solver settings and the number of MPI ranks. Additionally, there might be two different memory spaces: device and host. Keep in mind that besides the memory consumed by nekRS itself, several other components such as MPI, GSLIB, nek5000 (required for legacy support), and HYPRE also allocate memory.

As a rule of thumb, a good starting point is to assume 1GB of device memory and 0.2GB of host memory per 1000 elements (using N=7).

[turbulentphil]

Dear Stefan,

thanks a lot for your answer. This rule of thumb is already a good point to start!

I understand that estimating memory consumption can be complicated. Maybe there is a way to anyway include the output of a rough estimation (maybe even of a lower limit or so) in a future build. Even if this estimation wasn't perfect, I am sure the community/user would benefit from it.

Cheers,
Philipp

[turbulentphil]

Meanwhile I performed some test RBC simulations with N=7 and 640,000 spectral elements on JUWELS Booster and Cluster. I would like to share the results since this can be of help for others that come across the same question.

CUDA backend: Running the simulation on 4 nodes on JUWELS Booster required in total 551 GB device and 476 GB host memory.
CPU backend: Running the identical simulation again on 16 nodes of JUWELS Cluster required in total 1,472 GB memory.

This breaks down to
0.9 GB of device memory and 0.8 GB of host memory (CUDA backend), or
2.3 GB memory usage (CPU backend)
per 1,000 spectral elements at N=7.

[stgeke]

How did you obtain these memory figures?On 11 Jul 2024, at 20:08, turbulentphil ***@***.***> wrote: Meanwhile I performed some test RBC simulations with N=7 and 640,000 spectral elements on JUWELS Booster and Cluster. I would like to share the results since this can be of help for others that come across the same question. CUDA backend: Running the simulation on 4 nodes on JUWELS Booster required in total 551 GB device and 476 GB host memory. CPU backend: Running the identical simulation again on 16 nodes of JUWELS Cluster required in total 1,472 GB memory. This breaks down to 0.9 GB of device memory and 0.8 GB of host memory (CUDA backend), or 2.3 GB memory usage per 1,000 spectral elements (CPU backend) per 1,000 spectral elements at N=7. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[turbulentphil]

These numbers are based on the job reports for the two individual jobs on JUWELS Booster and Cluster, so they are not generated by NekRS but by the system instead.