Run large cases

[Sameer-hui]

Hey guys,
I have recently started using NekRS and would like to run my own large-scale case (E > 350,000) on NekRS. What are the key considerations and specific modifications needed for such cases? Currently, I am encountering an issue related to HYPRE. My platform is HIP, with 8 nodes and 32 processes. Below are the detailed specifications.

log：
================ ELLIPTIC SETUP VELOCITY ================
bID 1 -> bcType codedFixedValue
bID 2 -> bcType fixedGradient
bID 3 -> bcType zeroValue
key: BLOCK SOLVER, value: TRUE
key: DISCRETIZATION, value: CONTINUOUS
key: ELLIPTIC COEFF FIELD, value: TRUE
key: ELLIPTIC PRECO COEFF FIELD,value: TRUE
key: INITIAL GUESS, value: EXTRAPOLATION
key: MAXIMUM ITERATIONS, value: 500
key: PRECONDITIONER, value: JACOBI
key: REGULARIZATION METHOD, value: NONE
key: SOLVER, value: PCG
key: SOLVER TOLERANCE, value: 9.9999999999999995e-07
key: STRESSFORMULATION, value: FALSE

  timing gs: 2.37e-03s 3.88e-03s 3.44e-03s 3.46e-03s 3.44e-03s 
  used config: nbc+hybrid (MPI min/max/avg: 3.43e-04s 4.90e-04s 4.29e-04s / avg bi-bw: 11.8GB/s/rank)
  timing gs: 4.34e-03s 4.76e-03s 5.45e-03s 5.50e-03s 5.45e-03s 
  used config: pw+host (MPI min/max/avg: 3.40e-04s 4.72e-04s 4.22e-04s / avg bi-bw: 12.0GB/s/rank)
  testing Ax overlap 5.75e-03s 5.16e-03s (overlap enabled)
  building Jacobi preconditioner ... done (0.040861s)
  done (0.907927s)
  ================ ELLIPTIC SETUP PRESSURE ================
  key: COARSE SOLVER,                                  value: BOOMERAMG
  key: COARSE SOLVER LOCATION,                         value: CPU
  key: COARSE SOLVER PRECISION,                        value: FP32
  key: DISCRETIZATION,                                 value: CONTINUOUS
  key: ELLIPTIC COEFF FIELD,                           value: FALSE
  key: ELLIPTIC PRECO COEFF FIELD,                     value: FALSE
  key: INITIAL GUESS,                                  value: PROJECTION-ACONJ
  key: MAXIMUM ITERATIONS,                             value: 500
  key: MGSOLVER CYCLE,                                 value: VCYCLE+MULTIPLICATIVE
  key: MULTIGRID CHEBYSHEV DEGREE,                     value: 3
  key: MULTIGRID CHEBYSHEV MAX EIGENVALUE BOUND FACTOR,value: 1.1
  key: MULTIGRID COARSE SOLVE,                         value: TRUE
  key: MULTIGRID SMOOTHER,                             value: FOURTHOPTCHEBYSHEV+ASM
  key: PRECONDITIONER,                                 value: MULTIGRID
  key: SOLVER,                                         value: PGMRES+FLEXIBLE
  key: SOLVER TOLERANCE,                               value: 1.0000000000000000e-04
  
  timing gs: 8.07e-04s 1.11e-03s 1.24e-03s 1.24e-03s 1.25e-03s 
  used config: pw+host (MPI min/max/avg: 1.05e-04s 1.67e-04s 1.43e-04s / avg bi-bw: 11.8GB/s/rank)
  timing gs: 1.60e-03s 1.81e-03s 2.03e-03s 2.02e-03s 2.04e-03s 
  used config: pw+host (MPI min/max/avg: 9.61e-05s 3.36e-04s 1.74e-04s / avg bi-bw: 9.7GB/s/rank)
  testing Ax overlap 2.04e-03s 2.00e-03s (overlap enabled)
  building MG preconditioner ... 
  ============= BUILDING pMG7 ==================
  timing gs: 4.55e-04s 7.38e-04s 7.50e-04s 7.41e-04s 7.52e-04s 
  used config: pw+host (MPI min/max/avg: 6.39e-05s 8.42e-05s 7.63e-05s / avg bi-bw: 11.0GB/s/rank)
  timing gs: 4.99e-04s 8.37e-04s 8.08e-04s 8.02e-04s 8.15e-04s 
  used config: pw+early+hybrid (MPI min/max/avg: 6.83e-05s 1.05e-04s 9.21e-05s / avg bi-bw: 11.8GB/s/rank)
  timing gs: 1.16e-03s 1.35e-03s 1.46e-03s 1.46e-03s 1.47e-03s 
  used config: pw+host (MPI min/max/avg: 7.46e-05s 1.14e-04s 9.43e-05s / avg bi-bw: 11.5GB/s/rank)
  autotuning overlap in smoothSchwarz: 3.69e-03s 3.68e-03s (overlap enabled)
  estimating maxEigenvalue ... 2.9381 done (0.342153s)
  timing gs: 8.46e-04s 1.02e-03s 1.16e-03s 1.15e-03s 1.17e-03s 
  used config: pw+host (MPI min/max/avg: 6.23e-05s 8.96e-05s 7.50e-05s / avg bi-bw: 11.2GB/s/rank)
  autotuning overlap in ellipticOperator: 1.44e-03s 1.12e-03s (overlap enabled)
  ============= BUILDING pMG3 ==================
  meshParallelGatherScatterSetup N=3
  done (0.00199926s)
  timing gs: 1.23e-04s 2.92e-04s 3.05e-04s 3.71e-04s 3.11e-04s 
  used config: pw+host (MPI min/max/avg: 2.55e-05s 3.99e-05s 3.33e-05s / avg bi-bw: 4.7GB/s/rank)
  timing gs: 1.46e-04s 3.67e-04s 3.38e-04s 3.33e-04s 3.46e-04s 
  used config: pw+early+hybrid (MPI min/max/avg: 2.94e-05s 4.15e-05s 3.67e-05s / avg bi-bw: 7.4GB/s/rank)
  timing gs: 2.59e-04s 3.93e-04s 4.50e-04s 4.46e-04s 4.58e-04s 
  used config: pw+host (MPI min/max/avg: 2.98e-05s 4.17e-05s 3.62e-05s / avg bi-bw: 7.5GB/s/rank)
  autotuning overlap in smoothSchwarz: 1.19e-03s 1.32e-03s 
  estimating maxEigenvalue ... 3.13551 done (0.0621785s)
  timing gs: 1.99e-04s 3.20e-04s 3.73e-04s 3.75e-04s 3.87e-04s 
  used config: pw+host (MPI min/max/avg: 2.95e-05s 3.87e-05s 3.32e-05s / avg bi-bw: 4.7GB/s/rank)
  autotuning overlap in ellipticOperator: 3.81e-04s 3.58e-04s (overlap enabled)
  ============= BUILDING COARSE pMG1 ==================
  meshParallelGatherScatterSetup N=1
  done (0.000524466s)
  timing gs: 6.29e-05s 1.73e-04s 1.97e-04s 1.90e-04s 1.97e-04s 
  used config: pw+host (MPI min/max/avg: 7.83e-06s 1.26e-05s 1.01e-05s / avg bi-bw: 1.8GB/s/rank)
  building FEM matrix ... done.
  done (0.105711s)

error：
*** Error in nekrs': free(): invalid next size (normal): 0x0000000016845980 *** *** Error in nekrs': free(): invalid next size (normal): 0x000000000f04b8e0 ***
*** Error in nekrs': free(): invalid next size (normal): 0x000000000febe940 *** *** Error in nekrs': free(): invalid next size (normal): 0x000000002f5557d0 ***
*** Error in `nekrs': malloc(): memory corruption: 0x000000001e03ae70 ***

I apologize if my expression is unclear. I would be happy to offer more details if necessary. Thank you for any guidance you can provide.

Yours sincerely,
sameer

[yslan]

Could be a memory issue.
What GPU are you using? and how large is the GPU memory?

350000*7*7*7/32=3.7M

which might be too large for 16GB but it should be ok for 32GB.

HYPRE sometimes has runtime failure during setup, which could be bypassed by running different number of MPI ranks.

I'd also recommend to make sure you can run multiple nodes job with NekRS examples first.

Hope this helps

[Sameer-hui]

First and foremost, thank you very much for your response.I've successfully run the NekRS example with 8 nodes and 32 processes. However, when attempting to run my own large-scale case (lelg=546048), I encountered the same issue with more processes, such as 16 nodes and 64 processes, 32 nodes and 128 processes, etc. Moreover, my GPU has 16GB of VRAM.

In reference to your statement about "HYPRE sometimes experiencing runtime failures during setup, which might be resolved by using a different number of MPI ranks," could you kindly provide more guidance on how to proceed?

Once again, I sincerely appreciate your assistance.

[Sameer-hui]

Moreover,when running a large-scale case, apart from using more processes to avoid memory issues, are there any other aspects I should pay attention to? For instance, file parameter settings. I suspect there might be something I haven't configured correctly.

[Sameer-hui]

I suspect there might be an issue with my input file, as the problem disappeared when I replaced the mesh with the turbPipe example. However, new issues have arisen. My proficiency in setting up input files is still limited, and I've only applied basic boundary conditions. Please bear with my simplistic setup, as I am currently in the learning phase of how to configure it more effectively.
.par

[OCCA]
backend = HIP

[GENERAL] 
verbose=true
polynomialOrder = 7
#startFrom = "restart.fld"
stopAt = endTime
endTime = 200 

dt = 6e-3
timeStepper = tombo2

[PRESSURE]
residualTol = 1e-04

[VELOCITY]
boundaryTypeMap = v,O,W
density = 1.0
viscosity = 1/19000
residualTol = 1e-06

[SCALAR00]
boundaryTypeMap = t,O,I

.oudf

void velocityDirichletConditions(bcData *bc)
{
  bc->u = 0.0;
  bc->v = 0.0;
  bc->w = 5.0;
}

void pressureDirichletConditions(bcData *bc)
{
  bc->p = 0.0;
}

void scalarDirichletConditions(bcData *bc)
{
  bc->s = 0;
}

Hope to receive your guidance.
sameer

[yslan]

Short answer, you might need more GPU.

I found a logfile from another case that has 277k elements.
Running it with 32 ranks will take 7.3GB memory on each GPU

occa memory usage: 7.31484 GB

For your E546k case, it's about double 14.6GB plus and minus some mesh dependency and so on. I will say you are very close to hit the 16GB memory bound you have. I'd suggest to try polynomialOrder=6 to see if this is indeed a memory issue.

As for the HYPRE setup, for my case, running with different number of MPI ranks can bypass the issue.
I don't think this is the main reason for you, but if you really want to rule our the potential HYPRE issue, you can run it without it by

[PRESSURE]
preconditioner = jac

Note that this is for debugging purpose and will slow doen the code with a poor convergence.

Hope this helps

[Sameer-hui]

Thank you for your response. I attempted to run on 64 or even more GPUs, but encountered the same issue. However, when I tried to run using several input files from turbPipe along with my own grid file, the problem disappeared, although new issues arose later. I suspect it may be an issue with my input files, and I am trying to address the problem from the perspective of input files.
Sincerely appreciate your assistance.