Error in Building Coarse pmG1

[JunaidAhmed2520]

Hi neks, I was trying to replicate in NekRS, the case of channel flow as described in one of the Nek5000's tutorials and I am facing the following error message in the BUILDING COARSE pmG1 section:

================= INITIAL CONDITION ====================
XYZ   min/max: 0 0.2  0 0.005  0 0.5
U     min/max: 0.5 0.5  0 0  0 0
P     min/max: 0 0
S     min/max: 10 10
================= ELLIPTIC SETUP SCALAR00 ===============
bID 1 -> bcType codedFixedValue
bID 2 -> bcType zeroGradient
bID 3 -> bcType zeroGradient
bID 4 -> bcType codedFixedGradient
timing gs: 2.65e-03s 4.17e-03s 2.69e-03s 1.04e-02s 1.05e-02s 5.27e-03s 3.20e-03s 
used config: pw+hybrid (MPI min/max/avg: 1.16e-06s 1.32e-06s 1.24e-06s / avg bi-bw: 3.7GB/s/rank)
timing gs: 5.29e-04s 2.04e-03s 9.83e-03s 1.05e-02s 3.43e-03s 1.36e-03s 1.36e-03s 
used config: pw+device (MPI min/max/avg: 1.11e-05s 1.13e-05s 1.12e-05s / avg bi-bw: 0.4GB/s/rank)
testing Ax overlap 6.54e-03s 2.96e-03s (overlap enabled)
building Jacobi preconditioner ... done (0.00811798s)
done (1.21893s)
================ ELLIPTIC SETUP VELOCITY ================
bID 1 -> bcType codedFixedValue
bID 2 -> bcType fixedGradient
bID 3 -> bcType zeroNValue/zeroGradient
bID 4 -> bcType zeroValue
timing gs: 2.53e-03s 1.05e-02s 4.51e-03s 2.05e-03s 2.04e-03s 5.31e-03s 5.27e-03s 
used config: nbc+hybrid (MPI min/max/avg: 2.59e-06s 2.62e-06s 2.60e-06s / avg bi-bw: 5.3GB/s/rank)
timing gs: 5.12e-04s 2.95e-03s 2.73e-03s 1.05e-02s 9.92e-03s 2.89e-03s 1.41e-03s 
used config: pw+early+device (MPI min/max/avg: 1.13e-05s 1.15e-05s 1.14e-05s / avg bi-bw: 1.2GB/s/rank)
testing Ax overlap 8.28e-03s 1.67e-03s (overlap enabled)
building Jacobi preconditioner ... done (0.00756416s)
done (1.19086s)
================ ELLIPTIC SETUP PRESSURE ================
timing gs: 2.60e-03s 1.05e-02s 1.05e-02s 4.23e-03s 2.89e-03s 1.35e-03s 4.96e-03s 
used config: pw+device (MPI min/max/avg: 1.11e-05s 1.12e-05s 1.12e-05s / avg bi-bw: 0.3GB/s/rank)
timing gs: 2.47e-03s 9.89e-03s 6.99e-03s 2.71e-03s 8.58e-03s 5.28e-03s 5.27e-03s 
used config: pw+early+hybrid (MPI min/max/avg: 1.01e-06s 1.21e-06s 1.11e-06s / avg bi-bw: 3.1GB/s/rank)
testing Ax overlap 2.74e-03s 8.27e-03s 
building MG preconditioner ... 
============= BUILDING pMG7 ==================
timing gs: 8.68e-04s 2.22e-03s 2.03e-03s 9.86e-03s 1.05e-02s 1.07e-03s 1.35e-03s 
used config: pw+device (MPI min/max/avg: 1.10e-05s 1.11e-05s 1.10e-05s / avg bi-bw: 0.2GB/s/rank)
timing gs: 2.62e-03s 1.05e-02s 7.12e-03s 2.22e-03s 2.02e-03s 5.34e-03s 5.27e-03s 
used config: nbc+hybrid (MPI min/max/avg: 1.02e-06s 1.04e-06s 1.03e-06s / avg bi-bw: 2.5GB/s/rank)
timing gs: 6.42e-04s 1.05e-02s 1.01e-02s 7.43e-03s 3.15e-03s 1.38e-03s 5.23e-03s 
used config: pw+device (MPI min/max/avg: 1.10e-05s 1.11e-05s 1.10e-05s / avg bi-bw: 0.2GB/s/rank)
autotuning overlap in smoothSchwarz: 1.76e-02s 1.36e-02s (overlap enabled)
estimating maxEigenvalue ... 1.58957 done (0.287024s)
timing gs: 2.48e-03s 9.87e-03s 4.52e-03s 2.05e-03s 5.96e-03s 5.41e-03s 5.26e-03s 
used config: pw+early+hybrid (MPI min/max/avg: 9.32e-07s 1.10e-06s 1.02e-06s / avg bi-bw: 1.7GB/s/rank)
autotuning overlap in ellipticOperator: 2.74e-03s 8.27e-03s 
============= BUILDING pMG3 ==================
meshParallelGatherScatterSetup N=3
done (0.00480078s)
timing gs: 6.76e-04s 1.05e-02s 1.05e-02s 7.56e-03s 2.25e-03s 1.50e-03s 5.27e-03s 
used config: pw+device (MPI min/max/avg: 1.10e-05s 1.11e-05s 1.10e-05s / avg bi-bw: 0.0GB/s/rank)
timing gs: 2.65e-03s 1.05e-02s 5.21e-03s 2.69e-03s 1.05e-02s 4.92e-03s 4.94e-03s 
used config: pw+early+hybrid (MPI min/max/avg: 8.54e-07s 8.61e-07s 8.58e-07s / avg bi-bw: 0.7GB/s/rank)
timing gs: 2.61e-03s 1.05e-02s 1.05e-02s 6.07e-03s 2.88e-03s 1.01e-03s 4.93e-03s 
used config: pw+device (MPI min/max/avg: 1.08e-05s 1.08e-05s 1.08e-05s / avg bi-bw: 0.1GB/s/rank)
autotuning overlap in smoothSchwarz: 1.81e-02s 1.71e-02s (overlap enabled)
estimating maxEigenvalue ... 1.35961 done (0.283729s)
timing gs: 2.65e-03s 1.05e-02s 4.15e-03s 2.03e-03s 9.89e-03s 5.29e-03s 5.27e-03s 
used config: pw+early+hybrid (MPI min/max/avg: 7.78e-07s 8.29e-07s 8.03e-07s / avg bi-bw: 0.3GB/s/rank)
autotuning overlap in ellipticOperator: 2.94e-03s 7.96e-03s 
============= BUILDING COARSE pMG1 ==================
meshParallelGatherScatterSetup N=1
done (0.00670556s)
timing gs: 8.98e-07s 
used config: local
building FEM matrix ... done.
done (0.000733483s)
setup FEM solver ...

 Num MPI tasks = 2

 Num OpenMP threads = 1


BoomerAMG SETUP PARAMETERS:

 Max levels = 25
 Num levels = 1

 Strength Threshold = 0.250000
 Interpolation Truncation Factor = 0.000000
 Maximum Row Sum Threshold for Dependency Weakening = 0.900000

 Coarsening Type = PMIS 
 measures are determined locally


 No global partition option chosen.

 Interpolation = extended+i interpolation

Operator Matrix Information:

[gpu02:686983] *** Process received signal ***
[gpu02:686983] Signal: Floating point exception (8)
[gpu02:686983] Signal code: Integer divide-by-zero (1)
[gpu02:686983] Failing at address: 0x7ff66977bf63
[gpu02:686982] *** Process received signal ***
[gpu02:686982] Signal: Floating point exception (8)
[gpu02:686982] Signal code: Integer divide-by-zero (1)
[gpu02:686982] Failing at address: 0x7f1513ecff63
[gpu02:686983] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7ff669a59140]
[gpu02:686983] [ 1] /scratch/jahmed/Libraries/nekRS/lib/libnekrs-hypre.so(+0x77f63)[0x7ff66977bf63]
[gpu02:686983] [ 2] /scratch/jahmed/Libraries/nekRS/lib/libnekrs-hypre.so(+0x21760)[0x7ff669725760]
[gpu02:686983] [ 3] /scratch/jahmed/Libraries/nekRS/lib/libnekrs-hypre.so(_ZN12hypreWrapper11boomerAMG_tC1EiiPKxS2_PKdiP19ompi_communicator_tiiS4_i+0x8a3)[0x7ff669708533]
[gpu02:686983] [ 4] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_ZN10MGSolver_t13coarseLevel_t11setupSolverEPxiS1_S1_Pdb+0xee7)[0x7ff66a1c74a7]
[gpu02:686983] [ 5] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_Z22ellipticMultiGridSetupP10elliptic_tP8precon_t+0x1196)[0x7ff66a1a7da6]
[gpu02:686983] [ 6] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_Z27ellipticPreconditionerSetupP10elliptic_tP5ogs_t+0x273)[0x7ff66a1afab3]
[gpu02:686983] [ 7] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_Z18ellipticSolveSetupP10elliptic_t+0x2045)[0x7ff66a1bae95]
[gpu02:686983] [ 8] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_Z8nrsSetupP19ompi_communicator_tR9setupAideP5nrs_t+0x7c44)[0x7ff669facae4]
[gpu02:686983] [ 9] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tS1_iiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_iii+0x15bc)[0x7ff669f7836c]
[gpu02:686983] [10] nekrs(main+0xe3d)[0x562daa884aed]
[gpu02:686983] [11] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7ff669895d0a]
[gpu02:686983] [12] nekrs(_start+0x2a)[0x562daa8860fa]
[gpu02:686983] *** End of error message ***
[gpu02:686982] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f15141ad140]
[gpu02:686982] [ 1] /scratch/jahmed/Libraries/nekRS/lib/libnekrs-hypre.so(+0x77f63)[0x7f1513ecff63]
[gpu02:686982] [ 2] /scratch/jahmed/Libraries/nekRS/lib/libnekrs-hypre.so(+0x21760)[0x7f1513e79760]
[gpu02:686982] [ 3] /scratch/jahmed/Libraries/nekRS/lib/libnekrs-hypre.so(_ZN12hypreWrapper11boomerAMG_tC1EiiPKxS2_PKdiP19ompi_communicator_tiiS4_i+0x8a3)[0x7f1513e5c533]
[gpu02:686982] [ 4] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_ZN10MGSolver_t13coarseLevel_t11setupSolverEPxiS1_S1_Pdb+0xee7)[0x7f151491b4a7]
[gpu02:686982] [ 5] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_Z22ellipticMultiGridSetupP10elliptic_tP8precon_t+0x1196)[0x7f15148fbda6]
[gpu02:686982] [ 6] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_Z27ellipticPreconditionerSetupP10elliptic_tP5ogs_t+0x273)[0x7f1514903ab3]
[gpu02:686982] [ 7] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_Z18ellipticSolveSetupP10elliptic_t+0x2045)[0x7f151490ee95]
[gpu02:686982] [ 8] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_Z8nrsSetupP19ompi_communicator_tR9setupAideP5nrs_t+0x7c44)[0x7f1514700ae4]
[gpu02:686982] [ 9] /scratch/jahmed/Libraries/nekRS/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tS1_iiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_iii+0x15bc)[0x7f15146cc36c]
[gpu02:686982] [10] nekrs(main+0xe3d)[0x5587a64d4aed]
[gpu02:686982] [11] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f1513fe9d0a]
[gpu02:686982] [12] nekrs(_start+0x2a)[0x5587a64d60fa]
[gpu02:686982] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node gpu02 exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------

Could someone help me figure out how to debug this situation. I am happy to share any relevant information that might be required. Thanks in advance.

[stgeke]

Please try to reproduce it using the next branch.