Erroneous values in the velocity field, possible issue with the parallelization?

[joneuhauser]

Dear NekRS community,

I'm testing a fairly simple CHT channel case on current master (4f87e0e), with MPT 2.26 and GCC 12.2.0 on HLRS HAWK. Occa Mode: serial. I'm unable to reproduce the issue on occa Mode: CUDA, but I also can't run on GPU with this amount of ranks.

For reference, the t-mesh has 35200 elements, the v-mesh has 28800 elements.

In the output (snapshot files, opened with paraview 5.10) I notice spurious values on some element corners. It kind of looks like a gather-scatter operation gone wrong:

(These values are emphasised during a "move" operation in Paraview, so they are pretty easy to spot)

The location and amount of these spurious values depend on the number of ranks. Here's a histogram of the streamwise component of the streamwise velocity gradient after 10 timesteps (all values should be in the (-10, 10) bin):

n = 128	n = 512

i.e. the number of spurious values roughly quadruples when increasing the number of ranks by a factor of 4, which would suggest an issue with the parallelisation.

Some additional observations:

• issue is not limited to CHT, I've had had it happen on non-CHT runs (no scalar) as well
• I've had this issue happen when restarting from a known good turbulent field computed on the same mesh. There it's pretty easy to spot (and abort the run) because the velocity in some nodes is 50% higher than the otherwise max velocity. This suggest that it's not my initial condition that causes the problem, and it's copied from the examples anyway
• changing the CFL number doesn't help
• changing the mesh resolution sometimes makes the issue disappear for a given case
• compared to "good" cases, parRSB complains: Warning: Lanczos reached a residual of 22.422759 (target: 0.000010) after 50 x 50 iterations in Level=0! - but as far as I understand it, that warning just tells you that the parallel efficiency goes down a bit, not that the results are incorrect.
• it's not an output problem, the values are also visible in e.g. platform->linalg.max
• it's not depending on the MPI implementation, also reproduced with openmpi 4.1.5

Here's my case for reference:
MWE_Channel.zip

Did anyone else encounter such an issue? What was your solution?

[joneuhauser]

Looks like I copied a bug from the examples. This loop ought to start with 0:

nekRS/examples/turbChannel/turbChannel.udf

Line 46 in 4f87e0e

for (int i = 1; i < mesh->Nlocal; i++) {

Then it's obvious why the number of spurious values scales with the number of ranks! I don't really understand why they don't disappear over time though. Anyway, the issue is probably solved?