Changes to mu_ij if statements for performance improvements

[pabloseleson]

Changes made to the force calculation for performance improvement.

Results below based on the KalthoffWinkler example with:

• std::array<int, 3> num_cell = { 400, 800, 36 };
• double delta = 0.00150000001;
  running on 1 node of Frontier with 8 GPUs.

For the PMB model:

Changes to the force calculation reduces:

• force calculation time from 313.90 s to 294.73 s (~6.1% reduction)
• communication time from 5.12 s to 3.82 s (~25.4% reduction)
• total time from 339.62 s to 316.45 s (~6.8% reduction)

For the LPS model:

(a) Changes to the weighted volume calculation reduces communication time from 383.48 s to 371.92 s (~3% reduction)
(b) Changes to the dilatation calculation in addition to (a) reduces communication time from 383.48 s to 350.14 s (~8.7 % reduction)
(c) Changes to the force calculation in addition to (a) and (b) reduces:

• force calculation time from 394.40 s to 374.19 s (~5.1% reduction)
• communication time from 383.48 s to 350.57 s (~8.6% reduction)
• total time from 801.35 s to 769.50 s (~4% reduction)

[streeve]

Also worth checking how much of a slowdown this is for 1 CPU node

[pabloseleson]

Performance change on 1 CPU-node of Frontier (56 CPUs) after the added force modifications:

For the PMB model:

• force calculation time : 1280.99 s -> 1258.08 s
• communication time: 6.42 s -> 15.70 s
• total time: 1344.85 s -> 1347.19 s

For the LPS model:

• force calculation time : 3649.69 s -> 3486.72 s
• communication time: 55.31 s -> 68.27 s
• total time: 3795.51 s -> 3688.12 s

[streeve]

Last changes