Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Inconsistencies and improvements to SST model #2329

Open
wants to merge 43 commits into
base: develop
Choose a base branch
from

Conversation

rois1995
Copy link
Contributor

@rois1995 rois1995 commented Jul 25, 2024

Proposed Changes

Hi everyone,

I have found some inconsistencies with respect to the literature on the implementation of the Menter's SST model. I would like to use this branch as test bench for any corrections/improvements made to the SST model.

Implementation errors found:

  • Use of production limiter constant in the clipping of the cross-diffusion term for the computation of the F1 blending function in CTurbSSTVariable.cpp.
  • SST naming conventions (V in V2003m should stay for Vorticity production term.). Not yet assessed.
  • Wrong cross-diffusion term included into the residual of w in turb_sources.hpp. It should not be only the positive value as considered in the computation of the F1 blending function. As stated in https://doi.org/10.2514/3.12149. "CDkw is the
    positive portion of the cross-diffusion term in Eq (A2)" pag. 1604. Moreover, a clipping has been introduced for large negative values of this term, as suggested in Peng, Shia-Hui, Peter Eliasson, and Lars Davidson. "Examination of the shear stress transport assumption with a low-Reynolds number k-omega model for aerodynamic flows." Eq 17.
  • Boundary conditions errors at inlets, following the Issue Turbulent Kinetic Energy(TKE) on energy equation in SST model. #1851. A fix has been proposed following @pcarruscag suggestions for the supersonic inlet BC.
  • Boundary conditions definitions are different than the ones proposed in TMR. Maybe give the user the choice to use the BCs implemented in SU2 or the ones from TMR.

Changes to SST model proposed:

  • Inclusion of non modified versions of SST. In this case I think that more work is needed to be sure that the correct terms are taken into account everywhere in the code.
  • Give the user the possibility of changing the production limiter for TKE (not essential).
  • Change values of default turbulent to laminar viscosity to 1e-2 (NASA TMR reports that it should be in the range of 1e-2 to 1e-5, but in SU2 this is set to 10 as default).

I've seen the proposed changes to the lower limits of k and w in #2323 and I tried implementing it in my branch. However, if the implementation proposed in the respective PR is preferred then I will change mine.

Related Work

#2323 #1851

PR Checklist

Put an X by all that apply. You can fill this out after submitting the PR. If you have any questions, don't hesitate to ask! We want to help. These are a guide for you to know what the reviewers will be looking for in your contribution.

  • I am submitting my contribution to the develop branch.
  • My contribution generates no new compiler warnings (try with --warnlevel=3 when using meson).
  • My contribution is commented and consistent with SU2 style (https://su2code.github.io/docs_v7/Style-Guide/).
  • I used the pre-commit hook to prevent dirty commits and used pre-commit run --all to format old commits.
  • I have added a test case that demonstrates my contribution, if necessary.
  • I have updated appropriate documentation (Tutorials, Docs Page, config_template.cpp), if necessary.

@rois1995 rois1995 changed the title Inconsistencies and improvements to SST model [WIP] Inconsistencies and improvements to SST model Jul 25, 2024
Comment on lines 859 to 860
nPrandtl_Lam, /*!< \brief Number of species
addDoubleOption("FREESTREAM_TURB2LAMVISCRATIO", TurbIntensityAndViscRatioFreeStream[1], 10.0); Prandtl number. */
Copy link
Contributor

@bigfooted bigfooted Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this intentionally commented, or...? If it is used, I guess this should go to line 872

Suggested change
nPrandtl_Lam, /*!< \brief Number of species
addDoubleOption("FREESTREAM_TURB2LAMVISCRATIO", TurbIntensityAndViscRatioFreeStream[1], 10.0); Prandtl number. */
nPrandtl_Lam, /*!< \brief Number of species laminar Prandtl number. */
addDoubleOption("FREESTREAM_TURB2LAMVISCRATIO", TurbIntensityAndViscRatioFreeStream[1], 10.0); /*!<\brief Freestream mu_turb to mu_lam viscosity ratio */

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a wrong copy-paste from me. It should not be there in the first place.

@bigfooted
Copy link
Contributor

Great contribution, Thanks @rois1995 !

@pcarruscag
Copy link
Member

If you are looking into robustness aspects too you should get in touch with @emaberman and @YairMO, seems like they have some good ideas and between the free time of 3 people a lot more can get done :)

@YairMO
Copy link

YairMO commented Jul 25, 2024

Hi,

Regarding the cross-diffusion term (CD) that appears in Omega source term (residual). The SST model (1994/2003) is a high-Reynolds-number model. Namely, It can not predict correctly the sub-layer region (especially the correct profile of the TKE). Therefore, only a positive contribution is required. Moreover, since the SST model was design as a k-w and k-epsilon blending, the CD term "belongs" only to the k-epsilon "branch", that is why the CD term include the factor "1-F1". However, it may happen, that the factor "1-F1" is not a 100% safe guarantee. It may happen that "1-F1" is not zero in region where the CD term is negative (this happen due numerical errors). To avoid such a situation, it is a good idea to clip the CD term with zero. Otherwise, severe numerical robustness issues may rise.
Yes, it is different from Menter publications, but I think that clipping the CD term with zero is completely inline with Menter original idea (that is why, I think, he was including the factor "1-F1". But again the 1-F1 factor is not 100% percent "safe").

- Given option for cross diffusion limiting in W residual
@YairMO
Copy link

YairMO commented Jul 26, 2024

Hi,

The use of an Omega production limiter (about the cross-diffusion term) is correct for low-Reynolds-number (LRN) models (the approach described by Peng et al. is very naive; there are other more rigorous treatments). For high-Reynolds-number (HRN) models, the clipping should be zero, keeping the cross-diffusion term positive; thus, the current implementation is correct.

Indeed, it is not exactly as it appears in Menter's original publication. The factor (1-F1) aimed to promise that the cross-diffusion term will be activated only outside the boundary layer, where it is positive (the cross-diffusion term switches its sign deep inside the boundary layer). This was also recognized by Peng et al. (first paragraph above Eq. 17). However, it may happen that the factor (1-F1)=1 where the cross-diffusion term is negative. Usually, it may happen at the wake, very near the airfoil trailing edge, where the upper and lower boundary layers merge. It is due to the imperfection of the F1 function.

To summarize, the current implementation is correct, and it is perfect for HRN models.

@YairMO
Copy link

YairMO commented Jul 26, 2024

For the sake of clarity, "current implementation" refers to the current treatment of the production code

@emaberman
Copy link

What YairMO is saying, is that allowing negative cross diffusion values is incorrect for high Reynolds models and should not be an option, this is a fix used for low Reynolds models only

@rois1995
Copy link
Contributor Author

rois1995 commented Jul 26, 2024

Hi @YairMO, Hi @emaberman ,

thank you very much for your comments. I haven't found any suggestion in literature to clip to only positive values the cross-diffusion term in the w-equation. I understand that it might be more robust, but it is not the standard implementation of the SST model, which is the first thing that we need to achieve. Only then we can build on top of that to improve the robustness of SU2.

Nevertheless, I tried the SWBLI test case and I compared the results across 6 different combinations:

1- develop branch, no changes
2- develop branch, changes to Supersonic_inlet profile as suggested in #1851
3- my branch, with original CDkw implementation (should give exactly the same result as develop+modified BC)
3- my branch, with original CDkw implementation and using boundary conditions from TMR
4- my branch, with original CDkw implementation and using your suggestions for lower limits for k and w.
5- my branch, allowing negative values of CDkw
6- my branch, allowing negative values of CDkw and using boundary conditions from TMR
7- my branch, allowing negative values of CDkw and using your suggestions for lower limits for k and w.
8- my branch, allowing negative values of CDkw, using boundary conditions from TMR and using your suggestions for lower limits for k and w.

When my branch is used, then the changes to the supersonic inlet BC are already in place.

I haven't achieved convergence with 1, 2 and 3. More precisely, 1 diverged right away (after 30 iterations), while 2 and 3 gave "FGMRES - Orthogonalization Failed" after 900ish iterations.

Here you can see the residuals for the different combinations.

OrigCDkw

NegCDkw

Unfortunately I will be busy with the AIAA Conference next week, thus I don't know how much I will be able to work on this. The next test case will be the 2D airfoil near-wake from TMR.

@YairMO
Copy link

YairMO commented Jul 26, 2024

Hi rois1995,

First of all, enjoy your time in Las Vegas. Any paper that you are presenting?

As for our discussion about the cross-diffusion term, I've emailed the "source" (Menter). I believe he will make it clear.
It may be that he will be able to answer only in a while ...

@rois1995
Copy link
Contributor Author

rois1995 commented Sep 10, 2024

I have tried running 7 consecutive and identical simulations with 4C10T and the residuals are not consistent. Moreover, Run 2 and 6 diverged respectively after 251 and 430 simulations. Here I am showing the residuals of the TKE (I know that NK does not apply to the turbulence solver but it is cleaner to visualize than the residual for the density)

NonDeterministicRMSk

I also tried going back a few commits right before changing the Reynolds stress computation and the situation is even worse.

NonDeterministicRMSk_PreviousCommit

Also tried with the develop branch and same non-consistency, even though every run here has diverged.

Here i summarize the convergence of these runs, 1e4 iterations means that it has "converged":

Run Iterations Iterations (old commit) Iterations (develop)
1 1e4 1e4 41
2 251 1193 34
3 1e4 1286 51
4 1e4 95 47
5 1e4 458 53
6 430 1e4 54
7 1e4 254 84

@pcarruscag
Copy link
Member

Are you trying to reproduce these results https://su2code.github.io/vandv/swbli/? or using your own meshes?

@rois1995
Copy link
Contributor Author

I am trying with the .geo file used to create the meshes in the V&V folder. However, I am trying with a finer mesh, with a mesh sizing factor of f = 3.9762. I also tried the develop branch and, even though it always diverges, also there the iterations done are different. I have updated the previous comment.

With the coarser meshes I might have the same problems, but convergence is easier thus I think that I might need more trials. I'll start running some simulation with the 3rd level grid.

@pcarruscag
Copy link
Member

Are you initializing from coarser results? I just checked that the L3 mesh still works fine in develop initialized from L2 results.
The NK solver has some non determinism when using threads especially if the convergence is rough. Using a lot of cores also makes the ILU preconditioner less effective, so you may need to tweak linear solver iterations or CFL to compensate. For reference when I ran the case originally I had a 4 core laptop xD

@rois1995
Copy link
Contributor Author

rois1995 commented Sep 10, 2024

Every simulation starts from zero. I also tried the third grid level and these are the results after seven runs

NonDeterministicRMSk_Level3

Runs 2, 6 and 7 diverge at different iterations while Run 4 converges to the desired residual value (log(RMS(Res_rho)) < -11.5). Running on 4 cores and 10 threads is a problem in this case? Or has the same problems of using 40 cores?

@pcarruscag
Copy link
Member

The ILU issue depends on number of cores regardless of MPI/OpenMP split.
The non-determinism of the NK solver should be worse with more threads.
I recommend initializing from the coarse results.

@rois1995
Copy link
Contributor Author

I tried a couple of things:

  • with 20C2T and the results are worse: 4 out of 7 runs diverged, one right away (84 iterations) and the others after more than 1000 iterations.

  • restarting the simulation from the level 3 mesh didn't change the results: 2 out of 7 simulations diverged.

  • running without NK improved a quite a lot, but the inconsistencies remain. Not only that, but something strange can be seen looking at the first hundreds of iterations: the even runs are on top of each other and the same happens for the odd ones.

No Newton-Krylov
NonDeterministicRMSk_NoNK

No Newton-Krylov Zoomed
NonDeterministicRMSk_NoNKZoom

The peak in the residuals that you can see around 3k iterations is different for each run, and leads to slightly different residuals after 10k iterations.

  • Running with 40C1T causes divergence of the simulation at the same iteration for every run (using NK), and in this case the residuals are on top of each other.

I will try having a look into the atomic operations.

@pcarruscag
Copy link
Member

There are two sources for those, atomicAdd which is used in dot products with CSysVectors (so it affects the linear solvers). And in the NK solver to compute the norm of the solution.
The other less obvious source is SU2_OMP_CRITICAL, critical sections do not have a guaranteed ordering. We use these to compute the RMS residuals, and those influence the CFL adaptation.
AFAIK the only way around this is to store the result of each thread into a vector visible by all threads, and then have a single thread compute the total.

@rois1995
Copy link
Contributor Author

I can try having a look into this, but I am not sure of how much help I can be. Just from a quick look at the code I can see that the CSYSVEC_PARFOR has a NOWAIT instuction, whereas the SU2_OMP_FOR_STAT doesn't, and these are used right before the atomicAdd call respectively in the CSysVector and in the CNewtonIntegration. But I doubt that's the main cause of large discrepancies when using NK.

@pcarruscag
Copy link
Member

I think NK is just more sensitive because the linear system uses a nastier Jacobian matrix.

@rois1995
Copy link
Contributor Author

Then it's kind of difficult to evaluate the improvements of this branch. When comparing with the develop branch, the simulations with NK at least do not straight up diverge. When not using the NK solver, the residuals kind of improve, mostly when OMP is not used. These are the results on the mesh level 3.

MyBranchVsDevelop_ResRho

MyBranchVsDevelop_ResTKE

MyBranchVsDevelop_ResW

MyBranchVsDevelop_LogAVGCFL

There are some spikes in the residuals, which strangely enough coincide for rho and w but have opposite direction.

When OMP is not used, the develop branch stalls the turbulence residuals, whereas this branch keep on decreasing them.

However, changing the number of cores used, the develop branch improves, while this branch does not.

MyBranchVsDevelop_ResRho_Altri

MyBranchVsDevelop_ResTKE_Altri

The main differences between the develop and this branch are:

  • use of TMR boundary conditions
  • correction to Reynolds stresses computation
  • some corrections to SST model

@rois1995
Copy link
Contributor Author

rois1995 commented Sep 11, 2024

Having a better look into the comparison between this branch and the develop one for the simulation with 10C and 1T.

When looking at the solution, the residuals actually improve with this branch everywhere except at the bottom symmetry BC. Don't really know why, I am checking all of the changes to check.

ComparisonAbsResRho_MyBranchVsDevelop

And, even more, the Mach number seems to be wrong on the bottom symmetry, but not on the upper one.

UPDATE:

The correction to the Supersonic_Inlet_BC works just fine, but some how it screws up the Mach on the symmetry (and I do not why bu only at the bottom one). It might be related to #1851, thus I am looking into including the tke in the computation of the sound speed.

@rois1995
Copy link
Contributor Author

I have started implementing the computation of the speed of sound with the tke. Just to be sure that I am following the right path, I have pushed some changes. With these changes the linear solver diverges after 39 inner iterations, don't really know why. Any support is welcome.

@@ -124,7 +135,8 @@ FORCEINLINE CRoeVariables<nDim> roeAveragedVariables(Double gamma,
roeAvg.velocity(iDim) = (R*V.j.velocity(iDim) + V.i.velocity(iDim)) * D;
}
roeAvg.enthalpy = (R*V.j.enthalpy() + V.i.enthalpy()) * D;
roeAvg.speedSound = sqrt((gamma-1) * (roeAvg.enthalpy - 0.5*squaredNorm(roeAvg.velocity)));
roeAvg.tke = (R*V.j.tke() + V.i.tke()) * D;
roeAvg.speedSound = sqrt((gamma-1) * (roeAvg.enthalpy - 0.5*squaredNorm(roeAvg.velocity) - roeAvg.tke));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is enthalpy being computed with tke? If not and then you subtract tke here...
Also please control the scope of this PR, you have another 3 work in progress PRs that took time to review and now are just sitting.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, the enthalpy is computed just as E-P/rho, thus the kinetic energy should not be involved here. Regarding the other WIPs, they all involve the SST model (except for the recent SA one, but that should be finished), thus I think they should wait for this branch first.

Comment on lines +116 to +117
// const auto& gradientsTurb = turbSolution.GetGradient_Reconstruction();
// const auto& limitersTurb = turbSolution.GetLimiter_Primitive();

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
const auto iPoint = geometry.edges->GetNode(iEdge,0);
const auto jPoint = geometry.edges->GetNode(iEdge,1);

// cout << gatherVariables(iPoint, turbVars->GetSolution()) << endl;

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
@rois1995
Copy link
Contributor Author

As of now the problem is that the supersonic inlet has the correct values of Mach number (5) but on the symmetry BC this does not occur. It seems that on the symmetry BC the problem is the same as it was for the supersonic inlet before, but I still have to find a way to fix this. Moreover, this only happens to the lower symmetry BC, whereas at the top one the Mach is correct.

@YairMO
Copy link

YairMO commented Sep 13, 2024

Maybe I'm mixing it up, but I remember there was an issue with the wrong treatment (I'm not sure exactly what it was, perhaps gradient treatment) at Symmetry BC.

@rois1995
Copy link
Contributor Author

rois1995 commented Sep 13, 2024

I will look into this, but what I found is that the velocity component normal to the bottom symmetry BC is non-zero in this branch.

@YairMO
Copy link

YairMO commented Sep 13, 2024

I think this was one of the problems mentioned

@rois1995
Copy link
Contributor Author

rois1995 commented Sep 13, 2024

Yeah but this does not happen on the results with the develop branch and I have pulled the latest develop commits into this branch, thus it might not be the cause but the consequence of something else.

After the first iteration there is a vertical velocity at the bottom corner between the supersonic inlet and the symmetry BC. Maybe it can be related to #2285

VerticalVelocityFirst

@rois1995
Copy link
Contributor Author

rois1995 commented Sep 16, 2024

Part of the problem was that I was considering the TKE in the computation of the boundary conditions but not inside the domain since I was using the modified version of the SST model. With this, the vertical velocity at the inlet is in the range of 10^-12. The problem on the bottom symmetry still remains though. I am trying on the coarser mesh and I et FGMRES divergence after more than 1.3k iterations.

I don't know if I should also change all the boundary conditions code to take the TKE from the solver rather than from the config. I guess that it should be done.

@@ -801,7 +804,8 @@
bool viscous = config->GetViscous();
bool gravity = config->GetGravityForce();
bool turbulent = (config->GetKind_Turb_Model() != TURB_MODEL::NONE);
bool tkeNeeded = (turbulent && config->GetKind_Turb_Model() == TURB_MODEL::SST);
bool tkeNeeded = (turbulent && config->GetKind_Turb_Model() == TURB_MODEL::SST && !(config->GetSSTParsedOptions().modified));
// bool tkeNeeded = (turbulent && config->GetKind_Turb_Model() == TURB_MODEL::SST);

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
@@ -5137,7 +5175,8 @@
Density_e = GetFluidModel()->GetDensity();
StaticEnergy_e = GetFluidModel()->GetStaticEnergy();
Energy_e = StaticEnergy_e + 0.5 * Velocity2_e;
if (tkeNeeded) Energy_e += GetTke_Inf();
// if (tkeNeeded) Energy_e += GetTke_Inf();

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
@@ -5164,7 +5203,8 @@
Density_e = GetFluidModel()->GetDensity();
StaticEnergy_e = GetFluidModel()->GetStaticEnergy();
Energy_e = StaticEnergy_e + 0.5 * Velocity2_e;
if (tkeNeeded) Energy_e += GetTke_Inf();
// if (tkeNeeded) Energy_e += GetTke_Inf();

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
@@ -7258,7 +7302,8 @@
Velocity2 += Velocity[iDim]*Velocity[iDim];
}
Energy = P_Exit/(Density*Gamma_Minus_One) + 0.5*Velocity2;
if (tkeNeeded) Energy += GetTke_Inf();
// if (tkeNeeded) Energy += GetTke_Inf();

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants