Documenting features and roadmap (#1215)

---------

Co-authored-by: Michael Kuhn <[email protected]>

README.md

@@ -56,7 +56,7 @@ modern C++ compiler that supports the C++17 standard. Users wishing to execute
 the code on high-performance computing (HPC) systems will also need MPI
 libraries installed on their system. The code can also be compiled using MPI+X, 
 where X can be OpenMP for CPU shared memory parallelism,
-CUDA to target NVIDIA GPUs, HIP for AMD GPUs, or DPC++ for Intel GPUs.
+CUDA to target NVIDIA GPUs, ROCM for AMD GPUs, or SyCL for Intel GPUs.
 
 ### Contributing, reporting bugs, and requesting help
 

docs/sphinx/index.rst

@@ -26,9 +26,10 @@ development effort. As with other codes in the
 `Exawind <https://github.com/exawind>`_ ecosystem, AMR-wind shares the following
 objectives:
 
-- an open, well-documented implementation of the state-of-the-art computational
-  models for modeling wind farm flow physics at various fidelities that are
-  backed by a comprehensive verification and validation (V&V) process;
+- an open, well-documented implementation of the state-of-the-art
+  computational models for modeling wind farm flow physics at various
+  fidelities that are backed by a comprehensive verification and
+  validation (V&V) process (:ref:`capabilities`);
 
 - be capable of performing the highest-fidelity simulations of flowfields within
   wind farms; and

docs/sphinx/theory/theory.rst

@@ -1,8 +1,10 @@
 Theory Manual
-================================
+=============
+
+.. _governing_equations:
 
 Governing equations
-------------------------------------
+-------------------
 
 Conservation of fluid mass:
 
@@ -21,11 +23,19 @@ Tracer(s) advection:
 
 .. math:: \frac{\partial \rho s}{\partial t} + \nabla \cdot (\rho U s)  = 0
 
-Discretization 
--------------------------------------
-The numerical methdology used to solve the partial differential equations (PDEs)
-within AMR-Wind is documented in `Almgren et al. (JCP 1998)
-<https://ccse.lbl.gov/Publications/almgren/abchw.pdf>`_.
+.. _discretization:
+
+Discretization
+--------------
+
+The numerical methdology used to solve the partial differential
+equations (PDEs) within AMR-Wind is documented in `Almgren et
+al. (JCP 1998)
+<https://ccse.lbl.gov/Publications/almgren/abchw.pdf>`_. AMR-Wind uses
+`AMReX-Hydro
+<https://amrex-fluids.github.io/amrex-hydro/docs_html/Schemes.html>`_
+for many advection routines. The reader is referred to their
+documentation for implementation details.
 
 Time Step -- MOL
 ~~~~~~~~~~~~~~~~
@@ -135,8 +145,10 @@ Often for simulations involving walls, (e.g., channel flows, complex terrains et
 
    mapping.rst
 
+.. _multiphase:
+
 Multiphase flow modelling
-------------------------------------
+-------------------------
 
 AMR-Wind employs the volume-of-fluid method for simulating two-phase (water-air) flows. 
 More specifically, the volume fraction field is advected explicitly using a
@@ -148,8 +160,10 @@ is not modeled by AMR-Wind currently. For further detail, see
 `Kuhn, Deskos, Sprague (Computers & Fluids 2023)
 <https://doi.org/10.1016/j.compfluid.2022.105770>`_.
 
+.. _source_terms:
+
 Source terms
-------------------------------------
+------------
 
 Gravity Forcing
 ~~~~~~~~~~~~~~~~
@@ -174,6 +188,7 @@ Using the perturbational form implies that the hydrostatic pressure is removed f
 
 .. math:: p = p' - \int_z^{z_{max}} \rho_0 g dz + p(z = z_{max}) = p' - \int_z^{z_{max}} \rho_0 g dz
 
+.. _mesoscale_forcing:
 
 Mesoscale Forcing
 ~~~~~~~~~~~~~~~~~
@@ -196,17 +211,12 @@ approach can be used with NWP model output or observational data. For more
 information, see `Allaerts et al. (BLM 2020)
 <https://doi.org/10.1007/s10546-020-00538-5>`_
 
-The application of these forcing approaches is detailed in:
-
-.. toctree::
-   :glob:
-   :maxdepth: 2
-
-   ../user/inputs_ABL_meso_forcing.rst
+The application of these forcing approaches is detailed :ref:`here <inputs_meso_forcing>`.
 
+.. _turbulence:
 
 Turbulence Models
------------------------------
+-----------------
 
 LES models for subgrid scales
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -306,11 +316,13 @@ length scale is modified as follows
 Here the term :math:`H=1.5 dz` specifies the location at which the length scale switches to :math:`L=C_s\Delta` and :math:`\phi_M`
 is the atmospheric stability function. Currently, the implementation for the stability function uses a single global value. 
 The implementation of the non-linear model is split into two parts. The subgrid-scale viscosity term is directly used 
-within the ``AMR-wind`` diffusion framework. The last two terms in :math:`M_{ij}` are added as source-terms in the momentum equation. 
+within the ``AMR-Wind`` diffusion framework. The last two terms in :math:`M_{ij}` are added as source-terms in the momentum equation. 
+
+.. _wall_models:
 
 Wall models
 -----------
-The wall models described in this section are implemented in ``AMR-wind`` for
+The wall models described in this section are implemented in ``AMR-Wind`` for
 running wall-bounded flows (non-ABL cases).
 
 Log-law wall model
@@ -329,7 +341,7 @@ Given a horizontal velocity magnitude
 :math:`z = z_{\mathrm{ref}}`, :math:`u_\tau` can be computed using a
 non-linear solve to satisfy `[eq:loglaw] <#eq:loglaw>`__.
 
-In ``AMR-wind`` Newton-Raphson iterations are used with a convergence
+In ``AMR-Wind`` Newton-Raphson iterations are used with a convergence
 criterion of :math:`\lvert u_\tau^{n+1} - u_\tau^n \rvert < 10^{-5}`.
 For this, derivative of
 :math:`\frac{\partial u_{\mathrm{mag}}}{\partial {u_\tau}}` is used,

docs/sphinx/user/build.rst

@@ -28,7 +28,11 @@ your system. The main dependencies are listed below:
 
 #. `Intel OneAPI
    <https://software.intel.com/content/www/us/en/develop/tools/oneapi.html>`_
-   for building with DPC++.
+   for building with SyCL.
+
+#. `AMD GPU
+   <https://www.amd.com/en/products/software/rocm.html>`_
+   for building with ROCM.
 
 Building from source
 --------------------
@@ -84,6 +88,10 @@ Architecture options
 
    Enable `Intel OneAPI SyCL <https://software.intel.com/content/www/us/en/develop/tools/oneapi.html>`_ builds. Default: OFF
 
+.. cmakeval:: AMR_WIND_ENABLE_ROCM
+
+   Enable `AMD ROCM GPU <https://www.amd.com/en/products/software/rocm.html>`_ builds. Default: OFF
+
 Dependencies
 ~~~~~~~~~~~~~
 
@@ -135,13 +143,6 @@ Other AMR-Wind specific options
 
    Enable compiler warnings during build. Default: OFF
 
-.. cmakeval:: AMR_WIND_ENABLE_FORTRAN
-
-   Build Fortran interfaces. Default: OFF
-
-   AMR-Wind itself has no Fortran code. This option is unused if
-   :cmakeval:`AMR_WIND_USE_INTERNAL_AMREX` is set to ``OFF``.
-
 General CMake options
 ~~~~~~~~~~~~~~~~~~~~~~
 

docs/sphinx/user/features.rst

@@ -0,0 +1,186 @@
+.. _capabilities:
+
+Capabilities and Roadmap
+========================
+
+This section documents a non-exhaustive list of current AMR-Wind
+capabilities and roadmap for future capabilities.
+
+.. tip::
+
+   If your project relies on a capability that is not yet present in
+   AMR-Wind, please create an issue on the code project page.
+
+
+.. note::
+
+   This reflects the capabilities for AMR-Wind version 2.1.0 and above.
+
+
+Capabilities
+------------
+
+.. tip::
+
+   The capabilities are linked to the relevant input file references
+   (keyword `inp`) and documentation (keyword `doc`). Searching for
+   those keywords in the `test/test_files` directory will give
+   concrete examples of the feature usage.
+
+
+Methods and models
+~~~~~~~~~~~~~~~~~~
+
+* Numerical methods
+
+   * Advection: second order, piecewise parabolic, piecewise linear, WENO, Bell-Dawson-Shubin [:ref:`doc <discretization>`, :ref:`inp <inputs_incflo_advection>`]
+
+   * Diffusion: second order, explicit, Crank-Nicolson, and implicit [:ref:`doc <discretization>`, :ref:`inp <inputs_incflo_diffusion>`]
+
+   * Mesh refinement: static refinement for specified regions [:ref:`inp <inputs_static_refinement>`], adaptive mesh refinement [:ref:`inp <inputs_amr>`] (e.g., field based, curvature, q-criterion, vorticity [:ref:`inp <inputs_tagging>`])
+
+   * Mesh mapping for non-uniform cartesian grids [:ref:`doc <mapping>`, :ref:`inp <inputs_geometry>`]
+
+* Equations systems
+
+   * Incompressible and low Mach formulations of Navier-Stokes :ref:`[doc] <governing_equations>`
+
+   * Temperature
+
+   * Level set
+
+   * Subgrid scale kinetic energy :ref:`[doc] <turbulence>`
+
+   * Specific dissipation rate :ref:`[doc] <turbulence>`
+
+   * Passive scalar
+
+   * Source terms for these PDEs [:ref:`doc <source_terms>`, :ref:`inp <inputs_momentum_sources>`]
+
+* Turbulence modeling
+
+   * Large Eddy Simulation: constant Smagorinsky,  AMD, one equation :math:`k_{sgs}`, Kosovic [:ref:`doc <turbulence>`, :ref:`inp <inputs_turbulence>`]
+
+   * Wall models: log-law, constant stress, Schumann [:ref:`doc <wall_models>`, :ref:`inp <inputs_abl>`]
+
+   * Reynolds-Average Navier-Stokes: :math:`k`-:math:`\omega` SST (and IDDES variant) [:ref:`doc <turbulence>`, :ref:`inp <inputs_turbulence>`]
+
+* Transport models
+
+   * Constant transport coefficients [:ref:`inp <inputs_transport>`]
+
+   * Two phase transport (separate coefficients for each material) [:ref:`inp <inputs_transport>`]
+
+Flow physics
+~~~~~~~~~~~~
+
+* Wind energy physics
+
+   * Atmospheric boundary layer (ABL): various stability states (stable, unstable, neutral), precursor simulations with inflow boundary planes for wind farm simulations, anelastic formulation, mesoscale forcing, geostrophic forcing, Coriolis forcing, Monin-Obukhov similarity theory, gravity forcing, gravity wave damping [:ref:`inp <inputs_abl>`]
+
+   * Actuator turbine representations: Joukowsky disks, uniform disks, actuator line [:ref:`inp <inputs_actuator>`]
+
+   * Coupling with OpenFAST
+
+   * Coupling with Nalu-Wind for blade resolved simulations
+
+* Multiphase flows [:ref:`doc <multiphase>`]
+
+   * Prescribed flow cases for verification of volume-of-fluid transport: Zalesak disk, vortex patch
+
+   * Prescribed flow cases for verification of momentum equation coupled to volume-of-fluid transport: Zalesak disk scalar vel, vortex patch scalar vel
+
+   * Validation and demonstration cases: sloshing tank, dam break, breaking waves, falling or inertial droplet
+
+   * Methods to initialize volume-of-fluid field from an initial levelset field
+
+   * Monitors conservation of mass and momentum
+
+* Ocean wave forcing (for multiphase flows) [:ref:`inp <inputs_ocean_waves>`]
+
+   * Wave types: linear (monochromatic), Stokes (2nd- to 5th-order), irregular (input by modes files from HOS-Ocean)
+
+   * Relaxation zones force wave profile to generate waves at lower x boundary or force toward quiescent flat interface at upper x boundary. Wave profile can also be enforced (instead of numerical beach) at upper x boundary for periodic simulations.
+
+* Boundary conditions
+
+   * Periodic, outflow, inflow, walls, user-defined inflows [:ref:`inp <inputs_boundary_conditions>`]
+
+   * Wall models (e.g., wall functions, stress) [:ref:`doc <wall_models>`, :ref:`inp <inputs_abl>`]
+
+   * Inflow planes from precursor simulations [:ref:`doc <amrwind-abl-bndry-io>`, :ref:`inp <inputs_abl>`]
+
+   * Mesoscale forcing [:ref:`doc <mesoscale_forcing>`, :ref:`inp <inputs_meso_forcing>`]
+
+   * Synthetic turbulence [:ref:`inp <inputs_synthetic_turbulence>`]
+
+* Geometry
+
+   * Immersed boundary
+
+   * Coupling with Nalu-Wind for body-conforming meshes with overset methodology
+
+* Miscellaneous cases
+
+  * Verification and validation cases: method of manufactured solutions, convecting Taylor-Vortex, Rayleigh-Taylor, passive scalar, Burggraf flow, channel flow, Ekman spiral, vortex dipole, vortex ring
+
+* Postprocessing
+
+   * Visualization with VisIt, Paraview, yt
+
+   * Sampling of fields with planes, point probes, lines, volumes, lidar, and radar [:ref:`doc <post_processing>`, :ref:`inp <inputs_sampling>`]
+
+   * Sampling of fields at probes that follow free surface of liquid-gas flows [:ref:`inp <inputs_sampling_freesurface_sampler>`]
+
+   * Scalar outputs such as kinetic energy, enstrophy, total wave energy, and norms [:ref:`doc <post_processing>`, :ref:`inp <inputs_sampling>`]
+
+   * Turbulence averaging quantities such as Reynolds stresses [:ref:`inp <inputs_averaging>`]
+
+   * Field plane averaging and second and third order moments
+
+   * Derived fields and field operators such as vorticity, q-criterion, strain-rates, gradients, divergence, laplacian
+
+   * in-situ post-processing with Ascent
+
+High performance computing
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Highly parallelized and performance portable
+
+   * Shared memory parallelism with OpenMP threading
+
+   * Distributed memory parallelism with MPI
+
+   * Supports all major compilers (e.g., GCC, Intel, LLVM)
+
+   * Runs on all major GPU vendors (NVIDIA, AMD, Intel)
+
+   * Supported build systems: cmake, spack
+
+* Supported linear solvers
+
+   * native AMReX solvers such as MLMG [:ref:`inp <inputs_mlmg>`]
+
+   * hypre
+
+
+Roadmap
+-------
+
+The roadmap is an evolving, living document and does not purport to
+track every future capability. It is not a promise of future
+capabilities. The main use case is to inform users of
+potential upcoming new capabilities.
+
+Current development
+~~~~~~~~~~~~~~~~~~~
+
+* Inflow-outflow BCs to enable coupling amr-wind to ERF mesoscale modeling software
+
+* Temporal and spatial varying MMC forcing
+
+* Complex terrain
+
+   * Improved wall conditions, e.g., non-uniform roughness, temperature and heat fluxes
+
+   * Complex terrain though immersed boundary methods

docs/sphinx/user/inputs.rst

@@ -38,6 +38,7 @@ Section                 Description
 ======================= ============================================================
 ``geometry``            Computational domain information
 ``amr``                 Mesh refinement controls
+``tagging``             Static and dynamic refinement options
 ``time``                Simulation time controls
 ``io``                  Input/Output controls
 ``incflo``              CFD algorithm and physics controls
@@ -49,7 +50,6 @@ Section                 Description
 ``Momentum sources``    Activate Momentum source terms and their parameters
 ``Boundary conditions`` Boundary condition types and gradients
 ``MLMG options``        Multi-Level Multi-Grid Linear solver options
-``Tagging``             Static and dynamic refinement options
 ``Sampling``            Data probes to sample field data during simulations
 ``Averaging``           Time averaging and correlations
 ======================= ============================================================
@@ -71,6 +71,7 @@ This section documents the parameters available within each section.
 
    inputs_geometry.rst
    inputs_amr.rst
+   inputs_tagging.rst
    inputs_time.rst
    inputs_io.rst
    inputs_incflo.rst
@@ -80,10 +81,8 @@ This section documents the parameters available within each section.
    inputs_ABL.rst
    inputs_ABL_meso_forcing.rst
    inputs_SyntheticTurbulence.rst
-   inputs_Static_Refinement.rst
    inputs_Boundary_conditions.rst
    inputs_MLMG.rst
-   inputs_AMR_Tagging.rst
    inputs_Sampling.rst
    inputs_Averaging.rst
    inputs_KineticEnergy.rst