diff --git a/docs/source/backends/adios2.rst b/docs/source/backends/adios2.rst index de6b47aaad..cc1060a3d7 100644 --- a/docs/source/backends/adios2.rst +++ b/docs/source/backends/adios2.rst @@ -105,6 +105,11 @@ The default behavior may be restored by setting the :ref:`JSON parameter ` ``adios2.attribute_writing_ranks`` can be used to restrict attribute writing to only a select handful of ranks (most typically a single one). +The ADIOS2 backend of the openPMD-api will then ignore attributes from all other MPI ranks. + +.. tip:: + + Treat metadata specification as a collective operation in order to retain compatibility with HDF5, and then specify ``adios2.attribute_writing_ranks = 0`` in order to achieve best performance in ADIOS2. + +.. warning:: + + The ADIOS2 backend may also use attributes to encode openPMD groups (ref. "group table"). + The ``adios.attribute_writing_ranks`` key also applies to those attributes, i.e. also group creation must be treated as collective then (at least on the specified ranks). + Experimental group table feature -------------------------------- diff --git a/docs/source/details/adios2.json b/docs/source/details/adios2.json index d71061c0bc..c817eb0d7a 100644 --- a/docs/source/details/adios2.json +++ b/docs/source/details/adios2.json @@ -2,6 +2,7 @@ "adios2": { "engine": { "type": "sst", + "preferred_flush_target": "disk", "parameters": { "BufferGrowthFactor": "2.0", "QueueLimit": "2" @@ -17,6 +18,7 @@ } } ] - } + }, + "attribute_writing_ranks": 0 } } diff --git a/docs/source/details/adios2.toml b/docs/source/details/adios2.toml index 863a5021fa..20ef9e827f 100644 --- a/docs/source/details/adios2.toml +++ b/docs/source/details/adios2.toml @@ -1,5 +1,12 @@ +[adios2] +# ignore all attribute writes not issued on these ranks +# can also be a list if multiple ranks need to be given +# however rank 0 should be the most common option here +attribute_writing_ranks = 0 + [adios2.engine] type = "sst" +preferred_flush_target = "disk" [adios2.engine.parameters] BufferGrowthFactor = "2.0" diff --git a/docs/source/details/backendconfig.rst b/docs/source/details/backendconfig.rst index f1c47c78bd..57d577af10 100644 --- a/docs/source/details/backendconfig.rst +++ b/docs/source/details/backendconfig.rst @@ -113,7 +113,7 @@ A full configuration of the ADIOS2 backend: .. literalinclude:: adios2.toml :language: toml -All keys found under ``adios2.dataset`` are applicable globally as well as per dataset, keys found under ``adios2.engine`` only globally. +All keys found under ``adios2.dataset`` are applicable globally as well as per dataset, any other keys such as those found under ``adios2.engine`` only globally. Explanation of the single keys: * ``adios2.engine.type``: A string that is passed directly to ``adios2::IO:::SetEngine`` for choosing the ADIOS2 engine to be used. @@ -142,6 +142,11 @@ Explanation of the single keys: The openPMD-api will automatically use a fallback implementation for the span-based Put() API if any operator is added to a dataset. This workaround is enabled on a per-dataset level. The workaround can be completely deactivated by specifying ``{"adios2": {"use_span_based_put": true}}`` or it can alternatively be activated indiscriminately for all datasets by specifying ``{"adios2": {"use_span_based_put": false}}``. +* ``adios2.attribute_writing_ranks``: A list of MPI ranks that define metadata. ADIOS2 attributes will be written only from those ranks, any other ranks will be ignored. Can be either a list of integers or a single integer. + +.. hint:: + + Specifying ``adios2.attribute_writing_ranks`` can lead to serious serialization performance improvements at large scale. Operations specified inside ``adios2.dataset.operators`` will be applied to ADIOS2 datasets in writing as well as in reading. Beginning with ADIOS2 2.8.0, this can be used to specify decompressor settings: diff --git a/docs/source/details/mpi.rst b/docs/source/details/mpi.rst index b94e886fac..ea4ec0551e 100644 --- a/docs/source/details/mpi.rst +++ b/docs/source/details/mpi.rst @@ -42,6 +42,8 @@ Functionality Behavior Description If you want to support all backends equally, treat as a collective operation. Note that openPMD represents constant record components with attributes, thus inheriting this for ``::makeConstant``. + When treating attribute definitions as collective, we advise specifying the ADIOS2 :ref:`JSON/TOML key ` ``adios2.attribute_writing_ranks`` for metadata aggregation scalabilty, typically as ``adios2.attribute_writing_ranks = 0``. + .. [4] We usually open iterations delayed on first access. This first access is usually the ``flush()`` call after a ``storeChunk``/``loadChunk`` operation. If the first access is non-collective, an explicit, collective ``Iteration::open()`` can be used to have the files already open. Alternatively, iterations might be accessed for the first time by immediate operations such as ``::availableChunks()``.