Documentation

docs/source/backends/adios2.rst

@@ -105,6 +105,11 @@ The default behavior may be restored by setting the :ref:`JSON parameter <backen
 Best Practice at Large Scale
 ----------------------------
 
+ADIOS2 distinguishes between "heavy" data of arbitrary size (i.e. the "actual" data) and lightweight metadata.
+
+Heavy I/O
+.........
+
 A benefitial configuration depends heavily on:
 
 1. Hardware: filesystem type, specific file striping, network infrastructure and available RAM on the aggregator nodes.
@@ -135,6 +140,24 @@ The preferred backend usually depends on the system's native software stack.
 For fine-tuning at extreme scale or for exotic systems, please refer to the ADIOS2 manual and talk to your filesystem admins and the ADIOS2 authors.
 Be aware that extreme-scale I/O is a research topic after all.
 
+Metadata
+........
+
+ADIOS2 will implicitly aggregate metadata specified from parallel MPI processes.
+Duplicate specification of metadata is eliminated in this process.
+Unlike in HDF5, specifying metadata collectively is not required and is even detrimental to performance.
+The :ref:`JSON/TOML key <backendconfig>` ``adios2.attribute_writing_ranks`` can be used to restrict attribute writing to only a select handful of ranks (most typically a single one).
+The ADIOS2 backend of the openPMD-api will then ignore attributes from all other MPI ranks.
+
+.. tip::
+
+  Treat metadata specification as a collective operation in order to retain compatibility with HDF5, and then specify ``adios2.attribute_writing_ranks = 0`` in order to achieve best performance in ADIOS2.
+
+.. warning::
+
+  The ADIOS2 backend may also use attributes to encode openPMD groups (ref. "group table").
+  The ``adios.attribute_writing_ranks`` key also applies to those attributes, i.e. also group creation must be treated as collective then (at least on the specified ranks).
+
 Experimental group table feature
 --------------------------------
 

docs/source/details/adios2.json

@@ -2,6 +2,7 @@
   "adios2": {
     "engine": {
       "type": "sst",
+      "preferred_flush_target": "disk",
       "parameters": {
         "BufferGrowthFactor": "2.0",
         "QueueLimit": "2"
@@ -17,6 +18,7 @@
           }
         }
       ]
-    }
+    },
+    "attribute_writing_ranks": 0
   }
 }

docs/source/details/adios2.toml

@@ -1,5 +1,12 @@
+[adios2]
+# ignore all attribute writes not issued on these ranks
+# can also be a list if multiple ranks need to be given
+# however rank 0 should be the most common option here
+attribute_writing_ranks = 0
+
 [adios2.engine]
 type = "sst"
+preferred_flush_target = "disk"
 
 [adios2.engine.parameters]
 BufferGrowthFactor = "2.0"

docs/source/details/backendconfig.rst

@@ -113,7 +113,7 @@ A full configuration of the ADIOS2 backend:
 .. literalinclude:: adios2.toml
    :language: toml
 
-All keys found under ``adios2.dataset`` are applicable globally as well as per dataset, keys found under ``adios2.engine`` only globally.
+All keys found under ``adios2.dataset`` are applicable globally as well as per dataset, any other keys such as those found under ``adios2.engine`` only globally.
 Explanation of the single keys:
 
 * ``adios2.engine.type``: A string that is passed directly to ``adios2::IO:::SetEngine`` for choosing the ADIOS2 engine to be used.
@@ -142,6 +142,11 @@ Explanation of the single keys:
   The openPMD-api will automatically use a fallback implementation for the span-based Put() API if any operator is added to a dataset.
   This workaround is enabled on a per-dataset level.
   The workaround can be completely deactivated by specifying ``{"adios2": {"use_span_based_put": true}}`` or it can alternatively be activated indiscriminately for all datasets by specifying ``{"adios2": {"use_span_based_put": false}}``.
+* ``adios2.attribute_writing_ranks``: A list of MPI ranks that define metadata. ADIOS2 attributes will be written only from those ranks, any other ranks will be ignored. Can be either a list of integers or a single integer.
+
+.. hint::
+
+  Specifying ``adios2.attribute_writing_ranks`` can lead to serious serialization performance improvements at large scale.
 
 Operations specified inside ``adios2.dataset.operators`` will be applied to ADIOS2 datasets in writing as well as in reading.
 Beginning with ADIOS2 2.8.0, this can be used to specify decompressor settings:

docs/source/details/mpi.rst

@@ -42,6 +42,8 @@ Functionality                Behavior           Description
        If you want to support all backends equally, treat as a collective operation.
        Note that openPMD represents constant record components with attributes, thus inheriting this for ``::makeConstant``.
 
+       When treating attribute definitions as collective, we advise specifying the ADIOS2 :ref:`JSON/TOML key <backendconfig>` ``adios2.attribute_writing_ranks`` for metadata aggregation scalabilty, typically as ``adios2.attribute_writing_ranks = 0``.
+
 .. [4] We usually open iterations delayed on first access. This first access is usually the ``flush()`` call after a ``storeChunk``/``loadChunk`` operation. If the first access is non-collective, an explicit, collective ``Iteration::open()`` can be used to have the files already open.
        Alternatively, iterations might be accessed for the first time by immediate operations such as ``::availableChunks()``.