update abt-io documentation

docs/source/abtio.rst

@@ -1,13 +1,53 @@
 ABT-IO
 ======
 
-Executing I/O operations (:code:`read`, :code:`write`, etc.) from an Argobots execution
-stream blocks this ES until the I/O operation has completed. This is
-problematic if the ES is used to run a Mercury progress loop, or to execute Mercury RPC
-handlers, since it prevents any progress on network activities or the execution of
-other RPC handlers. ABT-IO was developed specifically to address this problem.
-It creates one or more ES to which I/O operations can be pushed, hence blocking these
-ES on I/O activities rather than the ES the operations originate from.
+ABT-IO is a Mochi library that provides wrappers for conventional POSIX I/O
+functions.  The wrappers are prefixed with :code:`abt_io_` as in
+:code:`abt_io_open()`, :code:`abt_io_read()`, and :code:`abt_io_write()`.
+These wrapper functions have the same arguments and semantics as their POSIX
+counterparts, except that the underlying I/O operations are offloaded to
+internal resources to be executed asynchronously.  ABT-IO presents both a
+blocking API and a non-blocking API to callers.  In the former case, the
+calling thread is suspended so that other user-level threads may
+gracefully execute until the I/O operation is complete.
+
+Rationale
+---------
+
+Executing I/O operations (:code:`read()`, :code:`write()`, etc.) directly
+from within an Argobots user-level thread (ULT) is technically legal but
+will block the underlying execution stream (ES) until the I/O operation has
+completed. This is problematic (e.g., it may limit concurrency, produce excessive
+latency, or even cause deadlock) if the ES is expected to execute other
+cooperatively-scheduled code. In Mochi for example, an ES is likely to be
+responsible for executing multiple RPC handlers or communication progress
+loops, and those capabilities will be starved while the ES is blocked
+awaiting completion of an I/O system call.
+
+ABT-IO was developed specifically to address this problem.  It creates one
+or more decoupled execution streams to which I/O operations are delegated so
+they can be executed there on behalf of the caller without perturbing the
+caller's user-level thread scheduling.  This also enables I/O concurrency to
+be tuned independently from the caller's level of execution concurrency.
+
+ABT-IO may also optionally be configured to use
+`io_uring <https://kernel.dk/io_uring.pdf>`_ on supported platforms to
+further accelerate I/O performance.
+
+References
+----------
+
+A description of the original implementation of ABT-IO can be found in
+Section 5.3 of `S.
+Seo et al., "Argobots: A Lightweight Low-Level Threading and Tasking
+Framework," in IEEE Transactions on Parallel and Distributed Systems, vol.
+29, no. 3, pp. 512-526, 1 March 2018, doi: 10.1109/TPDS.2017.2766062.
+<https://ieeexplore.ieee.org/document/8082139>`_
+
+
+ABT-IO Usage
+------------
+
 This section gives a list of tutorials on how to use ABT-IO.
 
 .. toctree::
@@ -16,3 +56,4 @@ This section gives a list of tutorials on how to use ABT-IO.
    abtio/01_init.rst
    abtio/02_io.rst
    abtio/03_async.rst
+   abtio/04_uring.rst

docs/source/abtio/01_init.rst

@@ -4,9 +4,42 @@ Initializing ABT-IO
 The following code shows how to initialize and finalize ABT-IO.
 ABT-IO must be initialized after Argobots and finalized before it.
 The :code:`abt_io_init` function takes the number of ES to create
-for handling I/O operations. Alternatively, the :code:`abt_io_init_pool`
-function can be used to make ABT-IO use an Argobots pool that was
-alread created.
+for handling I/O operations.
 
 .. literalinclude:: ../../../code/abtio/01_init/main.c
        :language: cpp
+
+More advanced tuning and configuration options can be controlled using the
+:code:`abt_io_init_ext` function rather than :code:`abt_io_init`.
+It accepts a json-formatted configuration specifier that supports the following parameters.
+
+.. code-block:: c
+
+    /* ------- abt-io configuration examples ------
+     *
+     * optional input field for convenience.  This will cause abt-io to create
+     * an internal service pool with N execution streams in it.
+     * --------------
+     * {"backing_thread_count": 16}
+     *
+     * The user may also pass in an explicit pool.  If so, the resulting json
+     * will look like this:
+     * --------------
+     * {"internal_pool_flag": 0}
+     *
+     * This is the fully resolved json description of an internal pool (may be
+     * passed in explicitly, or will be generated based on the
+     * backing_thread_count json parameter).
+     * --------------
+     * {"internal_pool_flag": 1,
+     *    "internal_pool":{
+     *       "kind":"fifo_wait",
+     *       "access":"mpmc",
+     *       "num_xstreams": 4
+     *    }
+     * }
+     */
+
+See :ref:`abtio_uring` for information about how to configure ABT-IO to use
+io_uring to execute I/O operations.
+

docs/source/abtio/02_io.rst

@@ -1,3 +1,5 @@
+.. _abtio_issuing:
+
 Issuing I/O operations
 ======================
 
@@ -27,13 +29,26 @@ The list of available I/O operations is the following.
 +-----------------+------------------------------+
 | abt_io_close    | close                        |
 +-----------------+------------------------------+
+| abt_io_ftruncate| ftruncate                    |
++-----------------+------------------------------+
+| abt_io_truncate | truncate                     |
++-----------------+------------------------------+
+| abt_io_lseek    | lseek                        |
++-----------------+------------------------------+
+| abt_io_fdatasync| fdatasync                    |
++-----------------+------------------------------+
+| abt_io_fallocate| fallocate                    |
++-----------------+------------------------------+
+| abt_io_stat     | stat                         |
++-----------------+------------------------------+
+| abt_io_statfs   | statfs                       |
++-----------------+------------------------------+
 
 .. important::
-   We highly recommend using the :code:`pwrite` and :code:`pread`
-   wrappers rather than the :code:`write` and :code:`read` wrappers,
-   especially if you either post I/O operations from multiple ES,
-   or setup ABT-IO to execute I/O operations in multiple ES.
-   :code:`abt_io_read` and :code:`abt_io_write` rely on the file
-   descriptor's internal cursor, and Argobots cannot guarantee
-   the order of I/O operations posted, which may lead to 
-   unpredictable reordering.
+   We highly recommend using the :code:`pwrite` and :code:`pread` wrappers
+   rather than the :code:`write`, :code:`read`, and :code:`lseek` wrappers.
+   The former functions include explicit offsets with each operation, while
+   the latter rely on implicit file positioning which will be unreliable and
+   unpredictable if current I/O operations are in flight.  ABT-IO inherently
+   does not serialize or otherwise enforce ordering among concurrent
+   operations.

docs/source/abtio/03_async.rst

@@ -8,32 +8,10 @@ examplifies this feature,
 .. literalinclude:: ../../../code/abtio/03_async/main.c
        :language: cpp
 
-The full list of non-blocking functions is the following.
-
-+--------------------+------------------------------+
-| ABT-IO function    | Corresponding POSIX function |
-+====================+==============================+
-| abt_io_open_nb     | open                         |
-+--------------------+------------------------------+
-| abt_io_pwrite_nb   | pwrite                       |
-+--------------------+------------------------------+
-| abt_io_write_nb    | write                        |
-+--------------------+------------------------------+
-| abt_io_pread_nb    | pread                        |
-+--------------------+------------------------------+
-| abt_io_read_nb     | read                         |
-+--------------------+------------------------------+
-| abt_io_mkostemp_nb | mkostemp                     |
-+--------------------+------------------------------+
-| abt_io_unlink_nb   | unlink                       |
-+--------------------+------------------------------+
-| abt_io_close_nb    | close                        |
-+--------------------+------------------------------+
-
-Compared with their blocking versions, these function take
-an additional parameter which is a pointer to the returned
-value (will be set when the operation has completed) and
-return a pointer to an :code:`abt_io_op_t` object.
+Every ABT-IO function described in :ref:`abtio_issuing` has a corresponding non-blocking variant. These
+non-blocking variants take an additional parameter which is a pointer to the
+returned value (will be set when the operation has completed) and return a
+pointer to an :code:`abt_io_op_t` object.
 
 The user can then use :code:`abt_io_op_wait` to wait for
 the operation to complete, and :code:`abt_io_op_free` to

docs/source/abtio/04_uring.rst

@@ -0,0 +1,54 @@
+.. _abtio_uring:
+
+Using io_uring with ABT-IO
+==========================
+
+Liburing support can be enabled at compile time by adding the
+:code:`+liburing` variant to the mochi-abt-io spack package build or by
+adding :code:`--enable-liburing` to the configure arguments if compiling by
+hand.
+
+Once ABT-IO has been compiled with liburing support, it can be enabled at run time by setting "num_urings" json configuration parameter to a
+value of 1 or higher.  ABT-IO will associate a dedicated internal execution
+stream with each uring to handle submission and completion of operations.
+If "num_urings" is > 1, then uring operations will be issued in round-robin
+fashion across the rings.  Any ABT-IO operations that are not supported
+by uring are serviced as usual by the normal ABT-IO pool.
+
+Optional uring flags can also be specified as follows::
+
+    {"liburing_flags":
+     ["IOSQE_ASYNC","IORING_SETUP_SQPOLL","IORING_SETUP_COOP_TASKRUN",
+     "IORING_SETUP_SINGLE_ISSUER", "IORING_SETUP_DEFER_TASKRUN"]}
+
+Please refer to the liburing documentation for an explanation of these
+flags.
+
+Recommended configuration
+-------------------------
+
+If you are using liburing then we recommend setting :code:`"num_urings":1`
+and :code:`"liburing_flags":["IOSQE_ASYNC"]`, and configuring the normal
+ABT-IO pool (used for any oprations not supported by liburing) to have just
+one execution stream.  For example::
+
+ {"internal_pool_flag": 1,
+    "internal_pool":{
+       "kind":"fifo_wait",
+       "access":"mpmc",
+       "num_xstreams": 1
+    },
+  "num_urings":1,
+  "liburing_flags":["IOSQE_ASYNC"]
+ }
+
+One uring is sufficient to saturate most storage devices as long as the
+kernel is instructed to use asynchronous mode by default so that the
+execution stream can put as many concurrent operations in flight as
+possible.  Note that the "IOSQE_ASYNC" flag is not supported on older Linux
+kernels, however.  One execution stream is also typicallly sufficient for
+the internal pool as it is only used to service a limited set of non-I/O
+operations (i.e., not the :code:`read` or :code:`write` operations).  You
+may find it helpful to increase this pool size if you anticipate servicing a
+large number of such operations, however.
+