✨ AsyncIterDataPipe for running concurrent tasks

An asynchronous iterable-style DataPipe for processing tasks concurrently! Composition over inheritance. Subclassing from collections.abc.AsyncIterable in Python's standard library. Added some basic API docstring, and have setup some extlinks and intersphinx mappings in the docs/_config.yml file for linking to terms in the Python glossary.

bambooflow/datapipes/__init__.py

@@ -0,0 +1,9 @@
+"""
+An asynchronous-style DataPipe is one that implements the
+:py:meth:`__aiter__ <object.__aiter__>` protocol, and represents an
+:py-term:`asynchronous iterable <asynchronous-iterable>` over data samples.
+This is well-suited for cases when I/O latency is slow, e.g. when waiting on
+network connections, or performing read operations on multiple files at once.
+"""
+
+from bambooflow.datapipes.aiter import AsyncIterDataPipe

bambooflow/datapipes/aiter.py

@@ -0,0 +1,23 @@
+"""
+Base classes for Asynchronous Iterable DataPipes.
+"""
+import collections
+
+
+class AsyncIterDataPipe(collections.abc.AsyncIterable):
+    """
+    Asynchronous iterable-style DataPipes.
+
+    All DataPipes that represent an asynchronous iterable of data samples
+    should subclass this. This style of DataPipes is particularly useful for
+    performing I/O-bound tasks such as streaming data from a network disk drive
+    or reading multiple files concurrently. ``AsyncIterDataPipe`` is
+    initialized in a lazy fashion, and its elements are computed only when
+    :py:meth:`__anext__ <object.__anext__>` is called on the async iterator of
+    an ``AsyncIterDataPipe``.
+    """
+
+    def __repr__(self) -> str:
+        # Instead of showing <bamboopipe. ... .AsyncIterableWrapper at 0x.....>,
+        # return the class name like <AsyncIterableWrapper>
+        return str(self.__class__.__qualname__)

docs/_config.yml

@@ -31,5 +31,15 @@ sphinx:
   config:
     myst_all_links_external: true
     html_show_copyright: false
+    extlinks:
+      py-term:
+        - 'https://docs.python.org/3/glossary.html#term-%s'
+        - '%s'
+    intersphinx_mapping:
+      python:
+        - 'https://docs.python.org/3/'
+        - null
   extra_extensions:
     - 'sphinx.ext.autodoc'
+    - 'sphinx.ext.extlinks'
+    - 'sphinx.ext.intersphinx'

docs/api.md

@@ -1,6 +1,10 @@
 # API Reference
 
+## Asynchronous-style DataPipes
+
 ```{eval-rst}
-.. automodule:: bambooflow
+.. automodule:: bambooflow.datapipes
     :members:
+.. autoclass:: bambooflow.datapipes.AsyncIterDataPipe
+    :show-inheritance:
 ```