Subprocesses behave differently on asyncio and Trio

[gschaffner]

I have been doing some work with subprocesses recently and encountered some backend-dependent behavioral differences. I'm opening a discussion thread first rather than issue threads because each of these/the potential fix in AnyIO for each of these is related to at least one of the others, and for some of them it's not clear whether it's worth the effort to do anything about them in AnyIO.

1. On Trio, anyio.abc.Process.[send_signal | terminate | kill] on an exited process is a no-op, but on asyncio it can raise ProcessLookupError (from BaseSubprocessTransport._check_proc). Reproducer:

   import subprocess
   import sys
   
   import anyio
   
   
   async def main() -> None:
       process = await anyio.open_process(
           (sys.executable, "-c", "pass"),
           stdin=subprocess.DEVNULL,
           stdout=subprocess.DEVNULL,
           stderr=subprocess.DEVNULL,
       )
   
       await anyio.sleep(1)
       # This bit will usually take place in another task:
       process.kill()
   
   
   anyio.run(main, backend="asyncio")
   # anyio.run(main, backend="trio")

   I think AnyIO should match Trio's behavior on asyncio, or if not that document the ProcessLookupError.

2. On Trio, anyio.abc.Process.returncode polls the process and thus can never return None for an exited process, but on asyncio it does not poll the process and can thus return None for an exited process.

   Process.returncode currently says

   The return code of the process. If the process has not yet terminated, this will be None.

   That it can also be None if the process has terminated may surprise people (and their code), especially if they have assumed that AnyIO has the same semantics as Trio. While AnyIO does not document .returncode as performing a poll, it also does not document that .returncode may return a stale value. Compare to subprocess.Popen.returncode, which documents that it does not poll and thus can return a stale value, and compare to trio.Process.returncode, which documents that it polls and thus will not return a stale value.

   It would be useful if AnyIO matched Trio's behavior on asyncio. It is useful to be able to poll a process; subprocess.Popen and trio.Process for instance expose API to let one poll.

   Reproducer:

   # This race condition can be reproduced deterministically, but to hit it
   # deterministically for the sake of testing this reproducer's code is non-realistic.
   
   import os
   import subprocess
   import sys
   
   import anyio
   
   
   async def main() -> None:
       process = await anyio.open_process(
           (sys.executable, "-c", "import time; time.sleep(0.1)"),
           stdin=subprocess.DEVNULL,
           stdout=subprocess.DEVNULL,
           stderr=subprocess.DEVNULL,
       )
       try:
           os.waitid(os.P_PID, process.pid, os.WEXITED | os.WNOWAIT)
       except ChildProcessError:
           pass
   
       assert process.returncode is not None
   
   
   anyio.run(main, backend="asyncio")
   # anyio.run(main, backend="trio")

3. On asyncio, we can get stuck (cancellation-invulnerable in some cases) waiting for a child process that is already dead, depending on the child's behavior. Reproducer:

   import subprocess
   import sys
   import textwrap
   
   import anyio
   from anyio import CancelScope
   
   child_args = (
       sys.executable,
       "-c",
       textwrap.dedent(
           """
           import subprocess
           import sys
   
   
           # This example is only for 5 seconds, but the child-of-child can
           # block us for arbitrarily long.
           subprocess.run((sys.executable, "-c", "import time; time.sleep(5)"))
           """
       ).removeprefix("\n"),
   )
   
   
   async def main() -> None:
       with anyio.move_on_after(1):
           t0 = anyio.current_time()
           try:
               await anyio.run_process(child_args)
           finally:
               print(anyio.current_time() - t0)
   
   
   async def main2() -> None:
       # Note that this isn't specific to run_process; more generally it happens with
       # Process.wait when at least one of stdout and stderr is subprocess.PIPE.
       async with await anyio.open_process(
           child_args, stdin=subprocess.DEVNULL, stderr=subprocess.STDOUT
       ) as process:
           with anyio.move_on_after(1):
               t0 = anyio.current_time()
               try:
                   try:
                       assert process.stdout is not None
                       async with process.stdout:
                           async for chunk in process.stdout:
                               ...
                       await process.wait()
                   except BaseException:
                       with CancelScope(shield=True):
                           process.kill()
                           print(
                               "this wait() will get stuck even after the child has exited..."
                           )
                           await process.wait()
                           print("stuck wait finished")
               finally:
                   print(anyio.current_time() - t0)
   
   
   anyio.run(main2, backend="asyncio")
   # anyio.run(main2, backend="trio")

   In both reproducers, note that our child gets sent SIGKILL, but even after it's been force-killed its Process.wait still gets stuck.

   Note that in both of these reproducers the stuck wait() is cancellation-invulnerable, but an unshielded wait() can get stuck too.

   (Also note that in this reproducer, if you KeyboardInterrupt the stuck wait() call, asyncio will typically also hit BaseSubprocessTransport.__del__ fails if the event loop is already closed, which can leak an orphan process python/cpython#114177 and fail to perform the last-chance cleanup it wanted to perform. That is not really relevant though, other than that it may serve to further confuse people who hit this bug.)

   Note that this is different than #757, because in this case the process is actually dead, whereas in #757 the process is alive but in a pathological state. Also, #757 is for pathological child behavior under both backends, whereas this is for non-pathological cases and works fine on the Trio backend.

   This is really an upstream asyncio bug/feature(?): asyncio proc.kill() and proc.wait() are counter intuitive python/cpython#119710. If asyncio deems this an asyncio feature, then I think that this is an AnyIO bug, as the behavior is different on the two backends and the behavior on the asyncio backend violates the anyio.abc.Process.wait docs. I think that AnyIO should behave in the way that people expect here, i.e. behave in the way that the Trio backend and subprocess.Popen both behave.

The following two are also backend-dependent behavioral differences, but these ones are caused by upstream asyncio bugs, not bugs in AnyIO itself. Unfortunately they still present to users as AnyIO bugs (i.e. they affect code that doesn't import asyncio), so an AnyIO-based library affected significantly by any of these bugs or needing poll support ((2)) can find themselves needing to do

if sniffio.current_async_library() == "asyncio":
    raise RuntimeError(
        "The current version of asyncio has bugs that can cause $FOO to misbehave. Please use the Trio backend instead."
    )

or something similar until it's fixed upstream (2027 or later (3.11 EoL), depending on upstream backports). And even so, having Trio-only API in an otherwise AnyIO-only package can impact users.

Note that if (3) is declared to be an asyncio bug rather than an asyncio feature then (3) would join (4) and (5) in this category.

4. On asyncio on Python >= 3.13, anyio.abc.Process.[send_signal | terminate | kill] can kill an unrelated process on Unix-likes. Trio does not have this issue.

   Under the hood this is an upstream asyncio bug: asyncio.subprocess.Process race condition kills an unrelated process on Unix python/cpython#127049.

5. On asyncio on Python < 3.13, anyio.abc.Process.[send_signal | terminate | kill] can cause Process.returncode to be incorrect (with a warning) on Unix-likes. Trio does not have this issue.

   Under the hood this is an upstream asyncio bug: With asyncio subprocess, send_signal() and the child process watcher will both call waitpid() python/cpython#87744.

   (This one I have not currently encountered, but I mention it here still as it is currently XOR with (4).)

Which of these are within the scope of AnyIO to address?