Clarify Docs About Daemon Threads

[ericsnowcurrently]

Feature or enhancement

Ideally, we would get rid of daemon threads (as a feature) eventually. See my DPO thread. In the meantime, it makes sense (to me) to at least clarify the docs about daemon threads and steer users away from using them.

That includes the following:

• clearly identify the case(s) where daemon threads might be a suitable solution
• explicitly point out that daemon threads do not detach at exit, nor cause the process to move into the background at exit
• a warning note indicating that daemon threads should be avoided
• a brief example of using non-daemon threads to accomplish the same thing
• (maybe) a "soft deprecation" of daemon threads

existing docs: https://docs.python.org/3/library/threading.html#thread-objects

Really the only case where daemon threads might be suitable is where:

• the target comes from an extension module
• it wraps a long-running call to a third-party library
• that call does not support being interrupted or stopped
• it does not support a timeout or running for a short time

Otherwise:

• the runtime can already interrupt Python code at shutdown
• if the user maintains the long-running function then they can make it interruptible
• if it is short-running (or can be called that way) in a loop then the user only needs to add a check for exit to each iteration
• if it supports timeouts then it can be likewise put in a loop with a check each iteration
• if it supports interruption/stopping then that can be triggered at exit

Examples

Note that many of these examples make use of threading._register_atexit(), which currently isn't public API, nor documented.

Short-running task in a loop

The code for a function that supports timeouts is essentially the same.

def background_task(task):
    stop = False
    def atexit():
        nonlocal stop
        stop = True
    threading._register_atexit(atexit)  # currently not public API
    def wrapper(*args, **kwargs):
        while not stop:
            task(*args, **kwargs)
    return wrapper

t = threading.Thread(target=background_task(mytask))
t.start()

# Implemented as a class:

class BackgroundTask(threading.Thread):
    def __init__(self, group=None, target=None, name=None, args=(), kwargs={}):
        super().__init__(group, target, name, args, kwargs)
        self._stop = False
    def stop(self):
        self._stop = True
    def run(self):
        threading._register_atexit(self.stop)  # currently not public API
        while not self._stop:
            self.target(*self.args, **self.kwargs)

t = BackgroundTask(target=mytask)
t.start()

Task that supports interruption or stopping

t = threading.Thread(target=mytask.run)
t.start()
threading._register_atexit(mytask.stop)

Long-running Python func from a third-party package

set_async_exc = ctypes.pythonapi.PyThreadState_SetAsyncExc
set_async_exc.argtypes = (ctypes.c_ulong, ctypes.py_object)
exc = ctypes.py_object(SystemExit)

def stop_thread(tid):
    # PyThreadState_SetAsyncExc(t.id, SystemExit)
    set_async_exc(tid, exc)

t = threading.Thread(target=mytask)
t.start()
threading._register_atexit(stop_thread, t.ident)

# Implemented as a class:

class BackgroundTask(threading.Thread):
    _set_async_exc = ctypes.pythonapi.PyThreadState_SetAsyncExc
    _set_async_exc.argtypes = (ctypes.c_ulong, ctypes.py_object)
    _exc = ctypes.py_object(SystemExit)

    def __init__(self, group=None, target=None, name=None, args=(), kwargs={}):
        super().__init__(group, target, name, args, kwargs)
    def stop(self):
        # PyThreadState_SetAsyncExc(t.id, SystemExit)
        self._set_async_exc(self.ident, self._exc)
    def run(self):
        threading._register_atexit(self.stop)
        super().run()

t = BackgroundTask(target=mytask)
t.start()

Long-running, uninterruptible, third-party task

About your only option is something like signal.pthread_kill():

t = threading.Thread(target=mytask)
t.start()

def stop():
    while t.is_alive():
        try:
            signal.pthread_kill(t.ident, signal.SIG_INT)
            t.join(0.1)
        except KeyboardInterrupt:
            pass
threading._register_atexit(stop)