[C API] Enhance PyErr_WriteUnraisable() API to pass an error message

[vstinner]

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Proposal:

I added a private _PyErr_WriteUnraisableMsg() API but it cannot be used in third party code, nor in stdlib extensions which try to avoid private/internal functions (like sqlite3). Moreover, _PyErr_WriteUnraisableMsg() API doesn't allow to fully customize the error message.

The limitation of _PyErr_WriteUnraisableMsg() affected PR #106674 which has to add ; consider using ... in the error message which is not great.

        _PyErr_WriteUnraisableMsg(
            "in PyMapping_HasKeyString(); consider using "
            "PyMapping_GetOptionalItemString() or PyMapping_GetItemString()",
            NULL);

@serhiy-storchaka suggested to add an API which allows string formatting, similar to PyErr_Format().

_PyErr_WriteUnraisableMsg() was added in issue #81010.

---

By the way, we should go through all calls to PyErr_WriteUnraisable() and add more context to more these logs easier to get for developers: give more context about what the issue is, how to fix it, where it occurred, etc.

Linked PRs

• gh-108082: Add PyErr_FormatUnraisable() function #111086
• gh-108082: C API: Add tests for PyErr_WriteUnraisable() #111455
• [3.12] gh-108082: C API: Add tests for PyErr_WriteUnraisable() (GH-111455) #111507
• gh-108082: Use PyErr_FormatUnraisable() #111580
• gh-108082: Remove _PyErr_WriteUnraisableMsg() #111643

[serhiy-storchaka]

I already started to work on this, but was not sure about API, so left it to your return.

I initially was going to add simple function:

void PyErr_FormatUnraisable(const char *format, ...);

With special case PyErr_FormatUnraisable(NULL, obj) equal to PyErr_WriteUnraisable(obj).

But it now creates the UnraisableHookArgs object which has field object. So we need more complex API:

void PyErr_FormatUnraisable(PyObject *obj, const char *format, ...);

With PyErr_WriteUnraisable(obj) equal to PyErr_FormatUnraisable(obj, "Exception ignored in: %R", obj) or special case PyErr_FormatUnraisable(obj, NULL).

Or we always set UnraisableHookArgs.object to None if use PyErr_FormatUnraisable(), then we can make the use simpler. Then custom message and object will be mutually exclusive, but seems that currently most of cases with custom message pass NULL as object.

Or we can keep PyErr_FormatUnraisable() simpler and add a separate function PyErr_FormatUnraisableObject() which allows to specify both custom message and object.

void PyErr_FormatUnraisable(const char *format, ...);
void PyErr_FormatUnraisableObject(PyObject *obj, const char *format, ...);

[vstinner]

I wrote PR #108863 to remove the private _PyErr_WriteUnraisableMsg() function from the public C API.

[serhiy-storchaka]

Few calls of _PyErr_WriteUnraisableMsg are left which pass both message and non-NULL object.

Modules/atexitmodule.c:140:            _PyErr_WriteUnraisableMsg("in atexit callback", the_func);
--
Modules/_threadmodule.c:1074:            _PyErr_WriteUnraisableMsg("in thread started by", boot->func);
--
Modules/_ctypes/callbacks.c:219:        _PyErr_WriteUnraisableMsg("on calling ctypes callback function",
Modules/_ctypes/callbacks.c-220-                                  callable);
--
Modules/_ctypes/callbacks.c:261:            _PyErr_WriteUnraisableMsg("on converting result "
Modules/_ctypes/callbacks.c-262-                                      "of ctypes callback function",
Modules/_ctypes/callbacks.c-263-                                      callable);
--
Modules/_ctypes/callbacks.c:273:                _PyErr_WriteUnraisableMsg("on converting result "
Modules/_ctypes/callbacks.c-274-                                          "of ctypes callback function",
Modules/_ctypes/callbacks.c-275-                                          callable);

They can be replaced with PyErr_FormatUnraisable(), but then object will be None. Or we can add new function specially for these cases, which passes both object and a format with arguments for a warning message. I do not know how important to have object.

[vstinner]

They can be replaced with PyErr_FormatUnraisable(), but then object will be None. Or we can add new function specially for these cases, which passes both object and a format with arguments for a warning message. I do not know how important to have object.

As the author of _PyErr_WriteUnraisableMsg(), I suggest to remove _PyErr_WriteUnraisableMsg() and replace all _PyErr_WriteUnraisableMsg() usage with PyErr_FormatUnraisable().

My intent for passing object to the hook was to delegate the formatting to give the hook the opportunity to catch exception on formatting the object, and maybe use tracemalloc to see where the object was allocated, etc.

But I don't think that these features are really useful and we should just skip/remove object, and just pass an error message to the hook.

[serhiy-storchaka]

As the author of _PyErr_WriteUnraisableMsg(), I suggest to remove _PyErr_WriteUnraisableMsg() and replace all _PyErr_WriteUnraisableMsg() usage with PyErr_FormatUnraisable().

I created #111643 for this.

[serhiy-storchaka]

Now we can replace more PyErr_Clear() with informative PyErr_FormatUnraisable().