Can free-threading wheel builds be made available on PyPI?

[rtobar]

I know from other issues/PRs (e.g., #40) that making cffi actually thread-safe is a big task. However, a small but easier step is to produce free-threaded wheels. As mentioned in https://docs.python.org/3.13/howto/free-threading-extensions.html#module-initialization

Extension modules need to explicitly indicate that they support running with the GIL disabled; otherwise importing the extension will raise a warning and enable the GIL at runtime.

Thus, providing FT wheels doesn't necessarily mean the extension supports it, it simply allows the package to be installed in a FT installation.

I do understand that providing such wheels could make some users believe that cffi is fully multi-threaded. As long as expectations are made clear that hopefully shouldn't be an issue, but that's very easy to say.

In my case I'm trying to provide FT builds of my library (with hopefully actual thread-safe code), and cffi is an optional dependency (rarely used though from what I know). I'm not running multi-threaded code in my tests (yet), and even if I did python would should re-enable the GIL when importing cffi.

[arigo]

As far as I can tell, CFFI will crash Python or corrupt memory if you force it to run in a free-threaded Python. That's why we don't provide free-threaded CFFIs.

[nitzmahone]

There are numerous issues to solve before CFFI has any real hope of "just working" under the t free-threaded ABI- even with the auto-GIL fallback. Things have come a long way since the last time I tried it around beta4- I poke at things every few months, since I want this to work just as badly as a lot of other folks... IIUC the biggest (but not only) blockers right now to getting it to work even with the GIL fallback are around the lack of limited/stable API/ABI support in the free-threaded builds, and that both CFFI and setuptools are fairly aggressive about trying to use those.

In any case, we've been playing with it, and will continue to do so, but we absolutely will not be publishing CFFI wheels for the t ABI until the full test suite can pass. Possibly not even then, depending if forcing the GIL on in a FT build is actually acceptable to anyone- it likely wouldn't be to me as a user of a free-threaded Python. There's a ton of quite old init and dispatch code that will need to be very carefully evaluated and stress-tested to function reliably in a free-threaded environment.

IMO unstable t-ABI wheels are just an attractive nuisance, providing a false sense of security to folks playing around that their code might actually be safe, and leading to a lot of bug report noise at this stage, since the typical failure mode is almost always either a hang or a segfault.

I'm going to pin this issue to (hopefully) head off duplicates.

PS, since I couldn't resist playing with it again: as of 3.13rc1 with a default-GIL-disabled Python build against the current main branch, things have improved dramatically. With a few small hacks to forcefully disable all limited API support in CFFI, the vast majority of the CFFI test suite passes on x86_64 both with the implicit GIL disable in place, and with it force-disabled. A couple of currently-invalid assertions around limited API support, and a handful of segfaults in, surprise, surprise, the threading tests. It's great that it's a lot closer to working than it was a couple months ago, but real-world stuff still immediately segfaults.

[nitzmahone]

(also see #126 for overall status on that stuff)

[pitrou]

In any case, we've been playing with it, and will continue to do so, but we absolutely will not be publishing CFFI wheels for the t ABI until the full test suite can pass.

This is understandable, but it seems CFFI can still be installed in a Python free-threaded build. It will just compile CFFI from source. Worse, under Windows the produced package will crash when importing:
apache/arrow#44804 (comment)

[arigo]

@pitrou If that's correct, then (1) why?? and (2) if someone can provide the necessary code to add to the CFFI sources to prevent that from happening, it would be nice.

[arigo]

Maybe this?

#ifdef Py_GIL_DISABLED
#error "see https://github.com/python-cffi/cffi/issues/119#issuecomment-2330374513"
#endif

[pitrou]

Yeah, I think that would be sufficient. Though, in the Windows case, it might be that cffi is being compiled with GIL enabled due to pypa/setuptools#4662

So perhaps adding a runtime check using bool(sysconfig.get_config_var('Py_GIL_DISABLED')) would be nice as well.

[arigo]

I think that CPython is meant to switch to a GIL-having mode when it tries to import an extension module that doesn't explicitly say "I support free-threading". If that's the case, then it should still work and not crash. If it doesn't, then it's a bug in either CPython or CFFI (it is probably in the interaction between the two, i.e. CFFI doing something that somehow makes the free-threading CPython unhappy). In that case, maybe CFFI should add a warning and not an error. For all I know it actually works in other contexts (e.g. not Windows).

[pitrou]

I think that CPython is meant to switch to a GIL-having mode when it tries to import an extension module that doesn't explicitly say "I support free-threading". If that's the case, then it should still work and not crash

Yes, but the extension module still needs to be compiled with the Py_GIL_DISABLED define enabled, because the ABI of free-threaded Python is not the same as regular Python. In this case, the setuptools bug means the define is not enabled, despite compiling for free-threaded Python.

More precisely, a Windows Python 3.13 install contains both a python3.13.exe and a python3.13t3.exe, and they share the same Python.h, which is why the define has to be enabled explicitly by the build tool (setuptools in this case).

[arigo]

How did I forget to include setuptools in my list of things to blame? I have no idea.

So you are saying that the C API is different, in such a way that if you try to load in python3.13t3.exe an extension module compiled without Py_GIL_DISABLED then it just crashes? That looks like a problem that should be fixed in CPython itself, to be honest. Keep in mind that in many use cases of CFFI, the user never imports a Python file, but directly loads an extension---the one produced by CFFI---that itself loads the extension from CFFI. If we can't access the C API from there without crashing, then there is nothing we can do.

[pitrou]

How did I forget to include setuptools in my list of things to blame? I have no idea.

😆

So you are saying that the C API is different, in such a way that if you try to load in python3.13t3.exe an extension module compiled without Py_GIL_DISABLED then it just crashes?

The ABI is different, yes. That's why the ABI tag is different too (cp313 vs cp313t).

That looks like a problem that should be fixed in CPython itself, to be honest.

Well, I don't know much about the free-threading internals, but IIRC they had to change the refcounting implementation to minimize performance regressions. Given that Py_INCREF and friend leak those details into the extension module code, this makes the ABI incompatible depending on the Py_GIL_DISABLED macro.

[arigo]

CPython could easily have come up with some hack that makes the module not load; for example, a #define in Python.h that makes one or a few key functions be named differently at the ABI level. Shrug.

I'd still (seriously) accept a CFFI PR that is a workaround for a setuptools bug exposing a CPython design mistake. That seems like a nonsensical enough proposition to be interesting, if only to remind me why I mostly left the Python ecosystem behind me.

[pitrou]

CPython could easily have come up with some hack that makes the module not load; for example, a #define in Python.h that makes one or a few key functions be named differently at the ABI level. Shrug.

That's a good point. I suppose that ship has sailed... (and once setuptools is fixed, things will be better hopefully!)