pytests crash on windows free-threaded interpreter #4685
Comments
* run free-threaded CI on MacOS and Windows * build against 3.13 instead of 3.13-dev on build-full * skip windows build, see #4685
|
I managed to trigger this locally and get it to happen with the visual studio debugger attached. Visual studio doesn't see debug symbols in the rust code, however, I can see that the crash is happening inside of It looks like there's only one place we call that function, here: Lines 109 to 110 in faa644a So somehow we're inside of a function that accepts a |
|
Here's the stack trace I see in the debugger
|
|
I found PyO3/maturin#2213 (comment) which has a workaround for the debugging symbols. I've attached a screenshot of the stack trace (I don't see a way to copy/paste it, sorry!)
|
|
Ultimately this is all happening inside of module import, so I think that means we have to have a valid thread state, and on 3.13 the GIL is enabled too. That said, the I'm using the python.org CPython build and it has compiler optimizations turned on so it's not straightforward for me to inspect the values of variables in the interpreter in the debugger. If there's some data corruption happening that messes with the Python thread state, I think I'd need to set a watchpoint somewhere in CPython internals, so I think my next step might be to compile a debug build of Python on my Windows machine and then try debugging with that. ping @colesbury, you may be interseted in this. I haven't actually proven it but this feels a little like a Windows-specific CPython bug. |
|
That's very strange. It might be worth checking if this is an issue specific to the Windows installer and try the test with |
|
So it turns out you can get the python.org installer to install debug binaries as well. Interestingly there is a valid reference to a thread state at the top of the call stack, but then at the bottom of the call stack it's NULL somehow. I tried setting a watchpoint on the value of the thread_local variable inside Unfortunately the uv Python doesn't work either, it dies trying to create a virtualenv: |
|
If it's easier to set breakpoints on functions, you could try setting one on |
|
If I set a breakpoint on the I also made the following modification to the macro, so that we immediately call diff --git a/pyo3-macros-backend/src/module.rs b/pyo3-macros-backend/src/module.rs
index d9fac3cb..0c093455 100644
--- a/pyo3-macros-backend/src/module.rs
+++ b/pyo3-macros-backend/src/module.rs
@@ -469,6 +469,7 @@ fn module_initialization(
#[doc(hidden)]
#[export_name = #pyinit_symbol]
pub unsafe extern "C" fn __pyo3_init() -> *mut #pyo3_path::ffi::PyObject {
+ let tstate = unsafe { #pyo3_path::ffi::PyInterpreterState_Get() };
unsafe { #pyo3_path::impl_::trampoline::module_init(|py| _PYO3_DEF.make_module(py, #gil_used)) }
}
});But then somehow the thread state inside the interpreter is NULL:
|
|
Ah, I know why this is happening! At first, only |
|
@davidhewitt helped me to find that we weren't accounting for the different library name on the free-threaded build inside of the pyo3 build configuration. In #4690 I attempted a fix, but I still see confusing behavior on Windows. With that PR I get past the error about the invalid thread state and then it dies later when it tries to initialize a module because the module name is a NULL pointer. Similar to the behavior when we weren't getting the right libpython, now I'm seeing that the module object is valid and filled in on the PyO3 side of the FFI boundary but then inside of One thing I'm confused about - is it normal on windows for both The call stack is here: https://gist.github.com/ngoldbaum/92e1dce5e5ddfef50d15a39d26a3f6ca And here's a screenshot of the VS debugger showing the invalid module pointer and the libpython dlls that are loaded: |
|
Now I understand why it's crashing - somehow the PyObject layout on the Rust side of the FFI boundary isn't correct. When I look on the rust side, And I see why it's not working, for whatever reason when Now how there isn't a |
|
Here's the issue: pyo3/pyo3-build-config/src/impl_.rs Lines 1108 to 1113 in faa644a So I guess we need a different way to communicate this information to |
|
Hmm, it looks like the comment there is out of date, So maybe that check needs to come with a Python version check? |
Bug Description
See this CI run on my fork of PyO3: https://github.com/ngoldbaum/pyo3/actions/runs/11694025826/job/32566839238
Relevant part of the CI output:
Steps to Reproduce
Install free-threaded python on Windows, build PyO3 against that Python and run
nox -f test-py.Backtrace
Your operating system and version
Windows (see github actions details)
Your Python version (
python --version)3.13.0t
Your Rust version (
rustc --version)1.82.0
Your PyO3 version
main branch
How did you install python? Did you use a virtualenv?
python.org installer
Additional Info
Not sure what's happening but I wanted to note this in an issue so I can come back and debug it on my Windows development environment.
The text was updated successfully, but these errors were encountered: