"unable to open database file" with Python 3.11 and PySide6 threads

[jeandet]

Hello,
With Python 3.11, I get "unable to open database file" quite quickly and "easily".
It turns out that my app has way too many "cache.db" opened. This looks like a leak, I tried to build a simple reproducer without any success yet.
I'm not sure how to track this down, it seems related to using diskcache from PySide 6 threads (QThread).
Could this be related to python/cpython#97641 and python/cpython#108015 ?
Do you have any idea how to investigate or better to solve this 😅?

The thread is implemented here and here in another package is the shared diskcache wrapper that seems to leak file descriptors.

[grantjenks]

How many threads are there?

try moving the cache reference to a single global reference. Opening and closing it a gain and again in fast succession from multiple threads is likely to cause issues.

[jeandet]

How many threads are there?
Something from 1 to 10, I have one thread per graph so it depend on usage.
The more I have threads the quicker I reach the maximum fd per process.

try moving the cache reference to a single global reference. Opening and closing it a gain and again in fast succession from multiple threads is likely to cause issues.
Unless I missed something it's already what I do.

To give more context, with python 3.10, 3.9 it works perfectly (several days). What I see with Python 3.11 is that sometimes a thread leaks a file descriptor (doesn't look systematic) so I need repeat quite a lot of requests to reach 1024 opened files.
The way it works, is for each graph I have a thread that listen for data requests. A data request correspond to any user interaction with the graph that modifies the X axis range (zoom, pan).
There is only one thread per graph and it is always the same, I mean I create only one thread at graph creation and this thread handles all data requests for this graph and this thread is supposed to use the single shared diskcache instance.

Is that possible that somehow diskcache or sqlite3 reopen the database because it lost the previous fd from within the same thread?

[grantjenks]

Is that possible that somehow diskcache or sqlite3 reopen the database because it lost the previous fd from within the same thread?

I don’t think so.

Sounds like a problem at a lower level than DiskCache.

[jeandet]

@grantjenks thanks for your help, I will continue investigating. I must be able to build a simple reproducer at some point.

[PhilipDeegan]

Hello,

Having done some digging into this.

I can state that for our case with SciQLop the file descriptor leak becomes apparent only on python3.11 as, in python3.11 SQLite added LRU caching to the connection so there are additional referrers to this connection that are not seen in python3.10, and for a reason unknown this case is always true (apart from two cases on app start)

So, with the additional references, and the close function not closing, the file descriptor leak appears, or so it seems.

I have validated this by essentially undoing the LRU cache on 3.11 by modifying the def _con function as shown

This is obviously not ideal, but just to prove a point.

it seems related to using diskcache from PySide 6 threads (QThread).

My guess right now is some incompatibility between threading.local() and QT threads, I am willing to test somethings locally to ensure compatibility if you have some thoughts @grantjenks ?

[erlend-aasland]

@PhilipDeegan, could you try building CPython with this patch? It should explicitly break the cycle:

diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c
index 0a6633972c..fcd8b36d41 100644
--- a/Modules/_sqlite/connection.c
+++ b/Modules/_sqlite/connection.c
@@ -410,8 +410,34 @@ clear_callback_context(callback_context *ctx)
 }
 
 static int
-connection_clear(pysqlite_Connection *self)
+cache_clear(pysqlite_Connection *self)
 {
+    PyObject *cache = self->statement_cache;
+    if (cache == NULL) {
+        return 0;
+    }
+    PyObject *clear = PyObject_GetAttrString(cache, "cache_clear");
+    if (clear == NULL) {
+        PyErr_WriteUnraisable(cache);
+        return -1;
+    }
+    PyObject *ret = PyObject_CallNoArgs(clear);
+    Py_DECREF(clear);
+    if (ret == NULL) {
+        PyErr_WriteUnraisable(cache);
+        return -1;
+    }
+    Py_DECREF(ret);
+    return 0;
+}
+
+static int
+connection_clear(PyObject *obj)
+{
+    pysqlite_Connection *self = (pysqlite_Connection *)obj;
+    if (cache_clear(self) < 0) {
+        PyErr_WriteUnraisable(obj);
+    }
     Py_CLEAR(self->statement_cache);
     Py_CLEAR(self->cursors);
     Py_CLEAR(self->blobs);

Also note that the Django issue linked in the OP was solved by explicit resource management; see the discussion for details.

[PhilipDeegan]

@erlend-aasland

I have compiled python 3.11.4 with the suggested patch and it does not appear to resolve the issue

As before, the gc edits to diskcache/core.py continues to prevent the FD leak somwhat.

Is there something else you might suggest?

[erlend-aasland]

Is there something else you might suggest?

Not from the top of my head; I'll see if I can reproduce this on my Mac and get back to you.

[jeandet]

I think I solved the underlying issue by implementing a basic/naive thread storage that works with Qt/PySide6 threads:
https://github.com/SciQLop/SciQLop/blob/main/SciQLop/plugins/speasy.py#L21-L38 and https://github.com/SciQLop/SciQLop/blob/main/SciQLop/plugins/speasy.py#L162-L163

Where _cache._data is the actual DiskCache instance.
I still don't get why the Python thread local storage get cleared all the time when using Qt/PySide6 threads.