Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-128505: Expose an interface to sqlite3_file_control #128507

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions Doc/library/sqlite3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -579,6 +579,61 @@ Module constants
.. deprecated-removed:: 3.12 3.14
The :data:`!version` and :data:`!version_info` constants.

.. _sqlite3-fcntl-constants:

.. data:: SQLITE_FCNTL_LOCKSTATE
SQLITE_FCNTL_GET_LOCKPROXYFILE
SQLITE_FCNTL_SET_LOCKPROXYFILE
SQLITE_FCNTL_LAST_ERRNO
SQLITE_FCNTL_SIZE_HINT
SQLITE_FCNTL_CHUNK_SIZE
SQLITE_FCNTL_FILE_POINTER
SQLITE_FCNTL_SYNC_OMITTED
SQLITE_FCNTL_WIN32_AV_RETRY
SQLITE_FCNTL_PERSIST_WAL
SQLITE_FCNTL_OVERWRITE
SQLITE_FCNTL_POWERSAFE_OVERWRITE
SQLITE_FCNTL_PRAGMA
SQLITE_FCNTL_BUSYHANDLER
SQLITE_FCNTL_MMAP_SIZE
SQLITE_FCNTL_TRACE
SQLITE_FCNTL_HAS_MOVED
SQLITE_FCNTL_SYNC
SQLITE_FCNTL_COMMIT_PHASETWO
SQLITE_FCNTL_WIN32_SET_HANDLE
SQLITE_FCNTL_WAL_BLOCK
SQLITE_FCNTL_ZIPVFS
SQLITE_FCNTL_RBU
SQLITE_FCNTL_VFS_POINTER
SQLITE_FCNTL_JOURNAL_POINTER
SQLITE_FCNTL_WIN32_GET_HANDLE
SQLITE_FCNTL_PDB
SQLITE_FCNTL_BEGIN_ATOMIC_WRITE
SQLITE_FCNTL_COMMIT_ATOMIC_WRITE
SQLITE_FCNTL_ROLLBACK_ATOMIC_WRITE
SQLITE_FCNTL_LOCK_TIMEOUT
SQLITE_FCNTL_DATA_VERSION
SQLITE_FCNTL_SIZE_LIMIT
SQLITE_FCNTL_CKPT_DONE
SQLITE_FCNTL_RESERVE_BYTES
SQLITE_FCNTL_CKPT_START
SQLITE_FCNTL_EXTERNAL_READER
SQLITE_FCNTL_CKSM_FILE
SQLITE_FCNTL_RESET_CACHE
SQLITE_FCNTL_NULL_IO

These constants are used for the :meth:`Connection.file_control` method.

The availability of these constants varies depending on the version of SQLite
Python was compiled with.

.. versionadded:: 3.14

.. seealso::

https://www.sqlite.org/c3ref/c_fcntl_begin_atomic_write.html
SQLite docs: Standard File Control Opcodes

.. _sqlite3-connection-objects:

Connection objects
Expand Down Expand Up @@ -1288,6 +1343,24 @@ Connection objects

.. versionadded:: 3.12

.. method:: file_control(op, val, /, name="main")

Invoke a file control method on the database.
Opcodes which take non-integer arguments are not supported.

:param int op:
The :ref:`SQLITE_FCNTL_* constant <sqlite3-fcntl-constants>` to invoke.

:param int arg:
The argument to pass to the operation.

:param str name:
the database name to operate against.

:rtype: int

.. versionadded:: 3.14

.. method:: serialize(*, name="main")

Serialize a database into a :class:`bytes` object. For an
Expand Down
26 changes: 26 additions & 0 deletions Lib/test/test_sqlite3/test_dbapi.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import sqlite3 as sqlite
import subprocess
import sys
import tempfile
import threading
import unittest
import urllib.parse
Expand Down Expand Up @@ -735,6 +736,31 @@ def test_database_keyword(self):
with contextlib.closing(sqlite.connect(database=":memory:")) as cx:
self.assertEqual(type(cx), sqlite.Connection)

@unittest.skipIf(sys.platform == "darwin", "skipped on macOS")
def test_wal_preservation(self):
with tempfile.TemporaryDirectory() as dirname:
path = os.path.join(dirname, "db.sqlite")
with contextlib.closing(sqlite.connect(path)) as cx:
cx.file_control(sqlite.SQLITE_FCNTL_PERSIST_WAL, 1)
cu = cx.cursor()
cu.execute("PRAGMA journal_mode = WAL")
cu.execute("CREATE TABLE foo (id int)")
cu.execute("INSERT INTO foo (id) VALUES (1)")
self.assertTrue(os.path.exists(path + "-wal"))
self.assertTrue(os.path.exists(path + "-wal"))

with contextlib.closing(sqlite.connect(path)) as cx:
cu = cx.cursor()
self.assertTrue(os.path.exists(path + "-wal"))
cu.execute("INSERT INTO foo (id) VALUES (2)")
self.assertFalse(os.path.exists(path + "-wal"))


def test_file_control_raises(self):
with memory_database() as cx:
with self.assertRaises(sqlite.ProgrammingError):
cx.file_control(sqlite.SQLITE_FCNTL_PERSIST_WAL, 1)


class CursorTests(unittest.TestCase):
def setUp(self):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
sqlite Connection objects now expose a method
:meth:`sqlite3.Connection.file_control`, which is a thin wrapper for
`sqlite3_file_control <https://www.sqlite.org/c3ref/file_control.html>`_.
hashbrowncipher marked this conversation as resolved.
Show resolved Hide resolved
95 changes: 94 additions & 1 deletion Modules/_sqlite/clinic/connection.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

117 changes: 117 additions & 0 deletions Modules/_sqlite/connection.c
Original file line number Diff line number Diff line change
Expand Up @@ -2173,6 +2173,122 @@ pysqlite_connection_create_collation_impl(pysqlite_Connection *self,
Py_RETURN_NONE;
}

static inline bool
is_int_fcntl(const int op)
{
switch (op) {
case SQLITE_FCNTL_LOCKSTATE:
case SQLITE_FCNTL_GET_LOCKPROXYFILE:
case SQLITE_FCNTL_SET_LOCKPROXYFILE:
case SQLITE_FCNTL_LAST_ERRNO:
case SQLITE_FCNTL_SIZE_HINT:
case SQLITE_FCNTL_CHUNK_SIZE:
case SQLITE_FCNTL_FILE_POINTER:
case SQLITE_FCNTL_SYNC_OMITTED:
case SQLITE_FCNTL_WIN32_AV_RETRY:
case SQLITE_FCNTL_PERSIST_WAL:
case SQLITE_FCNTL_OVERWRITE:
case SQLITE_FCNTL_POWERSAFE_OVERWRITE:
case SQLITE_FCNTL_PRAGMA:
case SQLITE_FCNTL_BUSYHANDLER:
case SQLITE_FCNTL_MMAP_SIZE:
case SQLITE_FCNTL_TRACE:
case SQLITE_FCNTL_HAS_MOVED:
case SQLITE_FCNTL_SYNC:
case SQLITE_FCNTL_COMMIT_PHASETWO:
case SQLITE_FCNTL_WIN32_SET_HANDLE:
case SQLITE_FCNTL_WAL_BLOCK:
case SQLITE_FCNTL_ZIPVFS:
case SQLITE_FCNTL_RBU:
case SQLITE_FCNTL_VFS_POINTER:
case SQLITE_FCNTL_JOURNAL_POINTER:
case SQLITE_FCNTL_WIN32_GET_HANDLE:
case SQLITE_FCNTL_PDB:
#if SQLITE_VERSION_NUMBER >= 3021000
case SQLITE_FCNTL_BEGIN_ATOMIC_WRITE:
case SQLITE_FCNTL_COMMIT_ATOMIC_WRITE:
case SQLITE_FCNTL_ROLLBACK_ATOMIC_WRITE:
#endif
#if SQLITE_VERSION_NUMBER >= 3023000
case SQLITE_FCNTL_LOCK_TIMEOUT:
#endif
#if SQLITE_VERSION_NUMBER >= 3025000
case SQLITE_FCNTL_DATA_VERSION:
#endif
#if SQLITE_VERSION_NUMBER >= 3028000
case SQLITE_FCNTL_SIZE_LIMIT:
#endif
#if SQLITE_VERSION_NUMBER >= 3031000
case SQLITE_FCNTL_CKPT_DONE:
#endif
#if SQLITE_VERSION_NUMBER >= 3032000
case SQLITE_FCNTL_RESERVE_BYTES:
case SQLITE_FCNTL_CKPT_START:
#endif
#if SQLITE_VERSION_NUMBER >= 3035000
case SQLITE_FCNTL_EXTERNAL_READER:
#endif
#if SQLITE_VERSION_NUMBER >= 3036000
case SQLITE_FCNTL_CKSM_FILE:
#endif
#if SQLITE_VERSION_NUMBER >= 3040000
case SQLITE_FCNTL_RESET_CACHE:
#endif
#if SQLITE_VERSION_NUMBER >= 3048000
case SQLITE_FCNTL_NULL_IO:
#endif
return true;
default:
return false;
}
}

/*[clinic input]
_sqlite3.Connection.file_control as pysqlite_connection_file_control

op: int
The SQLITE_FCNTL_* constant to invoke.
arg: int
The argument to pass to the operation.
/
name: str = "main"
The database name to operate against.

Invoke a file control method on the database.

Opcodes which take non-integer arguments are not supported.
[clinic start generated code]*/

static PyObject *
pysqlite_connection_file_control_impl(pysqlite_Connection *self, int op,
int arg, const char *name)
/*[clinic end generated code: output=8a9f04093fc1f59c input=8819ab1022e6a5ee]*/
{
if(!is_int_fcntl(op)) {
PyErr_Format(PyExc_ValueError, "unknown file control 'op': %d", op);
return NULL;
}

int val = arg;
int rc;

if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) {
return NULL;
}

Py_BEGIN_ALLOW_THREADS
rc = sqlite3_file_control(self->db, name, op, &val);
Py_END_ALLOW_THREADS

if (rc != SQLITE_OK) {
PyErr_SetString(self->ProgrammingError, sqlite3_errstr(rc));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ProgrammingError may not be the right fit for all opcodes. Ideally, we should use a variant of _pysqlite_seterror where we pass an error code instead of a SQLite database pointer. Perhaps I should make such an internal API available first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be happy to use any internal API that you build for this purpose.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Give me a couple of days; my CPython bandwidth is pretty low this week, but I'll try to cook something up within the week. I'll try to jot down my plan in a dedicated issue soon.

return NULL;
}

return PyLong_FromLong(val);
}


#ifdef PY_SQLITE_HAVE_SERIALIZE
/*[clinic input]
_sqlite3.Connection.serialize as serialize
Expand Down Expand Up @@ -2601,6 +2717,7 @@ static PyMethodDef connection_methods[] = {
PYSQLITE_CONNECTION_SET_AUTHORIZER_METHODDEF
PYSQLITE_CONNECTION_SET_PROGRESS_HANDLER_METHODDEF
PYSQLITE_CONNECTION_SET_TRACE_CALLBACK_METHODDEF
PYSQLITE_CONNECTION_FILE_CONTROL_METHODDEF
SETLIMIT_METHODDEF
GETLIMIT_METHODDEF
SERIALIZE_METHODDEF
Expand Down
Loading
Loading