Clean up materialised tables that Splink creates when we're not using linker.py (profiling)

[RobinL]

This PR ensures that we delete tables created by the profiling code so the database doesn't get littered with lots of temp tables.

This does mean they won't exist in the cache, meaning if for some reason want to call profile_columns twice, the second one call won't be able to use the cache. But I can't see when that'd matter.

example

import logging

import duckdb

import splink.comparison_library as cl
from splink import (
    DuckDBAPI,
    SettingsCreator,
    block_on,
    splink_datasets,
)
from splink.blocking_rule_library import block_on
from splink.datasets import splink_datasets
from splink.profile_data import profile_columns

settings = SettingsCreator(
    link_type="dedupe_only",
    comparisons=[
        cl.ExactMatch("first_name"),
        cl.ExactMatch("surname"),
        cl.ExactMatch("dob"),
        cl.ExactMatch("city"),
        cl.ExactMatch("email"),
    ],
    blocking_rules_to_generate_predictions=[block_on("first_name")],
    retain_matching_columns=True,
    retain_intermediate_calculation_columns=True,
)

df = splink_datasets.fake_1000
df2 = splink_datasets.fake_1000.copy()
con = duckdb.connect()

con.sql("CREATE TABLE my_table AS SELECT * FROM df")


db_api = DuckDBAPI(connection=con)
db_api.register_multiple_tables(["my_table", df2])
con.execute("SELECT name FROM sqlite_master WHERE type='table'").df()
# db_api.debug_mode = True
# logging.basicConfig(
#     format="%(message)s",
# )
# splink_logger = logging.getLogger("splink")
# logging.getLogger("splink").setLevel(1)

profile_columns(
    ["my_table", df2], column_expressions=["first_name", "surname"], db_api=db_api
)


con.execute("SELECT name FROM sqlite_master WHERE type='table'").df()

[RobinL]

This change has two effects:

1. It reduces memory usage, see script below
2. It means the table doesn't need to be cleaned up. The create table version creates a materialised table in the database. But since this is not a SplinkDataframe and is not registered in the intermediate table cache, it isn't cleaned up by delete_tables_created_by_splink_from_db() . That method iterates through SplinkDataFrames that Splink knows about and if 'created_by_splink' is set to True, allows them to be deleted.

Illustration of how register doesn't use memory but create table does

import os

import duckdb
import pandas as pd
import psutil


def get_size():
    process = psutil.Process(os.getpid())
    bytes = process.memory_info().rss  # in bytes
    factor = 1024
    for unit in ["", "K", "M", "G", "T", "P"]:
        if bytes < factor:
            return f"{bytes:.2f}{unit}B"
        bytes /= factor


print("Initial memory usage:", get_size())

df = pd.read_parquet(
    "/Users/robinlinacre/Documents/data_linking/ohid_docker/synthetic_1m.parquet"
)

print("Memory usage after loading DataFrame:", get_size())

con  = duckdb.connect()

con.register("registered_table", df)
print("Memory usage after registering DataFrame as a table in DuckDB:", get_size())

con.sql("CREATE TABLE my_table AS SELECT * FROM registered_table")

print("Memory usage after creating a new table from the registered table:", get_size())

Initial memory usage: 119.45MB
Memory usage after loading DataFrame: 865.04MB
Memory usage after registering DataFrame as a table in DuckDB: 867.31MB
Memory usage after creating a new table from the registered table: 1000.46MB

[RobinL]

In terms of what this does, see here

DuckDB also supports “registering” a DataFrame or Arrow object as a virtual table, comparable to a SQL VIEW. This is useful when querying a DataFrame/Arrow object that is stored in another way (as a class variable, or a value in a dictionary). Below is a Pandas example

[ADBond]

Yeah that makes sense. I originally switched to using tables when I implemented the DatabaseAPI as at that time it was difficult getting things working with some things being tables and some being views (see note under 'DuckDB registration'), and that was easiest for time being, but happy to go for the lighter approach if that works okay

[ADBond]

ah looks like there is still some sort of clash with views/tables. but yeah if that can be resolved i'm all in favour of the reduced-memory option

[RobinL]

Do you remember the details of this comment - is it intended to distinguish between the previous behaviour of register, or just a general comment to point out we don't need any error checking logic.

I think you also get error checking when using register:

con = duckdb.connect()
con.register("registered_table", [{'a': 1}])

> InvalidInputException: Invalid Input Error: Python Object list not suitable to be registered as a view

[ADBond]

I think this is just a general comment - it is around in Splink 3 already, and seems to originate (with a slight expansion) from quite a while back

[RobinL]

Moved linker.py to the database_api because otherwise it isn't accessible to methods like profile_columns which are using the dbapi but not the linker

[RobinL]

@ADBond apologies! Tests passing now. Feel free to merge the mypy one first, and i can update this in case it introduces a mypy failure!

[ADBond]

Great - all looks good to me 👍
Have merged mypy branch, so happy for you to merge this if there's no clash there