Framework for multi-tenancy support (#121)

* Introducing catalog table for managing key providers

This commit introduces a user catalog table, percona_tde.pg_tde_key_provider,
within the percona_tde schema, as part of the pg_tde extension. The purpose of
this table is to store essential provider information. The catalog accommodates
various key providers, present and future, utilizing a JSON type
options field to capture provider-specific details.

To facilitate the creation of key providers,
the commit introduces new SQL interfaces:

- pg_tde_add_key_provider(provider_type VARCHAR(10),
        provider_name VARCHAR(128), options JSON)
- pg_tde_add_key_provider_file(provider_name VARCHAR(128),
        file_path TEXT)
- pg_tde_add_key_provider_vault_v2(provider_name VARCHAR(128),
        vault_token TEXT, vault_url TEXT,
        vault_mount_path TEXT, vault_ca_path TEXT)

Additionally, the commit implements the C interface for catalog
interaction, detailed in the 'tde_keyring.h' file.

These changes lay the foundation for implementing multi-tenancy in pg_tde by
eliminating the necessity of a 'keyring.json' file for configuring a
cluster-wide key provider. With this enhancement, each database can have its
dedicated key provider, added via SQL interface, removing the need
for DBA intervention in TDE setup."

* Establishing a Framework for Master Key and Shared Cache Management

Up until now, pg_tde relied on a hard-coded master key name, primarily for
proof-of-concept purposes. This commit introduces a more robust infrastructure
for configuring the master key and managing a dynamic shared memory-based
master-key cache to enhance accessibility.

For user interaction, a new SQL interface is provided:
- pg_tde_set_master_key(master_key_name VARCHAR(255), provider_name VARCHAR(255));
This interface enables users to set a master key for a specific database and make
further enhancements toward implementing the multi-tenancy.

In addition to the public SQL interface, the commit optimizes the internal
master-key API. It introduces straightforward Get and Set functions,
handling locking, retrieval, caching, and seamlessly assigning a master key for
a database.

The commit also introduces a unified internal interface for requesting and
utilizing shared memory, contributing to a more cohesive and efficient
master key and cache management system.

* Revamping the Keyring API Interface and Integrating Master Key

This commit unifies the master-key and key-provider modules with the core of
pg_tde, marking a significant evolution in the architecture.
As part of this integration, the keyring API undergoes substantial changes
to enhance flexibility and remove unnecessary components such as the key cache.

As a result of the keyring refactoring, the file keyring is also rewritten,
offering a template for implementing additional key providers for the extension.
The modifications make the keyring API more pluggable, streamlining
interactions and paving the way for future enhancements.

* An Interface for Informing the Shared Memory Manager about Lock Requirements

This commit addresses PostgreSQL core's requirement for upfront information
regarding the number of locks the extension needs. Given the connection
between locks and the shared memory interface, a new callback routine
is introduced. This routine allows modules to specify
the number of locks they require.

In addition to this functionality, the commit includes code cleanups
and adjustments to nomenclature for improved clarity and consistency.

* Adjusting test cases

* Extension Initialization and Cleanup Mechanism

This commit enhances the extension by adding a new mechanism to facilitate
cleanup or setup procedures when the extension is installed in a database.
The core addition is a function "pg_tde_extension_initialize" invoked upon
executing the database's 'CREATE EXTENSION' command.

The commit introduces a callback registration mechanism to streamline
future development and ensure extensibility. This enables any module
to specify a callback function (registered using on_ext_install() ) to be
invoked during extension creation.

As of this commit, the callback functionality is explicitly utilized by the
master key module to handle the cleanup of the master key information file.
This file might persist in the database directory if the extension had been
previously deleted in the same database.

This enhancement paves the way for a more modular and maintainable extension
architecture, allowing individual modules to manage their specific
setup and cleanup tasks seamlessly."

* Adjusting Vault-V2 key provider to use new keyring architecture

.github/workflows/postgresql-16-src-meson-perf.yml

@@ -76,12 +76,13 @@ jobs:
         run: |  
           bin/initdb -D data
           echo "shared_preload_libraries = 'pg_tde'" >> data/postgresql.conf
-          echo "pg_tde.keyringConfigFile = '/tmp/keyring.json'" >> data/postgresql.conf
           bin/pg_ctl -D data start
           bin/createdb sbtest
           bin/createdb sbtest2
           bin/createuser sbtest -s
           bin/psql sbtest2 <<< "CREATE EXTENSION pg_tde;"
+          bin/psql sbtest2 <<< "SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');"
+          bin/psql sbtest2 <<< "SELECT pg_tde_set_master_key('test-db-master-key','file-vault');"
           cp -r ../src/contrib/pg_tde/sysbench .
         working-directory: inst
 

Makefile.in

@@ -34,6 +34,10 @@ src/keyring/keyring_config.o \
 src/keyring/keyring_file.o \
 src/keyring/keyring_vault.o \
 src/keyring/keyring_api.o \
+src/catalog/tde_keyring.o \
+src/catalog/tde_master_key.o \
+src/common/pg_tde_shmem.o \
+src/common/pg_tde_utils.o \
 src/pg_tde.o
 
 override PG_CPPFLAGS += @tde_CPPFLAGS@

expected/move_large_tuples.out

@@ -1,5 +1,17 @@
 -- test pg_tde_move_encrypted_data()
 CREATE EXTENSION pg_tde;
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_key_provider_file 
+------------------------------
+                            1
+(1 row)
+
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
 CREATE TABLE sbtest2(
 	  id SERIAL,
 	  k TEXT STORAGE PLAIN,

expected/multi_insert.out

@@ -1,6 +1,18 @@
 -- trigger multi_insert path
 -- 
 CREATE EXTENSION pg_tde;
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_key_provider_file 
+------------------------------
+                            1
+(1 row)
+
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
 CREATE TABLE albums (
     album_id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
     artist_id INTEGER,

expected/non_sorted_off_compact.out

@@ -1,6 +1,18 @@
 -- A test case for https://github.com/Percona-Lab/pg_tde/pull/21
 -- 
 CREATE EXTENSION pg_tde;
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_key_provider_file 
+------------------------------
+                            1
+(1 row)
+
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
 DROP TABLE IF EXISTS sbtest1;
 NOTICE:  table "sbtest1" does not exist, skipping
 CREATE TABLE sbtest1(

expected/pgtde_is_encrypted.out

@@ -1,4 +1,16 @@
 CREATE EXTENSION pg_tde;
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_key_provider_file 
+------------------------------
+                            1
+(1 row)
+
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
 CREATE TABLE test_enc(
 	  id SERIAL,
 	  k INTEGER DEFAULT '0' NOT NULL,

expected/toast_decrypt.out

@@ -1,4 +1,16 @@
 CREATE EXTENSION pg_tde;
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_key_provider_file 
+------------------------------
+                            1
+(1 row)
+
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
 CREATE TABLE src (f1 TEXT STORAGE EXTERNAL) USING pg_tde;
 INSERT INTO src VALUES(repeat('abcdeF',1000));
 SELECT * FROM src;

expected/toast_extended_storage.out

@@ -1,5 +1,17 @@
 -- test https://github.com/Percona-Lab/pg_tde/issues/63
 CREATE EXTENSION pg_tde;
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_key_provider_file 
+------------------------------
+                            1
+(1 row)
+
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
 CREATE TEMP TABLE src (f1 text) USING pg_tde;
 -- Crash on INSERT
 INSERT INTO src

expected/trigger_on_view.out

@@ -1,4 +1,16 @@
 CREATE extension pg_tde;
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_key_provider_file 
+------------------------------
+                            1
+(1 row)
+
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
 --
 -- 2 -- Test triggers on a join view
 --

expected/update_compare_indexes.out

@@ -1,4 +1,16 @@
 CREATE EXTENSION pg_tde;
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_key_provider_file 
+------------------------------
+                            1
+(1 row)
+
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
 DROP TABLE IF EXISTS pvactst;
 NOTICE:  table "pvactst" does not exist, skipping
 CREATE TABLE pvactst (i INT, a INT[], p POINT) USING pg_tde;

expected/vault_v2_test.out

@@ -0,0 +1,31 @@
+CREATE EXTENSION pg_tde;
+SELECT pg_tde_add_key_provider_vault_v2('vault-v2','ROOT_TOKEN','http://127.0.0.1:8200','secret',NULL);
+ pg_tde_add_key_provider_vault_v2 
+----------------------------------
+                                1
+(1 row)
+
+SELECT pg_tde_set_master_key('vault-v2-master-key','vault-v2');
+ pg_tde_set_master_key 
+-----------------------
+
+(1 row)
+
+CREATE TABLE test_enc(
+	  id SERIAL,
+	  k INTEGER DEFAULT '0' NOT NULL,
+	  PRIMARY KEY (id)
+	) USING pg_tde;
+INSERT INTO test_enc (k) VALUES (1);
+INSERT INTO test_enc (k) VALUES (2);
+INSERT INTO test_enc (k) VALUES (3);
+SELECT * from test_enc;
+ id | k 
+----+---
+  1 | 1
+  2 | 2
+  3 | 3
+(3 rows)
+
+DROP TABLE test_enc;
+DROP EXTENSION pg_tde;

meson.build

@@ -26,6 +26,10 @@ pg_tde_sources = files(
         'src/keyring/keyring_vault.c',
         'src/keyring/keyring_api.c',
 
+        'src/catalog/tde_keyring.c',
+        'src/catalog/tde_master_key.c',
+        'src/common/pg_tde_shmem.c',
+        'src/common/pg_tde_utils.c',
         'src/pg_tde.c',
 )
 

pg_tde--1.0.sql

@@ -3,6 +3,62 @@
 -- complain if script is sourced in psql, rather than via CREATE EXTENSION
 \echo Use "CREATE EXTENSION pg_tde" to load this file. \quit
 
+-- pg_tde catalog tables
+CREATE SCHEMA percona_tde;
+-- Note: The table is created using heap storage becasue we do not want this table
+-- to be encrypted by pg_tde. This table is used to store key provider information
+-- and we do not want to encrypt this table using pg_tde.
+CREATE TABLE percona_tde.pg_tde_key_provider(provider_id SERIAL,
+        keyring_type VARCHAR(10) CHECK (keyring_type IN ('file', 'vault-v2')),
+        provider_name VARCHAR(255) UNIQUE NOT NULL, options JSON, PRIMARY KEY(provider_id)) using heap;
+
+-- If you want to add new provider types, you need to make appropriate changes
+-- in include/catalog/tde_keyring.h and src/catalog/tde_keyring.c files.
+
+SELECT pg_catalog.pg_extension_config_dump('percona_tde.pg_tde_key_provider', '');
+-- Key Provider Management
+
+CREATE OR REPLACE FUNCTION pg_tde_add_key_provider(provider_type VARCHAR(10), provider_name VARCHAR(128), options JSON)
+RETURNS INT
+AS $$
+    INSERT INTO percona_tde.pg_tde_key_provider (keyring_type, provider_name, options) VALUES (provider_type, provider_name, options) RETURNING provider_id;
+$$
+LANGUAGE SQL;
+
+CREATE OR REPLACE FUNCTION pg_tde_add_key_provider_file(provider_name VARCHAR(128), file_path TEXT)
+RETURNS INT
+AS $$
+-- JSON keys in the options must be matched to the keys in
+-- load_file_keyring_provider_options function.
+
+    SELECT pg_tde_add_key_provider('file', provider_name,
+                json_object('type' VALUE 'file', 'path' VALUE COALESCE(file_path,'')));
+$$
+LANGUAGE SQL;
+
+CREATE OR REPLACE FUNCTION pg_tde_add_key_provider_vault_v2(provider_name VARCHAR(128),
+                                                        vault_token TEXT,
+                                                        vault_url TEXT,
+                                                        vault_mount_path TEXT,
+                                                        vault_ca_path TEXT)
+RETURNS INT
+AS $$
+-- JSON keys in the options must be matched to the keys in
+-- load_vaultV2_keyring_provider_options function.
+    SELECT pg_tde_add_key_provider('vault-v2', provider_name,
+                            json_object('type' VALUE 'vault-v2',
+                            'url' VALUE COALESCE(vault_url,''),
+                            'token' VALUE COALESCE(vault_token,''),
+                            'mountPath' VALUE COALESCE(vault_mount_path,''),
+                            'caPath' VALUE COALESCE(vault_ca_path,'')));
+$$
+LANGUAGE SQL;
+
+CREATE FUNCTION pg_tde_get_keyprovider(provider_name text)
+RETURNS VOID
+AS 'MODULE_PATHNAME'
+LANGUAGE C;
+-- Table access method
 CREATE FUNCTION pg_tdeam_handler(internal)
 RETURNS table_am_handler
 AS 'MODULE_PATHNAME'
@@ -18,9 +74,20 @@ RETURNS boolean
 AS 'MODULE_PATHNAME'
 LANGUAGE C;
 
+CREATE FUNCTION pg_tde_set_master_key(master_key_name VARCHAR(255), provider_name VARCHAR(255))
+RETURNS VOID
+AS 'MODULE_PATHNAME'
+LANGUAGE C;
+
+CREATE FUNCTION pg_tde_extension_initialize()
+RETURNS VOID
+AS 'MODULE_PATHNAME'
+LANGUAGE C;
+
+
 -- Access method
 CREATE ACCESS METHOD pg_tde TYPE TABLE HANDLER pg_tdeam_handler;
 COMMENT ON ACCESS METHOD pg_tde IS 'pg_tde table access method';
 
--- Opclasses
-
+-- Per database extension initialization
+SELECT pg_tde_extension_initialize();

sql/move_large_tuples.sql

@@ -1,6 +1,9 @@
 -- test pg_tde_move_encrypted_data()
 CREATE EXTENSION pg_tde;
 
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+
 CREATE TABLE sbtest2(
 	  id SERIAL,
 	  k TEXT STORAGE PLAIN,

sql/multi_insert.sql

@@ -2,6 +2,9 @@
 -- 
 CREATE EXTENSION pg_tde;
 
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+
 CREATE TABLE albums (
     album_id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
     artist_id INTEGER,

sql/non_sorted_off_compact.sql

@@ -1,6 +1,10 @@
 -- A test case for https://github.com/Percona-Lab/pg_tde/pull/21
 -- 
 CREATE EXTENSION pg_tde;
+
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+
 DROP TABLE IF EXISTS sbtest1;
 CREATE TABLE sbtest1(
 	  id SERIAL,

sql/pgtde_is_encrypted.sql

@@ -1,5 +1,8 @@
 CREATE EXTENSION pg_tde;
 
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+
 CREATE TABLE test_enc(
 	  id SERIAL,
 	  k INTEGER DEFAULT '0' NOT NULL,

sql/toast_decrypt.sql

@@ -1,5 +1,8 @@
 CREATE EXTENSION pg_tde;
 
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+
 CREATE TABLE src (f1 TEXT STORAGE EXTERNAL) USING pg_tde;
 INSERT INTO src VALUES(repeat('abcdeF',1000));
 SELECT * FROM src;

sql/toast_extended_storage.sql

@@ -1,6 +1,9 @@
 -- test https://github.com/Percona-Lab/pg_tde/issues/63
 CREATE EXTENSION pg_tde;
 
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+
 CREATE TEMP TABLE src (f1 text) USING pg_tde;
 -- Crash on INSERT
 INSERT INTO src

sql/trigger_on_view.sql

@@ -1,5 +1,8 @@
 CREATE extension pg_tde;
 
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+
 --
 -- 2 -- Test triggers on a join view
 --

sql/update_compare_indexes.sql

@@ -1,5 +1,8 @@
 CREATE EXTENSION pg_tde;
 
+SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
+SELECT pg_tde_set_master_key('test-db-master-key','file-vault');
+
 DROP TABLE IF EXISTS pvactst;
 CREATE TABLE pvactst (i INT, a INT[], p POINT) USING pg_tde;
 INSERT INTO pvactst SELECT i, array[1,2,3], point(i, i+1) FROM generate_series(1,1000) i;

sql/vault_v2_test.sql

@@ -0,0 +1,20 @@
+CREATE EXTENSION pg_tde;
+
+SELECT pg_tde_add_key_provider_vault_v2('vault-v2','ROOT_TOKEN','http://127.0.0.1:8200','secret',NULL);
+SELECT pg_tde_set_master_key('vault-v2-master-key','vault-v2');
+
+CREATE TABLE test_enc(
+	  id SERIAL,
+	  k INTEGER DEFAULT '0' NOT NULL,
+	  PRIMARY KEY (id)
+	) USING pg_tde;
+
+INSERT INTO test_enc (k) VALUES (1);
+INSERT INTO test_enc (k) VALUES (2);
+INSERT INTO test_enc (k) VALUES (3);
+
+SELECT * from test_enc;
+
+DROP TABLE test_enc;
+
+DROP EXTENSION pg_tde;