From 0bfa62cfa460858bfbef31f3bba60a7c744bc1a8 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 20 Aug 2024 16:02:31 -0400 Subject: [PATCH 01/16] Reedit of PGD major rollimg upgrades topic - do not merge --- .../5/upgrades/upgrading_major_rolling.mdx | 121 ++++++++++-------- 1 file changed, 68 insertions(+), 53 deletions(-) diff --git a/product_docs/docs/pgd/5/upgrades/upgrading_major_rolling.mdx b/product_docs/docs/pgd/5/upgrades/upgrading_major_rolling.mdx index 63518c6e23a..140bd3aa3a1 100644 --- a/product_docs/docs/pgd/5/upgrades/upgrading_major_rolling.mdx +++ b/product_docs/docs/pgd/5/upgrades/upgrading_major_rolling.mdx @@ -9,7 +9,7 @@ redirects: ## Upgrading Postgres major versions -Upgrading a Postgres database's major version to access improved features, performance enhancements, and security updates is a common administration task. Doing the same for a Postgres Distributed (PGD) cluster deployed with Trusted Postgres Architect (TPA) is essentially the same process but performed as a rolling upgrade. +Upgrading a Postgres database's major version to access improved features, performance enhancements, and security updates is a common administration task. Doing the same for an EDB Postgres Distributed (PGD) cluster deployed with Trusted Postgres Architect (TPA) is essentially the same process but performed as a rolling upgrade. The rolling upgrade process allows updating individual cluster nodes to a new major Postgres version while maintaining cluster availability and operational continuity. This approach minimizes downtime and ensures data integrity by allowing the rest of the cluster to remain operational as each node is upgraded sequentially. @@ -25,7 +25,7 @@ To do this, connect to one of the nodes using SSH and run the `pgd show-nodes` c sudo -u postgres pgd show-nodes ``` -The `pgd show-nodes` command shows you all the nodes in your PGD cluster and the subgroup to which each node belongs. Then you want to find out which node is the write leader in each subgroup by running: +The `pgd show-nodes` command shows you all the nodes in your PGD cluster and the subgroup to which each node belongs. Then you want to find out which node is the write leader in each subgroup: ```bash sudo -u postgres pgd show-groups @@ -40,72 +40,80 @@ You now have enough information to determine your upgrade order, one subgroup at ### Perform the upgrade on each node !!! Note -To help prevent data loss, ensure that your databases and configuration files are backed up before starting the upgrade process. +To help prevent data loss, before starting the upgrade process, ensure that your databases and configuration files are backed up. !!! Using the [preliminary order](#prepare-the-upgrade), perform the following steps on each node while connected via SSH: * **Confirm the current Postgres version** - * View versions from PGD by running `sudo -u postgres pgd show-version`. + * View versions from PGD: + + `sudo -u postgres pgd show-version`. * Ensure that the expected major version is running. * **Verify that the target node isn't the write leader** - * Check whether the target node is the write leader for the group you're upgrading using `sudo -u postgres pgd show-groups`. - * If the target node is the current write leader for the group/subgroup you're upgrading, perform a [planned switchover](#perform-a-planned-switchover) to another node. - * `sudo -u postgres pgd switchover --group-name --node-name ` + * Check whether the target node is the write leader for the group you're upgrading: + + `sudo -u postgres pgd show-groups` + * If the target node is the current write leader for the group/subgroup you're upgrading, perform a [planned switchover](#perform-a-planned-switchover) to another node: + + `sudo -u postgres pgd switchover --group-name --node-name ` * **Stop Postgres on the target node** - * Stop the Postgres service on the current node by running `sudo systemctl stop postgres`. - * The target node is no longer actively participating as a node in the cluster. + * Stop the Postgres service on the current node: + + `sudo systemctl stop postgres` + + The target node is no longer actively participating as a node in the cluster. * **Install PGD and utilities** - * Install PGD and its utilities compatible with the Postgres version you're upgrading to. - * `sudo apt install edb-bdr-pg edb-bdr-utilities` + * Install PGD and its utilities compatible with the Postgres version you're upgrading to: + `sudo apt install edb-bdr-pg edb-bdr-utilities` * **Initialize the new Postgres instance** - * Create a directory that will house the database files for the new version of PostgreSQL: - * `sudo mkdir -p /opt/postgres/datanew` - * Ensure that user postgres has ownership permissions (chown) to the directory. - * Initialize a new PostgreSQL database cluster in the directory you just created. - * This step involves using the `initdb` command provided by the newly installed version of PostgreSQL. - * Replace `` with the path to the bin directory of the newly installed PostgreSQL version: `sudo -u postgres /initdb -D /opt/postgres/datanew`. - * You may need to run this command as the postgres user or another user with appropriate permissions. - * Make sure to include the `--data-checksums` flag to ensure the cluster uses data checksums. + * Create a directory to house the database files for the new version of PostgreSQL: + `sudo mkdir -p /opt/postgres/datanew` + * Ensure that the user postgres has ownership permissions to the directory using `chown`. + * Initialize a new PostgreSQL database cluster in the directory you just created. This step involves using the `initdb` command provided by the newly installed version of PostgreSQL. Include the `--data-checksums` flag to ensure the cluster uses data checksums. + + `sudo -u postgres /initdb -D /opt/postgres/datanew --data-checksums` + + Replace `` with the path to the bin directory of the newly installed PostgreSQL version. + + You may need to run this command as the postgres user or another user with appropriate permissions. * **Migrate configuration to the new Postgres version** * Locate the following configuration files in your current PostgreSQL data directory: - * `postgresql.conf` - * The main configuration file containing settings related to the database system. - * `postgresql.auto.conf` - * Contains settings set by PostgreSQL, such as those modified by the `ALTER SYSTEM` command. - * `pg_hba.conf` - * Manages client authentication, specifying which users can connect to which databases from which hosts. - * The entire `conf.d` directory (if present) - * Allows for organizing configuration settings into separate files for better manageability. + * `postgresql.conf` — The main configuration file containing settings related to the database system. + * `postgresql.auto.conf`— Contains settings set by PostgreSQL, such as those modified by the `ALTER SYSTEM` command. + * `pg_hba.conf` — Manages client authentication, specifying which users can connect to which databases from which hosts. + * The entire `conf.d` directory (if present) — Allows for organizing configuration settings into separate files for better manageability. * Copy these files and the `conf.d` directory to the new data directory you created for the upgraded version of PostgreSQL. * **Verify the Postgres service is inactive** * Before proceeding, it's important to ensure that no PostgreSQL processes are active for both the old and the new data directories. This verification step prevents any data corruption or conflicts during the upgrade process. - * Use the `sudo systemctl status postgres` command to verify that Postgres was stopped. - * If it isn't stopped, run `systemctl stop postgres` and verify again that it was stopped. + + Use the `sudo systemctl status postgres` command to verify that Postgres was stopped. If it isn't stopped, run `systemctl stop postgres` and verify again that it was stopped. * **Swap PGDATA directories for version upgrade** - * Rename `/opt/postgres/data` to `/opt/postgres/dataold` and `/opt/postgres/datanew` to `/opt/postgres/data`. - * This step readies your system for the next crucial phase: running pg_upgrade to finalize the PostgreSQL version transition. + * Rename `/opt/postgres/data` to `/opt/postgres/dataold` and `/opt/postgres/datanew` to `/opt/postgres/data`. + + This step readies your system for the next crucial phase: running pg_upgrade to finalize the PostgreSQL version transition. * **Verify upgrade feasibility** * The `bdr_pg_upgrade` tool offers a `--check` option designed to perform a preliminary scan of your current setup, identifying any potential issues that could hinder the upgrade process. - * You need to run this check from an upgrade directory with ownership given to user `postgres`, such as `/home/upgrade/`, so that the upgrade log files created by `bdr_pg_upgrade` can be stored. - * To initiate the safety check, append the `--check` option to your `bdr_pg_upgrade` command. - * This operation simulates the upgrade process without making any changes, providing insights into any compatibility issues, deprecated features, or configuration adjustments required for a successful upgrade. + + You need to run this check from an upgrade directory with ownership given to user postgres, such as `/home/upgrade/`, so that the upgrade log files created by `bdr_pg_upgrade` can be stored. To initiate the safety check, append the `--check` option to your `bdr_pg_upgrade` command. + + This operation simulates the upgrade process without making any changes, providing insights into any compatibility issues, deprecated features, or configuration adjustments required for a successful upgrade. * Address any warnings or errors indicated by this check to ensure an uneventful transition to the new version. @@ -117,23 +125,27 @@ Using the [preliminary order](#prepare-the-upgrade), perform the following steps * **Update the Postgres service configuration** * Update the service configuration to reflect the new PostgreSQL version by updating the version number in the `postgres.service` file: - * `sudo sed -i -e 's///g' /etc/systemd/system/postgres.service` + + `sudo sed -i -e 's///g' /etc/systemd/system/postgres.service` * Refresh the system's service manager to apply these changes: - * `sudo systemctl daemon-reload` + + `sudo systemctl daemon-reload` * **Restart Postgres** - * Proceed to restart the PostgreSQL service. - * `systemctl start postgres` + * Proceed to restart the PostgreSQL service: + + `systemctl start postgres` * **Validate the new Postgres version** - * Verify that your PostgreSQL instance is now upgraded by again running `sudo -u postgres pgd show-version`. + * Verify that your PostgreSQL instance is now upgraded: + + `sudo -u postgres pgd show-version` * **Clean up post-upgrade** - * Run `vacuumdb` with the `ANALYZE` option immediately after the upgrade but before introducing a heavy production load. - * Running this command minimizes the immediate performance impact, preparing the database for more accurate testing. + * Run `vacuumdb` with the `ANALYZE` option immediately after the upgrade but before introducing a heavy production load. Running this command minimizes the immediate performance impact, preparing the database for more accurate testing. * Remove the old version's data directory, `/opt/postgres/dataold`. ### Reconcile the upgrade with TPA @@ -143,18 +155,21 @@ TPA needs to continue to manage the deployment effectively after all the nodes h Follow these steps to update the configuration and redeploy the PGD cluster through TPA. * **Update the `config.yml`** - * Change the `config.yml` of the TPA-managed cluster to the new version. - * `cluster_vars: postgres_version: ''` + * Change the `config.yml` of the TPA-managed cluster to the new version: + + `cluster_vars: postgres_version: ''` * **Use `tpaexec` to redeploy the PGD cluster with the updated `config.yml`** - * `tpaexec deploy ` + * Use this the `deploy` option: + + `tpaexec deploy ` The worked example that follows shows upgrading the Postgres major version from 15 to 16 on a PGD 5 cluster deployed with TPA in detail. ## Worked example -This worked example starts with a TPA-managed PGD cluster deployed using the [AWS Quickstart](https://www.enterprisedb.com/docs/pgd/latest/quickstart/quick_start_aws/). The cluster has three nodes: kaboom, kaolin, and kaftan, all running Postges 15. +This worked example starts with a TPA-managed PGD cluster deployed using the [AWS quick start](/pgd/latest/quickstart/quick_start_aws/). The cluster has three nodes: kaboom, kaolin, and kaftan, all running Postgres 15. This example starts with kaboom. @@ -164,7 +179,7 @@ Some steps of this process involve running commands as the Postgres owner. We re ### Confirm the current Postgres version -SSH into kaboom to confirm the major version of Postgres is expected by running: +SSH into kaboom to confirm the major version of Postgres is expected: ```bash sudo -u postgres pgd show-version @@ -305,14 +320,14 @@ sudo mv /opt/postgres/datanew /opt/postgres/data ``` !!! Important -If something goes wrong at some point during the procedure, you may want to rollback/revert a node to the older major version. To do this, rename directories again so that the current data directory, `/opt/postgres/data`, becomes `/opt/postgres/datafailed` and the old data directory, `/opt/postgres/dataold`, becomes the current data directory: +If something goes wrong at some point during the procedure, you may want to roll back/revert a node to the older major version. To do this, rename directories again so that the current data directory, `/opt/postgres/data`, becomes `/opt/postgres/datafailed` and the old data directory, `/opt/postgres/dataold`, becomes the current data directory: ```bash sudo mv /opt/postgres/data /opt/postgres/datafailed sudo mv /opt/postgres/dataold /opt/postgres/data ``` -This rolls back/reverts the node back to the previous major version of Postgres. +This rolls back/reverts the node to the previous major version of Postgres. !!! ### Verify upgrade feasibility @@ -367,7 +382,7 @@ Checking for new cluster tablespace directories ok ``` !!! Note -If you didn't initialize Postgres 16 with checksums using the `--data-checksums` option, but did initialize checksums with your Postgres 15 instance, an error tells you about the incompatibility: +If you didn't initialize Postgres 16 with checksums using the `--data-checksums` option but did initialize checksums with your Postgres 15 instance, an error tells you about the incompatibility: ```bash old cluster uses data checksums but the new one does not @@ -506,7 +521,7 @@ sudo systemctl daemon-reload ### Restart Postgres -Start the modified Postgres service by running: +Start the modified Postgres service: ```bash sudo systemctl start postgres @@ -560,11 +575,11 @@ Upgrading the target node is now complete. After completing the upgrade on kaboom, run the same steps on kaolin and kaftan. -If you followed along with this example and kaftan is the write leader, to ensure availability, you must [perform a planned switchover](#perform-a-planned-switchover) to another, already upgraded node before running the upgrade steps on kaftan. +If you followed along with this example and kaftan is the write leader, to ensure availability, you must [perform a planned switchover](#perform-a-planned-switchover) to another node that was already upgraded before running the upgrade steps on kaftan. #### Check Postgres versions across the cluster -After completing the upgrade on all nodes, while connected to one of the nodes, you can once again check your versions: +After completing the upgrade on all nodes, while connected to one of the nodes, you can check your versions again: ```bash sudo -u postgres pgd show-version @@ -585,7 +600,7 @@ This output shows that all the nodes are successfully upgraded to the new Postgr #### Reconcile with TPA -After all the nodes are upgraded, you still need to [reconcile](https://www.enterprisedb.com/docs/tpa/latest/reference/reconciling-local-changes/) the upgraded version of Postgres with TPA so you can continue to use TPA to manage the cluster in the future. +After all the nodes are upgraded, you still need to [reconcile](/tpa/latest/reference/reconciling-local-changes/) the upgraded version of Postgres with TPA so you can continue to use TPA to manage the cluster in the future. To do this, return to the command line where your TPA cluster directory resides. In this worked example, the TPA cluster directory is `/home/ubuntu/democluster` on the instance where you originally deployed the cluster using TPA. From c99faf1233a2d765afde0c50856717c1fd61f951 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Wed, 21 Aug 2024 16:26:14 -0400 Subject: [PATCH 02/16] Re-edit of PGD group 15 --- product_docs/docs/pgd/5/backup.mdx | 28 +++++++++++------------ product_docs/docs/pgd/5/compatibility.mdx | 4 ++-- product_docs/docs/pgd/5/index.mdx | 9 ++++---- product_docs/docs/pgd/5/known_issues.mdx | 6 ++--- product_docs/docs/pgd/5/parallelapply.mdx | 18 +++++++-------- 5 files changed, 31 insertions(+), 34 deletions(-) diff --git a/product_docs/docs/pgd/5/backup.mdx b/product_docs/docs/pgd/5/backup.mdx index f90209d9f9a..6754a9716f4 100644 --- a/product_docs/docs/pgd/5/backup.mdx +++ b/product_docs/docs/pgd/5/backup.mdx @@ -35,7 +35,7 @@ so a restore effectively resets all sequence kinds to the value of `bdr.default_sequence_kind` at time of restore. To create a post-restore script to reset the precise sequence kind -for each sequence, you might want to use an SQL script like this: +for each sequence, you might want to use a SQL script like this: ```sql SELECT 'SELECT bdr.alter_sequence_set_kind('''|| @@ -44,7 +44,7 @@ FROM bdr.sequences WHERE seqkind != 'local'; ``` -If pg_dump is run using `bdr.crdt_raw_value = on`, then you can reload the +If you run pg_dump using `bdr.crdt_raw_value = on`, then you can reload the dump only with `bdr.crdt_raw_value = on`. Technical Support recommends the use of physical backup techniques for @@ -52,7 +52,7 @@ backup and recovery of PGD. ### Physical backup -You can take physical backups of a node in a EDB Postgres Distributed cluster using +You can take physical backups of a node in an EDB Postgres Distributed cluster using standard PostgreSQL software, such as [Barman](https://www.enterprisedb.com/docs/supported-open-source/barman/). @@ -60,7 +60,7 @@ You can perform a physical backup of a PGD node using the same procedure that applies to any PostgreSQL node. A PGD node is just a PostgreSQL node running the BDR extension. -Consider these specific points to consider when applying +Consider these specific points when applying PostgreSQL backup techniques to PGD: - PGD operates at the level of a single database, while a physical @@ -83,15 +83,15 @@ PostgreSQL backup techniques to PGD: ### Eventual consistency -The nodes in a EDB Postgres Distributed cluster are *eventually consistent*, but not +The nodes in an EDB Postgres Distributed cluster are *eventually consistent* but not *entirely consistent*. A physical backup of a given node provides point-in-time recovery capabilities limited to the states -actually assumed by that node. +actually assumed by that node. The following example shows how two nodes in the same EDB Postgres Distributed cluster might not (and usually don't) go through the same sequence of states. -Consider a cluster with two nodes `N1` and `N2` that's initially in +Consider a cluster with two nodes, `N1` and `N2`, that's initially in state `S`. If transaction `W1` is applied to node `N1`, and at the same time a non-conflicting transaction `W2` is applied to node `N2`, then node `N1` goes through the following states: @@ -120,9 +120,9 @@ applied to `N2` until `T2`. PostgreSQL PITR is designed around the assumption of changes arriving from a single master in COMMIT order. Thus, PITR is possible by scanning through changes until one particular point in time (PIT) is reached. -With this scheme, you can restore one node to a single point in time +With this scheme, you can restore one node to a single PIT from its viewpoint, for example, `T1`. However, that state doesn't include other -data from other nodes that committed near that time but had not yet +data from other nodes that committed near that time but had not yet arrived on the node. As a result, the recovery might be considered to be partially inconsistent, or at least consistent for only one replication origin. @@ -146,7 +146,7 @@ but that's reached as different points for each origin separately. The WAL stream is read until requested origins have found their PIT. All changes are applied up until that point, except that -any transaction records are not marked as commmited for an origin after the PIT on that origin is +any transaction records aren't marked as committed for an origin after the PIT on that origin is reached. You end up with one LSN "stopping point" in WAL, but you also have one single @@ -174,7 +174,7 @@ recovery_target_time = T1 You need to specify the list of replication origins that are restored to `T1` in one of two ways. You can use a separate `multi_recovery.conf` file by way of -a new parameter `recovery_target_origins`: +a new parameter, `recovery_target_origins`: ``` recovery_target_origins = '*' @@ -224,7 +224,7 @@ of a single PGD node, optionally plus WAL archives: - Restore a single PostgreSQL node from a physical backup of one of the PGD nodes. - If you have WAL archives associated with the backup, create a suitable - `postgresql.conf` and start PostgreSQL in recovery to replay up to the latest + `postgresql.conf`, and start PostgreSQL in recovery to replay up to the latest state. You can specify an alternative `recovery_target` here if needed. - Start the restored node, or promote it to read/write if it was in standby recovery. Keep it fenced from any surviving nodes! @@ -245,7 +245,7 @@ To clean up leftover PGD metadata: You must explicitly remove replication origins with a separate step because they're recorded persistently in a system catalog. They're therefore included in the backup and in the restored instance. They -aren't removed automatically when dropping the BDR extension, because +aren't removed automatically when dropping the BDR extension because they aren't explicitly recorded as its dependencies. To track progress of incoming replication in a crash-safe way, @@ -270,7 +270,7 @@ If a physical backup was created with `pg_basebackup`, replication slots are omitted from the backup. Some other backup methods might preserve replications slots, likely in -outdated or invalid states. Once you restore the backup, use this to drop all replication slots: +outdated or invalid states. Once you restore the backup, use these commands to drop all replication slots: ``` SELECT pg_drop_replication_slot(slot_name) diff --git a/product_docs/docs/pgd/5/compatibility.mdx b/product_docs/docs/pgd/5/compatibility.mdx index af364077922..873ac4b4cdf 100644 --- a/product_docs/docs/pgd/5/compatibility.mdx +++ b/product_docs/docs/pgd/5/compatibility.mdx @@ -1,10 +1,10 @@ --- -title: PGD Compatibility by PostgreSQL Version +title: PGD compatibility by PostgreSQL version navTitle: Compatibility description: Compatibility of EDB Postgres Distributed with different versions of PostgreSQL --- -The following table shows which major versions of PostgreSQL are compatible with each version of EDB Postgres Distributed (PGD). +The following table shows the major versions of PostgreSQL and each version of EDB Postgres Distributed (PGD) they are compatible with. | Postgres
Version | PGD 5 | PGD 4 | PGD 3.7 | PGD 3.6 | |----------------------|--------------|--------------|------------------|------------------| diff --git a/product_docs/docs/pgd/5/index.mdx b/product_docs/docs/pgd/5/index.mdx index 093a247b6dd..a501ea26290 100644 --- a/product_docs/docs/pgd/5/index.mdx +++ b/product_docs/docs/pgd/5/index.mdx @@ -51,11 +51,11 @@ EDB Postgres Distributed (PGD) provides multi-master replication and data distri By default, EDB Postgres Distributed uses asynchronous replication, applying changes on the peer nodes only after the local commit. You can configure additional levels of synchronicity between different nodes, groups of nodes, or all nodes by configuring [Group Commit](durability/group-commit), [CAMO](durability/camo), or -[Eager](consistency/eager) Replication. +[Eager Replication](consistency/eager). ## Compatibility -EDB Postgres Distributed 5 is compatible with +EDB Postgres Distributed 5 is compatible with the package versions shown in the table. Package | Versions -------- | ------------ @@ -64,8 +64,7 @@ EDB Postgres Extended Server | 12-16 EDB Postgres Advanced Server | 12-16 !!! Note Postgres 16 support -Postgres 16 support is only available in EDB Postgres Distributed 5.3 and later +Postgres 16 support is available only in EDB Postgres Distributed 5.3 and later. !!! -For feature compatibility with compatible servers, see [Choosing a Postgres distribution](/pgd/latest/planning/choosing_server). - +For feature compatibility with compatible servers, see [Choosing a Postgres distribution](planning/choosing_server). diff --git a/product_docs/docs/pgd/5/known_issues.mdx b/product_docs/docs/pgd/5/known_issues.mdx index 633290321d2..296c602a67b 100644 --- a/product_docs/docs/pgd/5/known_issues.mdx +++ b/product_docs/docs/pgd/5/known_issues.mdx @@ -38,7 +38,7 @@ release. - Changing the CAMO partners in a CAMO pair isn't currently possible. It's possible only to add or remove a pair. - Adding or removing a pair doesn't need a restart of Postgres or even a + Adding or removing a pair doesn't require a restart of Postgres or even a reload of the configuration. - Group Commit can't be combined with [CAMO](durability/camo/) or [Eager @@ -48,7 +48,7 @@ release. - Transactions using Eager Replication can't yet execute DDL. The TRUNCATE command is allowed. - - Parallel apply isn't currently supported in combination with Group Commit. Make sure to disable it when using Group Commit by either: +- Parallel Apply isn't currently supported in combination with Group Commit. Make sure to disable it when using Group Commit by either: - Setting `num_writers` to 1 for the node group using [`bdr.alter_node_group_config`](/pgd/latest/reference/nodes-management-interfaces/#bdralter_node_group_config). - Using the GUC `bdr.writers_per_subscription`. See [Configuration of generic replication](/pgd/latest/reference/pgd-settings/#generic-replication). @@ -69,7 +69,7 @@ release. - When using [`bdr.add_commit_scope`](/pgd/latest/reference/functions#bdradd_commit_scope), - if a new commit scope is added which has the same name as a commit scope on + if a new commit scope is added that has the same name as a commit scope on any group, then the commit scope silently overwrites the commit scope but retains the original group the scope was associated with (if any). To modify a commit scope safely, use diff --git a/product_docs/docs/pgd/5/parallelapply.mdx b/product_docs/docs/pgd/5/parallelapply.mdx index 6bcea831369..933f8aabb3e 100644 --- a/product_docs/docs/pgd/5/parallelapply.mdx +++ b/product_docs/docs/pgd/5/parallelapply.mdx @@ -70,11 +70,11 @@ performance. The `nprovisional_waits` value reflects the number of operations on the same tuples being performed by concurrent apply transactions. These are provisional -waits that aren't actually waiting yet but could. +waits that aren't actually waiting yet but could. If a tuple's write needs to wait until it can be safely applied, it's counted in `ntuple_waits`. Fully applied transactions that waited before being committed -are counted in `ncommit_waits`. +are counted in `ncommit_waits`. ### Disabling Parallel Apply @@ -95,20 +95,20 @@ resulting error could manifest as a deadlock: - That other transaction executed on the origin node before the pending transaction did. - The pending transaction took out a lock request. -Additionally, handling the error could increase replication lag, due to a +Additionally, handling the error could increase replication lag due to a combination of the time taken: - To detect the deadlock -- For the client to roll back its transaction -- For indirect garbage collection of the changes that were already applied -- To redo the work +- For the client to roll back its transaction +- For indirect garbage collection of the changes that were already applied +- To redo the work This is where Parallel Apply’s deadlock mitigation, introduced in PGD 5.2, can help. For any transaction, Parallel Apply looks at transactions already scheduled for any row (tuple) that the current transaction wants to write. If it finds one, the row is marked as needing to wait until the other transaction is committed before applying its change to the row. This approach ensures that rows -are written in the correct order. +are written in the correct order. -### Parallel Apply Support +### Parallel Apply support Up to and including PGD 5.1, don't use Parallel Apply with Group Commit, CAMO, and Eager Replication. Disable Parallel Apply in these scenarios. If, using PGD 5.1 or earlier, you're experiencing a large number of deadlocks, you might also want to disable Parallel Apply or consider upgrading. From PGD 5.2, Parallel Apply works with CAMO. It isn't compatible with Group Commit or Eager Replication, so disable it if Group Commit or Eager Replication are in use. - - From 5e2b4729ed5d7f51a6085e7f0c65aff7492d1a4d Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Thu, 22 Aug 2024 11:17:06 +0100 Subject: [PATCH 03/16] Update product_docs/docs/pgd/5/parallelapply.mdx --- product_docs/docs/pgd/5/parallelapply.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/parallelapply.mdx b/product_docs/docs/pgd/5/parallelapply.mdx index 933f8aabb3e..fe4289cc90c 100644 --- a/product_docs/docs/pgd/5/parallelapply.mdx +++ b/product_docs/docs/pgd/5/parallelapply.mdx @@ -70,7 +70,7 @@ performance. The `nprovisional_waits` value reflects the number of operations on the same tuples being performed by concurrent apply transactions. These are provisional -waits that aren't actually waiting yet but could. +waits that aren't actually waiting yet but could start waiting. If a tuple's write needs to wait until it can be safely applied, it's counted in `ntuple_waits`. Fully applied transactions that waited before being committed From 9570427779d029d343dc4cf4fd51ba0d92703239 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 21 Aug 2024 14:42:29 +0100 Subject: [PATCH 04/16] First refresh for 3.0 Signed-off-by: Dj Walker-Morgan --- .../docs/livecompare/3/advanced_usage.mdx | 139 ++++++ .../docs/livecompare/3/bdr_support.mdx | 373 +++++++++++++++ .../docs/livecompare/3/command_line_usage.mdx | 239 ++++++++++ product_docs/docs/livecompare/3/index.mdx | 59 +++ product_docs/docs/livecompare/3/licenses.mdx | 359 ++++++++++++++ .../docs/livecompare/3/oracle_support.mdx | 238 ++++++++++ .../livecompare/3/rel_notes/3.0_rel_notes.mdx | 27 ++ .../docs/livecompare/3/rel_notes/index.mdx | 14 + .../docs/livecompare/3/requirements.mdx | 93 ++++ .../livecompare/3/settings/connections.mdx | 444 ++++++++++++++++++ .../docs/livecompare/3/settings/filters.mdx | 327 +++++++++++++ .../docs/livecompare/3/settings/general.mdx | 412 ++++++++++++++++ .../docs/livecompare/3/settings/index.mdx | 12 + .../livecompare/3/supported_technologies.mdx | 54 +++ 14 files changed, 2790 insertions(+) create mode 100644 product_docs/docs/livecompare/3/advanced_usage.mdx create mode 100644 product_docs/docs/livecompare/3/bdr_support.mdx create mode 100644 product_docs/docs/livecompare/3/command_line_usage.mdx create mode 100644 product_docs/docs/livecompare/3/index.mdx create mode 100644 product_docs/docs/livecompare/3/licenses.mdx create mode 100644 product_docs/docs/livecompare/3/oracle_support.mdx create mode 100644 product_docs/docs/livecompare/3/rel_notes/3.0_rel_notes.mdx create mode 100644 product_docs/docs/livecompare/3/rel_notes/index.mdx create mode 100644 product_docs/docs/livecompare/3/requirements.mdx create mode 100644 product_docs/docs/livecompare/3/settings/connections.mdx create mode 100644 product_docs/docs/livecompare/3/settings/filters.mdx create mode 100644 product_docs/docs/livecompare/3/settings/general.mdx create mode 100644 product_docs/docs/livecompare/3/settings/index.mdx create mode 100644 product_docs/docs/livecompare/3/supported_technologies.mdx diff --git a/product_docs/docs/livecompare/3/advanced_usage.mdx b/product_docs/docs/livecompare/3/advanced_usage.mdx new file mode 100644 index 00000000000..6c16fae833a --- /dev/null +++ b/product_docs/docs/livecompare/3/advanced_usage.mdx @@ -0,0 +1,139 @@ +--- +navTitle: Advanced usage +title: Advanced usage +originalFilePath: advanced_usage.md + +--- + +When LiveCompare runs, it creates a folder called `lc_session_` in the working directory. This folder contains the following files: + +- `lc__.log` — Log file for the session. + +- `summary_.out` — A list of all tables that were processed. For each table, it shows the time LiveCompare took to process the table, the total number of rows and how many rows were processed, how many differences were found in the table, and the maximum number of ignored columns, if any. + + To get the complete summary, you can also execute the following query against the output database: + + ```postgresql + select * + from .vw_table_summary + where session_id = ; + ``` + +- `differences_.out` — Useful information about any differences. This file isn't generated if there are no differences. + + The following is an example of a difference list: + +```text ++-------------------+-------------------------+-----------------+---------------------+ +| table_name | table_pk_column_names | difference_pk | difference_status | +|-------------------+-------------------------+-----------------+---------------------| +| public.categories | category | (7) | P | +| public.categories | category | (10) | P | +| public.categories | category | (17) | P | +| public.categories | category | (18) | P | ++-------------------+-------------------------+-----------------+---------------------+ +``` + +To get the full list of differences with all details, you can execute the following query against the output database: + + ```postgresql + select * + from ; + ``` + + To understand how LiveCompare consensus worked to decide which databases are divergent, the view `vw_consensus` can provide details on the consensus algorithm: + + ```postgresql + select * + from ; + ``` + +- `apply_on_the_first_.sql` — If there are any differences, this file shows a DML command to apply on the first database to make it consistent with all other databases. The following is an example of a script for the differences shown in the table: + + ```postgresql + BEGIN; + + DELETE FROM public.categories WHERE (category) = 7; + UPDATE public.categories SET categoryname = $lc1$Games Changed$lc1$ WHERE (category) = 10; + INSERT INTO public.categories (category,categoryname) VALUES (17, $lc1$Test 1$lc1$); + INSERT INTO public.categories (category,categoryname) VALUES (18, $lc1$Test 2$lc1$); + + COMMIT; + ``` + + LiveCompare generates this script. To fix the inconsistencies in the first database, execute the script in it. + + LiveCompare generates a similar `apply_on_*.sql` script for each database that has inconsistent data. + +## Aborting comparisons + +Before starting the comparison session, LiveCompare tries all connections. If the number of reachable connections isn't at least two, then LiveCompare aborts the whole session and gives an error message. If at least two connections are reachable, then LiveCompare proceeds with the comparison session. For all connections, LiveCompare writes a flag `connection_reachable` in the `connections` table in the cache database. + +For all reachable connections, LiveCompare does some sanity checks around the database technologies and the setting `logical_replication_mode`. If any of the sanity checks fail, then LiveCompare aborts the comparison and gives an error message. + +Considering the tables available on all reachable connections, LiveCompare builds the list of tables to compare, taking into account the table filter. If a specific table doesn't exist on at least two connections, then the comparison on that specific table is aborted. + +LiveCompare initially gathers metadata from all tables. This step is called *setup*. If any errors happen during the setup, for example, the user doesn't have access to a specific table, then it's called a *setup error*. If `abort_on_setup_error` is enabled, then LiveCompare aborts the whole comparison session, and the program finishes with an error message. Otherwise, only the table having the error has its table comparison aborted, and LiveCompare moves on to the next table. + +For each table that LiveCompare starts the table comparison on, LiveCompare first checks the table definition on all reachable connections. If the tables don't have the same columns and column data types, LiveCompare applies `column_intersection`. If there are no columns to compare, then LiveCompare aborts the table comparison. + +## Comparison key + +For each table being compared, when gathering the table metadata, LiveCompare builds the comparison key to use in the table comparison, following these rules: + +1. Use the custom comparison key if configured. + +1. Alternatively, use PK if available. + +1. Alternatively, if the table has `UNIQUE` indexes, among the `UNIQUE` indexes that have all `NOT NULL` columns, use the `UNIQUE` index with fewer columns. + +1. If none of these are possible, try to use all `NOT NULL` columns as a comparison key. `NULL` columns are also considered if `ignore_nullable = false`. + +If you decide to use strategies 1 or 4 as a comparison key, then LiveCompare also checks for uniqueness on the key. If uniqueness isn't possible, then LiveCompare aborts the comparison on that table. You can disable this behavior by using `check_uniqueness_enforcement = false`. + +## Differences to fix + +LiveCompare can identify and provide fixes for the following differences: + +- A row exists in the majority of the data connections. The fix is an `INSERT` on the divergent databases. +- A row doesn't exist in the majority of the data connections. The fix is a `DELETE` on the divergent databases. +- A row exists in all databases, but some column values mismatch. The fix is an `UPDATE` on the divergent databases. + +The default setting is `difference_statements = all`, which means that LiveCompare tries to apply all three DML types (`INSERT`, `UPDATE`, and `DELETE`) for each difference it finds. But you can specify the type of DML for LiveCompare to consider when providing difference fixes. Change the value of the setting `difference_statements`to any of these values: + +- `all` (default): Fixes `INSERT`, `UPDATE`, and `DELETE` DML types. +- `inserts`: Fixes only `INSERT` DML types. +- `updates`: Fixes only `UPDATE` DML types. +- `deletes`: Fixes only `DELETE` DML types. +- `inserts_updates`: Fixes only `INSERT` and `UPDATE` DML types. +- `inserts_deletes`: Fixes only `INSERT` and `DELETE` DML types. +- `updates_deletes`: Fixes only `UPDATE` and `DELETE` DML types. + +When `difference_statements` has the values `all`, `updates`, `inserts_updates`, or `updates_deletes`, then you can tell LiveCompare to ignore any `UPDATE` that sets `NULL` to a column. + +## Difference log + +The table `difference_log` stores all information about differences every time LiveCompare checks them. You can run LiveCompare in recheck mode multiple times, so this table shows how the difference evolved over the time window +in which LiveCompare was rechecking it. + +- **Detected (D)**: The difference was just detected. In recheck and fix modes, LiveCompare marks all Permanent and Tie differences as Detected so it can recheck them. + +- **Permanent (P)**: After rechecking the difference, if data is still divergent, LiveCompare marks the difference as Permanent. + +- **Tie (T)**: This entry is the same as Permanent, but there isn't enough consensus to determine the connections that are the majority. + +- **Absent (A)**: If, upon a recheck, LiveCompare finds that the difference doesn't exist anymore, that is, the row is now consistent between both databases, then LiveCompare marks the difference as Absent. + +- **Volatile (V)**: If, upon a recheck, `xmin` changed on an inconsistent row, then LiveCompare marks the difference as Volatile. + +- **Ignored (I)**: You can stop difference recheck of certain differences by manually calling the function `.accept_divergence(session_id, table_name, difference_pk)` in the output PostgreSQL connection. For example: + +```postgresql +SELECT livecompare.accept_divergence( + 2 -- session_id + , 'public.categories' -- table_name + , $$(10)$$ -- difference_pk +); +``` diff --git a/product_docs/docs/livecompare/3/bdr_support.mdx b/product_docs/docs/livecompare/3/bdr_support.mdx new file mode 100644 index 00000000000..7585dd5b5fb --- /dev/null +++ b/product_docs/docs/livecompare/3/bdr_support.mdx @@ -0,0 +1,373 @@ +--- +navTitle: PGD support +title: EDB Postgres Distributed support +originalFilePath: bdr_support.md + +--- + +You can use LiveCompare against EDB Postgres Distributed (PGD, formerly known as BDR) nodes as well as non-PGD nodes. + +Setting `logical_replication_mode = bdr` makes the tool assume that all databases being compared belong to the same PGD cluster. Then you can specify node names as connections and replication sets to filter tables. + +For example, suppose you can connect to any node in the PGD cluster, which we'll refer to as the initial connection. By initially connecting to this node, LiveCompare can check PGD metadata and retrieve connection information from all other nodes. + +Now suppose you want to compare three PGD nodes. As LiveCompare can connect to any node starting from the initial connection, you don't need to define `dsn` or any connection information for the data connections. You only need to +define `node_name`. LiveCompare searches in PGD metadata about the connection information for that node and then connects to the node. + +For LiveCompare to connect to all other nodes by fetching PGD metadata, LiveCompare must be able to connect to +them using the same DSN from the PGD view `bdr.node_summary` in the field `interface_connstr`. In this case, we recommend running LiveCompare on the same machine as the initial connection as the postgres user. If that's not possible, then define the `dsn` attribute in all data connections. + +You can also specify replication sets as table filters. LiveCompare uses PGD metadata to build the table list, considering only tables that belong to the replication sets you defined in the `replication_sets` setting. + +For example, you can create an `.ini` file to compare three PGD nodes: + +```ini +[General Settings] +logical_replication_mode = bdr +max_parallel_workers = 4 + +[Initial Connection] +dsn = port=5432 dbname=live user=postgres + +[Node1 Connection] +node_name = node1 + +[Node2 Connection] +node_name = node2 + +[Node3 Connection] +node_name = node3 + +[Output Connection] +dsn = port=5432 dbname=liveoutput user=postgres + +[Table Filter] +replication_sets = set_name = 'bdrgroup' +``` + +You can also tell LiveCompare to compare all active nodes in the PGD cluster. To do so: + +1. Under `General Settings`, enable `all_bdr_nodes = on`. +1. Under `Initial Connection`, specify an initial connection. + +Additional data connections aren't required. + +For example: + +```ini +[General Settings] +logical_replication_mode = bdr +max_parallel_workers = 4 +all_bdr_nodes = on + +[Initial Connection] +dsn = port=5432 dbname=live user=postgres + +[Output Connection] +dsn = port=5432 dbname=liveoutput user=postgres + +[Table Filter] +replication_sets = set_name = 'bdrgroup' +``` + +When `all_bdr_nodes = on`, LiveCompare uses the `Initial Connection` setting to fetch the list of all PGD nodes. While additional data connections aren't required, if set, they're appended to the list of data connections. For example, you can compare a whole PGD cluster against a single Postgres connection, which is useful in migration projects: + +```ini +[General Settings] +logical_replication_mode = bdr +max_parallel_workers = 4 +all_bdr_nodes = on + +[Initial Connection] +dsn = port=5432 dbname=live user=postgres + +[Old Connection] +dsn = host=oldpg port=5432 dbname=live user=postgres + +[Output Connection] +dsn = port=5432 dbname=liveoutput user=postgres + +[Table Filter] +replication_sets = set_name = 'bdrgroup' +``` + +The settings `node_name` and `replication_sets` are supported for the following technologies: + +- PGD 1, 2, 3, and 4 +- pglogical 2 and 3 + +To enable pglogical metadata fetch instead of PGD, set `logical_replication_mode = pglogical` instead of `logical_replication_mode = bdr`. + +## PGD witness nodes + +Using replication sets in PGD, you can configure specific tables to include in the PGD replication. You can also specify the nodes to receive data from these tables by configuring the node to subscribe to the replication +set the table belongs to. This setting allows for different architectures such as PGD sharding and the use of PGD witness nodes. + +A PGD witness is a regular PGD node that doesn't replicate any DML from other nodes. The purpose of the witness is to provide quorum in Raft Consensus voting. (For details on the PGD witness node, see [Witness nodes](/pgd/latest/node_management/witness_nodes/) in the PGD documentation.) Replication set configuration determines whether the witness replicates DDLs. This means that there are two types of PGD witnesses: + +- A completely empty node, without any data nor tables +- A node that replicates DDL from other nodes, so it has empty tables + +In the first case, even if the PGD witness is included in the comparison (either manually under `[Connections]` or using `all_bdr_nodes = on`), because the witness doesn't have any tables, the following message is logged: + +``` +Table public.tbl does not exist on connection node1 +``` + +In the second case, the table exists on the PGD witness. However, it's not correct to report data missing on the witness as divergences. So, for each table, LiveCompare checks the following information on each node included in the comparison: + +- The replication sets that the node subscribes to +- The replication sets that the table is associated with +- The replication sets, if any, you defined in the filter `replication_sets` under `Table Filter` + +If the intersection among all three lists of replication sets is empty, which is the case for the PGD witness, then LiveCompare logs this message: + +``` +Table public.tbl is not subscribed on connection node1 +``` + +In both cases, the comparison for that table proceeds on the nodes where the table exists, and the table is replicated according to the replication sets configuration. + +## Differences in a PGD cluster + +LiveCompare makes changes only to the local node. It's important that corrective changes don't get replicated to other nodes. + +When `logical_replication_mode = bdr`, LiveCompare first checks if a replication origin called `bdr_local_only_origin` already exists. (You can configure the name of the replication origin by adjusting the setting `difference_fix_replication_origin`.) If a replication origin called `bdr_local_only_origin` doesn't exist, then LiveCompare creates it on all PGD connections. + +!!! Important + PGD 3.6.18 introduced the new preexisting `bdr_local_only_origin` replication origin to use for applying local-only transactions. If LiveCompare is connected to PGD 3.6.18, it doesn't create this replication origin. + +LiveCompare generates apply scripts considering the following: + +- Set the current transaction to use the replication origin `bdr_local_only_origin`, so any DML executed has `xmin` associated with `bdr_local_only_origin`. +- Set the current transaction datetime to be far in the past, so if there are any PGD conflicts with real DML being executed on the database, LiveCompare DML always loses the conflict. + +After applying a LiveCompare fix script to a PGD node, you can get exactly the rows that were inserted or updated by LiveCompare using the following query. Replace `mytable` with the name of any table. + +```postgresql +with lc_origin as ( + select roident + from pg_replication_origin + where roname = 'bdr_local_only_origin' +) +select t.* +from mytable t +inner join lc_origin r +on r.roident = bdr.pg_xact_origin(t.xmin); +``` + +Deleted rows are no longer visible. + +LiveCompare requires at least a PostgreSQL user with bdr_superuser privileges to properly fetch metadata. + +All of these steps involving replication origins applied only to the output script if the PostgreSQL user has bdr_superuser or PostgreSQL superuser privileges. Otherwise, LiveCompare generates fixes without associating any replication origin. (Transaction replication is still disabled using `SET LOCAL bdr.xact_replication = off`.) However, we recommend using a replication origin when applying the DML scripts. Otherwise, LiveCompare has the same precedence as a regular user application regarding conflict resolution. Also, as there isn't any replication origin associated with the fix, you can't use the query to list all rows fixed by LiveCompare. + +Between PGD 3.6.18 and PGD 3.7.0, the following functions are used: + +- `bdr.difference_fix_origin_create()`: Executed by LiveCompare to create the replication origin specified in `difference_fix_replication_origin` (by default, set to `bdr_local_only_origin`), if this replication origin doesn't exist. +- `bdr.difference_fix_session_setup()`: Included in the generated DML script so the transaction is associated with the replication origin specified in `difference_fix_replication_origin`. +- `bdr.difference_fix_xact_set_avoid_conflict()`: Included in the generated DML script so the transaction is set far in the past (`2010-01-01`). The fix transaction applied by LiveCompare always loses any conflict. + +These functions require a bdr_superuser rather than a PostgreSQL superuser. Starting with PGD 3.7.0, those functions are deprecated. In that case, if running as a PostgreSQL superuser, LiveCompare uses the following functions to perform the same actions: + +- `pg_replication_origin_create(origin_name)`; +- `pg_replication_origin_session_setup()`; +- `pg_replication_origin_xact_setup()`. + +If a PostgreSQL superuser isn't being used, then LiveCompare includes only the following in the generated DML transaction: + +``` +SET LOCAL bdr.xact_replication = off; +``` + +## Conflicts in PGD + +LiveCompare has an execution mode called `conflicts`. This execution mode is specific for PGD clusters. It works only in PGD 3.6, PGD 3.7, PGD 4, and PGD 5 clusters. + +While `compare` mode is used to compare all content of tables as a whole, `conflicts` mode focuses just in tuples/tables that are related to existing conflicts that are registered in `bdr.apply_log`, in case of PGD 3.6, or in +`bdr.conflict_history`, in case of PGD 3.7, PGD 4, and PGD 5. + +`conflicts` execution mode is expected to run much faster than `compare` mode because it inspects only specific tuples from specific tables. However, it's not as complete as `compare` mode for the same reason. + +The main objective of this execution mode is to check that the automatic conflict resolution that's being done by PGD is consistent among nodes, that is, after PGD resolves conflicts, the cluster is in a consistent state. + +Although, for the general use case, automatic conflict resolution ensures cluster consistency, there are a few known cases where automatic conflict resolution can result in divergent tuples among nodes. So the `conflicts` execution mode from LiveCompare can help with checking and ensuring consistency, providing a good balance between time and result. + +### Conflict example + +Suppose on `node3`, you execute the following query: + +``` +SELECT c.reloid::regclass, + s.origin_name, + c.local_time, + c.key_tuple, + c.local_tuple, + c.remote_tuple, + c.apply_tuple, + c.conflict_type, + c.conflict_resolution +FROM bdr.conflict_history c +INNER JOIN bdr.subscription_summary s +ON s.sub_id = c.sub_id; +``` + +You can see the following conflict in `bdr.conflict_history`: + +``` +reloid | tbl +origin_name | node2 +local_time | 2021-05-13 19:17:43.239744+00 +key_tuple | {"a":null,"b":3,"c":null} +local_tuple | +remote_tuple | +apply_tuple | +conflict_type | delete_missing +conflict_resolution | skip +``` + +This conflict means that when the `DELETE` arrived from `node2` to `node3`, there was no row with `b = 3` in table `tbl`. However, the `INSERT` might have arrived from `node1` to `node3` later, which then added the row with `b = 3` to `node3`. So this is the current situation on `node3`: + +``` +bdrdb=# SELECT * FROM tbl WHERE b = 3; + a | b | c +---+---+----- + x | 3 | foo +(1 row) +``` + +While on nodes `node1` and `node2`, you see this: + +``` +bdrdb=# SELECT * FROM tbl WHERE b = 3; + a | b | c +---+---+--- +(0 rows) +``` + +The PGD cluster is divergent. + +To detect and fix such divergence, you can execute LiveCompare in `compare` mode. However, depending on the size of the comparison set (suppose table `tbl` is very large), that can take a long time, even hours. + +This situation is one in in which `conflicts` mode can be helpful. In this case, the `delete_missing` conflict is visible only from `node3`, but LiveCompare can extract the PK values from the conflict logged rows (`key_tuple`, `local_tuple`, `remote_tuple`, and `apply_tuple`) and perform an automatic cluster-wide comparison only on the affected table, already filtering by the PK values. The comparison then checks the current row version in all nodes in the cluster. + +Create a `check.ini` file to set `all_bdr_nodes = on`, that is, to tell LiveCompare to compare all nodes in the cluster: + +``` +[General Settings] +logical_replication_mode = bdr +max_parallel_workers = 2 +all_bdr_nodes = on + +[Initial Connection] +dsn = dbname=bdrdb + +[Output Connection] +dsn = dbname=liveoutput +``` + +To run LiveCompare in `conflicts` mode: + +``` +livecompare check.ini --conflicts +``` + +After the execution, in the console output, you see something like this: + +``` +Elapsed time: 0:00:02.443557 +Processed 1 conflicts about 1 tables from 3 connections using 2 workers. +Found 1 divergent conflicts in 1 tables. +Processed 1 rows in 1 tables from 3 connections using 2 workers. +Found 1 inconsistent rows in 1 tables. +``` + +Inside folder `./lc_session_X/` (`X` is the number of the current comparison session), LiveCompare writes the file `conflicts_DAY.out` (replacing `DAY` in the name of the file with the current day). The file shows the main information about all divergent conflicts. + +If you connect to database `liveoutput`, you can see more details about the conflicts, for example, using this query: + +``` +SELECT * +FROM livecompare.vw_conflicts +WHERE session_id = 1 + AND conflict_id = 1 +ORDER BY table_name, + local_time, + target_node; +``` + +The output is something like this: + +``` +session_id | 1 +table_name | public.tbl +conflict_id | 1 +connection_id | node3 +origin_node | node2 +target_node | node3 +local_time | 2021-05-13 19:17:43.239744+00 +key_tuple | {"a": null, "b": 3, "c": null} +local_tuple | +remote_tuple | +apply_tuple | +conflict_type | delete_missing +conflict_resolution | skip +conflict_pk_value_list | {(3)} +difference_log_id_list | {1} +is_conflict_divergent | t +``` + +The `is_conflict_divergent = true` means that LiveCompare compared the conflict and found the nodes to be currently divergent in the tables and rows reported by the conflict. The view `livecompare.vw_conflicts` shows information about all conflicts, including the non-divergent ones. + +LiveCompare also generates the DML script `./lc_session_X/apply_on_the_node3_DAY.sql` (where `DAY` in the name of the file with the current day): + +``` +BEGIN; + +SET LOCAL bdr.xact_replication = off; +SELECT pg_replication_origin_session_setup('bdr_local_only_origin'); +SELECT pg_replication_origin_xact_setup('0/0', '2010-01-01'::timestamptz);; + +SET LOCAL ROLE postgres; +DELETE FROM public.tbl WHERE (b) = (3); + +COMMIT; +``` + +LiveCompare is suggesting to `DELETE` the row where `b = 3` from `node3` because the row doesn't exist on the other two rows. By default, LiveCompare suggests the DML to fix based on the majority of the nodes. + +Running this DML script against `node3` makes the PGD cluster consistent again: + +``` +psql -h node3 -f ./lc_session_X/apply_on_the_node3_DAY.sql +``` + +As the `--conflicts` mode comparison is much faster than a full `--compare`, we strongly recommend scheduling a `--conflicts` comparison session more often to ensure conflict resolution is providing cluster-wide consistency. + +!!! Note + To see the data in `bdr.conflict_history` in PGD 3.7 or `bdr.apply_log` in PGD 3.6, run LiveCompare with a user that's a bdr_superuser or a PostgreSQL superuser. + +To be able to see the data in `bdr.conflict_history` +in PGD 3.7+ or `bdr.apply_log` in PGD 3.6, run LiveCompare with a +user that's bdr_superuser or a PostgreSQL superuser. + +### Conflicts Filter + +You can also tell LiveCompare to filter the conflicts by any of the columns in either `bdr.conflicts_history` or `bdr.apply_log`. For example: + +```ini +[Conflicts Filter] +conflicts = table_name = 'public.tbl' and conflict_type = 'delete_missing' +``` + +## Mixing technologies + +Metadata for `node_name` and `replication_sets` are fetched in the initial connection. So it must be a pglogical- and/or PGD-enabled database. + +The list of tables is built in the first data connection. So the `replication_sets` condition must be valid in the first connection. + +You can perform mixed-technology comparisons, for example: + +- PGD 1 node versus PGD 3 node +- PGD 4 node versus vanilla Postgres instance +- Vanilla Postgres instance versus pglogical node diff --git a/product_docs/docs/livecompare/3/command_line_usage.mdx b/product_docs/docs/livecompare/3/command_line_usage.mdx new file mode 100644 index 00000000000..5cd59e1d777 --- /dev/null +++ b/product_docs/docs/livecompare/3/command_line_usage.mdx @@ -0,0 +1,239 @@ +--- +navTitle: Command line usage +title: Command-line usage +originalFilePath: command_line_usage.md + +--- + +## Compare mode + +Copy any `/etc/livecompare/template*.ini` to use in your project and adjust as necessary. See [Settings](settings/). + +``` +cp /etc/livecompare/template_basic.ini my_project.ini + +livecompare my_project.ini +``` + +While LiveCompare executes, N+1 progress bars appear, where N is the number of processes. (You can specify the number of processes in the settings.) The first progress bar shows overall execution. The other progress bars show the current table being processed by a specific process. + +The information being shown for each table is, from left to right: + +- Number of the process +- Table name +- Status, which can be the ID of the comparison round followed by the current table chunk. + + `p1/1` means the table wasn't split. A status of `setup` means the table is being analyzed (checking row count and splitting if necessary). + +- Number of rows processed +- Number of total rows being considered in this comparison round +- Time elapsed +- Estimated time to complete +- Speed in records per second + +While the program is executing, you can cancel it at any time by pressing **Ctrl-C**. A message like the following appears: + +```text +Manually stopping session 6... You can resume the session with: + +livecompare my_project.ini 6 +``` + +!!! Important + If LiveCompare is running in the background or running in another shell, you can still softly stop it. It keeps the `PID` of the master process inside the session folder (`lc_session_6` in the example) in a file named `livemaster.pid`. You can then invoke `kill -2 ` to softly stop it. + +Then, at any time you can resume a previously canceled session, for example: + +``` +livecompare my_project.ini 6 +``` + +When the program ends, if it found no inconsistencies, the output is similar to the following: + +```text +Saved file lc_session_5/summary_20190514.out with the complete table summary. +You can also get the table summary by connecting to the output database and executing: +select * from livecompare.vw_table_summary where session_id = 5; + +Elapsed time: 0:02:10.970954 +Processed 3919015 rows in 6 tables using 3 processes. +Found 0 inconsistent rows in 0 tables. +``` + +If any inconsistencies were found, the output looks like this: + +```text +Comparison finished, waiting for remaining difference checks... + +Outstanding differences: + ++--------------+-------------------+-----------------+------------------+----------------------+-------------------+---------------------------+ +| session_id | table_name | elapsed_time | num_total_rows | num_processed_rows | num_differences | max_num_ignored_columns | +|--------------+-------------------+-----------------+------------------+----------------------+-------------------+---------------------------| +| 6 | public.categories | 00:00:00.027864 | 18 | 18 | 4 | | ++--------------+-------------------+-----------------+------------------+----------------------+-------------------+---------------------------+ + +Saved file lc_session_6/summary_20200129.out with the complete table summary. +You can also get the table summary by connecting to the output database and executing: +select * from livecompare.vw_table_summary where session_id = 6; + +Elapsed time: 0:00:50.149987 +Processed 172718 rows in 8 tables from 3 connections using 2 workers. +Found 4 inconsistent rows in 1 tables. + +Saved file lc_session_6/differences_20200129.out with the list of differences per table. +You can also get a list of differences per table with: +select * from livecompare.vw_differences where session_id = 6; +Too see more details on how LiveCompare determined the differences: +select * from livecompare.vw_consensus where session_id = 6; + +Script lc_session_6/apply_on_the_first_20200129.sql was generated, which can be applied to the first connection and make it consistent with the majority of connections. +You can also get this script with: +select difference_fix_dml from livecompare.vw_difference_fix where session_id = 6 and connection_id = 'first'; +``` + +## Recheck mode + +In a PGD environment, any divergence that PGD finds can later not exist, as the replication caught up due to eventual consistency. Depending on several factors, replication lag can cause LiveCompare to report false positives. + +To overcome that, in a later moment when replication lag has decreased or data has already caught up, you can manually execute a recheck only on the differences that were previously found. This execution mode is called *recheck*. You can execute it like this: + +``` +livecompare my_project.ini 6 --recheck +``` + +In this mode, LiveCompare generates separate recheck logs and updates all reports that already exist in the `lc_session_X` directory. + +!!! Important + If resuming a `compare` or executing under `recheck`, LiveCompare checks whether the settings and connections attributes are the same as when the session was created. If any divergence is found, it quits the execution and gives a message. + +## Conflicts mode + +To run LiveCompare in `conflicts` mode, invoke it with: + +``` +livecompare my_project.ini --conflicts +``` + +For more details about the `conflicts` mode, see [PGD support](bdr_support/). + +## Dry-run mode + +!!! Tip "New Feature" + LiveCompare dry-run mode support is available for LiveCompare version 2.2.0 and later. +!!! + +For example, suppose you have the following INI file: + +```ini +[General Settings] +logical_replication_mode = off +difference_tie_breakers = first + +[First Connection] +dsn = dbname=testb + +[Second Connection] +dsn = dbname=testdb2 + +[Third Connection] +dsn = dbname=testdb3 + +[Output Connection] +dsn = dbname=liveoutpu + +[Table Filter] +schemas = schema_name = 'public' +``` + +As the DSN under `Output Connection` (the LiveCompare cache database) is incorrect, running LiveCompare initially fails with: + +``` +Output connection is not reachable. +``` + +After fixing this, then the output connection is now reachable. But suppose that only one of the data connections is set correctly. In that case, LiveCompare fails again with: + +``` +At least two reachable connections are required. +Following connections are unreachable: first, third. +Following connections are reachable: second. +``` + +LiveCompare can start a comparison with at least two data connections available. So you go ahead and fix the third connection. But LiveCompare still fails with: + +``` +A difference_tie_breakers host is not a reachable connection: first. +``` + +This happens because the example set `difference_tie_breakers = first`, and any connection set as a tie breaker or source of truth needs to be reachable. + +After fixing all those issues, then LiveCompare can start the comparison. + +However, when setting up a comparison from scratch, you can check beforehand whether LiveCompare will abort with a configuration error. Further checks of this nature are all shown in the order LiveCompare performs them. + +You can do this with the `--dry-run` mode, which: + +- Prints all execution aborts that will happen due to configuration issues. +- Prints the list of connections with some details, including if it's reachable. +- Prints the table filter. +- After applying the table filter, prints the list of tables that are common to the reachable connections. + +Here's one sample output, given the example `.ini` file, and all configuration errors regarding unreachable connections: + +``` +$ livecompare test.ini --dry-run +EnterpriseDB LiveCompare 2.2.0, dry-run mode + + +Output connection is not reachable. + +At least two reachable connections are required. +Following connections are unreachable: first, third. +Following connections are reachable: second. + +A difference_tie_breakers host is not a reachable connection: first. + + +Connections + ++--------+--------------+-----------+---------------+---------------------+-----------+---------------+-------------------+-------------+ +| ID | Technology | Version | PGD Version | Pglogical Version | Initial | Tie Breaker | Source of Truth | Reachable | +|--------+--------------+-----------+---------------+---------------------+-----------+---------------+-------------------+-------------| +| second | postgresql | 110015 | - | - | False | False | False | True | +| first | postgresql | - | - | - | False | True | False | False | +| third | postgresql | - | - | - | False | False | False | False | +| output | postgresql | - | - | - | - | - | - | False | ++--------+--------------+-----------+---------------+---------------------+-----------+---------------+-------------------+-------------+ + + +Table Filter + +publications = '' +replication_sets = '' +schemas = schema_name = 'public' +tables = '' + + +Tables + ++---------------------+--------------+-----------------+-------------------------+ +| Table Name | Row Filter | Column Filter | Custom Comparison Key | +|---------------------+--------------+-----------------+-------------------------| +| public.categories | - | - | - | +| public.cust_hist | - | - | - | +| public.customers | - | - | - | +| public.departments | - | - | - | +| public.dept_emp | - | - | - | +| public.dept_manager | - | - | - | +| public.employees | - | - | - | +| public.inventory | - | - | - | +| public.orderlines | - | - | - | +| public.orders | - | - | - | +| public.products | - | - | - | +| public.reorder | - | - | - | +| public.salaries | - | - | - | +| public.tbl | - | - | - | +| public.titles | - | - | - | ++---------------------+--------------+-----------------+-------------------------+ +``` diff --git a/product_docs/docs/livecompare/3/index.mdx b/product_docs/docs/livecompare/3/index.mdx new file mode 100644 index 00000000000..6a532499509 --- /dev/null +++ b/product_docs/docs/livecompare/3/index.mdx @@ -0,0 +1,59 @@ +--- +navigation: + - index + - rel_notes + - requirements + - supported_technologies + - command_line_usage + - advanced_usage + - bdr_support + - oracle_support + - settings + - licenses +title: LiveCompare +originalFilePath: index.md +--- + +LiveCompare is designed to compare any number of databases to verify they're identical. The tool compares the databases and generates a comparison report, a list of differences, and handy DML scripts so you can optionally +apply the DML and fix the inconsistencies in any of the databases. + +By default, the comparison set includes all tables in the database. LiveCompare allows checking of multiple tables concurrently (multiple worker processes) and is highly configurable to allow checking just a few tables or just a section of rows in a table. + +Each database comparison is called a *comparison session*. When the program starts for the first time, it starts a new session and starts comparing table by table. In standalone mode, once all tables are compared, the program stops and generates all reports. You can start and stop LiveCompare without losing context information, so you can run it at convenient times. + +Each table comparison operation is called a *comparison round*. If the table is too big, LiveCompare splits the table into multiple comparison rounds that are also executed in parallel, alongside other tables that are being carried on by other workers at the same time. + +In standalone mode, the initial comparison round for a table starts from the beginning of the table (oldest existing PK) to the end of the table (newest existing PK). New rows inserted after the round starts are ignored. LiveCompare sorts the PK columns to get min and max PK from each table. For each PK column that's unsortable, LiveCompare casts its content to `string`. In PostgreSQL, you achieve this by using `::text`. In Oracle, use `to_char`. + +When executing the comparison algorithm, each worker requires N+1 database connections, where N is the number of databases being compared. The extra required connection is to an output/reporting database, where the program cache is kept too, enabling you to stop and resume a comparison session. + +You can manually recheck any differences found by the comparison algorithm at a later, convenient time. We recommend doing this to allow a replication consistency check. Upon the difference recheck, replication might have caught up on that specific row and the difference doesn't exist anymore, so the difference is removed. Otherwise it's marked as permanent. + +At the end of the execution, the program generates a DML script so you can review it and fix differences one by one. Or you can apply the entire DML script to fix all permanent differences. + +You can potentially use LiveCompare to ensure logical data integrity at the row level, for example, for these scenarios: + +- Database technology migration (Oracle x Postgres). +- Server migration or upgrade (old server x new server). +- Physical replication (primary x standby). +- After failover incidents, for example to compare the new primary data against the old, isolated primary data. +- In case of an unexpected split-brain situation after a failover. If the old primary wasn't properly fenced and the application wrote data into it, you can use LiveCompare to know exactly the data that's present in the old primary and isn't present in the new primary. If they want, the DBA can use the DML script that LiveCompare generates to apply those data into the new primary. +- Logical replication. Three kinds of logical replication technologies are supported: Postgres native logical replication, pglogical, and EDB Postgres Distributed (PGD, formerly known as BDR). + +## Comparison performance + +LiveCompare is optimized for use on production systems and has various parameters for tuning. Comparison rounds are read-only workloads. An example use case compared 43,109,165 rows in 6 tables in 9m 17s with 4 connections and 4 workers, giving comparison performance of approximately 77k rows per second, or 1 billion rows in <4 hours. + +This use case is a general use case. For low-load, testing, migration, and other specific scenarios, you might be able to improve speed by changing the `data_fetch_mode` setting to use server-side cursors. In our experiments, each kind of server-side cursors provides an increase in performance on use cases involving either small or large tables. + +## Security considerations for the user + +For PostgreSQL 13 and earlier, LiveCompare requires a user that can read all data being compared. PostgreSQL 14 introduced a new role, pg_read_all_data, that can be used for LiveCompare. + +When `logical_replication_mode = bdr`, LiveCompare requires a user with the bdr_superuser role. When `logical_replication_mode = pglogical`, LiveCompare requires a user with the `pglogical_superuser` role. + +To apply the DML scripts in PGD, all divergent connections (potentially all data connections) require a user with the bdr_superuser role to disable `bdr.xact_replication`. + +If PGD is being used, LiveCompare associates all fixed rows with a replication origin called `bdr_local_only_origin`. LiveCompare also applies the DML with the transaction datetime far in the past, so if there are any PGD conflicts with real DML being executed on the database, LiveCompare DML always loses the conflict. + +With the default setting of `difference_fix_start_query`, the transaction in apply scripts changes role to the owner of the table to prevent database users from gaining access to the role applying fixes by writing malicious triggers. As a result, the user for the divergent connection needs to be able to switch role to the table owner. diff --git a/product_docs/docs/livecompare/3/licenses.mdx b/product_docs/docs/livecompare/3/licenses.mdx new file mode 100644 index 00000000000..57bfa5be7b5 --- /dev/null +++ b/product_docs/docs/livecompare/3/licenses.mdx @@ -0,0 +1,359 @@ +--- +title: Licenses +originalFilePath: licenses.md + +--- + +## TQDM + +`tqdm` is a product of collaborative work. Unless otherwise stated, all authors (see commit logs) retain copyright +for their respective work, and release the work under the MIT licence (text below). + +Exceptions or notable authors are listed below in reverse chronological order: + +- files: \* + MPLv2.0 2015-2020 (c) Casper da Costa-Luis + [casperdcl](https://github.com/casperdcl). +- files: tqdm/\_tqdm.py + MIT 2016 (c) [PR #96] on behalf of Google Inc. +- files: tqdm/\_tqdm.py setup.py README.rst MANIFEST.in .gitignore + MIT 2013 (c) Noam Yorav-Raphael, original author. + +[PR #96]: https://github.com/tqdm/tqdm/pull/96 + +### Mozilla Public License (MPL) v. 2.0 - Exhibit A + +This Source Code Form is subject to the terms of the +Mozilla Public License, v. 2.0. +If a copy of the MPL wasn't distributed with this file, +you can obtain one at . + +### MIT License (MIT) + +Copyright (c) 2013 noamraph + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +## oracledb + +LICENSE AGREEMENT FOR python-oracledb + +Copyright © 2016, 2024 Oracle and/or its affiliates. + +This software is dual-licensed to you under the Universal Permissive License (UPL) 1.0 as shown at https://oss.oracle.com/licenses/upl and Apache License 2.0 as shown at http://www.apache.org/licenses/LICENSE-2.0. You may choose either license. + +If you elect to accept the software under the Apache License, Version 2.0, the following applies: + +Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at + +https://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. + +The Universal Permissive License (UPL), Version 1.0 + +Subject to the condition set forth below, permission is hereby granted to any person obtaining a copy of this software, associated documentation and/or data (collectively the “Software”), free of charge and under any and all copyright rights in the Software, and any and all patent rights owned or freely licensable by each licensor hereunder covering either (i) the unmodified Software as contributed to or provided by such licensor, or (ii) the Larger Works (as defined below), to deal in both + +a. the Software, and +b. any piece of software and/or hardware listed in the lrgrwrks.txt file if one is included with the Software (each a “Larger Work” to which the Software is contributed by such licensors), + +without restriction, including without limitation the rights to copy, create derivative works of, display, perform, and distribute the Software and make, use, sell, offer for sale, import, export, have made, and have sold the Software and the Larger Work(s), and to sublicense the foregoing rights on either these or other terms. + +This license is subject to the following condition: + +The above copyright notice and either this complete permission notice or at a minimum a reference to the UPL must be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +### Apache license + +``` + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ +``` + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + +## Psycopg2 + +psycopg2 is free software: you can redistribute it and/or modify it +under the terms of the GNU Lesser General Public License as published +by the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +psycopg2 is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public +License for more details. + +In addition, as a special exception, the copyright holders give +permission to link this program with the OpenSSL library (or with +modified versions of OpenSSL that use the same license as OpenSSL), +and distribute linked combinations including the two. + +You must obey the GNU Lesser General Public License in all respects for +all of the code used other than OpenSSL. If you modify file(s) with this +exception, you may extend this exception to your version of the file(s), +but you are not obligated to do so. If you do not wish to do so, delete +this exception statement from your version. If you delete this exception +statement from all source files in the program, then also delete it here. + +You should have received a copy of the GNU Lesser General Public License +along with psycopg2 (see the doc/ directory.) +If not, see . + +### Alternative licenses + +The following BSD-like license applies (at your option) to the files following +the pattern `psycopg/adapter*.{h,c}` and `psycopg/microprotocol*.{h,c}`: + + Permission is granted to anyone to use this software for any purpose, + including commercial applications, and to alter it and redistribute it + freely, subject to the following restrictions: + +1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this + software in a product, an acknowledgment in the product documentation + would be appreciated but is not required. + +2. Altered source versions must be plainly marked as such, and must not + be misrepresented as being the original software. + +3. This notice may not be removed or altered from any source distribution. + +## Tabulate + + Copyright (c) 2011-2020 Sergey Astanin and contributors + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +© 2020 GitHub, Inc. + +## OmniDB + +MIT License + +Portions Copyright (c) 2015-2019, The OmniDB Team +Portions Copyright (c) 2017-2019, 2ndQuadrant Limited + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/product_docs/docs/livecompare/3/oracle_support.mdx b/product_docs/docs/livecompare/3/oracle_support.mdx new file mode 100644 index 00000000000..9065fb3b874 --- /dev/null +++ b/product_docs/docs/livecompare/3/oracle_support.mdx @@ -0,0 +1,238 @@ +--- +navTitle: Oracle support +title: Oracle support +originalFilePath: oracle_support.md + +--- + +You can use LiveCompare to compare data from an Oracle database against any number of PostgreSQL or PGD databases. + +For example, you can define `technology = oracle` in a data connection. You can then use other settings to define the connection to Oracle: + +- `host` +- `port` +- `service` +- `user` +- `password` + +All other data connections must be PostgreSQL. + +Here's a simple example of comparison between an Oracle database and a PostgreSQL database: + +```ini +[General Settings] +logical_replication_mode = off +max_parallel_workers = 4 +oracle_user_tables_only = on +oracle_ignore_unsortable = on +column_intersection = on +force_collate = C +difference_tie_breakers = oracle + +[Oracle Connection] +technology = oracle +host = 127.0.0.1 +port = 1521 +service = XE +user = LIVE +password = live + +[Postgres Connection] +technology = postgresql +dsn = dbname=liveoracle user=william + +[Output Connection] +dsn = dbname=liveoutput user=william + +[Table Filter] +schemas = schema_name = 'live' +``` + +Here, the `schema_name` in Oracle is the user table sandbox. All table names are schema qualified by default: + +- Postgres: ` . ` +- Oracle: ` . Connection`, with `` being any single-word string starting with an +alphabetic character. In this case, whatever you use as `Name` is called +the *connection ID* of the data connection. Each data +connection must also have a unique connection ID in the list of data connections. + +If `logical_replication_mode = bdr` and `all_bdr_nodes = on`, then you don't need +to specify any data connection. LiveCompare builds the +data connection list by fetching PGD metadata from the `Initial Connection`. + +### `technology` +RDBMS technology. Currently possible values are `postgresql` or + `oracle`. +### `node_name` +Name of the node in the cluster. Requires + `logical_replication_mode` set to `pglogical` or `bdr` and also requires that + the `Initial Connection` is filled. If `node_name` is set, then `dsn`, `host`, + `port`, `dbname`, and `user` settings are all ignored. +### `dsn` +PostgreSQL connection string. If `dsn` is set, then `host`, `port`, + `dbname`, and `user` are ignored. The `dsn` setting can also have all other + [parameter key words allowed by libpq](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS). +### `host` +Server address. Leave empty to use the Unix socket connection. +### `port` +Port. Default: `5432`. +### `dbname` +Database name. Default: `postgres`. +### `service` +Service name, used in Oracle connections. Default +`XE`. +### `user` +Database user. Default: `postgres`. + +### `password` +Plain text password. We don't recommend using this. However, it might be required in some legacy connections. + +### `application_name` +Application name. Can be used even if you set `dsn` + or `node_name` instead of all other connection information. +Default: `livecompare_`. + +### `start_query` +Arbitrary query that's executed each time a connection to a + database is open. + +### `fetch_fk_metadata` +Specifies whether LiveCompare gathers metadata about foreign keys + on the connection. +Default: `on`. + +## Table Filter + +If omitted or left empty, this section from the `.ini` file means that +LiveCompare executes against all tables in the first database. + +If you want LiveCompare to execute against a specific set of tables, there +are different ways to specify this: + +### `publications` +You can filter specific publications, and LiveCompare uses + only the tables associated with those publications. You can use the variable + `publication_name` to build the conditional expression, for example: + +```ini +publications = publication_name = 'livepub' +``` + +Requires `logical_replication_mode = native`. + +### `replication_sets` +When using pglogical or PGD, you can filter specific + replication sets, and LiveCompare works only on the tables associated with + those replication sets. You can use the variable `set_name` to build the + conditional expression, for example: + +```ini +replication_sets = set_name in ('default', 'bdrgroup') +``` + +Requires `logical_replication_mode = pglogical` or +`logical_replication_mode = bdr`. + +### `schemas` +You can filter specific schemas, and LiveCompare works only on + the tables that belong to those schemas. You can use the variable `schema_name` + to build the conditional expression, for example: + +```ini +schemas = schema_name != 'badschema' +``` + +### `tables` +The variable `table_name` can help you build a conditional + expression to filter only the tables you want LiveCompare to work on, for + example: + +```ini +tables = table_name not like '%%account' +``` + +In any conditional expression, escape the `%` character as `%%`. + +The table name must be schema-qualified, unless `schema_qualified_table_names` +is disabled. For example, you can filter only a specific list of +tables: + +``` +tables = table_name in ('myschema1.mytable1', 'myschema2.mytable2') +``` + +If you disable the general setting `schema_qualified_table_names`, then you +must also set an appropriate `search_path` for Postgres in the connection +`start_query` setting, for example: + +``` +[General Setting] +... +schema_qualified_table_names = off + +[My Connection] +... +start_query = SET search_path TO myschema1, myschema2 + +[Table Filter] +tables = table_name in ('mytable1', 'mytable2') +``` + +!!! Important + If two or more schemas that were set on `search_path` contain a table with the same name, just the first one found is considered in the comparison. + +The `Table Filter` section can have a mix of `publications`, `replication_sets`, +`schemas`, and `tables` filters. LiveCompare considers the set of tables +that are in the intersection of all filters you specified. For example: + +```ini +[Table Filter] +publications = publication_name = 'livepub' +replication_sets = set_name in ('default', 'bdrgroup') +schemas = schema_name != 'badschema' +tables = table_name not like '%%account' +``` + +The table filter is applied in the first database to +build the table list. If a table exists in the first database and is being +considered in the filter, but it doesn't exist in any other database, then you +something like this is added to the logs, and the comparison for that specific +table is skipped: + +```text +2019-06-17 11:52:41,403 - ERROR - live_table.py - 55 - GetMetaData - P1: livecompare_second_1: Table public.test does not exist +2019-06-17 11:52:41,410 - ERROR - live_round.py - 201 - Initialize - P1: Table public.test does not exist on second connection. Aborting comparison +``` + +Similarly, if a table exists in any other database but doesn't exist in the +first database, then it isn't considered in the comparison, even if you +didn't apply any table filter. + +A comparison for a specific table is also skipped if the table column names +aren't exactly the same (unless `column_intersection` is enabled), and in the +same order. An appropriate message is added to the log file as well. + +Currently LiveCompare doesn't check if data types or constraints are the same +on both tables. + +!!! Important + `conflicts` mode doesn't make use of the table filter. + +## Row Filter + +In this section, you can apply a row-level filter to any table, so LiveCompare +works only on the rows that satisfy the row filter. + +You can write a list of tables under this section, one table per line. All +table names must be schema qualified unless `schema_qualified_table_names` is +disabled. For example: + +```ini +[Row Filter] +public.table1 = id = 10 +public.table2 = logdate >= '2000-01-01' +``` + +In this case, for the table `public.table1`, LiveCompare works only in the +rows that satisfy the clause `id = 10`. For the table `public.table2`, +only rows that satisfy `logdate >= '2000-01-01` are considered in the +comparison. + +If you disable the general setting `schema_qualified_table_names`, then you +must also set an appropriate `search_path` for Postgres in the connection +`start_query` setting, for example: + +``` +[General Setting] +... +schema_qualified_table_names = off + +[My Connection] +... +start_query = SET search_path TO public + +[Row Filter] +table1 = id = 10 +table2 = logdate >= '2000-01-01' +``` + +Any kind of SQL condition (same as you put in the `WHERE` clause) is +accepted in the same line as the table row filter. For example, if you have a +large table and want to compare only a specific number of IDs, you can +create a temporary table with all the IDs. Then you can use an `IN` clause to +emulate a `JOIN`, like this: + +``` +[Row Filter] +public.large_table = id IN (SELECT id2 FROM temp_table) +``` + +If a row filter is written incorrectly, then LiveCompare tries to apply the +filter but fails. So the comparison for this specific table is skipped, +and an exception is written to the log file. + +If a table is listed in the `Row Filter` section but somehow got filtered out +by the `Table Filter`, then the row filter for this table is silently +ignored. + +!!! Important + `conflicts` mode doesn't make use of the row filter. + +### Using current timestamp in Row Filter + +The `Row Filter` is applied differently depending on the `data_fetch_mode`: + +- On Postgres, setting `data_fetch_mode` to `server_side_cursors_with_hold` or + `server_side_cursors_without_hold` causes the `Row Filter` to be applied + only at the beginning of the table comparison, when the query is executed. This + means that using a server-side cursor to fetch data ensures the data is seen as + a snapshot of how it was beginning of the comparison. +- On Postgres, setting `data_fetch_mode` to `prepared_statements` (the default) + includes the `Row Filter` in the prepared query, which is then + executed at every data buffer that's fetched. This means that, if the query + uses `now()`, `CURRENT_TIMESTAMP`, or `SYSDATE` (on EDB Postgres Advanced Server) on the `Row Filter`, + then when the prepared statement executes, Postgres reevaluates the + current timestamp. + +So, suppose you're using `now()`, `CURRENT_TIMESTAMP`, or `SYSDATE` on the `Row Filter`, +for example: + +``` +[Row Filter] +public.table3 = logdate < CURRENT_TIMESTAMP +``` + +In this case, you must also use a server-side cursor to ensure the current +timestamp is evaluated only at the beginning of the queries. In other words, +`data_fetch_mode` must be set to a value different from +`prepared_statements`. + +On Oracle, the `data_fetch_mode` setting is ignored, and +the query is executed at the beginning. Then data is fetched by way of the client-side +cursor. This approach ensures data is seen as a snapshot of how it was at the beginning +of the comparison. This is a client-side cursor, but the behavior is similar to +using a server-side cursor in Postgres. + +## Column Filter + +In this section, you can apply a column-level filter to any table, so LiveCompare +works only on the columns that aren't part of the column filter. + +You can write a list of tables under this section, one table per line. All +table names must be schema qualified unless `schema_qualified_table_names` is +disabled. For example, suppose that both `public.table1` and `public.table2` have +the columns `column1`, `column2`, `column3`, `column4`, and `column5`: + +```ini +[Column Filter] +public.table1 = column1, column3 +public.table2 = column1, column5 +``` + +In this case, for the table `public.table1`, LiveCompare works only in the +columns `column2`, `column4`, and `column5`, filtering out `column1` and `column3`. +For the table `public.table2`, only the columns `column2`, `column3`, and +`column4` are considered in the comparison, filtering out `column1` and `column5`. + +If you disable the general setting `schema_qualified_table_names`, then you +must also set an appropriate `search_path` for Postgres in the connection +`start_query` setting, for example: + +``` +[General Setting] +... +schema_qualified_table_names = off + +[My Connection] +... +start_query = SET search_path TO public + +[Column Filter] +table1 = column1, column3 +table2 = column1, column5 +``` + +If absent column names are given in the column filter, that is, the column doesn't +exist in the given table, then LiveCompare logs a message about the missing columns +and ignores them. It uses just the valid ones, if any. + +If a table is listed in the `Column Filter` section but somehow got filtered +out by the `Table Filter`, then the column filter for this table is +silently ignored. + +!!! Important + If a column specified in a `Column Filter` is part of the table PK, then it isn't ignored in the comparison. LiveCompare logs that and ignores the filter of such a column. + +!!! Important + `conflicts` mode doesn't make use of the column filter. + +## Comparison Key + +!!! Tip "New feature" + LiveCompare comparison key support is available in LiveCompare version 2.0 and later. +!!! + +Similar to the `Column Filter`, in this section you can also specify a list +of columns per table. These columns are considered as a comparison key for +the specific table, even if the table has a primary key or `UNIQUE` constraint. + +For example: + +```ini +[Comparison Key] +public.table1 = col_a, col_b +public.table2 = c1, c2 +``` + +In this example, for table `public.table1`, the comparison key is +columns `col_a` and `col_b`. For table `public.table2`, columns `c1` and `c2` are +considered as a comparison key. + +The same behavior about missing columns or filtered out or missing tables that +are explained in [Column Filter](#column-filter), also apply to the comparison +key. Similarly, the `Comparison Key` section is ignored in `conflicts` mode. + +## Conflicts Filter + +In this section, you can specify a filter to use in `--conflicts` mode while +fetching conflicts from PGD nodes. You can build any SQL conditional expression +and use these fields in the expression: + +- `origin_node`: The upstream node of the subscription. +- `target_node`: The downstream node of the subscription. +- `local_time`: The timestamp when the conflict occurred in the node. +- `conflict_type`: The type of conflict. +- `conflict_resolution`: The resolution that was applied. +- `nspname`: Schema name of the involved relation. +- `relname`: Relation name of the involved relation. + +You must use the `conflicts` attribute under the section. For example: + +``` +[Conflicts Filter] +conflicts = conflict_type = 'update_missing' AND nspname = 'my_schema' +``` + +If you add this piece of configuration to your `.ini` file, LiveCompare fetches +only conflicts that are of type `update_missing` and related to tables under +the schema `my_schema` while querying for conflicts in each of the PGD nodes. + +!!! Important + This section is exclusively for `--conflicts` mode. diff --git a/product_docs/docs/livecompare/3/settings/filters.mdx b/product_docs/docs/livecompare/3/settings/filters.mdx new file mode 100644 index 00000000000..0906b3b4789 --- /dev/null +++ b/product_docs/docs/livecompare/3/settings/filters.mdx @@ -0,0 +1,327 @@ +--- +title: Filter Settings +navTitle: Filters +originalFilePath: settings.md +deepToC: true +description: How to filter tables, rows, columns, and conflicts in LiveCompare. +--- + + +## Table Filter + +If omitted or left empty, this section from the `.ini` file means that +LiveCompare executes against all tables in the first database. + +If you want LiveCompare to execute against a specific set of tables, there +are different ways to specify this: + +### `publications` +You can filter specific publications, and LiveCompare uses + only the tables associated with those publications. You can use the variable + `publication_name` to build the conditional expression, for example: + +```ini +publications = publication_name = 'livepub' +``` + +Requires `logical_replication_mode = native`. + +### `replication_sets` +When using pglogical or PGD, you can filter specific + replication sets, and LiveCompare works only on the tables associated with + those replication sets. You can use the variable `set_name` to build the + conditional expression, for example: + +```ini +replication_sets = set_name in ('default', 'bdrgroup') +``` + +Requires `logical_replication_mode = pglogical` or +`logical_replication_mode = bdr`. + +### `schemas` +You can filter specific schemas, and LiveCompare works only on + the tables that belong to those schemas. You can use the variable `schema_name` + to build the conditional expression, for example: + +```ini +schemas = schema_name != 'badschema' +``` + +### `tables` +The variable `table_name` can help you build a conditional + expression to filter only the tables you want LiveCompare to work on, for + example: + +```ini +tables = table_name not like '%%account' +``` + +In any conditional expression, escape the `%` character as `%%`. + +The table name must be schema-qualified, unless `schema_qualified_table_names` +is disabled. For example, you can filter only a specific list of +tables: + +``` +tables = table_name in ('myschema1.mytable1', 'myschema2.mytable2') +``` + +If you disable the general setting `schema_qualified_table_names`, then you +must also set an appropriate `search_path` for Postgres in the connection +`start_query` setting, for example: + +``` +[General Setting] +... +schema_qualified_table_names = off + +[My Connection] +... +start_query = SET search_path TO myschema1, myschema2 + +[Table Filter] +tables = table_name in ('mytable1', 'mytable2') +``` + +!!! Important + If two or more schemas that were set on `search_path` contain a table with the same name, just the first one found is considered in the comparison. + +The `Table Filter` section can have a mix of `publications`, `replication_sets`, +`schemas`, and `tables` filters. LiveCompare considers the set of tables +that are in the intersection of all filters you specified. For example: + +```ini +[Table Filter] +publications = publication_name = 'livepub' +replication_sets = set_name in ('default', 'bdrgroup') +schemas = schema_name != 'badschema' +tables = table_name not like '%%account' +``` + +The table filter is applied in the first database to +build the table list. If a table exists in the first database and is being +considered in the filter, but it doesn't exist in any other database, then you +something like this is added to the logs, and the comparison for that specific +table is skipped: + +```text +2019-06-17 11:52:41,403 - ERROR - live_table.py - 55 - GetMetaData - P1: livecompare_second_1: Table public.test does not exist +2019-06-17 11:52:41,410 - ERROR - live_round.py - 201 - Initialize - P1: Table public.test does not exist on second connection. Aborting comparison +``` + +Similarly, if a table exists in any other database but doesn't exist in the +first database, then it isn't considered in the comparison, even if you +didn't apply any table filter. + +A comparison for a specific table is also skipped if the table column names +aren't exactly the same (unless `column_intersection` is enabled), and in the +same order. An appropriate message is added to the log file as well. + +Currently LiveCompare doesn't check if data types or constraints are the same +on both tables. + +!!! Important + `conflicts` mode doesn't make use of the table filter. + +## Row Filter + +In this section, you can apply a row-level filter to any table, so LiveCompare +works only on the rows that satisfy the row filter. + +You can write a list of tables under this section, one table per line. All +table names must be schema qualified unless `schema_qualified_table_names` is +disabled. For example: + +```ini +[Row Filter] +public.table1 = id = 10 +public.table2 = logdate >= '2000-01-01' +``` + +In this case, for the table `public.table1`, LiveCompare works only in the +rows that satisfy the clause `id = 10`. For the table `public.table2`, +only rows that satisfy `logdate >= '2000-01-01` are considered in the +comparison. + +If you disable the general setting `schema_qualified_table_names`, then you +must also set an appropriate `search_path` for Postgres in the connection +`start_query` setting, for example: + +``` +[General Setting] +... +schema_qualified_table_names = off + +[My Connection] +... +start_query = SET search_path TO public + +[Row Filter] +table1 = id = 10 +table2 = logdate >= '2000-01-01' +``` + +Any kind of SQL condition (same as you put in the `WHERE` clause) is +accepted in the same line as the table row filter. For example, if you have a +large table and want to compare only a specific number of IDs, you can +create a temporary table with all the IDs. Then you can use an `IN` clause to +emulate a `JOIN`, like this: + +``` +[Row Filter] +public.large_table = id IN (SELECT id2 FROM temp_table) +``` + +If a row filter is written incorrectly, then LiveCompare tries to apply the +filter but fails. So the comparison for this specific table is skipped, +and an exception is written to the log file. + +If a table is listed in the `Row Filter` section but somehow got filtered out +by the `Table Filter`, then the row filter for this table is silently +ignored. + +!!! Important + `conflicts` mode doesn't make use of the row filter. + +### Using current timestamp in Row Filter + +The `Row Filter` is applied differently depending on the `data_fetch_mode`: + +- On Postgres, setting `data_fetch_mode` to `server_side_cursors_with_hold` or + `server_side_cursors_without_hold` causes the `Row Filter` to be applied + only at the beginning of the table comparison, when the query is executed. This + means that using a server-side cursor to fetch data ensures the data is seen as + a snapshot of how it was beginning of the comparison. +- On Postgres, setting `data_fetch_mode` to `prepared_statements` (the default) + includes the `Row Filter` in the prepared query, which is then + executed at every data buffer that's fetched. This means that, if the query + uses `now()`, `CURRENT_TIMESTAMP`, or `SYSDATE` (on EDB Postgres Advanced Server) on the `Row Filter`, + then when the prepared statement executes, Postgres reevaluates the + current timestamp. + +So, suppose you're using `now()`, `CURRENT_TIMESTAMP`, or `SYSDATE` on the `Row Filter`, +for example: + +``` +[Row Filter] +public.table3 = logdate < CURRENT_TIMESTAMP +``` + +In this case, you must also use a server-side cursor to ensure the current +timestamp is evaluated only at the beginning of the queries. In other words, +`data_fetch_mode` must be set to a value different from +`prepared_statements`. + +On Oracle, the `data_fetch_mode` setting is ignored, and +the query is executed at the beginning. Then data is fetched by way of the client-side +cursor. This approach ensures data is seen as a snapshot of how it was at the beginning +of the comparison. This is a client-side cursor, but the behavior is similar to +using a server-side cursor in Postgres. + +## Column Filter + +In this section, you can apply a column-level filter to any table, so LiveCompare +works only on the columns that aren't part of the column filter. + +You can write a list of tables under this section, one table per line. All +table names must be schema qualified unless `schema_qualified_table_names` is +disabled. For example, suppose that both `public.table1` and `public.table2` have +the columns `column1`, `column2`, `column3`, `column4`, and `column5`: + +```ini +[Column Filter] +public.table1 = column1, column3 +public.table2 = column1, column5 +``` + +In this case, for the table `public.table1`, LiveCompare works only in the +columns `column2`, `column4`, and `column5`, filtering out `column1` and `column3`. +For the table `public.table2`, only the columns `column2`, `column3`, and +`column4` are considered in the comparison, filtering out `column1` and `column5`. + +If you disable the general setting `schema_qualified_table_names`, then you +must also set an appropriate `search_path` for Postgres in the connection +`start_query` setting, for example: + +``` +[General Setting] +... +schema_qualified_table_names = off + +[My Connection] +... +start_query = SET search_path TO public + +[Column Filter] +table1 = column1, column3 +table2 = column1, column5 +``` + +If absent column names are given in the column filter, that is, the column doesn't +exist in the given table, then LiveCompare logs a message about the missing columns +and ignores them. It uses just the valid ones, if any. + +If a table is listed in the `Column Filter` section but somehow got filtered +out by the `Table Filter`, then the column filter for this table is +silently ignored. + +!!! Important + If a column specified in a `Column Filter` is part of the table PK, then it isn't ignored in the comparison. LiveCompare logs that and ignores the filter of such a column. + +!!! Important + `conflicts` mode doesn't make use of the column filter. + +## Comparison Key + +!!! Tip "New feature" + LiveCompare comparison key support is available in LiveCompare version 2.0 and later. +!!! + +Similar to the `Column Filter`, in this section you can also specify a list +of columns per table. These columns are considered as a comparison key for +the specific table, even if the table has a primary key or `UNIQUE` constraint. + +For example: + +```ini +[Comparison Key] +public.table1 = col_a, col_b +public.table2 = c1, c2 +``` + +In this example, for table `public.table1`, the comparison key is +columns `col_a` and `col_b`. For table `public.table2`, columns `c1` and `c2` are +considered as a comparison key. + +The same behavior about missing columns or filtered out or missing tables that +are explained in [Column Filter](#column-filter), also apply to the comparison +key. Similarly, the `Comparison Key` section is ignored in `conflicts` mode. + +## Conflicts Filter + +In this section, you can specify a filter to use in `--conflicts` mode while +fetching conflicts from PGD nodes. You can build any SQL conditional expression +and use these fields in the expression: + +- `origin_node`: The upstream node of the subscription. +- `target_node`: The downstream node of the subscription. +- `local_time`: The timestamp when the conflict occurred in the node. +- `conflict_type`: The type of conflict. +- `conflict_resolution`: The resolution that was applied. +- `nspname`: Schema name of the involved relation. +- `relname`: Relation name of the involved relation. + +You must use the `conflicts` attribute under the section. For example: + +``` +[Conflicts Filter] +conflicts = conflict_type = 'update_missing' AND nspname = 'my_schema' +``` + +If you add this piece of configuration to your `.ini` file, LiveCompare fetches +only conflicts that are of type `update_missing` and related to tables under +the schema `my_schema` while querying for conflicts in each of the PGD nodes. + +!!! Important + This section is exclusively for `--conflicts` mode. diff --git a/product_docs/docs/livecompare/3/settings/general.mdx b/product_docs/docs/livecompare/3/settings/general.mdx new file mode 100644 index 00000000000..477bcceedd3 --- /dev/null +++ b/product_docs/docs/livecompare/3/settings/general.mdx @@ -0,0 +1,412 @@ +--- +title: General Settings +navTitle: General +originalFilePath: settings.md +deepToC: true +description: All the general settings available for LiveCompare 3.0 +--- + +## General settings + +### `logical_replication_mode` + +Affects how the program interprets connections and table filter settings and also the requirements to check for in the connections before starting the comparison. Currently the possible values are: + +- `off`: Assumes there's no logical replication between the databases. + +- `native`: Assumes there's native logical replication between the databases. Enables the use of the `Table Filter -> publications` setting to specify the list of tables to use. Requires PostgreSQL 10+ on all databases. + +- `pglogical`: Assumes there's pglogical replication between the databases. Enables the use of the `Table Filter -> replication_sets` setting to specify the list of tables to use. Also enables the use of `node_name` to specify the data connections, which requires setting the `Initial Connection` that's used to retrieve DSN information of the nodes. Requires the `pglogical` extensions to be installed on all databases. + +- `bdr`: Assumes all data connections are nodes from the same PGD cluster. + Enables use of the `Table Filter -> replication_sets` setting to specify the list + of tables to use. Also enables the use of `node_name` to + specify the data connections, which requires setting the `Initial Connection` + that's used to retrieve DSN information of the nodes. Requires `pglogical` + and `bdr` extensions installed on all databases. + +### `all_bdr_nodes` + +If `logical_replication_mode` is set to `bdr`, then you can specify only the Initial Connection and let LiveCompare build the connection list based on the current list of active PGD nodes. +Default: `off`. + +### `max_parallel_workers` + +Number of parallel processes to consider. Each +process works on a table from the queue. +Default: `2`. + +!!! Important +Each process keeps N+1 open connections: one to each data connection and another one to the output database. +!!! + +### `buffer_size` + +Number of rows to retrieve from the tables on every data fetch operation. +Default: `4096`. + +### `log_level` + +Verbosity level in the log file. Possible values: `debug`, `info`, `warning`, or `error`. +Default: `info`. + +### `data_fetch_mode` + +Affects how LiveCompare fetches data from the database. + +- `prepared_statements`
+ Uses prepared statements (a query with `LIMIT`) for + data fetch. Only a very small amount of data (`buffer_size = 4096` rows by + default) is fetched each time, so it has the smallest impact of all three modes, + and for the same reason it's the safer fetch mode. Allows asynchronous data + fetch (defined by `parallel_data_fetch`). For the general use case, this + fetch method provides good performance, but a performance decrease can be + felt for large tables. This is the default and strongly recommended when + server load is medium-high. + +- `server_side_cursors_with_hold`
+ Uses server-side cursors `WITH HOLD` for + data fetch. As table data is retrieved in a single transaction, it holds + back `xmin` and can cause bloat and replication issues and also prevent + `VACUUM` from running well. Also, the `WITH HOLD` clause tells Postgres to + materialize the query (workers can hang for a few seconds waiting for the + data to materialize), so the whole table data consumes RAM and can be + stored on Postgres side disk as temporary files. You can reduce all that impact by increasing `buffer_size` a little. Allows + asynchronous data fetch (defined by `parallel_data_fetch`). For the general + use case, this fetch method doesn't provide any benefits when compared to + `prepared_stataments`, but for multiple small tables it's faster. However, + this mode is recommended only when load is very low, for example, on tests + and migration scenarios. + +- `server_side_cursors_without_hold`
+ Uses server-side cursors + `WITHOUT HOLD` for data fetch. As `server_side_cursors_with_hold`, this + mode can also hold back `xmin`, thus it potentially can cause bloat, `VACUUM`, + and replication issues on Postgres. However, such impact is higher because + `WITHOUT HOLD` cursors require an open transaction for the whole comparison + session (this requirement will be lifted in later versions). As the snapshot is held + for the whole comparison session, comparison results might be helpful + depending on your use case. As the query isn't materialized, memory usage + and temp file generation remains low. Asynchronous data fetch isn't + allowed. In terms of performance, this mode is slower for the general use + case, but for large tables it can be the faster. We recommend it when load + on the database is low-medium. + +!!! Important + The choice of the right `data_fetch_mode` for the right scenario + is very important. Using prepared statements has the smallest footprint on the + database server, so it's the safest approach, and it's good for the general use + case. Another point is that prepared statements allow LiveCompare to always see + the latest version of the rows, which might not happen when using server-side + cursors on a busy database. So we recommend using `prepared_statements` for + production, high-load servers and either `server_side_cursors_*` setting for + testing, migration scenarios, and low-load servers. The best strategy + probably mixes `server_side_cursors_without_hold` for very large tables and + `prepared_statements` for the remaining tables. The following table shows + a comparison of the cost/benefit ratio. + +| | prepared_statements | server_side_cursors_with_hold | server_side_cursors_without_hold | +| ------------------ | :-----------------: | :---------------------------: | :------------------------------: | +| xmin hold | very low | medium | high | +| xmin released per | buffer | chunk | whole comparison session | +| temp files | very low | very high | low | +| memory | very low | high | low | +| allows async conns | yes | yes | no | +| fastest for | general | small tables | large tables | +| recommended load | high | very low | low-medium | + +!!! Note "Note about Oracle" + For Oracle, the `data_fetch_mode` setting is completely + ignored, and data is always fetched from Oracle using a direct query. + Data is taken in chunks of `buffer_size` through the client-side cursor. + +### ~~`parallel_chunk_rows`~~ +Removed in LiveCompare 3.0. If this setting is present in any `.ini` file, remove it before using the file with LiveCompare 3.0. + +### `parallel_data_fetch` +Specifies whether data fetch is performed in parallel (that is, + using async connections to the databases). Improves performance of multi-way + comparison. If any data connections aren't PostgreSQL, then this setting is + automatically disabled. It's allowed only when + `data_fetch_mode = prepared_statements` or + `data_fetch_mode = server_side_cursors_with_hold`. + Default: `on`. + +### `comparison_algorithm`: +Affects how LiveCompare works through table rows to +compare data. Using hashes is faster than full-row comparison. It can assume one of the following values: + +- `full_row`
Disables row comparison using hashes. Full comparison, in this + case, is performed by comparing the row column by column. For + comparisons involving Oracle 10g database, `full_row` is the only valid value for the + `comparison_algorithm` parameter. + +- `row_hash`
Enables row comparison using hashes and enables table + splitting. Tables are split so each worker compares a maximum of + `parallel_chunk_rows` per table. Data row is hashed in PostgreSQL, so the + comparison is faster than `full_row`. However, if the hash for a specific row + doesn't match, then for that specific row, LiveCompare falls back + to the `full_row` algorithm (that is, compare row by row). If any data connection + isn't PostgreSQL, then LiveCompare uses a row hash that's defined as the MD5 + hash of the concatenated column values of the row being considered, a + *common hash* among the database technologies being compared. + +- `block_hash`
Works the same as `row_hash`, but instead of comparing row + by row, LiveCompare builds a *block hash*, that is, a hash of the hashes of all + rows in the data buffer that was just fetched (maximum of `buffer_size` + rows). Conceptually it works like a two-level Merkle tree. If the block hash + matches, then LiveCompare advances the whole block, which is why this + comparison algorithm is faster than `row_hash`. If block hash doesn't + match, then LiveCompare falls back to `row_hash` and performs the comparison row + by row in the buffer to find the divergent rows. This is the default value. + +### `min_time_between_heart_beats` +Time in seconds to wait before logging a + *heart beat* message to the log. Each worker tracks it separately per round + part being compared. Default: 30 seconds. + +### `min_time_between_round_saves` +Time in seconds to wait before updating each + round state when the comparison algorithm is in progress. A round save can + happen only during a heart beat, so `min_time_between_round_saves` must be greater + than or equal to `min_time_between_heart_beats`. When the round + finishes, LiveCompare always updates the round state for that table. + Default: 60 seconds. + +!!! Important +If you cancel execution of LiveCompare by pressing **Ctrl-C** +and start it again, then LiveCompare resumes the round for that table, +starting from the point where the round state was saved. +!!! + +### `comparison_cost_limit` +If > 0, corresponds to a number of rows each worker + processes before taking a nap of `comparison_cost_delay` seconds. Defaults + to 0, meaning that each worker processes rows without taking a nap. + +### `comparison_cost_delay` +If `comparison_cost_limit > 0`, then this setting + specifies how long each worker sleeps. Default: `0.0`. + +### `stop_after_time` +Time in seconds after which LiveCompare + stop as if you press **Ctrl-C**. You can resume the comparison session that was + interrupted, if not finished yet, by passing the session + ID as an argument in the command line. Default: `stop_after_time = 0`, which + means that automatic interruption is disabled. + +### `consensus_mode` +Consensus algorithm used by LiveCompare to determine which + data connections are divergent. Possible values are `off`, `simple_majority`, + `quorum_based`, or `source_of_truth`. If `consensus_mode = source_of_truth`, then + `difference_sources_of_truth` must be filled. Default: `simple_majority`. + +!!! Note +If `consensus_mode = off` and `comparison_algorithm = block_hash`, LiveCompare won't switch to a more granular comparison algorithm. This significantly improves the performance when divergences are found for a specific table, at the cost of not having the information about which rows are divergent. The user can subsequently run LiveCompare again adding the divergent tables, if any, to the `[Table Filter]` and compare only the divergent tables to get the information about which rows are divergent. +!!! + +### `difference_required_quorum` +If `consensus_mode = quorum_based`, then this + setting specifies the minimum quorum required to decide which connections are + divergent. Must be a number between 0.0 and 1.0. 0.0 means no connection is + required, and 1.0 means all connections are required. Both cases are extreme + and we don't recommend using them. The default value is 0.5, and we recommend using a + value close to that. + +### `difference_sources_of_truth` +Comma-separated list of connections names (or node names, if +`logical_replication_mode = bdr` and `all_bdr_nodes = on`) to consider as the source of truth. It's used only when `consensus_mode = source_of_truth`. For + example: `difference_sources_of_truth = node1,node2`. In this example, + either the sections `node1 Connection` and `node2 Connection` must be + defined in the `.ini` file or `all_bdr_nodes = on` and only the `Initial + Connection` is defined, while `node1` and `node2` must be valid PGD node + names. + +### `difference_tie_breakers` +Comma-separated list of connection names (or node + names, if `logical_replication_mode = bdr` and `all_bdr_nodes = on`) to + be considered as tie breakers whenever the consensus algorithm finds a tie + situation. For example: `difference_tie_breakers = node1,node2`. In this + example, either the sections `node1 Connection` and `node2 Connections` must + be defined in the `.ini` file or `all_bdr_nodes = on` and only the `Initial + Connection` is defined, while `node1` and `node2` must be valid PGD node + names. Default: Don't consider any connection as tie breaker. + +### `difference_statements` +Controls the kind of DML statements for + LiveCompare to generate. The value of `difference_statements` can + be one of: + +- `all` (default) +- `inserts` +- `updates` +- `deletes` +- `inserts_updates` +- `inserts_deletes` +- `updates_deletes` + +### `difference_allow_null_updates` +Determines whether commands like `UPDATE SET + col = NULL` are allowed in the difference report. +Default: `on`. + +### `difference_statement_order` +Controls order of DML statements that + LiveCompare generates. The value of `difference_statement_order` + can be one of: + +- `delete_insert_update` +- `delete_update_insert` (default) +- `insert_update_delete` +- `insert_delete_update` +- `update_insert_delete` +- `update_delete_insert` + +### `difference_fix_replication_origin` +When working with PGD databases, for + difference, LiveCompare creates a specific replication origin if it doesn't + exist yet. It then uses the replication origin to create an apply script with DML + fixes. The setting `difference_fix_replication_origin` specifies the name of + the replication origin used by LiveCompare. If you don't set any value + for this setting, then LiveCompare sets + `difference_fix_replication_origin = bdr_local_only_origin`. The + replication origin that LiveCompare creates isn't dropped to allow verification + after the comparison. However, if needed, you can manually drop the replication origin + later. Requires `logical_replication_mode = bdr`. + +!!! Important + PGD 3.6.18 introduced the new pre-created `bdr_local_only_origin` replication origin to use for applying local-only transactions. So if LiveCompare is connected to PGD 3.6.18, it doesn't create this replication origin, and we recommend you don't try to drop this replication origin. + +### `difference_fix_start_query` +Arbitrary query that's executed at the beginning of the apply script generated by LiveCompare. Additionally, if a PGD comparison is being performed and the `difference_fix_start_query` is empty, then LiveCompare also automatically does the following: + +- If the divergent connection is PGD 3.6.7, adds +`SET LOCAL bdr.xact_replication = off;` +- Adds commands that set up transaction to use the replication origin +specified in `difference_fix_replication_origin` + +### `show_progress_bars` +Determines whether to show progress bars in the console output. Disabling this setting might be useful for batch executions. +Default: `on`. + +### `output_schema` +In the output connection, the schema where the comparison report tables are created. +Default: `livecompare`. + +### `hash_column_name` +Every data fetch contains a specific column that's + the hash of all actual columns in the row. This setting specifies the name of + this column. +Default: `livecompare_hash`. + +### `rownumber_column_name` +Some fetches need to use the `row_number()` function + value inside a query column. This setting specifies the name of this column. +Default: `livecompare_rownumber`. + +### `fetch_row_origin` +When this setting is enabled, LiveCompare fetches the + origin name for each divergent row, which might be useful for debugging + purposes. To be enabled, requires `logical_replication_mode` set + to `pglogical` or `bdr`. Default: `off`. + +### `column_intersection` +When this setting is enabled, for a given table that's + being compared, LiveCompare works only on the intersection of columns from + the table on all connections, ignoring extra columns that might exist on any of + the connections. When this setting is disabled, LiveCompare checks if + columns are equivalent on the table on all connections and aborts the comparison + of the table if there are any column mismatches. Default: `off`. + +!!! Important +If a table has PK, then the PK columns aren't allowed to be different, even if `column_intersection = on`. +!!! + +### `ignore_nullable` +For a specific table comparison, if LiveCompare is using a + comparison key different from the primary key, then LiveCompare requires all + columns to be `NOT NULL` if `ignore_nullable` is enabled (default). You can override + that behavior by setting `ignore_nullable = off`, which + allows LiveCompare to consider null-able columns in the comparison, which in some + corner cases can produce false positives. + +### `check_uniqueness_enforcement` +If LiveCompare is using a user-defined + comparison key or using all columns in the table as a comparison key, then + LiveCompare checks for table uniqueness on the comparison key if setting + `check_uniqueness_enforcement` is enabled (default). + +### `oracle_ignore_unsortable` +When enabled, tells LiveCompare to ignore columns + with Oracle unsortable data types (BLOB, CLOB, NCLOB, BFILE) if column isn't + part of the table PK. If enabling this setting, we recommend also enabling + `column_intersection`. + +### `oracle_user_tables_only` +When enabled, tells LiveCompare to fetch table + metadata only from the Oracle logged-in user. This approach is faster because it reads, + for example, from `sys.user_tables` and `sys.user_tab_columns` instead of + `sys.all_tables` and `sys.all_tab_columns`. +Default: `off`. + +### `oracle_fetch_fk_metadata` +When enabled, tells LiveCompare to fetch foreign-key + metadata, which can be a slow operation. Overrides the value of the setting + `fetch_fk_metadata` on the Oracle connection. +Default: `off`. + +### `schema_qualified_table_names` +Table names are treated as schema qualified + when this setting is enabled. Disabling it allows comparing tables without + using schema-qualified table names. On Oracle x Postgres comparisons, it + requires also enabling `oracle_user_tables_only`. On Postgres x Postgres, + it allows for comparisons of tables that are under different schemas, even in + the same database. Also, when `schema_qualified_table_names` is enabled, + `Table Filter -> tables`, `Row Filter`, and `Column Filter` allow table name + without the schema name. +Default: `on`. + +### ~~`force_collate`~~ +Removed in LiveCompare 3.0. If this setting is present in any `.ini` file, remove it before using the file with LiveCompare 3.0. + +### `work_directory` +Path to the `LiveCompare` working directory. The session + folder containing output files is created in this directory. +Default: `.` (current directory). + +### `abort_on_setup_error` +When enabled, if LiveCompare encounters any error when + trying to set up a table comparison round, the whole comparison session is + aborted. +Default: `off`. + +!!! Important + Setting `abort_on_setup_error` is considered only during `compare` mode. In `recheck` mode, LiveCompare always aborts at the first error in setup. + +### `custom_dollar_quoting_delimiter` +When LiveCompare finds differences, it + outputs the DML using dollar quoting on strings. The default behavior is to create + a random string to compose it. If you want by any means to use a custom one, you + can set this parameter as the delimiter to use. You need to set only the + constant, not the `$` symbols around the constant. +Default: `off`, which means LiveCompare uses an `md5` hash of the word `LiveCompare`. + +### `session_replication_role_replica` +When enabled, LiveCompare uses the + `session_replication_role` PostgreSQL setting as `replica` in the output apply + scripts. That's useful if you want to prevent firing triggers and rules while + applying DML in the nodes with divergences. Enabling it requires a PostgreSQL + superuser. Otherwise, it has no effect. +Default: `off`. + +### `split_updates` +When enabled, LiveCompare splits `UPDATE` divergences. + That is, instead of generating an `UPDATE` DML, it generates corresponding + `DELETE` and `INSERT` in the apply script. +Default: `off`. + +### `float_point_round` +An integer to specify decimal digits that LiveCompare + rounds when comparing float-point values coming from the database +Default: `-1`, which disables float-point rounding. + diff --git a/product_docs/docs/livecompare/3/settings/index.mdx b/product_docs/docs/livecompare/3/settings/index.mdx new file mode 100644 index 00000000000..bc297c9bd2e --- /dev/null +++ b/product_docs/docs/livecompare/3/settings/index.mdx @@ -0,0 +1,12 @@ +--- +title: Settings +navTitle: Settings +indexCards: simple +navigation: +- general +- connections +- filters +--- + +LiveCompare settings are defined in the `.ini` files that you use to configure LiveCompare. The settings are divided into three categories: general settings, connection settings, and filter settings. + diff --git a/product_docs/docs/livecompare/3/supported_technologies.mdx b/product_docs/docs/livecompare/3/supported_technologies.mdx new file mode 100644 index 00000000000..435714385e2 --- /dev/null +++ b/product_docs/docs/livecompare/3/supported_technologies.mdx @@ -0,0 +1,54 @@ +--- +navTitle: Supported technologies +title: Supported technologies +originalFilePath: supported_technologies.md + +--- + +LiveCompare can connect to and compare data from a list of technologies, +including PostgreSQL, EDB Postgres Distributed (PGD, formerly known as BDR), and Oracle. + +LiveCompare has three kinds of connections: + +- **Initial** (optional): Used to fetch metadata about pglogical or PGD + connections. Required if data connections are pglogical or PGD, and if + `replication_sets` or `node_name` settings are used. Requires + `logical_replication_mode = pglogical` or `logical_replication_mode = bdr`. + A pglogical- or PGD-enabled database is required. +- **Data**: The actual database connection that the tool connects to to perform + data comparison. The first connection in the list is used to solve `Table + Filter` and `Row Filter`, and is also used with the `Initial + Connection' to gather information about PGD nodes. If + `logical_replication_mode = bdr` and `all_bdr_nodes = on`, then LiveCompare + considers all PGD nodes that are part of the same PGD cluster as the `Initial + Connection`. In this case, you don't need to define data connections + individually. The fix can be potentially applied in all data connections, as comparison + and consensus decisions work per row. +- **Output** (mandatory): Where LiveCompare creates a schema called + `livecompare`, some tables, and views. This is required to keep progress and + reporting data about comparison sessions. It must be a PostgreSQL or + 2ndQPostgres connection. + +The table shows versions and details about supported technologies and the context in which you can use them in LiveCompare. + +| Technology | Versions | Possible connections | +|-------------------------------------|----------------------------------------|---------------------------| +| PostgreSQL | 10[^1], 11[^1], 12, 13, 14, 15, and 16 | Data and output | +| EDB Postgres Extended (PGE) | 10[^1], 11[^1], 12, 13, 14, 15, and 16 | Data and output | +| EDB Postgres Advanced Server (EPAS) | 11[^1], 12, 13, 14, 15, and 16 | Data and output | +| pglogical | 2 and 3 | Initial, data, and output | +| EDB Postgres Distributed (PGD) | 1, 2, 3, 4, and 5 | Initial, data, and output | +| Oracle | 11g, 12c, 18c, 19c, and 21c | A single data connection | + +[^1]:LiveCompare only supports the use of LiveCompare with older versions of Postgres (PostgreSQL, EDB Postgres Extended, or EDB Postgres Advanced Server) that have reached end of life in support of performing a major version upgrade to a newer supported version of Postgres. + +!!! Note +EDB no longer tests LiveCompare with Oracle 10g. As such, it's no longer on the list of Oracle versions officially supported with LiveCompare. Oracle 10g was previously supported and is expected to continue to work in most cases. However, some limitations may exist. One known limitation is that LiveCompare requires the `comparison_algorithm` parameter to be set to `full_row` (for example, `comparison_algorithm = full_row`). +!!! + +## PgBouncer support + +You can use LiveCompare against nodes through PgBouncer. However, you must use +`pool_mode=session` because LiveCompare uses prepared statements on PostgreSQL, +which isn't possible when `pool_mode` is either `transaction` or +`statement`. From 912ed5eb3f2bd02fec6e93e571f87d9f7e42cab3 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Wed, 21 Aug 2024 15:22:55 +0100 Subject: [PATCH 05/16] Removed filters from connections.mdx --- .../livecompare/3/settings/connections.mdx | 318 ------------------ 1 file changed, 318 deletions(-) diff --git a/product_docs/docs/livecompare/3/settings/connections.mdx b/product_docs/docs/livecompare/3/settings/connections.mdx index 33cff518faa..82007222226 100644 --- a/product_docs/docs/livecompare/3/settings/connections.mdx +++ b/product_docs/docs/livecompare/3/settings/connections.mdx @@ -124,321 +124,3 @@ Specifies whether LiveCompare gathers metadata about foreign keys on the connection. Default: `on`. -## Table Filter - -If omitted or left empty, this section from the `.ini` file means that -LiveCompare executes against all tables in the first database. - -If you want LiveCompare to execute against a specific set of tables, there -are different ways to specify this: - -### `publications` -You can filter specific publications, and LiveCompare uses - only the tables associated with those publications. You can use the variable - `publication_name` to build the conditional expression, for example: - -```ini -publications = publication_name = 'livepub' -``` - -Requires `logical_replication_mode = native`. - -### `replication_sets` -When using pglogical or PGD, you can filter specific - replication sets, and LiveCompare works only on the tables associated with - those replication sets. You can use the variable `set_name` to build the - conditional expression, for example: - -```ini -replication_sets = set_name in ('default', 'bdrgroup') -``` - -Requires `logical_replication_mode = pglogical` or -`logical_replication_mode = bdr`. - -### `schemas` -You can filter specific schemas, and LiveCompare works only on - the tables that belong to those schemas. You can use the variable `schema_name` - to build the conditional expression, for example: - -```ini -schemas = schema_name != 'badschema' -``` - -### `tables` -The variable `table_name` can help you build a conditional - expression to filter only the tables you want LiveCompare to work on, for - example: - -```ini -tables = table_name not like '%%account' -``` - -In any conditional expression, escape the `%` character as `%%`. - -The table name must be schema-qualified, unless `schema_qualified_table_names` -is disabled. For example, you can filter only a specific list of -tables: - -``` -tables = table_name in ('myschema1.mytable1', 'myschema2.mytable2') -``` - -If you disable the general setting `schema_qualified_table_names`, then you -must also set an appropriate `search_path` for Postgres in the connection -`start_query` setting, for example: - -``` -[General Setting] -... -schema_qualified_table_names = off - -[My Connection] -... -start_query = SET search_path TO myschema1, myschema2 - -[Table Filter] -tables = table_name in ('mytable1', 'mytable2') -``` - -!!! Important - If two or more schemas that were set on `search_path` contain a table with the same name, just the first one found is considered in the comparison. - -The `Table Filter` section can have a mix of `publications`, `replication_sets`, -`schemas`, and `tables` filters. LiveCompare considers the set of tables -that are in the intersection of all filters you specified. For example: - -```ini -[Table Filter] -publications = publication_name = 'livepub' -replication_sets = set_name in ('default', 'bdrgroup') -schemas = schema_name != 'badschema' -tables = table_name not like '%%account' -``` - -The table filter is applied in the first database to -build the table list. If a table exists in the first database and is being -considered in the filter, but it doesn't exist in any other database, then you -something like this is added to the logs, and the comparison for that specific -table is skipped: - -```text -2019-06-17 11:52:41,403 - ERROR - live_table.py - 55 - GetMetaData - P1: livecompare_second_1: Table public.test does not exist -2019-06-17 11:52:41,410 - ERROR - live_round.py - 201 - Initialize - P1: Table public.test does not exist on second connection. Aborting comparison -``` - -Similarly, if a table exists in any other database but doesn't exist in the -first database, then it isn't considered in the comparison, even if you -didn't apply any table filter. - -A comparison for a specific table is also skipped if the table column names -aren't exactly the same (unless `column_intersection` is enabled), and in the -same order. An appropriate message is added to the log file as well. - -Currently LiveCompare doesn't check if data types or constraints are the same -on both tables. - -!!! Important - `conflicts` mode doesn't make use of the table filter. - -## Row Filter - -In this section, you can apply a row-level filter to any table, so LiveCompare -works only on the rows that satisfy the row filter. - -You can write a list of tables under this section, one table per line. All -table names must be schema qualified unless `schema_qualified_table_names` is -disabled. For example: - -```ini -[Row Filter] -public.table1 = id = 10 -public.table2 = logdate >= '2000-01-01' -``` - -In this case, for the table `public.table1`, LiveCompare works only in the -rows that satisfy the clause `id = 10`. For the table `public.table2`, -only rows that satisfy `logdate >= '2000-01-01` are considered in the -comparison. - -If you disable the general setting `schema_qualified_table_names`, then you -must also set an appropriate `search_path` for Postgres in the connection -`start_query` setting, for example: - -``` -[General Setting] -... -schema_qualified_table_names = off - -[My Connection] -... -start_query = SET search_path TO public - -[Row Filter] -table1 = id = 10 -table2 = logdate >= '2000-01-01' -``` - -Any kind of SQL condition (same as you put in the `WHERE` clause) is -accepted in the same line as the table row filter. For example, if you have a -large table and want to compare only a specific number of IDs, you can -create a temporary table with all the IDs. Then you can use an `IN` clause to -emulate a `JOIN`, like this: - -``` -[Row Filter] -public.large_table = id IN (SELECT id2 FROM temp_table) -``` - -If a row filter is written incorrectly, then LiveCompare tries to apply the -filter but fails. So the comparison for this specific table is skipped, -and an exception is written to the log file. - -If a table is listed in the `Row Filter` section but somehow got filtered out -by the `Table Filter`, then the row filter for this table is silently -ignored. - -!!! Important - `conflicts` mode doesn't make use of the row filter. - -### Using current timestamp in Row Filter - -The `Row Filter` is applied differently depending on the `data_fetch_mode`: - -- On Postgres, setting `data_fetch_mode` to `server_side_cursors_with_hold` or - `server_side_cursors_without_hold` causes the `Row Filter` to be applied - only at the beginning of the table comparison, when the query is executed. This - means that using a server-side cursor to fetch data ensures the data is seen as - a snapshot of how it was beginning of the comparison. -- On Postgres, setting `data_fetch_mode` to `prepared_statements` (the default) - includes the `Row Filter` in the prepared query, which is then - executed at every data buffer that's fetched. This means that, if the query - uses `now()`, `CURRENT_TIMESTAMP`, or `SYSDATE` (on EDB Postgres Advanced Server) on the `Row Filter`, - then when the prepared statement executes, Postgres reevaluates the - current timestamp. - -So, suppose you're using `now()`, `CURRENT_TIMESTAMP`, or `SYSDATE` on the `Row Filter`, -for example: - -``` -[Row Filter] -public.table3 = logdate < CURRENT_TIMESTAMP -``` - -In this case, you must also use a server-side cursor to ensure the current -timestamp is evaluated only at the beginning of the queries. In other words, -`data_fetch_mode` must be set to a value different from -`prepared_statements`. - -On Oracle, the `data_fetch_mode` setting is ignored, and -the query is executed at the beginning. Then data is fetched by way of the client-side -cursor. This approach ensures data is seen as a snapshot of how it was at the beginning -of the comparison. This is a client-side cursor, but the behavior is similar to -using a server-side cursor in Postgres. - -## Column Filter - -In this section, you can apply a column-level filter to any table, so LiveCompare -works only on the columns that aren't part of the column filter. - -You can write a list of tables under this section, one table per line. All -table names must be schema qualified unless `schema_qualified_table_names` is -disabled. For example, suppose that both `public.table1` and `public.table2` have -the columns `column1`, `column2`, `column3`, `column4`, and `column5`: - -```ini -[Column Filter] -public.table1 = column1, column3 -public.table2 = column1, column5 -``` - -In this case, for the table `public.table1`, LiveCompare works only in the -columns `column2`, `column4`, and `column5`, filtering out `column1` and `column3`. -For the table `public.table2`, only the columns `column2`, `column3`, and -`column4` are considered in the comparison, filtering out `column1` and `column5`. - -If you disable the general setting `schema_qualified_table_names`, then you -must also set an appropriate `search_path` for Postgres in the connection -`start_query` setting, for example: - -``` -[General Setting] -... -schema_qualified_table_names = off - -[My Connection] -... -start_query = SET search_path TO public - -[Column Filter] -table1 = column1, column3 -table2 = column1, column5 -``` - -If absent column names are given in the column filter, that is, the column doesn't -exist in the given table, then LiveCompare logs a message about the missing columns -and ignores them. It uses just the valid ones, if any. - -If a table is listed in the `Column Filter` section but somehow got filtered -out by the `Table Filter`, then the column filter for this table is -silently ignored. - -!!! Important - If a column specified in a `Column Filter` is part of the table PK, then it isn't ignored in the comparison. LiveCompare logs that and ignores the filter of such a column. - -!!! Important - `conflicts` mode doesn't make use of the column filter. - -## Comparison Key - -!!! Tip "New feature" - LiveCompare comparison key support is available in LiveCompare version 2.0 and later. -!!! - -Similar to the `Column Filter`, in this section you can also specify a list -of columns per table. These columns are considered as a comparison key for -the specific table, even if the table has a primary key or `UNIQUE` constraint. - -For example: - -```ini -[Comparison Key] -public.table1 = col_a, col_b -public.table2 = c1, c2 -``` - -In this example, for table `public.table1`, the comparison key is -columns `col_a` and `col_b`. For table `public.table2`, columns `c1` and `c2` are -considered as a comparison key. - -The same behavior about missing columns or filtered out or missing tables that -are explained in [Column Filter](#column-filter), also apply to the comparison -key. Similarly, the `Comparison Key` section is ignored in `conflicts` mode. - -## Conflicts Filter - -In this section, you can specify a filter to use in `--conflicts` mode while -fetching conflicts from PGD nodes. You can build any SQL conditional expression -and use these fields in the expression: - -- `origin_node`: The upstream node of the subscription. -- `target_node`: The downstream node of the subscription. -- `local_time`: The timestamp when the conflict occurred in the node. -- `conflict_type`: The type of conflict. -- `conflict_resolution`: The resolution that was applied. -- `nspname`: Schema name of the involved relation. -- `relname`: Relation name of the involved relation. - -You must use the `conflicts` attribute under the section. For example: - -``` -[Conflicts Filter] -conflicts = conflict_type = 'update_missing' AND nspname = 'my_schema' -``` - -If you add this piece of configuration to your `.ini` file, LiveCompare fetches -only conflicts that are of type `update_missing` and related to tables under -the schema `my_schema` while querying for conflicts in each of the PGD nodes. - -!!! Important - This section is exclusively for `--conflicts` mode. From d667285e348423372685f724e8ada771a4b914ff Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Wed, 21 Aug 2024 15:27:30 +0100 Subject: [PATCH 06/16] Update tpa requirements snippet - needs more validation. --- product_docs/docs/livecompare/3/requirements.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/product_docs/docs/livecompare/3/requirements.mdx b/product_docs/docs/livecompare/3/requirements.mdx index a60f1ebf3b3..4fd84abd7b6 100644 --- a/product_docs/docs/livecompare/3/requirements.mdx +++ b/product_docs/docs/livecompare/3/requirements.mdx @@ -56,8 +56,8 @@ cluster_vars: postgres_version: '13' postgresql_flavour: postgresql repmgr_failover: manual - tpa_2q_repositories: - - products/livecompare/release + edb_repositories: + - enterprise packages: common: - edb-livecompare From 0e34df79d0287a498c87da5bde3a101c451c1123 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 21 Aug 2024 18:07:09 +0100 Subject: [PATCH 07/16] Fixups on Oracle support page Signed-off-by: Dj Walker-Morgan --- .../docs/livecompare/3/oracle_support.mdx | 36 ------------------- 1 file changed, 36 deletions(-) diff --git a/product_docs/docs/livecompare/3/oracle_support.mdx b/product_docs/docs/livecompare/3/oracle_support.mdx index 9065fb3b874..39784a64e74 100644 --- a/product_docs/docs/livecompare/3/oracle_support.mdx +++ b/product_docs/docs/livecompare/3/oracle_support.mdx @@ -26,7 +26,6 @@ max_parallel_workers = 4 oracle_user_tables_only = on oracle_ignore_unsortable = on column_intersection = on -force_collate = C difference_tie_breakers = oracle [Oracle Connection] @@ -62,7 +61,6 @@ max_parallel_workers = 4 oracle_user_tables_only = on oracle_ignore_unsortable = on column_intersection = on -force_collate = C difference_tie_breakers = oracle schema_qualified_table_names = off @@ -157,19 +155,6 @@ replication_sets = set_name = 'bdrgroup' LiveCompare works on PostgreSQL databases out-of-the-box. You don't need to install any additional software. But to be able to connect to Oracle, LiveCompare does requires additional software. -### oracledb Python module - -You need the Python module [oracledb](https://oracle.github.io/python-oracledb/) installed and available on your system so that LiveCompare can connect to an Oracle database. - -Currently, oracledb isn't installable from Linux distribution repositories, so follow -[the instructions on the oracledb website](https://python-oracledb.readthedocs.io/en/latest/user_guide/installation.html) to install it on your system. - -We recommend executing LiveCompare under the postgres operating system user. Then you can install the `oracledb` module through PIP only for the `postgres` user, using the following command: - -``` -pip3 install --user oracledb --upgrade -``` - ## Differences If LiveCompare finds any difference, it generates a DML script to apply only on the PostgreSQL connections. No DML script to apply on the Oracle connection is generated. @@ -192,27 +177,6 @@ oracle_ignore_unsortable = on column_intersection = on ``` -## Incompatible collation - -On Oracle, generally the following initialization parameters are set: - -``` -NLS_COMP = BINARY -NLS_SORT = BINARY -``` - -This means that, regardless of the `NLS_LANG` and other language settings, all `ORDER BY` operations in Oracle are performed using the character binary code. - -In Postgres, the equivalent collation that shows the same behavior is the `C` collation. If your Postgres database was initialized in a different collation, then by default LiveCompare might find issues when sorting PK values. This can lead to false positives. - -To work around that, you can force a collation (say, the `C` collation) in Postgres so it matches the same sort behavior from Oracle: - -```ini -force_collate = C -``` - -If LiveCompare detects that the comparison session involves Oracle and PostgreSQL, then LiveCompare already sets `force_collate = C`, unless you set it to another value. - ## Common hash By default, LiveCompare has `comparison_algorithm = block_hash`, even when comparing PostgreSQL to Oracle. However, a *common hash* is built following these rules: From 9e5c807810b4001ef54dee1d98151601fb392949 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 21 Aug 2024 18:24:33 +0100 Subject: [PATCH 08/16] More fixups and separate installaton section Signed-off-by: Dj Walker-Morgan --- product_docs/docs/livecompare/3/index.mdx | 1 + .../docs/livecompare/3/installation.mdx | 60 +++++++++++++++++++ .../docs/livecompare/3/requirements.mdx | 2 +- .../docs/livecompare/3/settings/general.mdx | 14 +++-- 4 files changed, 70 insertions(+), 7 deletions(-) create mode 100644 product_docs/docs/livecompare/3/installation.mdx diff --git a/product_docs/docs/livecompare/3/index.mdx b/product_docs/docs/livecompare/3/index.mdx index 6a532499509..983abf0ff60 100644 --- a/product_docs/docs/livecompare/3/index.mdx +++ b/product_docs/docs/livecompare/3/index.mdx @@ -3,6 +3,7 @@ navigation: - index - rel_notes - requirements + - installation - supported_technologies - command_line_usage - advanced_usage diff --git a/product_docs/docs/livecompare/3/installation.mdx b/product_docs/docs/livecompare/3/installation.mdx new file mode 100644 index 00000000000..3a83f30f158 --- /dev/null +++ b/product_docs/docs/livecompare/3/installation.mdx @@ -0,0 +1,60 @@ +--- +title: LiveCompare Installation +navTitle: Installation +--- + +Before installing, consult the [LiveCompare requirements](requirements) to ensure that your system meets the necessary prerequisites. + +## Installation from EDB repositories + +You can install LiveCompare from the EnterpriseDB's Repos 2.0 repository. + +If you have not already installed the appropriate EnterpriseDB repository, follow the instructions in the [EnterpriseDB repository installation guide](/repos/getting_started/). + +Then, to install LiveCompare, run the following command: + +### RHEL 8, or othe RPM-based distributions: + +```bash +sudo dnf install edb-livecompare +``` + +### Debian 11, Ubuntu, or other Debian-based distributions: + +```bash +sudo apt-get install edb-livecompare +``` + +### SLES 12 SP5 and 15 SP3: + +```bash +sudo zypper install edb-livecompare +``` + +## Installation with manual download + +If you prefer to download the LiveCompare package manually, you can download the package from the [EnterpriseDB website](https://www.enterprisedb.com/software-downloads). + +After downloading the package, you can install it using the following command: + +### RHEL 8, or othe RPM-based distributions: + +```bash +sudo rpm -i edb-livecompare-.rpm +``` + +### Debian 11, Ubuntu, or other Debian-based distributions: + +```bash +sudo dpkg -i edb-livecompare-.deb +``` + +### SLES 12 SP5 and 15 SP3: + +```bash +sudo zypper install edb-livecompare-.rpm +``` + + + + diff --git a/product_docs/docs/livecompare/3/requirements.mdx b/product_docs/docs/livecompare/3/requirements.mdx index 4fd84abd7b6..e62445ff82a 100644 --- a/product_docs/docs/livecompare/3/requirements.mdx +++ b/product_docs/docs/livecompare/3/requirements.mdx @@ -35,7 +35,7 @@ EDB no longer tests LiveCompare with Oracle 10g. As such, it's no longer on the ## Installation -You can install LiveCompare from the EnterpriseDB `products/livecompare` repository. For details, see the [EDB customer portal](https://techsupport.enterprisedb.com/customer_portal/sw/livecompare/). +You can install LiveCompare from the EnterpriseDB's Repos 2.0 repository. For details, see the [LiveCompare installation guide](/livecompare/latest/installation/). diff --git a/product_docs/docs/livecompare/3/settings/general.mdx b/product_docs/docs/livecompare/3/settings/general.mdx index 477bcceedd3..37185b022cf 100644 --- a/product_docs/docs/livecompare/3/settings/general.mdx +++ b/product_docs/docs/livecompare/3/settings/general.mdx @@ -6,19 +6,21 @@ deepToC: true description: All the general settings available for LiveCompare 3.0 --- -## General settings - ### `logical_replication_mode` Affects how the program interprets connections and table filter settings and also the requirements to check for in the connections before starting the comparison. Currently the possible values are: -- `off`: Assumes there's no logical replication between the databases. +- `off`
+Assumes there's no logical replication between the databases. -- `native`: Assumes there's native logical replication between the databases. Enables the use of the `Table Filter -> publications` setting to specify the list of tables to use. Requires PostgreSQL 10+ on all databases. +- `native`
+Assumes there's native logical replication between the databases. Enables the use of the `Table Filter -> publications` setting to specify the list of tables to use. Requires PostgreSQL 10+ on all databases. -- `pglogical`: Assumes there's pglogical replication between the databases. Enables the use of the `Table Filter -> replication_sets` setting to specify the list of tables to use. Also enables the use of `node_name` to specify the data connections, which requires setting the `Initial Connection` that's used to retrieve DSN information of the nodes. Requires the `pglogical` extensions to be installed on all databases. +- `pglogical`
+Assumes there's pglogical replication between the databases. Enables the use of the `Table Filter -> replication_sets` setting to specify the list of tables to use. Also enables the use of `node_name` to specify the data connections, which requires setting the `Initial Connection` that's used to retrieve DSN information of the nodes. Requires the `pglogical` extensions to be installed on all databases. -- `bdr`: Assumes all data connections are nodes from the same PGD cluster. +- `bdr`
+Assumes all data connections are nodes from the same PGD cluster. Enables use of the `Table Filter -> replication_sets` setting to specify the list of tables to use. Also enables the use of `node_name` to specify the data connections, which requires setting the `Initial Connection` From 79305049bab27773e76c2478abb76b164954e72f Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 22 Aug 2024 14:46:09 +0100 Subject: [PATCH 09/16] Remove LiveCompare V1 docs, leaving only a stub for archive docs Signed-off-by: Dj Walker-Morgan --- .../docs/livecompare/1/advanced_usage.mdx | 152 ---- .../docs/livecompare/1/bdr_support.mdx | 477 ------------ .../docs/livecompare/1/command_line_usage.mdx | 139 ---- product_docs/docs/livecompare/1/index.mdx | 107 +-- product_docs/docs/livecompare/1/licenses.mdx | 358 --------- .../docs/livecompare/1/oracle_support.mdx | 309 -------- .../1/rel_notes/1.17.0_rel_notes.mdx | 12 - .../1/rel_notes/1.18.0_rel_notes.mdx | 17 - .../1/rel_notes/1.18.1_rel_notes.mdx | 10 - .../docs/livecompare/1/rel_notes/index.mdx | 25 - .../docs/livecompare/1/requirements.mdx | 90 --- product_docs/docs/livecompare/1/settings.mdx | 689 ------------------ .../livecompare/1/supported_technologies.mdx | 49 -- src/constants/products.js | 2 +- src/constants/updates.js | 8 + 15 files changed, 11 insertions(+), 2433 deletions(-) delete mode 100644 product_docs/docs/livecompare/1/advanced_usage.mdx delete mode 100644 product_docs/docs/livecompare/1/bdr_support.mdx delete mode 100644 product_docs/docs/livecompare/1/command_line_usage.mdx delete mode 100644 product_docs/docs/livecompare/1/licenses.mdx delete mode 100644 product_docs/docs/livecompare/1/oracle_support.mdx delete mode 100644 product_docs/docs/livecompare/1/rel_notes/1.17.0_rel_notes.mdx delete mode 100644 product_docs/docs/livecompare/1/rel_notes/1.18.0_rel_notes.mdx delete mode 100644 product_docs/docs/livecompare/1/rel_notes/1.18.1_rel_notes.mdx delete mode 100644 product_docs/docs/livecompare/1/rel_notes/index.mdx delete mode 100644 product_docs/docs/livecompare/1/requirements.mdx delete mode 100644 product_docs/docs/livecompare/1/settings.mdx delete mode 100644 product_docs/docs/livecompare/1/supported_technologies.mdx diff --git a/product_docs/docs/livecompare/1/advanced_usage.mdx b/product_docs/docs/livecompare/1/advanced_usage.mdx deleted file mode 100644 index e4e67253c0c..00000000000 --- a/product_docs/docs/livecompare/1/advanced_usage.mdx +++ /dev/null @@ -1,152 +0,0 @@ ---- -navTitle: Advanced usage -title: Advanced Usage -originalFilePath: advanced_usage.md - ---- - -After the end of execution of LiveCompare, you will notice it created a folder -called `lc_session_` in the working directory. This folder contains -the following files: - -- `lc__.log`: log file for the session; - -- `summary_.out`: shows a list of all tables that were processed, - and for each table it shows the time LiveCompare took to process the table, the - total number of rows and how many rows were processed, how many - differences were found in the table, and also the maximum number of ignored columns, - if any. - -To get the complete summary, you can also execute the following query against -the output database: - -```postgresql -select * -from .vw_table_summary -where session_id = ; -``` - -- `differences_.out`: if there are any differences, this file - shows useful information about each difference. This file is not generated if - there are no differences. - -For example, the difference list could be like this: - -```text -+-------------------+-------------------------+-----------------+---------------------+ -| table_name | table_pk_column_names | difference_pk | difference_status | -|-------------------+-------------------------+-----------------+---------------------| -| public.categories | category | (7) | P | -| public.categories | category | (10) | P | -| public.categories | category | (17) | P | -| public.categories | category | (18) | P | -+-------------------+-------------------------+-----------------+---------------------+ -``` - -To get the full list of differences with all details, you can also execute the -following query against the output database: - -```postgresql -select * -from ; -``` - -To understand how LiveCompare consensus worked to decide which databases are -divergent, then the view `vw_consensus` can provide details on the consensus -algorithm: - -```postgresql -select * -from ; -``` - -- `apply_on_the_first_.sql`: if there are any differences, this - file will show a DML command to be applied on the **first** database, to make - the **first** database consistent all other databases. For example, for the - differences above, this script could be: - -```postgresql -BEGIN; - -DELETE FROM public.categories WHERE (category) = 7; -UPDATE public.categories SET categoryname = $lc1$Games Changed$lc1$ WHERE (category) = 10; -INSERT INTO public.categories (category,categoryname) VALUES (17, $lc1$Test 1$lc1$); -INSERT INTO public.categories (category,categoryname) VALUES (18, $lc1$Test 2$lc1$); - -COMMIT; -``` - -LiveCompare generates this script automatically. In order to fix the -inconsistencies in the **first** database, you can simply execute the script in -the **first** database. - -LiveCompare generates a similar `apply_on_*.sql` script for each database that -has inconsistent data. - -## Which differences to fix - -LiveCompare is able to identify and provide fixes for the following differences: - -- A row exists in the majority of the data connections. The fix will be an - `INSERT` on the divergent databases; -- A row does not exist in the majority of the data connections. The fix will be - a `DELETE` on the divergent databases; -- A row exists in all databases, but some column values mismatch. The fix will - be an `UPDATE` on the divergent databases. - -By default `difference_statements = all`, which means that LiveCompare will try -to apply all 3 DML types (`INSERT`, `UPDATE` and `DELETE`) for each difference -it finds. But it is possible to specify which type of DML LiveCompare should -consider when providing difference fixes, by changing the value of -the setting `difference_statements`, which can be: - -- `all` (default): Fixes `INSERT`s, `UPDATE`s and `DELETE`s; -- `inserts`: Fixes only `INSERT`s; -- `updates`: Fixes only `UPDATE`s; -- `deletes`: Fixes only `DELETE`s; -- `inserts_updates`: Fixes only `INSERT`s and `UPDATE`s; -- `inserts_deletes`: Fixes only `INSERT`s and `DELETE`s; -- `updates_deletes`: Fixes only `UPDATE`s and `DELETE`s. - -When `difference_statements` has the values `all`, `updates`, `inserts_updates` -or `updates_deletes`, then it is possible to tell LiveCompare to ignore any -`UPDATE`s that would set `NULL` to a column. - -## Difference log - -Table `difference_log` stores all information about differences every time -LiveCompare checked them. Users can run LiveCompare in re-check mode multiple -times, so this table shows how the difference has evolved over the time window -where LiveCompare was re-checking it. - -- **Detected (D)**: The difference was just detected. In re-check and fix modes, - LiveCompare will mark all Permanent and Tie differences as Detected in order to - re-check them. - -- **Permanent (P)**: After having re-checked the difference, if data is still - divergent, LiveCompare marks the difference as **Permanent**. - -- **Tie (T)**: Same as Permanent, but there is not enough consensus to determine - which connections are the majority. - -- **Absent (A)**: If upon a re-check LiveCompare finds that the difference does - not exist anymore (the row is now consistent between both databases), then - LiveCompare marks the difference as **Absent**. - -- **Volatile (V)**: If upon a re-check `xmin` has changed on an inconsistent - row, then LiveCompare marks the difference as **Volatile**. - -- **Ignored (I)**: Users can stop difference re-check of certain differences by - manually calling the function - `.accept_divergence(session_id, table_name, difference_pk)` - in the Output PostgreSQL connection. For example: - -```postgresql -SELECT livecompare.accept_divergence( - 2 -- session_id - , 'public.categories' -- table_name - , $$(10)$$ -- difference_pk -); -``` diff --git a/product_docs/docs/livecompare/1/bdr_support.mdx b/product_docs/docs/livecompare/1/bdr_support.mdx deleted file mode 100644 index 8ca3fe652af..00000000000 --- a/product_docs/docs/livecompare/1/bdr_support.mdx +++ /dev/null @@ -1,477 +0,0 @@ ---- -navTitle: BDR support -title: BDR Support -originalFilePath: bdr_support.md - ---- - -LiveCompare can be used against BDR nodes, as well as non-BDR nodes. - -Setting `logical_replication_mode = bdr` will make the tool assume that all -databases being compared belong to the same BDR cluster. Then you can specify -node names as connections, and replication sets to filter tables. - -For example, consider you are able to connect to any node in the BDR cluster. -Let's call this `Initial Connection`. By initially connection to this node, -LiveCompare is able to check BDR metadata and retrieve connection information -from all other nodes. - -Now consider you want to compare 3 BDR nodes. As LiveCompare is able to connect -to any node starting from the `Initial Connection`, you do not need to define -`dsn` or any connection information for the data connections. You just need to -define `node_name`. LiveCompare searches in BDR metadata about the connection -information for that node, and then connects to the node. - -Please note that, for LiveCompare to be able to connect to all other nodes by -fetching BDR metadata, it is required that LiveCompare is able to connect to -them using the same DSN from BDR view `bdr.node_summary`, field -`interface_connstr`. In this case it is recommended to run LiveCompare on the -same machine as the `Initial Connection`, as `postgres` user. If that's not -possible, then please define the `dsn` attribute in all data connections. - -You can also specify replication sets as table filters. LiveCompare will use -BDR metadata to build the table list, considering only tables that belong to the -replication set(s) you defined in the `replication_sets` setting. - -For example, you can create an `.ini` file to compare 3 BDR nodes: - -```ini -[General Settings] -logical_replication_mode = bdr -max_parallel_workers = 4 -parallel_chunk_rows = 1000000 - -[Initial Connection] -dsn = port=5432 dbname=live user=postgres - -[Node1 Connection] -node_name = node1 - -[Node2 Connection] -node_name = node2 - -[Node3 Connection] -node_name = node3 - -[Output Connection] -dsn = port=5432 dbname=liveoutput user=postgres - -[Table Filter] -replication_sets = set_name = 'bdrgroup' -``` - -It is also possible to tell LiveCompare to compare all active nodes in the BDR -cluster. For that purpose just do the following: - -- In `General Settings`, enable `all_bdr_nodes = on`; -- Specify an `Initial Connection`; -- Additional data connections are not required. - -For example: - -```ini -[General Settings] -logical_replication_mode = bdr -max_parallel_workers = 4 -parallel_chunk_rows = 1000000 -all_bdr_nodes = on - -[Initial Connection] -dsn = port=5432 dbname=live user=postgres - -[Output Connection] -dsn = port=5432 dbname=liveoutput user=postgres - -[Table Filter] -replication_sets = set_name = 'bdrgroup' -``` - -When `all_bdr_nodes = on`, LiveCompare uses the `Initial Connection` to fetch -the list of all BDR nodes. Additional data connections are not required; -although if set, will be appended to the list of data connections. For example, -it would be possible to compare a whole BDR cluster against a single Postgres -connection, useful in migration projects: - -```ini -[General Settings] -logical_replication_mode = bdr -max_parallel_workers = 4 -parallel_chunk_rows = 1000000 -all_bdr_nodes = on - -[Initial Connection] -dsn = port=5432 dbname=live user=postgres - -[Old Connection] -dsn = host=oldpg port=5432 dbname=live user=postgres - -[Output Connection] -dsn = port=5432 dbname=liveoutput user=postgres - -[Table Filter] -replication_sets = set_name = 'bdrgroup' -``` - -Settings `node_name` and `replication_sets` are supported for the following -technologies: - -- BDR 1, 2 and 3; -- pglogical 2 and 3. - -Please note that to enable pglogical metadata fetch instead of BDR, just set -`logical_replication_mode = pglogical` instead of -`logical_replication_mode = bdr`. - -## BDR Witness nodes - -Using replication sets in BDR, it's possible to configure specific tables to be -included in the BDR replication, and also specify which nodes should receive -data from such tables, by configuring the node to subscribe to the replication -set the table belongs to. This allows for different architectures such as BDR -Sharding and the use of BDR Witness nodes. - -A BDR Witness is a regular BDR node which doesn't replicate any DML from other -nodes. The purpose of the Witness is to provide quorum in Raft Consensus voting -(for more details on the BDR Witness node, check BDR documentation). Depending -on how replication sets were configured, the Witness may or may not replicate -DDL. Which means that there are 2 types of BDR Witnesses: - -- A completely empty node, without any data nor tables; or -- A node that replicates DDL from other nodes, hence having empty tables. - -In the first case, even if the BDR Witness is included in the comparison (either -manually under `[Connections]` or using `all_bdr_nodes = on`), as the Witness -doesn't have any tables, the following message will be logged: - -``` -Table public.tbl does not exist on connection node1 -``` - -In the second case, on the other hand, the table exists on the BDR Witness. -However, it would not be correct to report data missing on the Witness as -divergences. So, for each table, LiveCompare checks the following information on -each node included in the comparison: - -- The replication sets that the node subscribes; -- The replication sets that the table is associated with; -- The replication sets, if any, the user defined in filter `replication_sets` - under `Table Filter`. - -If the intersection among all 3 lists of replication sets is empty, which is the -case for the BDR Witness, then LiveCompare will log this: - -``` -Table public.tbl is not subscribed on connection node1 -``` - -In both cases, the comparison for that specific table proceeds on the nodes -where the table exists, and the table is replicated according to the replication -sets configuration. - -## Differences in a BDR cluster - -LiveCompare will make changes to the local node only; it is important that -corrective changes do not get replicated to other nodes. - -When `logical_replication_mode = bdr`, LiveCompare will initially check if a -replication origin called `bdr_local_only_origin` already exists (the name of -the replication origin can be configured by adjusting the setting -`difference_fix_replication_origin`). If a replication origin called -`bdr_local_only_origin` does not exist yet, then LiveCompare creates it on all -BDR connections. - -**IMPORTANT**: Please note that BDR 3.6.18 introduced the new pre-created -`bdr_local_only_origin` replication origin to be used for applying local-only -transactions. So if LiveCompare is connected to BDR 3.6.18, it won't create this -replication origin. - -LiveCompare will generate apply scripts considering the following: - -- Set the current transaction to use the replication origin - `bdr_local_only_origin`, so any DML executed will have `xmin` associated to - `bdr_local_only_origin`; -- Set the current transaction datetime to be far in the past, so if there are - any BDR conflicts with real DML being executed on the database, LiveCompare DML - always loses the conflict. - -After applying LiveCompare fix script to a BDR node, it -will be possible to get exactly which rows were inserted or updated by -LiveCompare using the following query (replace `mytable` with the name of any -table): - -```postgresql -with lc_origin as ( - select roident - from pg_replication_origin - where roname = 'bdr_local_only_origin' -) -select t.* -from mytable t -inner join lc_origin r -on r.roident = bdr.pg_xact_origin(t.xmin); -``` - -(Note that deleted rows are no longer visible.) - -Please note that LiveCompare requires at least a PostgreSQL user with -`bdr_superuser` privileges in order to properly fetch metadata. - -All steps above involving replication origins only applied to output script, if -the PostgreSQL user has `bdr_superuser` or PostgreSQL superuser privileges. -Otherwise, LiveCompare will generate fixes without associating any replication -origin (transaction replication is still disabled using -`SET LOCAL bdr.xact_replication = off`). However, it is recommended to use a -replication origin when applying the DML scripts, because otherwise LiveCompare -will have the same precedence as a regular user application regarding conflict -resolution. Also, as there will not be any replication origin associated to the -fix, the query above to list all rows fixed by LiveCompare can not be used. - -Between BDR 3.6.18 and BDR 3.7.0, the following functions are used: - -- `bdr.difference_fix_origin_create()`: Executed by LiveCompare to create the - replication origin specified in `difference_fix_replication_origin` (by default - set to `bdr_local_only_origin`), if this replication origin does not exist; -- `bdr.difference_fix_session_setup()`: Included in the generated DML script so - the transaction is associated with the replication origin specified in - `difference_fix_replication_origin`; -- `bdr.difference_fix_xact_set_avoid_conflict()`: Included in the generated DML - script so the transaction is set far in the past (`2010-01-01`), so the fix - transaction applied by LiveCompare always loses a conflict, if any. - -The functions above require a `bdr_superuser` rather than a PostgreSQL -superuser. Starting from BDR 3.7.0, those functions are deprecated. LiveCompare -then will, if running as a PostgreSQL superuser, use the following functions -instead, to perform the same actions as above: - -- `pg_replication_origin_create(origin_name)`; -- `pg_replication_origin_session_setup()`; -- `pg_replication_origin_xact_setup()`. - -If a PostgreSQL superuser is not being used, then LiveCompare will include only -the following in the generated DML transaction: - -``` -SET LOCAL bdr.xact_replication = off; -``` - -## Conflicts in BDR - -LiveCompare has an execution mode called `conflicts`. This execution mode is -specific for BDR clusters. It will only work in BDR 3.6 or BDR 3.7 clusters. - -While `compare` mode is used to compare all content of tables as a whole, -`conflicts` mode will focus just in tuples/tables that are related to existing -conflicts that are registered in `bdr.apply_log`, in case of BDR 3.6, or in -`bdr.conflict_history`, in case of BDR 3.7. - -Having said that, `conflicts` execution mode is expected to run much faster than -`compare` mode, because it will just inspect specific tuples from specific -tables. At the same time, it's not as complete as `compare` mode, because of the -same reason. - -The main objective of this execution mode is to check that the automatic -conflict resolution which is being done by BDR is consistent among nodes, i.e., -after BDR resolving conflicts the cluster is in a consistent state. - -Although, for the general use case, automatic conflict resolution ensures -cluster consistency, there are a few known cases where automatic conflict -resolution can result in divergent tuples among nodes. So the `conflicts` -execution mode from LiveCompare can help checking and ensuring consistency, with -a good balance between time vs result. - -### Conflict example - -Imagine on `node3` we execute the following query: - -``` -SELECT c.reloid::regclass, - s.origin_name, - c.local_time, - c.key_tuple, - c.local_tuple, - c.remote_tuple, - c.apply_tuple, - c.conflict_type, - c.conflict_resolution -FROM bdr.conflict_history c -INNER JOIN bdr.subscription_summary s -ON s.sub_id = c.sub_id; -``` - -We can see the following conflict in `bdr.conflict_history`: - -``` -reloid | tbl -origin_name | node2 -local_time | 2021-05-13 19:17:43.239744+00 -key_tuple | {"a":null,"b":3,"c":null} -local_tuple | -remote_tuple | -apply_tuple | -conflict_type | delete_missing -conflict_resolution | skip -``` - -Which means that when the `DELETE` arrived from `node2` to `node3`, there was no -row with `b = 3` in table `tbl`. However, the `INSERT` might have arrived from -`node1` to `node3` later, which then added the row with `b = 3` to `node3`. So -this is the current situation on `node3`: - -``` -bdrdb=# SELECT * FROM tbl WHERE b = 3; - a | b | c ----+---+----- - x | 3 | foo -(1 row) -``` - -While on nodes `node1` and `node2`, this is what we see: - -``` -bdrdb=# SELECT * FROM tbl WHERE b = 3; - a | b | c ----+---+--- -(0 rows) -``` - -The BDR cluster is divergent. - -Now in order to detect and fix such divergence, we could execute LiveCompare in -`compare` mode, but depending on the size of the comparison set (imagine table -`tbl` is very large), that can take a long time, even hours. - -This is exactly the situation where `conflicts` mode can be helpful. In this -case, the `delete_missing` conflict is visible only from `node3`, but -LiveCompare is able to extract the PK values from the conflict logged rows -(`key_tuple`, `local_tuple`, `remote_tuple` and `apply_tuple`) and perform an -automatic cluster-wide comparison only on the affected table, already filtering -by the PK values. The comparison will then check the current row version in all -nodes in the cluster. - -So we create a `check.ini` file to set `all_bdr_nodes = on`, i.e., to tell -LiveCompare to compare all nodes in the cluster: - -``` -[General Settings] -logical_replication_mode = bdr -max_parallel_workers = 2 -all_bdr_nodes = on - -[Initial Connection] -dsn = dbname=bdrdb - -[Output Connection] -dsn = dbname=liveoutput -``` - -To run LiveCompare in `conflicts` mode: - -``` -2ndq-livecompare check.ini --conflicts -``` - -After the execution, in the console output, you will see something like this: - -``` -Elapsed time: 0:00:02.443557 -Processed 1 conflicts about 1 tables from 3 connections using 2 workers. -Found 1 divergent conflicts in 1 tables. -Processed 1 rows in 1 tables from 3 connections using 2 workers. -Found 1 inconsistent rows in 1 tables. -``` - -Inside folder `./lc_session_X/` (being `X` the number of the current comparison -session), LiveCompare will write the file `conflicts_DAY.out` (replacing `DAY` -in the name of the file with the current day), showing the main information -about all divergent conflicts. - -If you connect to database `liveoutput`, you will be able to see more details -about the conflicts, for example using this query: - -``` -SELECT * -FROM livecompare.vw_conflicts -WHERE session_id = 1 - AND conflict_id = 1 -ORDER BY table_name, - local_time, - target_node; -``` - -You will see something like this: - -``` -session_id | 1 -table_name | public.tbl -conflict_id | 1 -connection_id | node3 -origin_node | node2 -target_node | node3 -local_time | 2021-05-13 19:17:43.239744+00 -key_tuple | {"a": null, "b": 3, "c": null} -local_tuple | -remote_tuple | -apply_tuple | -conflict_type | delete_missing -conflict_resolution | skip -conflict_pk_value_list | {(3)} -difference_log_id_list | {1} -is_conflict_divergent | t -``` - -The `is_conflict_divergent = true` means that LiveCompare has compared the -conflict and found the nodes to be currently divergent in the tables and rows -reported by the conflict. View `livecompare.vw_conflicts` shows information -about all conflicts, including the non-divergent ones. - -LiveCompare will also automatically generate DML script -`./lc_session_X/apply_on_the_node3_DAY.sql` (replacing `DAY` in the name of the -file with the current day): - -``` -BEGIN; - -SET LOCAL bdr.xact_replication = off; -SELECT pg_replication_origin_session_setup('bdr_local_only_origin'); -SELECT pg_replication_origin_xact_setup('0/0', '2010-01-01'::timestamptz);; - -SET LOCAL ROLE postgres; -DELETE FROM public.tbl WHERE (b) = (3); - -COMMIT; -``` - -LiveCompare is suggesting to `DELETE` the row where `b = 3` from `node3`, -because on the other 2 nodes the row does not exist. By default, LiveCompare -suggest the DML to fix based on the majority of the nodes. - -If you run this DML script against `node3`: - -``` -psql -h node3 -f ./lc_session_X/apply_on_the_node3_DAY.sql -``` - -You will get the BDR cluster consistent again. - -As the `--conflicts` mode comparison is much faster than a full `--compare`, it -is highly recommended to schedule a `--conflicts` comparison session more often, -to ensure conflict resolution is providing cluster-wide consistency. - -Please note that, in order to be able to see the data in `bdr.conflict_history` -in BDR 3.7 or `bdr.apply_log` in BDR 3.6, you should run LiveCompare with an -user that is `bdr_superuser` or is a PostgreSQL superuser. - -## Mixing technologies - -Please note that metadata for `node_name` and `replication_sets` are fetched in -the `Initial Connection`. So it should be a pglogical- and/or BDR-enabled -database. - -The list of tables is built in the first data connection. So the -`replication_sets` condition should be valid in the first connection. - -It is possible to perform mixed technology comparisons, for example: - -- BDR 1 node versus BDR 3 node; -- BDR 3 node versus vanilla Postgres instance; -- Vanilla Postgres instance versus pglogical node. diff --git a/product_docs/docs/livecompare/1/command_line_usage.mdx b/product_docs/docs/livecompare/1/command_line_usage.mdx deleted file mode 100644 index e3ba7cb9ad3..00000000000 --- a/product_docs/docs/livecompare/1/command_line_usage.mdx +++ /dev/null @@ -1,139 +0,0 @@ ---- -navTitle: Command line usage -title: Command-line Usage -originalFilePath: command_line_usage.md - ---- - -## Compare mode - -Copy any `/etc/2ndq-livecompare/template*.ini` to use in your project and adjust -as necessary (see the section `Settings` below). - -``` -cp /etc/2ndq-livecompare/template_basic.ini my_project.ini - -2ndq-livecompare my_project.ini -``` - -During the execution of LiveCompare, you will see `N+1` progress bars, `N` being -the number of processes (you can specify the number of processes in the -settings). The first progress bar shows overall execution while the other -progress bars show the current table being processed by a specific process. - -The information being shown for each table is, from left to right: - -- Number of the process -- Table name -- Status, which may be the ID of the comparison round followed by the current - table chunk (`p1/1` means the table was not split). If the status says - `setup`, it means the table is being analyzed (checking row count and splitting - if necessary) -- Number of rows processed -- Number of total rows being considered in this comparison round -- Time elapsed -- Estimated time to complete -- Speed in records per second - -If a table has more rows than the `parallel_chunk_rows` setting (see more -details below), then a hash function will be used to determine which job will -consider each row. This can slow down the comparison individually, but the -comparison as a whole may benefit from parallelism for the given table. - -While the program is executing, you can cancel it at any time by pressing -`Ctrl-c`. You will see a message like this: - -```text -Manually stopping session 6... You can resume the session with: - -2ndq-livecompare my_project.ini 6 -``` - -**Important**: If LiveCompare is running in background or running in another shell, -you can still softly stop it. It will keep the `PID` of the master process inside the -session folder (in the example `lc_session_6`), in a file named `livemaster.pid`. You -can then invoke `kill -2 ` to softly stop it. - -Then, at any time you can resume a previously canceled session, for example: - -``` -2ndq-livecompare my_project.ini 6 -``` - -When the program ends, if it found no inconsistencies, you will see an output -like this: - -```text -Saved file lc_session_5/summary_20190514.out with the complete table summary. -You can also get the table summary by connecting to the output database and executing: -select * from livecompare.vw_table_summary where session_id = 5; - -Elapsed time: 0:02:10.970954 -Processed 3919015 rows in 6 tables using 3 processes. -Found 0 inconsistent rows in 0 tables. -``` - -But if any inconsistencies were found, the output will look like this: - -```text -Comparison finished, waiting for remaining difference checks... - -Outstanding differences: - -+--------------+-------------------+-----------------+------------------+----------------------+-------------------+---------------------------+ -| session_id | table_name | elapsed_time | num_total_rows | num_processed_rows | num_differences | max_num_ignored_columns | -|--------------+-------------------+-----------------+------------------+----------------------+-------------------+---------------------------| -| 6 | public.categories | 00:00:00.027864 | 18 | 18 | 4 | | -+--------------+-------------------+-----------------+------------------+----------------------+-------------------+---------------------------+ - -Saved file lc_session_6/summary_20200129.out with the complete table summary. -You can also get the table summary by connecting to the output database and executing: -select * from livecompare.vw_table_summary where session_id = 6; - -Elapsed time: 0:00:50.149987 -Processed 172718 rows in 8 tables from 3 connections using 2 workers. -Found 4 inconsistent rows in 1 tables. - -Saved file lc_session_6/differences_20200129.out with the list of differences per table. -You can also get a list of differences per table with: -select * from livecompare.vw_differences where session_id = 6; -Too see more details on how LiveCompare determined the differences: -select * from livecompare.vw_consensus where session_id = 6; - -Script lc_session_6/apply_on_the_first_20200129.sql was generated, which can be applied to the first connection and make it consistent with the majority of connections. -You can also get this script with: -select difference_fix_dml from livecompare.vw_difference_fix where session_id = 6 and connection_id = 'first'; -``` - -## Re-check mode - -In a BDR environment, any divergence that BDR finds can be later non-existing -as the replication caught up due to eventual consistency. Depending on several -factors, replication lag can cause LiveCompare to report false positives. - -To overcome that, in a later moment when replication lag has decreased or data -has already caught up, users can manually execute a re-check only on the -differences that were previously found. This execution mode is called "recheck" -and can be executed like this: - -``` -2ndq-livecompare my_project.ini 6 --recheck -``` - -In this mode, LiveCompare will generate separate recheck logs and update all -reports that are already existing in the `lc_session_X` directory. - -**Important**: If resuming a `compare` or executing under `recheck`, -LiveCompare will check if the settings and connections attributes are the same as -when the session was created. If any divergence found, it will quit the execution -with proper message. - -## Conflicts mode - -To run LiveCompare in `conflicts` mode, you should invoke it with: - -``` -2ndq-livecompare my_project.ini --conflicts -``` - -For more details about the `conflicts` mode, check BDR Support chapter. diff --git a/product_docs/docs/livecompare/1/index.mdx b/product_docs/docs/livecompare/1/index.mdx index 9958301778d..775ee129f5a 100644 --- a/product_docs/docs/livecompare/1/index.mdx +++ b/product_docs/docs/livecompare/1/index.mdx @@ -1,112 +1,9 @@ --- -navigation: - - index - - rel_notes - - requirements - - supported_technologies - - command_line_usage - - advanced_usage - - bdr_support - - oracle_support - - settings - - licenses title: LiveCompare originalFilePath: index.md - --- -LiveCompare is designed to compare any number of databases to verify they are -identical. The tool compares any number databases and generates a comparison -report, a list of differences and handy DML scripts so the user can optionally -apply the DML and fix the inconsistencies in any of the databases. - -By default, the comparison set will include all tables in the database. -LiveCompare allows checking of multiple tables concurrently (multiple worker -processes) and is highly configurable to allow checking just a few tables or -just a section of rows within a table. - -Each database comparison is called a "comparison session". When the program -starts for the first time, it will start a new session and start comparing table -by table. In standalone mode, once all tables are compared, the program stops -and generates all reports. LiveCompare can be stopped and started without losing -context information, so it can be run at convenient times. - -Each table comparison operation is called a "comparison round". If the table is -too big, LiveCompare will split the table into multiple comparison rounds that -will also be executed in parallel, alongside with other tables that are being -carried on by other workers at the same time. - -In standalone mode, the initial comparison round for a table starts from the -beginning of the table (oldest existing PK) to the end of the table (newest -existing PK). New rows inserted after the round started are ignored. LiveCompare -will sort the PK columns in order to get min and max PK from each table. For each -PK column which is unsortable, LiveCompare will cast it's content to `string`. In -PostgreSQL that's achieved by using `::text` and in Oracle by using `to_char`. - -When executing the comparison algorithm, each worker requires N+1 database -connections, being N the number of databases being compared. The extra required -connection is to an output/reporting database, where the program cache is kept -too, so the user is able to stop/resume a comparison session. - -Any differences found by the comparison algorithm can be manually re-checked by -the user at a later convenient time. This is recommended to be done to allow a -replication consistency check. Upon the difference re-check, maybe replication -caught up on that specific row and the difference does not exist anymore, so the -difference is removed, otherwise it is marked as permanent. - -At the end of the execution the program generates a DML script so the user can -review it, and fix differences one by one, or simply apply the entire DML script -so all permanent differences are fixed. - -LiveCompare can be potentially used to ensure logical data integrity at -row-level; for example, for these scenarios: - -- Database technology migration (Oracle x Postgres); -- Server migration or upgrade (old server x new server); -- Physical replication (primary x standby); -- After failover incidents, for example to compare the new primary data against - the old, isolated primary data; -- In case of an unexpected split-brain situation after a failover. If the old - primary was not properly fenced and the application wrote data into it, it is - possible to use LiveCompare to know exactly which data is present in the old - primary and is not present in the new primary. If desired, the DBA can use the - DML script that LiveCompare generates to apply those data into the new primary; -- Logical replication. Three kind of logical replication technologies are - supported: Postgres native logical replication, pglogical and BDR. - -## Comparison Performance - -LiveCompare has been optimized for use on production systems and has various -parameters for tuning, described later. Comparison rounds are read-only -workloads. An example use case compared 43,109,165 rows in 6 tables in 9m 17s -with 4 connections and 4 workers, giving comparison performance of approximately -77k rows per second, or 1 billion rows in <4 hours. - -The use case above can be considered a general use case. For low-load, testing, -migration and other specific scenarios, it might be possible to improve speed by -changing the `data_fetch_mode` setting to use server-side cursors. Each kind of -server side cursors, in our experiments, provides an increase in performance on -use cases involving either small or large tables. - -## Security Considerations for the User - -When `logical_replication_mode = bdr`, LiveCompare requires a user that has been -granted the `bdr_superuser` role. When `logical_replication_mode = pglogical`, -LiveCompare requires a user that has been granted the `pglogical_superuser` -role. - -To apply the DML scripts in BDR, then all divergent connections (potentially all -data connections) require a user that has been granted the `bdr_superuser` in -order to disable `bdr.xact_replication`. +LiveCompare v1 is no longer supported. Please consider upgrading to the latest [LiveCompare](/livecompare/latest/). -If BDR is being used, LiveCompare will associate all fixed rows with a -replication origin called `bdr_local_only_origin`. LiveCompare will also apply -the DML with the transaction datetime far in the past, so if there are any BDR -conflicts with real DML being executed on the database, LiveCompare DML always -loses the conflict. +Documentation for LiveCompare v1 is archived and available from the EDB documenation archive - [here](https://docs-archive.enterprisedb.com/livecompare-index/). -With the default setting of `difference_fix_start_query`, the transaction in -apply scripts will change role to the owner of the table in order to prevent -database users from gaining access to the role applying fixes by writing -malicious triggers. As a result the user for the divergent connection needs to -have ability to switch role to the table owner. diff --git a/product_docs/docs/livecompare/1/licenses.mdx b/product_docs/docs/livecompare/1/licenses.mdx deleted file mode 100644 index ed424ef6c20..00000000000 --- a/product_docs/docs/livecompare/1/licenses.mdx +++ /dev/null @@ -1,358 +0,0 @@ ---- -title: 'Licenses' -originalFilePath: appendix_b.md -redirects: -- /livecompare/1/appendix_b.mdx ---- - -## TQDM - -`tqdm` is a product of collaborative work. -Unless otherwise stated, all authors (see commit logs) retain copyright -for their respective work, and release the work under the MIT licence -(text below). - -Exceptions or notable authors are listed below -in reverse chronological order: - -- files: \* - MPLv2.0 2015-2020 (c) Casper da Costa-Luis - [casperdcl](https://github.com/casperdcl). -- files: tqdm/\_tqdm.py - MIT 2016 (c) [PR #96] on behalf of Google Inc. -- files: tqdm/\_tqdm.py setup.py README.rst MANIFEST.in .gitignore - MIT 2013 (c) Noam Yorav-Raphael, original author. - -[PR #96]: https://github.com/tqdm/tqdm/pull/96 - -### Mozilla Public Licence (MPL) v. 2.0 - Exhibit A - -This Source Code Form is subject to the terms of the -Mozilla Public License, v. 2.0. -If a copy of the MPL was not distributed with this file, -You can obtain one at . - -### MIT License (MIT) - -Copyright (c) 2013 noamraph - -Permission is hereby granted, free of charge, to any person obtaining a copy of -this software and associated documentation files (the "Software"), to deal in -the Software without restriction, including without limitation the rights to -use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software is furnished to do so, -subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS -FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR -COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER -IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN -CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - -## cx_Oracle - -LICENSE AGREEMENT FOR CX_ORACLE - -Copyright © 2016, 2020, Oracle and/or its affiliates. All rights reserved. - -Copyright © 2007-2015, Anthony Tuininga. All rights reserved. - -Copyright © 2001-2007, Computronix (Canada) Ltd., Edmonton, Alberta, Canada. All rights reserved. - -Redistribution and use in source and binary forms, with or without modification, - are permitted provided that the following conditions are met: - -Redistributions of source code must retain the above copyright notice, -this list of conditions, and the disclaimer that follows. -Redistributions in binary form must reproduce the above copyright notice, -this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. -Neither the names of the copyright holders nor the names of any contributors may be used to endorse or promote products derived from this software without specific prior written permission. -DISCLAIMER: THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS *AS IS* AND ANY EXPRESS OR IMPLIED WARRANTIES, - INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. - IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -Computronix ® is a registered trademark of Computronix (Canada) Ltd. - -© Copyright 2016, 2020, Oracle and/or its affiliates. All rights reserved. Portions Copyright © 2007-2015, Anthony Tuininga. All rights reserved. Portions Copyright © 2001-2007, Computronix (Canada) Ltd., Edmonton, Alberta, Canada. All rights reserved Revision 10e5c258. - -### Apache License - -``` - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ -``` - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - -1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - -2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - -3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - -4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - -5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - -6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - -7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - -8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - -9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - -## Psycopg2 - -psycopg2 is free software: you can redistribute it and/or modify it -under the terms of the GNU Lesser General Public License as published -by the Free Software Foundation, either version 3 of the License, or -(at your option) any later version. - -psycopg2 is distributed in the hope that it will be useful, but WITHOUT -ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public -License for more details. - -In addition, as a special exception, the copyright holders give -permission to link this program with the OpenSSL library (or with -modified versions of OpenSSL that use the same license as OpenSSL), -and distribute linked combinations including the two. - -You must obey the GNU Lesser General Public License in all respects for -all of the code used other than OpenSSL. If you modify file(s) with this -exception, you may extend this exception to your version of the file(s), -but you are not obligated to do so. If you do not wish to do so, delete -this exception statement from your version. If you delete this exception -statement from all source files in the program, then also delete it here. - -You should have received a copy of the GNU Lesser General Public License -along with psycopg2 (see the doc/ directory.) -If not, see . - -### Alternative licenses - -The following BSD-like license applies (at your option) to the files following -the pattern `psycopg/adapter*.{h,c}` and `psycopg/microprotocol*.{h,c}`: - - Permission is granted to anyone to use this software for any purpose, - including commercial applications, and to alter it and redistribute it - freely, subject to the following restrictions: - -1. The origin of this software must not be misrepresented; you must not - claim that you wrote the original software. If you use this - software in a product, an acknowledgment in the product documentation - would be appreciated but is not required. - -2. Altered source versions must be plainly marked as such, and must not - be misrepresented as being the original software. - -3. This notice may not be removed or altered from any source distribution. - -## Tabulate - - Copyright (c) 2011-2020 Sergey Astanin and contributors - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -"Software"), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -© 2020 GitHub, Inc. - -## OmniDB - -MIT License - -Portions Copyright (c) 2015-2019, The OmniDB Team -Portions Copyright (c) 2017-2019, 2ndQuadrant Limited - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/product_docs/docs/livecompare/1/oracle_support.mdx b/product_docs/docs/livecompare/1/oracle_support.mdx deleted file mode 100644 index 272506f3bb7..00000000000 --- a/product_docs/docs/livecompare/1/oracle_support.mdx +++ /dev/null @@ -1,309 +0,0 @@ ---- -navTitle: Oracle support -title: Oracle Support -originalFilePath: oracle_support.md - ---- - -LiveCompare can be used to compare data from an Oracle database against any -number of PostgreSQL or BDR databases. - -For example, you can define `technology = oracle` in a data connection. Other -settings can then be used to define the connection to Oracle: - -- `host` -- `port` -- `service` -- `user` -- `password` - -All other data connections are required to be PostgreSQL. - -Here is a simple example of comparison between an Oracle database versus a -PostgreSQL database: - -```ini -[General Settings] -logical_replication_mode = off -full_comparison_mode = on -max_parallel_workers = 4 -oracle_user_tables_only = on -oracle_ignore_unsortable = on -column_intersection = on -force_collate = C -difference_tie_breakers = oracle - -[Oracle Connection] -technology = oracle -host = 127.0.0.1 -port = 1521 -service = XE -user = LIVE -password = live - -[Postgres Connection] -technology = postgresql -dsn = dbname=liveoracle user=william - -[Output Connection] -dsn = dbname=liveoutput user=william - -[Table Filter] -schemas = schema_name = 'live' -``` - -Here the `schema_name` in Oracle is the user table sandbox. All table names are -schema-qualified by default: - -- Postgres: ` . ` -- Oracle: ` . = 3.6 and <= 3.8 -- PostgreSQL / EDB Postgres Extended 9.5+ / EPAS 11+ (on the output connection) -- PostgreSQL / EDB Postgres Extended 9.4+ / EPAS 11+ or Oracle 10g+ (on the data connections being compared) - -LiveCompare requires at least Debian 10, Ubuntu 16.04, or CentOS/RHEL 7. - -LiveCompare can be installed from the EnterpriseDB `products/livecompare` -repository. More details can be found in: - - - -LiveCompare installs on top of: - -- The latest Python version for Ubuntu, Debian and CentOS/RHEL 8, as provided by - the `python3` packages; or -- Python 3.6 for CentOS/RHEL 7, as provided by the `python-36` packages. - -On CentOS/RHEL distributions, LiveCompare also requires the EPEL repository. -More details can be found in: - - - -Specifically on CentOS/RHEL version 7, the Python component `tqdm` is too old -(< 4.16.0). It is possible to install the latest `tqdm` using `pip` or `pip3` -for the user that is running LiveCompare: - -``` -pip install --user tqdm --upgrade -``` - -## LiveCompare with TPAexec - -The following sample config for `TPAexec` can be used to build a server with -`LiveCompare` and `PostgreSQL 11`: - -```yaml ---- -architecture: M1 -cluster_name: livecompare_m1 -cluster_tags: {} - -cluster_vars: - postgres_coredump_filter: '0xff' - postgres_version: '13' - postgresql_flavour: postgresql - repmgr_failover: manual - tpa_2q_repositories: - - products/livecompare/release - packages: - common: - - 2ndq-livecompare - use_volatile_subscriptions: true - -locations: -- Name: main - -instance_defaults: - image: tpa/redhat - platform: docker - vars: - ansible_user: root - -instances: -- Name: livem1node1 - location: main - node: 1 - role: primary - published_ports: - - 5401:5432 -- Name: livem1node2 - location: main - node: 2 - role: replica - upstream: livem1node1 - published_ports: - - 5402:5432 - -``` - -More details about TPAexec can be found in: - - diff --git a/product_docs/docs/livecompare/1/settings.mdx b/product_docs/docs/livecompare/1/settings.mdx deleted file mode 100644 index e9430551c18..00000000000 --- a/product_docs/docs/livecompare/1/settings.mdx +++ /dev/null @@ -1,689 +0,0 @@ ---- -title: Settings -originalFilePath: settings.md - ---- - -## General Settings - -- `logical_replication_mode`: Affects how the program interprets connections and - table filter settings (see more details below), and also what requirements to - check for in the connections before starting the comparison. Currently the - possible values are: - - ``` - - `off`: Assumes there is no logical replication between the databases; - - - `native`: Assumes there is native logical replication between the - databases. Enables the usage of the `Table Filter -> publications` - setting to specify the list of tables to be used. Requires PostgreSQL 10+ on - all databases. - - - `pglogical`: Assumes there is pglogical replication between the databases. - Enables the usage of the `Table Filter -> replication_sets` setting to - specify the list of tables to be used. Also enables the usage of `node_name` - to specify the data connections, which require setting the `Initial - Connection` that is used to retrieve DSN information of the nodes. Requires - the `pglogical` extensions to be installed on all databases. - - - `bdr`: Assumes all data connections are nodes from the same BDR cluster. - Enables usage of `Table Filter -> replication_sets` setting to specify list - of tables to be used. Also enables usage of `node_name` to - specify the data connections, which require setting `Initial Connection` - that is used to retrieve DSN information of the nodes. Requires `pglogical` - and `bdr` extensions installed on all databases. - ``` - -- `all_bdr_nodes`: If `logical_replication_mode` is set to `bdr`, then it is - possible to specify only the Initial Connection (see below) and let LiveCompare - build the connection list based on the current list of active BDR nodes. - Default: `off`. - -- `max_parallel_workers`: Number of parallel processes to be considered. Each - process will work on a table from the queue. Default: `2`. - -**Important**: Each process will keep N+1 open connections: 1 to each data -connection and another 1 to the output database. - -- `buffer_size`: Number of rows to be retrieved from the tables on every data - fetch operation. Default: `4096`. - -- `log_level`: Verbosity level in the log file. Possible values: `debug`, - `info`, `warning` or `error`. Default: `info`. - -- `data_fetch_mode`: Affects how LiveCompare fetches data from the database. - - - `prepared_statements`: Uses prepared statements (a query with `LIMIT`) for - data fetch. Only a very small amount of data (`buffer_size = 4096` rows by - default) is fetched each time, so it has the smallest impact of all 3 modes, - and by the same reason it's the safer fetch mode. Allows asynchronous data - fetch (defined by `parallel_data_fetch`). For the general use case, this - fetch method provides a good performance, but a performance decrease can be - felt for large tables. This is the default and strongly recommended when - server load is medium-high. - - - `server_side_cursors_with_hold`: Uses server-side cursors `WITH HOLD` for - data fetch. As table data is retrieved in a single transaction, it holds - back `xmin` and can cause bloat and replication issues, and also prevent - `VACUUM` from running well. Also, the `WITH HOLD` clause tells Postgres to - materialize the query (workers may hang for a few seconds waiting for the - data to be materialized), so the whole table data consumes RAM and can be - stored on Postgres side disk as temporary files. All that impact can be - decreased by using `parallel_chunk_rows` (set to 10000000 by default), and - speed can be improved by increasing `buffer_size` a little. Allows - asynchronous data fetch (defined by `parallel_data_fetch`). For the general - use case, this fetch method doesn't provide any benefits when compared to - `prepared_stataments`, but for multiple small tables it's faster. However, - this mode is recommended only when load is very low, for example on tests - and migration scenarios. - - - `server_side_cursors_without_hold`: Uses server-side cursors - `WITHOUT HOLD` for data fetch. As `server_side_cursors_with_hold`, this - mode can also hold back `xmin`, thus potentially can cause bloat, `VACUUM` - and replication issues on Postgres, but such impact is higher because - `WITHOUT HOLD` cursors require an open transaction for the whole comparison - session (this will be lifted in further versions). As the snapshot is held - for the whole comparison session, comparison results might be helpful - depending on your use case. As the query is not materialized, memory usage - and temp file generation remains low. Asynchronous data fetch is not - allowed. In terms of performance, this mode is slower for the general use - case, but for large tables it can be the faster. It's recommended when load - on the database is low-medium. - -**Important**: the choice of the right `data_fetch_mode` for the right scenario -is very important. Using prepared statements has the smallest footprint on the -database server, so it's the safest approach, and it's good for the general use -case. Another point is that prepared statements allow LiveCompare to always see -the latest version of the rows, which may not happen when using server-side -cursors on a busy database. So it's recommended to use `prepared_statements` for -production, high load servers; and either `server_side_cursors_*` settings for -testing, migration scenarios, and low load servers. The best strategy would -probably mix `server_side_cursors_without_hold` for very large tables, and -`prepared_statements` for the remaining tables. Refer to the table below for -a comparison on the cost/benefit ratio: - -| | prepared_statements | server_side_cursors_with_hold | server_side_cursors_without_hold | -| ------------------ | :-----------------: | :---------------------------: | :------------------------------: | -| xmin hold | very low | medium | high | -| xmin released per | buffer | chunk | whole comparison session | -| temp files | very low | very high | low | -| memory | very low | high | low | -| allows async conns | yes | yes | no | -| fastest for | general | small tables | large tables | -| recommended load | high | very low | low-medium | - -**Note about Oracle**: For Oracle, the `data_fetch_mode` setting is completely -ignored, and data will always be fetch from Oracle using direct queries with -`LIMIT`, without using prepared statements or cursors. - -- `parallel_chunk_rows`: Minimum number of rows required to consider splitting a - table into multiple chunks for parallel comparison. A hash is used to fetch - data, so workers don't clash with each other. Each table chunk will have no more - than `parallel_chunk_rows` rows. Setting it to any value < 1 disables table - splitting. If any connections are not PostgreSQL, then table splitting is - disabled automatically by LiveCompare. Default: 10000000. - -- `parallel_data_fetch`: If data fetch should be performed in parallel (i.e., - using async connections to the databases). Improves performance of multi-way - comparison. If any data connections are not PostgreSQL, then this setting is - automatically disabled. It's only allowed when - `data_fetch_mode = prepared_statements` or - `data_fetch_mode = server_side_cursors_with_hold`. - Default: `on`. - -- `comparison_algorithm`: Affects how LiveCompare works through table rows to - compare data. Using hashes is faster than full row comparison. It can assume one - of the following values: - - ``` - - `full_row`: Disables row comparison using hashes. Full comparison, in this - case, is performed by comparing the row column by column. - - - `row_hash`: Enables row comparison using hashes and enables table - splitting. Tables are split so each worker compares a maximum of - `parallel_chunk_rows` per table. Data row is hashed in PostgreSQL, so the - comparison is faster than `full_row`. However, if for a specific row the - hash does not match, then for that specific row, LiveCompare will fallback - to `full_row` algorithm (i.e., compare row by row). If any data connections - is not PostgreSQL, then LiveCompare uses a row hash that's defined as the MD5 - hash of the concatenated column values of the row, being considered a - "common hash" among the database technologies being compared. - - - `block_hash`: Works the same as `row_hash`, but instead of comparing row - by row, LiveCompare builds a "block hash", i.e., a hash of the hashes of all - rows in the data buffer that was just fetched (maximum of `buffer_size` - rows). Conceptually it works like a 2-level Merkle Tree. If the block hash - matches, then LiveCompare advances the whole block (this is why this - comparison algorithm is faster than `row_hash`). If block hash does not - match, then LiveCompare falls back to `row_hash` and performs comparison row - by row in the buffer to find the divergent rows. This is the default value. - ``` - -- `min_time_between_round_saves`: Time in seconds to wait before updating each - round state when the comparison algorithm is in progress. Note that when the - round finishes, LiveCompare always updates the round state for that table. - Default: 60 seconds. - -**Important**: If the user cancels execution of LiveCompare by hitting `Ctrl-c` -and starts it again, then LiveCompare will resume the round for that table, -starting from the point where the round state was saved. - -- `stop_after_time`: Time in seconds after which LiveCompare will automatically - stop itself as if the user had hit `Ctrl-c`. The comparison session that was - interrupted, if not finished yet, can be resumed again by passing the session - ID as argument in the command line. Default is `stop_after_time = 0`, which - means that automatic interruption is disabled. - -- `consensus_mode`: Consensus algorithm used by LiveCompare to determine which - data connections are divergent. Possible values are `simple_majority`, - `quorum_based` or `source_of_truth`. If `consensus_mode = source_of_truth` then - `difference_sources_of_truth` must be filled. Default is `simple_majority`. - -- `difference_required_quorum`: If `consensus_mode = quorum_based`, then this - setting specified the minimum quorum is required to decide which connections are - divergent. Should be a number between 0.0 and 1.0 (0.0 means no connection is - required while 1.0 means all connections are required, both cases are extreme - and should not be used). The default value is 0.5, and we recommend using a - value close to that. - -- `difference_sources_of_truth`: Comma-separated list of connections names (or - node names, if `logical_replication_mode = bdr` and `all_bdr_nodes = on`) that - should be considered as source of truth. It is only used when `consensus_mode = - source_of_truth`. For example: `difference_sources_of_truth = node1,node2`. In - this example, either the sections `node1 Connection` and `node2 Connection` - should be defined in the .ini file or `all_bdr_nodes = on` and only the `Initial - Connection` is defined, while `node1` and `node2` should be valid BDR node - names. - -- `difference_tie_breakers`: Comma-separated list of connection names (or node - names, if `logical_replication_mode = bdr` and `all_bdr_nodes = on`) that should - be considered as tie breakers whenever the consensus algorithm finds a tie - situation. For example: `difference_tie_breakers = node1,node2`. In this - example, either the sections `node1 Connection` and `node2 Connections` should - be defined in the .ini file or `all_bdr_nodes = on` and only the `Initial - Connection` is defined, while `node1` and `node2` should be valid BDR node - names. Default is to not consider any connection as tie breaker. - -- `difference_statements`: Controls what kind of DML statements will be - generated by LiveCompare. The value of `difference_statements` can - be one of: - - ``` - - `all` (default) - - `inserts` - - `updates` - - `deletes` - - `inserts_updates` - - `inserts_deletes` - - `updates_deletes` - ``` - -- `difference_allow_null_updates`: Determines whether commands like `UPDATE SET - col = NULL` will be allowed in the difference report. Default: - `on`. - -- `difference_statement_order`: Controls order of DML statements that will be - generated by LiveCompare. The value of `difference_statement_order` - can be one of: - - ``` - - `delete_insert_update` - - `delete_update_insert` (default) - - `insert_update_delete` - - `insert_delete_update` - - `update_insert_delete` - - `update_delete_insert` - ``` - -- `difference_fix_replication_origin`: When working with BDR databases, for - difference LiveCompare will create a specific replication origin if it doesn't - exist yet, then use the replication origin to create apply script with DML - fixes. The setting `difference_fix_replication_origin` specifies the name of - the replication origin used by LiveCompare. If the user doesn't set any value - for this setting, then LiveCompare will automatically set - `difference_fix_replication_origin = bdr_local_only_origin`. Note that the - replication origin that LiveCompare creates is not dropped to allow verification - after the comparison, but if needed the replication origin can be manually - dropped later. Requires `logical_replication_mode = bdr`. - -**IMPORTANT**: Please note that BDR 3.6.18 introduced the new pre-created -`bdr_local_only_origin` replication origin to be used for applying local-only -transactions. So if LiveCompare is connected to BDR 3.6.18, it won't create this -replication origin, and it is recommended that the user should not try to drop -this replication origin. - -- `difference_fix_start_query`: Arbitrary query that is executed at the - beginning of the apply script generated by LiveCompare. Additionally if a BDR comparison - is being performed and the `difference_fix_start_query` is empty, then - LiveCompare also automatically does the following: - - ``` - - If the divergent connection is BDR 3.6.7, add - `SET LOCAL bdr.xact_replication = off;` - - Add commands that setup transaction to use the replication origin - specified in `difference_fix_replication_origin`. - ``` - -- `show_progress_bars`: Determines whether or not progress bars should be shown - in the console output. Disabling this setting might be useful for batch - executions. Default: `on`. - -- `output_schema`: In the output connection, the schema where the comparison - report tables will be created. Default: `livecompare`. - -- `hash_column_name`: Every data fetch will contain a specific column which is - the hash of all actual columns in the row. This setting specifies the name of - this column. Default: `livecompare_hash`. - -- `rownumber_column_name`: Some fetches need to use the `row_number()` function - value inside a query column. This setting specifies the name of this column. - Default: `livecompare_rownumber`. - -- `fetch_row_origin`: When this setting is enabled, LiveCompare fetches the - origin name for each divergent row, which might be useful for debugging - purposes. Default: `off`. To be enabled, requires `logical_replication_mode` set - to `pglogical` or `bdr`. - -- `column_intersection`: When this setting is enabled, for a given table that is - being compared, LiveCompare will only work on the intersection of columns from - the table on all connections, ignoring extra columns that might exist on any of - the connections. When this setting is disabled, LiveCompare will check if - columns are equivalent on the table on all connections, and abort the comparison - of the table if there are any column mismatches. Default: `off`. - -**Important**: If table has PK, then the PK columns are not allowed to be -different, even if `column_intersection = on`. - -- `oracle_ignore_unsortable`: When enabled, tells LiveCompare to ignore columns - with Oracle unsortable data types (BLOB, CLOB, NCLOB, BFILE) if column is not - part of the table PK. If enabling this setting, it is recommended to also enable - `column_intersection`. - -- `oracle_user_tables_only`: When enabled, tells LiveCompare to fetch table - metadata only from the Oracle logged in user, which is faster because it reads, - for example, from `sys.user_tables` and `sys.user_tab_columns` instead of - `sys.all_tables` and `sys.all_tab_columns`. Default: `off`. - -- `oracle_fetch_fk_metadata`: When enabled, tells LiveCompare to fetch foreign - key metadata, which can be a slow operation. Overrides the value of the setting - `fetch_fk_metadata` on the Oracle connection. Default: `off`. - -- `schema_qualified_table_names`: Table names are treated as schema-qualified - when this setting is enabled. Disabling it allows comparison of tables without - using schema-qualified table names: on Oracle x Postgres comparisons, it - requires also enabling `oracle_user_tables_only`, while on Postgres x Postgres, - it allows for comparisons of tables that are under different schemas, even in - the same database. Also, when `schema_qualified_table_names` is enabled, - `Table Filter -> tables`, `Row Filter` and `Column Filter` allow table name - without the schema name. Default: `on`. - -- `force_collate`: When set to a value other than `off` and to a valid collation - name, forces the specified collation name in `ORDER BY` operations in all - Postgres databases being compared. Useful when comparing Postgres databases with - different collation or when comparing Oracle versus Postgres databases (in this - case users should set `force_collate = C`). Will assume value `C` if comparing - mixed technologies (like Oracle vs PostgreSQL) and no collation is specified. - Default: `off`. - -- `work_directory`: path to the `LiveCompare` working directory. The session - folder containing output files will be created in such directory. Default: - `.` (current directory). - -- `abort_on_setup_error`: when enabled, if LiveCompare hits any error when - trying to setup a table comparison round, the whole comparison session is - aborted. Default: `off`. - -**Important**: Setting `abort_on_setup_error` is only considered during -`compare` mode. In `recheck` mode, LiveCompare always aborts at the first error -in setup. - -- `custom_dollar_quoting_delimiter`: when LiveCompare finds differences, it will - output the DML using dollar quoting on strings. The default behavior is create - a random string to compose it. If you want by any means use a custom one, you - can set this parameter as the delimiter to be used. You just need to set the - constant, not the `$` symbols around the constant. Defaults to `off`, which - means LiveCompare will use a `md5` hash of the word `LiveCompare`. - -- `session_replication_role_replica`: when enabled LiveCompare will use - `session_replication_role` PostgreSQL setting as `replica` in the output apply - scripts. That's useful if you want to prevent firing triggers and rules while - applying DML in the nodes with divergences. Enabling it requires a PostgreSQL - super user, otherwise will take no effect. Defaults to `off`. - -- `split_updates`: when enabled LiveCompare will split `UPDATE` divergences, - i.e., instead of generating a `UPDATE` DML, it will generate corresponding - `DELETE` and `INSERT` in the apply script. Defaults to `off`. - -- `float_point_round`: an integer to specify decimal digits that LiveCompare - should round when comparing float point values coming from the database. Default - is -1, which disables float point rounding. - -## Initial Connection - -The initial connection is used only when `logical_replication_mode` is set to -`pglogical` or `bdr`, and is used only when the program starts, to fetch DSN -from node names, if the user has set data connections using only the `node_name` -setting. - -- `technology`: RDBMS technology. Currently the only possible value is - `postgresql`. -- `dsn`: PostgreSQL connection string. If `dsn` is set, then `host`, `port`, - `dbname` and `user` are ignored. The `dsn` setting can also have all other - [parameter key words allowed by libpq](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS). -- `host`: Server address. Leave empty to use the Unix socket connection. -- `port`: Port. Default: `5432`. -- `dbname`: Database name. Default: `postgres`. -- `user`: Database user. Default: `postgres`. -- `application_name`. Application name. Can be used even if the user set `dsn` - instead of all other connection information. Default: `livecompare_initial`. - -## Output Connection - -The output connection specifies where LiveCompare will create the comparison -report tables. - -- `technology`: RDBMS technology. Currently the only possible value is - `postgresql`. -- `dsn`: PostgreSQL connection string. If `dsn` is set, then `host`, `port`, - `dbname` and `user` are ignored. The `dsn` setting can also have all other - [parameter key words allowed by libpq](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS). -- `host`: Server address. Leave empty to use the Unix socket connection. -- `port`: Port. Default: `5432`. -- `dbname`: Database name. Default: `postgres`. -- `user`: Database user. Default: `postgres`. -- `application_name`. Application name. Can be used even if the user set `dsn` - instead of all other connection information. Default: `livecompare_output`. - -## Data Connection - -A "data connection" is a connection section similar to the `Initial Connection` -and the `Output Connection`, but LiveCompare effectively fetches and compares -data on the data connections. - -Similarly to the `Initial Connection` and `Output Connection`, a "data -connection" is defined in a named section. The section name should be of the -form `Name Connection`, being `Name` any single-worded string starting with an -alphabetic character. In this case, whatever the user fills in `Name` is called -the "Connection ID" of the data connection. It is also required that each data -connection has an unique Connection ID in the whole list of data connections. - -If `logical_replication_mode = bdr` and `all_bdr_nodes = on`, then the user is -not required to specify any data connection, because LiveCompare will build the -data connection list by fetching BDR metadata from the `Initial Connection`. - -- `technology`: RDBMS technology. Currently possible values are `postgresql` or - `oracle`. -- `node_name`: Name of the node in the cluster. Requires - `logical_replication_mode` set to `pglogical` or `bdr`, and also requires that - the `Initial Connection` is filled. If `node_name` is set, then `dsn`, `host` - `port`, `dbname` and `user` settings are all ignored. -- `dsn`: PostgreSQL connection string. If `dsn` is set, then `host`, `port`, - `dbname` and `user` are ignored. The `dsn` setting can also have all other - [parameter key words allowed by libpq](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS). -- `host`: Server address. Leave empty to use the Unix socket connection. -- `port`: Port. Default: `5432`. -- `dbname`: Database name. Default: `postgres`. -- `service`: Service name, used in Oracle connections. Default: `XE`. -- `user`: Database user. Default: `postgres`. -- `password`: Plain text password. We recommend not to use this, but in some - legacy connections it might be required. -- `application_name`. Application name. Can be used even if the user set `dsn` - or `node_name` instead of all other connection information. Default: - `livecompare_`. -- `start_query`: Arbitrary query that is executed each time a connection to a - database is open. -- `fetch_fk_metadata`: If LiveCompare should gather metadata about foreign keys - on the connection. Default: `on`. - -## Table Filter - -If omitted or left empty, this section from the `.ini` file will mean that -LiveCompare should be executed against **all** tables in the **first** database. - -If you want LiveCompare to be executed against a specific set of tables, there -are different ways to specify this: - -- `publications`: You can filter specific publications, and LiveCompare will use - only the tables associated to those publications. The variable - `publication_name` can be used to build the conditional expression, for example: - -```ini -publications = publication_name = 'livepub' -``` - -Requires `logical_replication_mode = native`. - -- `replication_sets`: When using pglogical or BDR, you can filter specific - replication sets, and LiveCompare will work only on the tables associated to - those replication sets. The variable `set_name` can be used to build the - conditional expression, for example: - -```ini -replication_sets = set_name in ('default', 'bdrgroup') -``` - -Requires `logical_replication_mode = pglogical` or -`logical_replication_mode = bdr`. - -- `schemas`: You can filter specific schemas, and LiveCompare will work only on - the tables that belong to those schemas. The variable `schema_name` can be used - to build the conditional expression, for example: - -```ini -schemas = schema_name != 'badschema' -``` - -- `tables`: The variable `table_name` can help you build a conditional - expression to filter only the tables you want LiveCompare to work on, for - example: - -```ini -tables = table_name not like '%%account' -``` - -Please note that, in any conditional expression, the `%` character should be -escaped as `%%`. - -The table name should be schema-qualified, unless `schema_qualified_table_names` -is disabled. For example, it's possible to filter only a specific list of -tables: - -``` -tables = table_name in ('myschema1.mytable1', 'myschema2.mytable2') -``` - -If you have disabled general setting `schema_qualified_table_names`, then you -should also set an appropriate `search_path` for Postgres in the connection -`start_query` setting, for example: - -``` -[General Setting] -... -schema_qualified_table_names = off - -[My Connection] -... -start_query = SET search_path TO myschema1, myschema2 - -[Table Filter] -tables = table_name in ('mytable1', 'mytable2') -``` - -**IMPORTANT**: Please note that if two or more schemas that were set on `search_path` -contains a table if the same name, just the first one found will be considered -in the comparison. - -The `Table Filter` section can have a mix of `publications`, `replication_sets`, -`schemas` and `tables` filters, and LiveCompare will consider the set of tables -that are in the intersection of all filters you specified. For example: - -```ini -[Table Filter] -publications = publication_name = 'livepub' -replication_sets = set_name in ('default', 'bdrgroup') -schemas = schema_name != 'badschema' -tables = table_name not like '%%account' -``` - -Also please note that the table filter is applied in the first database, to -build the table list. If a table exists in the first database and is being -considered in the filter, but does not exist in any other database, then you -will see something like this in the logs, and the comparison for that specific -table will be skipped. - -```text -2019-06-17 11:52:41,403 - ERROR - live_table.py - 55 - GetMetaData - P1: livecompare_second_1: Table public.test does not exist -2019-06-17 11:52:41,410 - ERROR - live_round.py - 201 - Initialize - P1: Table public.test does not exist on second connection. Aborting comparison -``` - -Similarly, if a table exists in any other database but does not exist in the -first database, then it won't be considered in the comparison, even if you -didn't apply any table filter. - -A comparison for a specific table will also be skipped if the table column names -are not exactly the same (unless `column_intersection` is enabled), and in the -same order. An appropriate message will be included in the log file as well. - -Currently LiveCompare does not check if data types nor constraints are the same -on both tables. - -**IMPORTANT**: please note that `conflicts` mode doesn't make use of table filter. - -## Row Filter - -In this section you can apply a row-level filter to any table, so LiveCompare -will work only on the rows that satisfy the row filter. - -You can write a list of tables under this section, one table per line (all -table names should be schema qualified unless `schema_qualified_table_names` is -disabled), for example: - -```ini -[Row Filter] -public.table1 = id = 10 -public.table2 = logdate >= '2000-01-01' -``` - -In this case, for the table `public.table1`, LiveCompare will work only in the -rows that satisfy the clause `id = 10`, while for the table `public.table2`, -only rows that satisfy `logdate >= '2000-01-01` will be considered in the -comparison. - -If you have disabled general setting `schema_qualified_table_names`, then you -should also set an appropriate `search_path` for Postgres in the connection -`start_query` setting, for example: - -``` -[General Setting] -... -schema_qualified_table_names = off - -[My Connection] -... -start_query = SET search_path TO public - -[Row Filter] -table1 = id = 10 -table2 = logdate >= '2000-01-01' -``` - -Any kind of SQL condition (same as you would put in the `WHERE` clause) is -accepted, in the same line, as the table row filter. For example, if you have a -large table and want to compare only a specific number of IDs, it's possible to -create a temporary table with all the IDs. Then you can use an `IN` clause to -emulate a `JOIN`, like this: - -``` -[Row Filter] -public.large_table = id IN (SELECT id2 FROM temp_table) -``` - -If a row filter is written incorrectly, then LiveCompare will try to apply the -filter but will fail. So the comparison for this specific table will be skipped, -and an exception will be written to the log file. - -If a table is listed in the `Row Filter` section, but somehow got filtered out -by the `Table Filter`, then the row filter for this table will be silently -ignored. - -**IMPORTANT**: please note that `conflicts` mode doesn't make use of row filter. - -## Column Filter - -In this section you can apply a column-level filter to any table, so LiveCompare -will work only on the columns that are not part of the column filter. - -You can write a list of tables under this section, one table per line (all -table names should be schema qualified unless `schema_qualified_table_names` is -disabled). For example, considering both `public.table1` and `public.table2` have -the columns `column1`, `column2`, `column3`, `column4` and `column5`: - -```ini -[Column Filter] -public.table1 = column1, column3 -public.table2 = column1, column5 -``` - -In this case, for the table `public.table1`, LiveCompare will work only in the -columns `column2`, `column4` and `column5`, filtering out `column1` and `column3`, -while for the table `public.table2`, only the columns `column2`, `column3` and -`column4` will be considered in the comparison, filtering out `column1` and `column5`. - -If you have disabled general setting `schema_qualified_table_names`, then you -should also set an appropriate `search_path` for Postgres in the connection -`start_query` setting, for example: - -``` -[General Setting] -... -schema_qualified_table_names = off - -[My Connection] -... -start_query = SET search_path TO public - -[Column Filter] -table1 = column1, column3 -table2 = column1, column5 -``` - -If absent column names are given in the column filter, that is, column doesn't -exist in the given table, then LiveCompare will log a message about the columns -that could not be found and ignore them, using just the valid ones, if any. - -If a table is listed in the `Column Filter` section, but somehow got filtered -out by the `Table Filter`, then the column filter for this table will be -silently ignored. - -**IMPORTANT**: Please note that if a column specified in a `Column Filter` is -part of the table PK, then it won't be ignored in the comparison. LiveCompare -will log that and ignore the filter of such column. - -**IMPORTANT**: please note that `conflicts` mode doesn't make use of column filter. - -## Conflicts Filter - -In this section you can specify a filter to be used in `--conflicts` mode while -fetching conflicts from BDR nodes. You can build any SQL conditional expression, -and use below fields in the expression: - -- `origin_node`: the upstream node of the subscription -- `target_node`: the downstream node of the subscription -- `local_time`: the timestamp when the conflict occurred in the node -- `conflict_type`: the type of the conflict -- `conflict_resolution`: the resolution which was applied -- `nspname`: schema name of the involved relation -- `relname`: relation name of the involved relation - -You must use `conflicts` attribute under the section. Please find an example below: - -``` -[Conflicts Filter] -conflicts = conflict_type = 'update_missing' AND nspname = 'my_schema' -``` - -By adding above piece of configuration to your INI file, LiveCompare would fetch -just conflicts that are of type `update_missing`, and related to tables under -schema `my_schema` while querying for conflicts in each of the BDR nodes. - -**IMPORTANT**: Please note that this section is exclusive for `--conflicts` mode. diff --git a/product_docs/docs/livecompare/1/supported_technologies.mdx b/product_docs/docs/livecompare/1/supported_technologies.mdx deleted file mode 100644 index 789a627133c..00000000000 --- a/product_docs/docs/livecompare/1/supported_technologies.mdx +++ /dev/null @@ -1,49 +0,0 @@ ---- -navTitle: Supported technologies -title: Supported Technologies -originalFilePath: supported_technologies.md - ---- - -LiveCompare is able to connect to and compare data from a list of technologies -including PostgreSQL, BDR and Oracle. - -In LiveCompare there are 3 kinds of connections: - -- **Initial** (optional): Used to fetch metadata about pglogical or BDR - connections. Required if data connections are pglogical or BDR, and if - `replication_sets` or `node_name` settings are used. Requires - `logical_replication_mode = pglogical` or `logical_replication_mode = bdr`. It - is required to be a pglogical- or BDR-enabled database. -- **Data**: The actual database connection that the tool will connect to perform - data comparison. The first connection in the list is used to solve `Table - Filter` and `Row Filter`, and is also used in conjunction with the Initial - connection to gather information about BDR nodes. If - `logical_replication_mode = bdr` and `all_bdr_nodes = on`, then LiveCompare will - consider all BDR nodes that are part of the same BDR cluster as the `Initial - Connection`. In this case it is not necessary to define Data connections - individually. The fix can be potentially applied in all Data connections, as comparison - and consensus decisions work per row. -- **Output** (mandatory): Where LiveCompare will create a schema called - `livecompare`, some tables and views. This is required to keep progress and - reporting data about comparison sessions. It is required to be a PostgreSQL or - 2ndQPostgres connection. - -Below you can find about versions and details about supported technologies and -in which context they can be used in LiveCompare. - -| Technology | Versions | Connections | -| ------------------------------ | --------------------------- | ------------------------ | -| PostgreSQL | 10, 11, 12 and 13 | Data and/or Output | -| EDB PostgreSQL Extended | 10, 11, 12 and 13 | Data and/or Output | -| EDB PostgreSQL Advanced (EPAS) | 11, 12 and 13 | Data and/or Output | -| pglogical | 2 and 3 | Initial and/or Data | -| BDR | 1, 2 and 3 | Initial and/or Data | -| Oracle | 10g, 11g, 12c and 18c | A single Data connection | - -## PgBouncer Support - -LiveCompare can be used against nodes through PgBouncer, but only if using -`pool_mode=session` because LiveCompare uses prepared statements on PostgreSQL, -and it would not be possible if `pool_mode` were either `transaction` or -`statement`. diff --git a/src/constants/products.js b/src/constants/products.js index b7d5115462e..a37c7e3a7cc 100644 --- a/src/constants/products.js +++ b/src/constants/products.js @@ -34,7 +34,7 @@ export const products = { edb_sqlpatch: { name: "EDB SQL Patch", iconName: IconNames.TOOLS }, language_pack: { name: "Language Pack", iconName: IconNames.TOOLS }, lasso: { name: "Lasso" }, - livecompare: { name: "LiveCompare" }, + livecompare: { name: "LiveCompare", iconName: IconNames.INTEGRATION }, "Migration Handbook": { name: "Migration Handbook" }, migration_portal: { name: "Migration Portal", diff --git a/src/constants/updates.js b/src/constants/updates.js index ad517bb77c1..a9ea842f910 100644 --- a/src/constants/updates.js +++ b/src/constants/updates.js @@ -1,6 +1,14 @@ import IconNames from "../components/icon/iconNames"; export const updates = [ + { + title: "LiveCompare 3.0", + icon: IconNames.INTEGRATION, + description: + "LiveCompare 3.0 is now available with improved performance, easier to configure Oracle support, and enhanced documentation. ", + url: "/livecompare/latest/", + moreUrl: "/livecompare/latest/rel_notes/3.0_rel_notes/", + }, { title: "EDB Postgres Enterprise Manager 9.7", icon: IconNames.EDB_PEM, From 9cf920c645bfd68d0b4d6644d7ed1e80e93b21e4 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Thu, 22 Aug 2024 13:04:09 -0400 Subject: [PATCH 10/16] Update backup.mdx per DJ's correction --- product_docs/docs/pgd/5/backup.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/backup.mdx b/product_docs/docs/pgd/5/backup.mdx index 6754a9716f4..01f1ebbe6bc 100644 --- a/product_docs/docs/pgd/5/backup.mdx +++ b/product_docs/docs/pgd/5/backup.mdx @@ -127,7 +127,7 @@ arrived on the node. As a result, the recovery might be considered to be partially inconsistent, or at least consistent for only one replication origin. -To request this, use the standard syntax: +With PostgreSQL PITR, you can use the standard syntax: ``` recovery_target_time = T1 From b499ee13145922c26cdd544bcfb93dfefb7a3663 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Thu, 22 Aug 2024 13:05:27 -0400 Subject: [PATCH 11/16] Update backup.mdx with DJ's comments --- product_docs/docs/pgd/5/backup.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/backup.mdx b/product_docs/docs/pgd/5/backup.mdx index 01f1ebbe6bc..6d5fdc5c381 100644 --- a/product_docs/docs/pgd/5/backup.mdx +++ b/product_docs/docs/pgd/5/backup.mdx @@ -282,4 +282,5 @@ you can add a `WHERE slot_name LIKE 'bdr%'` clause, but this is rarely useful. !!! Warning - Never run this on a live PGD node. + Never use these commands to drop replication slots on a live PGD node + From bb613d5036695f9bbbfb230a0a5a5892a0bdafd4 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Thu, 22 Aug 2024 13:07:17 -0400 Subject: [PATCH 12/16] Update parallelapply.mdx with DJ's comments --- product_docs/docs/pgd/5/parallelapply.mdx | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/product_docs/docs/pgd/5/parallelapply.mdx b/product_docs/docs/pgd/5/parallelapply.mdx index fe4289cc90c..726f0b79f61 100644 --- a/product_docs/docs/pgd/5/parallelapply.mdx +++ b/product_docs/docs/pgd/5/parallelapply.mdx @@ -26,9 +26,8 @@ bdr.writers_per_subscription = 2 This configuration gives each subscription two writers. However, in some circumstances, the system might allocate up to eight writers for a subscription. -You can only change -[`bdr.max_writers_per_subscription`](/pgd/latest/reference/pgd-settings#bdrmax_writers_per_subscription) -with a server restart. +Changing [`bdr.max_writers_per_subscription`](/pgd/latest/reference/pgd-settings#bdrmax_writers_per_subscription) +requires a server restart to take effect. You can change [`bdr.writers_per_subscription`](/pgd/latest/reference/pgd-settings#bdrwriters_per_subscription) From f2a0f229c3063fcbdf9ba455b56b5cd976980ed5 Mon Sep 17 00:00:00 2001 From: Jordan Counts <4984714+jcounts@users.noreply.github.com> Date: Thu, 22 Aug 2024 15:01:08 -0500 Subject: [PATCH 13/16] Update supported_technologies.mdx formatting fix --- product_docs/docs/livecompare/3/supported_technologies.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/livecompare/3/supported_technologies.mdx b/product_docs/docs/livecompare/3/supported_technologies.mdx index 435714385e2..76504e90382 100644 --- a/product_docs/docs/livecompare/3/supported_technologies.mdx +++ b/product_docs/docs/livecompare/3/supported_technologies.mdx @@ -18,7 +18,7 @@ LiveCompare has three kinds of connections: - **Data**: The actual database connection that the tool connects to to perform data comparison. The first connection in the list is used to solve `Table Filter` and `Row Filter`, and is also used with the `Initial - Connection' to gather information about PGD nodes. If + Connection` to gather information about PGD nodes. If `logical_replication_mode = bdr` and `all_bdr_nodes = on`, then LiveCompare considers all PGD nodes that are part of the same PGD cluster as the `Initial Connection`. In this case, you don't need to define data connections From 71c6cc7f2b23b6fcd9e6b5c64378ebdcff491894 Mon Sep 17 00:00:00 2001 From: Jordan Counts <4984714+jcounts@users.noreply.github.com> Date: Thu, 22 Aug 2024 15:07:08 -0500 Subject: [PATCH 14/16] Update oracle_support.mdx - removes Requirements section No additional software is needed anymore (no need to install oracle drivers and oracle python package is now included in all installs) --- product_docs/docs/livecompare/3/oracle_support.mdx | 4 ---- 1 file changed, 4 deletions(-) diff --git a/product_docs/docs/livecompare/3/oracle_support.mdx b/product_docs/docs/livecompare/3/oracle_support.mdx index 39784a64e74..ecaab121097 100644 --- a/product_docs/docs/livecompare/3/oracle_support.mdx +++ b/product_docs/docs/livecompare/3/oracle_support.mdx @@ -151,10 +151,6 @@ dsn = port=5432 dbname=liveoutput user=postgres replication_sets = set_name = 'bdrgroup' ``` -## Requirements - -LiveCompare works on PostgreSQL databases out-of-the-box. You don't need to install any additional software. But to be able to connect to Oracle, LiveCompare does requires additional software. - ## Differences If LiveCompare finds any difference, it generates a DML script to apply only on the PostgreSQL connections. No DML script to apply on the Oracle connection is generated. From 5fd6f33312e74fb35b24f812267eebdb0f5ebb31 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Fri, 23 Aug 2024 12:37:35 +0100 Subject: [PATCH 15/16] Rename directory and add readme for clarity Signed-off-by: Dj Walker-Morgan --- docs-for-docs/README.md | 6 ++++++ {docs => docs-for-docs}/agreements/branch-and-release.md | 0 .../agreements/doc-release-tag-format.md | 0 {docs => docs-for-docs}/agreements/product-version-fork.md | 0 .../agreements/release-notes-guidelines.md | 0 {docs => docs-for-docs}/agreements/screenshot_policy.md | 0 {docs => docs-for-docs}/how-tos/avoid-breaking-links.md | 0 {docs => docs-for-docs}/how-tos/markdown-tips.md | 0 {docs => docs-for-docs}/how-tos/sync-cnp-docs.md | 0 .../overviews/legacy-redirects/README.md | 0 {docs => docs-for-docs}/overviews/search/README.md | 0 11 files changed, 6 insertions(+) create mode 100644 docs-for-docs/README.md rename {docs => docs-for-docs}/agreements/branch-and-release.md (100%) rename {docs => docs-for-docs}/agreements/doc-release-tag-format.md (100%) rename {docs => docs-for-docs}/agreements/product-version-fork.md (100%) rename {docs => docs-for-docs}/agreements/release-notes-guidelines.md (100%) rename {docs => docs-for-docs}/agreements/screenshot_policy.md (100%) rename {docs => docs-for-docs}/how-tos/avoid-breaking-links.md (100%) rename {docs => docs-for-docs}/how-tos/markdown-tips.md (100%) rename {docs => docs-for-docs}/how-tos/sync-cnp-docs.md (100%) rename {docs => docs-for-docs}/overviews/legacy-redirects/README.md (100%) rename {docs => docs-for-docs}/overviews/search/README.md (100%) diff --git a/docs-for-docs/README.md b/docs-for-docs/README.md new file mode 100644 index 00000000000..330fc7cb736 --- /dev/null +++ b/docs-for-docs/README.md @@ -0,0 +1,6 @@ +# Docs for Docs + +This is the documentation for the documenters who need to know how to write documentation for EDB products. + +The Style Guide is a good place to start, but thats not here. That is currently in the advocacy_docs directory under community/contributing; this makes [the style guide available online](https://www.enterprisedb.com/docs/community/contributing/) along with the current guide on working with the repo and style guides for release notes. + diff --git a/docs/agreements/branch-and-release.md b/docs-for-docs/agreements/branch-and-release.md similarity index 100% rename from docs/agreements/branch-and-release.md rename to docs-for-docs/agreements/branch-and-release.md diff --git a/docs/agreements/doc-release-tag-format.md b/docs-for-docs/agreements/doc-release-tag-format.md similarity index 100% rename from docs/agreements/doc-release-tag-format.md rename to docs-for-docs/agreements/doc-release-tag-format.md diff --git a/docs/agreements/product-version-fork.md b/docs-for-docs/agreements/product-version-fork.md similarity index 100% rename from docs/agreements/product-version-fork.md rename to docs-for-docs/agreements/product-version-fork.md diff --git a/docs/agreements/release-notes-guidelines.md b/docs-for-docs/agreements/release-notes-guidelines.md similarity index 100% rename from docs/agreements/release-notes-guidelines.md rename to docs-for-docs/agreements/release-notes-guidelines.md diff --git a/docs/agreements/screenshot_policy.md b/docs-for-docs/agreements/screenshot_policy.md similarity index 100% rename from docs/agreements/screenshot_policy.md rename to docs-for-docs/agreements/screenshot_policy.md diff --git a/docs/how-tos/avoid-breaking-links.md b/docs-for-docs/how-tos/avoid-breaking-links.md similarity index 100% rename from docs/how-tos/avoid-breaking-links.md rename to docs-for-docs/how-tos/avoid-breaking-links.md diff --git a/docs/how-tos/markdown-tips.md b/docs-for-docs/how-tos/markdown-tips.md similarity index 100% rename from docs/how-tos/markdown-tips.md rename to docs-for-docs/how-tos/markdown-tips.md diff --git a/docs/how-tos/sync-cnp-docs.md b/docs-for-docs/how-tos/sync-cnp-docs.md similarity index 100% rename from docs/how-tos/sync-cnp-docs.md rename to docs-for-docs/how-tos/sync-cnp-docs.md diff --git a/docs/overviews/legacy-redirects/README.md b/docs-for-docs/overviews/legacy-redirects/README.md similarity index 100% rename from docs/overviews/legacy-redirects/README.md rename to docs-for-docs/overviews/legacy-redirects/README.md diff --git a/docs/overviews/search/README.md b/docs-for-docs/overviews/search/README.md similarity index 100% rename from docs/overviews/search/README.md rename to docs-for-docs/overviews/search/README.md From ea89c53a5aa2893a75389ba5f0aad1253b13898f Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Fri, 23 Aug 2024 14:02:35 +0100 Subject: [PATCH 16/16] Remove Jira tickets from Release Notes --- .../livecompare/3/rel_notes/3.0_rel_notes.mdx | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/product_docs/docs/livecompare/3/rel_notes/3.0_rel_notes.mdx b/product_docs/docs/livecompare/3/rel_notes/3.0_rel_notes.mdx index 7e6606fb08e..44e6d62ba80 100644 --- a/product_docs/docs/livecompare/3/rel_notes/3.0_rel_notes.mdx +++ b/product_docs/docs/livecompare/3/rel_notes/3.0_rel_notes.mdx @@ -11,17 +11,17 @@ LiveCompare 3.0 includes the following new features, enhancements, bug fixes, an | Type | Description | Addresses | |:----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| -| Platform Support | LiveCompare is now supported for use on ARM64 based Debian 12 platforms. [LIV-283] | | -| Platform Support | LiveCompare now supports the use of Python 3.10. [LIV-56] | | -| Enhancement | LiveCompare tables in the cache database (`Output Connection`) are now `UNLOGGED`, which improves performance of writing to the cache database. [LIV-260] | #103499 / 35636 #103402 / 35637 | -| Enhancement | Added a new `consensus_mode = off` option where, if `comparison_algorithm = block_hash`, won't switch to a more granular comparison algorithm. This significantly improves the performance when divergences are found for a specific table, at the cost of not having the information about which rows are divergent. The user can subsequently run LiveCompare again adding the divergent tables, if any, to the `[Table Filter]` and compare only the divergent tables to get the information about which rows are divergent. [LIV-282] | #103499 / 35636 #103402 / 35637 | +| Platform Support | LiveCompare is now supported for use on ARM64 based Debian 12 platforms. | | +| Platform Support | LiveCompare now supports the use of Python 3.10. | | +| Enhancement | LiveCompare tables in the cache database (`Output Connection`) are now `UNLOGGED`, which improves performance of writing to the cache database. | #103499 / 35636 #103402 / 35637 | +| Enhancement | Added a new `consensus_mode = off` option where, if `comparison_algorithm = block_hash`, won't switch to a more granular comparison algorithm. This significantly improves the performance when divergences are found for a specific table, at the cost of not having the information about which rows are divergent. The user can subsequently run LiveCompare again adding the divergent tables, if any, to the `[Table Filter]` and compare only the divergent tables to get the information about which rows are divergent. | #103499 / 35636 #103402 / 35637 | | Enhancement | Replaced the `cx_Oracle` Python module with the new `oracledb` Python module, which is slightly faster and no longer requires the Oracle Instant Client. | #103402 / 35637 | -| Bug Fix | Fixed an issue where `[Column Filter]` was not being honored when `comparison_algorithm = block_hash` was set. [LIV-290] | #103499 / 35636 | -| Bug Fix | Fixed an issue where the PK sort retrieved from the database wasn't being honored in Python memory, dragging performance down. [LIV-277] | #103499 / 35636 #103402 / 35637 | -| Deprecation | Removed the Table Splitting feature, which could drag performance down and was obsolete since LiveCompare version 2.1. The `parallel_chunk_rows` setting was removed as a byproduct of this update. [LIV-279] | | -| Deprecation | Removed the `force_collate` setting as a byproduct of fixing the PK sorting issue.[LIV-277] | #35656 | +| Bug Fix | Fixed an issue where `[Column Filter]` was not being honored when `comparison_algorithm = block_hash` was set. | #103499 / 35636 | +| Bug Fix | Fixed an issue where the PK sort retrieved from the database wasn't being honored in Python memory, dragging performance down. | #103499 / 35636 #103402 / 35637 | +| Deprecation | Removed the Table Splitting feature, which could drag performance down and was obsolete since LiveCompare version 2.1. The `parallel_chunk_rows` setting was removed as a byproduct of this update. | | +| Deprecation | Removed the `force_collate` setting as a byproduct of fixing the PK sorting issue. | #35656 | !!! Note LiveCompare 3.0 removed support for the `parallel_chunk_rows` and `force_collate` settings. If these settings are included in any existing .ini files, be sure to remove these settings prior to using the .ini files with LiveCompare 3.0. -!!! \ No newline at end of file +!!!