Skip to content

Commit

Permalink
Merge pull request #5965 from EnterpriseDB/docs/pgd_reedit_14
Browse files Browse the repository at this point in the history
Reedit of PGD major rolling upgrades topic
  • Loading branch information
ebgitelman authored Aug 23, 2024
2 parents ec1f3a5 + 0bfa62c commit 827e4f7
Showing 1 changed file with 68 additions and 53 deletions.
121 changes: 68 additions & 53 deletions product_docs/docs/pgd/5/upgrades/upgrading_major_rolling.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ redirects:

## Upgrading Postgres major versions

Upgrading a Postgres database's major version to access improved features, performance enhancements, and security updates is a common administration task. Doing the same for a Postgres Distributed (PGD) cluster deployed with Trusted Postgres Architect (TPA) is essentially the same process but performed as a rolling upgrade.
Upgrading a Postgres database's major version to access improved features, performance enhancements, and security updates is a common administration task. Doing the same for an EDB Postgres Distributed (PGD) cluster deployed with Trusted Postgres Architect (TPA) is essentially the same process but performed as a rolling upgrade.

The rolling upgrade process allows updating individual cluster nodes to a new major Postgres version while maintaining cluster availability and operational continuity. This approach minimizes downtime and ensures data integrity by allowing the rest of the cluster to remain operational as each node is upgraded sequentially.

Expand All @@ -25,7 +25,7 @@ To do this, connect to one of the nodes using SSH and run the `pgd show-nodes` c
sudo -u postgres pgd show-nodes
```

The `pgd show-nodes` command shows you all the nodes in your PGD cluster and the subgroup to which each node belongs. Then you want to find out which node is the write leader in each subgroup by running:
The `pgd show-nodes` command shows you all the nodes in your PGD cluster and the subgroup to which each node belongs. Then you want to find out which node is the write leader in each subgroup:

```bash
sudo -u postgres pgd show-groups
Expand All @@ -40,72 +40,80 @@ You now have enough information to determine your upgrade order, one subgroup at
### Perform the upgrade on each node

!!! Note
To help prevent data loss, ensure that your databases and configuration files are backed up before starting the upgrade process.
To help prevent data loss, before starting the upgrade process, ensure that your databases and configuration files are backed up.
!!!

Using the [preliminary order](#prepare-the-upgrade), perform the following steps on each node while connected via SSH:

* **Confirm the current Postgres version**
* View versions from PGD by running `sudo -u postgres pgd show-version`.
* View versions from PGD:

`sudo -u postgres pgd show-version`.
* Ensure that the expected major version is running.


* **Verify that the target node isn't the write leader**
* Check whether the target node is the write leader for the group you're upgrading using `sudo -u postgres pgd show-groups`.
* If the target node is the current write leader for the group/subgroup you're upgrading, perform a [planned switchover](#perform-a-planned-switchover) to another node.
* `sudo -u postgres pgd switchover --group-name <group_name> --node-name <new_write_leader_node_name>`
* Check whether the target node is the write leader for the group you're upgrading:

`sudo -u postgres pgd show-groups`
* If the target node is the current write leader for the group/subgroup you're upgrading, perform a [planned switchover](#perform-a-planned-switchover) to another node:

`sudo -u postgres pgd switchover --group-name <group_name> --node-name <new_write_leader_node_name>`


* **Stop Postgres on the target node**
* Stop the Postgres service on the current node by running `sudo systemctl stop postgres`.
* The target node is no longer actively participating as a node in the cluster.
* Stop the Postgres service on the current node:

`sudo systemctl stop postgres`

The target node is no longer actively participating as a node in the cluster.


* **Install PGD and utilities**
* Install PGD and its utilities compatible with the Postgres version you're upgrading to.
* `sudo apt install edb-bdr<new_pgd_version_number>-pg<new_postgres_version_number> edb-bdr-utilities`
* Install PGD and its utilities compatible with the Postgres version you're upgrading to:

`sudo apt install edb-bdr<new_pgd_version_number>-pg<new_postgres_version_number> edb-bdr-utilities`

* **Initialize the new Postgres instance**
* Create a directory that will house the database files for the new version of PostgreSQL:
* `sudo mkdir -p /opt/postgres/datanew`
* Ensure that user postgres has ownership permissions (chown) to the directory.
* Initialize a new PostgreSQL database cluster in the directory you just created.
* This step involves using the `initdb` command provided by the newly installed version of PostgreSQL.
* Replace `<path_to_postgres_bin>` with the path to the bin directory of the newly installed PostgreSQL version: `sudo -u postgres <path_to_postgres_bin>/initdb -D /opt/postgres/datanew`.
* You may need to run this command as the postgres user or another user with appropriate permissions.
* Make sure to include the `--data-checksums` flag to ensure the cluster uses data checksums.
* Create a directory to house the database files for the new version of PostgreSQL:

`sudo mkdir -p /opt/postgres/datanew`
* Ensure that the user postgres has ownership permissions to the directory using `chown`.
* Initialize a new PostgreSQL database cluster in the directory you just created. This step involves using the `initdb` command provided by the newly installed version of PostgreSQL. Include the `--data-checksums` flag to ensure the cluster uses data checksums.

`sudo -u postgres <path_to_postgres_bin>/initdb -D /opt/postgres/datanew --data-checksums`

Replace `<path_to_postgres_bin>` with the path to the bin directory of the newly installed PostgreSQL version.

You may need to run this command as the postgres user or another user with appropriate permissions.

* **Migrate configuration to the new Postgres version**
* Locate the following configuration files in your current PostgreSQL data directory:
* `postgresql.conf`
* The main configuration file containing settings related to the database system.
* `postgresql.auto.conf`
* Contains settings set by PostgreSQL, such as those modified by the `ALTER SYSTEM` command.
* `pg_hba.conf`
* Manages client authentication, specifying which users can connect to which databases from which hosts.
* The entire `conf.d` directory (if present)
* Allows for organizing configuration settings into separate files for better manageability.
* `postgresql.conf` &mdash; The main configuration file containing settings related to the database system.
* `postgresql.auto.conf`&mdash; Contains settings set by PostgreSQL, such as those modified by the `ALTER SYSTEM` command.
* `pg_hba.conf` &mdash; Manages client authentication, specifying which users can connect to which databases from which hosts.
* The entire `conf.d` directory (if present) &mdash; Allows for organizing configuration settings into separate files for better manageability.
* Copy these files and the `conf.d` directory to the new data directory you created for the upgraded version of PostgreSQL.


* **Verify the Postgres service is inactive**
* Before proceeding, it's important to ensure that no PostgreSQL processes are active for both the old and the new data directories. This verification step prevents any data corruption or conflicts during the upgrade process.
* Use the `sudo systemctl status postgres` command to verify that Postgres was stopped.
* If it isn't stopped, run `systemctl stop postgres` and verify again that it was stopped.

Use the `sudo systemctl status postgres` command to verify that Postgres was stopped. If it isn't stopped, run `systemctl stop postgres` and verify again that it was stopped.


* **Swap PGDATA directories for version upgrade**
* Rename `/opt/postgres/data` to `/opt/postgres/dataold` and `/opt/postgres/datanew` to `/opt/postgres/data`.
* This step readies your system for the next crucial phase: running pg_upgrade to finalize the PostgreSQL version transition.
* Rename `/opt/postgres/data` to `/opt/postgres/dataold` and `/opt/postgres/datanew` to `/opt/postgres/data`.

This step readies your system for the next crucial phase: running pg_upgrade to finalize the PostgreSQL version transition.


* **Verify upgrade feasibility**
* The `bdr_pg_upgrade` tool offers a `--check` option designed to perform a preliminary scan of your current setup, identifying any potential issues that could hinder the upgrade process.
* You need to run this check from an upgrade directory with ownership given to user `postgres`, such as `/home/upgrade/`, so that the upgrade log files created by `bdr_pg_upgrade` can be stored.
* To initiate the safety check, append the `--check` option to your `bdr_pg_upgrade` command.
* This operation simulates the upgrade process without making any changes, providing insights into any compatibility issues, deprecated features, or configuration adjustments required for a successful upgrade.

You need to run this check from an upgrade directory with ownership given to user postgres, such as `/home/upgrade/`, so that the upgrade log files created by `bdr_pg_upgrade` can be stored. To initiate the safety check, append the `--check` option to your `bdr_pg_upgrade` command.

This operation simulates the upgrade process without making any changes, providing insights into any compatibility issues, deprecated features, or configuration adjustments required for a successful upgrade.
* Address any warnings or errors indicated by this check to ensure an uneventful transition to the new version.


Expand All @@ -117,23 +125,27 @@ Using the [preliminary order](#prepare-the-upgrade), perform the following steps

* **Update the Postgres service configuration**
* Update the service configuration to reflect the new PostgreSQL version by updating the version number in the `postgres.service` file:
* `sudo sed -i -e 's/<old_version_number>/<new_version_number>/g' /etc/systemd/system/postgres.service`

`sudo sed -i -e 's/<old_version_number>/<new_version_number>/g' /etc/systemd/system/postgres.service`
* Refresh the system's service manager to apply these changes:
* `sudo systemctl daemon-reload`

`sudo systemctl daemon-reload`


* **Restart Postgres**
* Proceed to restart the PostgreSQL service.
* `systemctl start postgres`
* Proceed to restart the PostgreSQL service:

`systemctl start postgres`


* **Validate the new Postgres version**
* Verify that your PostgreSQL instance is now upgraded by again running `sudo -u postgres pgd show-version`.
* Verify that your PostgreSQL instance is now upgraded:

`sudo -u postgres pgd show-version`


* **Clean up post-upgrade**
* Run `vacuumdb` with the `ANALYZE` option immediately after the upgrade but before introducing a heavy production load.
* Running this command minimizes the immediate performance impact, preparing the database for more accurate testing.
* Run `vacuumdb` with the `ANALYZE` option immediately after the upgrade but before introducing a heavy production load. Running this command minimizes the immediate performance impact, preparing the database for more accurate testing.
* Remove the old version's data directory, `/opt/postgres/dataold`.

### Reconcile the upgrade with TPA
Expand All @@ -143,18 +155,21 @@ TPA needs to continue to manage the deployment effectively after all the nodes h
Follow these steps to update the configuration and redeploy the PGD cluster through TPA.

* **Update the `config.yml`**
* Change the `config.yml` of the TPA-managed cluster to the new version.
* `cluster_vars: postgres_version: '<new_version_number>'`
* Change the `config.yml` of the TPA-managed cluster to the new version:

`cluster_vars: postgres_version: '<new_version_number>'`


* **Use `tpaexec` to redeploy the PGD cluster with the updated `config.yml`**
* `tpaexec deploy <cluster_name>`
* Use this the `deploy` option:

`tpaexec deploy <cluster_name>`

The worked example that follows shows upgrading the Postgres major version from 15 to 16 on a PGD 5 cluster deployed with TPA in detail.

## Worked example

This worked example starts with a TPA-managed PGD cluster deployed using the [AWS Quickstart](https://www.enterprisedb.com/docs/pgd/latest/quickstart/quick_start_aws/). The cluster has three nodes: kaboom, kaolin, and kaftan, all running Postges 15.
This worked example starts with a TPA-managed PGD cluster deployed using the [AWS quick start](/pgd/latest/quickstart/quick_start_aws/). The cluster has three nodes: kaboom, kaolin, and kaftan, all running Postgres 15.

This example starts with kaboom.

Expand All @@ -164,7 +179,7 @@ Some steps of this process involve running commands as the Postgres owner. We re

### Confirm the current Postgres version

SSH into kaboom to confirm the major version of Postgres is expected by running:
SSH into kaboom to confirm the major version of Postgres is expected:

```bash
sudo -u postgres pgd show-version
Expand Down Expand Up @@ -305,14 +320,14 @@ sudo mv /opt/postgres/datanew /opt/postgres/data
```

!!! Important
If something goes wrong at some point during the procedure, you may want to rollback/revert a node to the older major version. To do this, rename directories again so that the current data directory, `/opt/postgres/data`, becomes `/opt/postgres/datafailed` and the old data directory, `/opt/postgres/dataold`, becomes the current data directory:
If something goes wrong at some point during the procedure, you may want to roll back/revert a node to the older major version. To do this, rename directories again so that the current data directory, `/opt/postgres/data`, becomes `/opt/postgres/datafailed` and the old data directory, `/opt/postgres/dataold`, becomes the current data directory:

```bash
sudo mv /opt/postgres/data /opt/postgres/datafailed
sudo mv /opt/postgres/dataold /opt/postgres/data
```

This rolls back/reverts the node back to the previous major version of Postgres.
This rolls back/reverts the node to the previous major version of Postgres.
!!!

### Verify upgrade feasibility
Expand Down Expand Up @@ -367,7 +382,7 @@ Checking for new cluster tablespace directories ok
```

!!! Note
If you didn't initialize Postgres 16 with checksums using the `--data-checksums` option, but did initialize checksums with your Postgres 15 instance, an error tells you about the incompatibility:
If you didn't initialize Postgres 16 with checksums using the `--data-checksums` option but did initialize checksums with your Postgres 15 instance, an error tells you about the incompatibility:

```bash
old cluster uses data checksums but the new one does not
Expand Down Expand Up @@ -506,7 +521,7 @@ sudo systemctl daemon-reload

### Restart Postgres

Start the modified Postgres service by running:
Start the modified Postgres service:

```bash
sudo systemctl start postgres
Expand Down Expand Up @@ -560,11 +575,11 @@ Upgrading the target node is now complete.

After completing the upgrade on kaboom, run the same steps on kaolin and kaftan.

If you followed along with this example and kaftan is the write leader, to ensure availability, you must [perform a planned switchover](#perform-a-planned-switchover) to another, already upgraded node before running the upgrade steps on kaftan.
If you followed along with this example and kaftan is the write leader, to ensure availability, you must [perform a planned switchover](#perform-a-planned-switchover) to another node that was already upgraded before running the upgrade steps on kaftan.

#### Check Postgres versions across the cluster

After completing the upgrade on all nodes, while connected to one of the nodes, you can once again check your versions:
After completing the upgrade on all nodes, while connected to one of the nodes, you can check your versions again:

```bash
sudo -u postgres pgd show-version
Expand All @@ -585,7 +600,7 @@ This output shows that all the nodes are successfully upgraded to the new Postgr

#### Reconcile with TPA

After all the nodes are upgraded, you still need to [reconcile](https://www.enterprisedb.com/docs/tpa/latest/reference/reconciling-local-changes/) the upgraded version of Postgres with TPA so you can continue to use TPA to manage the cluster in the future.
After all the nodes are upgraded, you still need to [reconcile](/tpa/latest/reference/reconciling-local-changes/) the upgraded version of Postgres with TPA so you can continue to use TPA to manage the cluster in the future.

To do this, return to the command line where your TPA cluster directory resides. In this worked example, the TPA cluster directory is `/home/ubuntu/democluster` on the instance where you originally deployed the cluster using TPA.

Expand Down

1 comment on commit 827e4f7

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.