From 7238268e4947146bc168ef678c9a993273b7a25b Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Tue, 5 Nov 2024 10:55:56 -0800 Subject: [PATCH 01/19] Upgrade rewrites for v24.3 - [DOC-11170] Document version skipping - [DOC-8495] Incorrect details about alpha releases - [DOC-8412] Add details about monitoring upgrade progress - [DOC-11046] Discuss major and minor version upgrades separately - [DOC-11650] v24.3 upgrade instructions - Modularize Cloud and CRDB upgrade pages to use includes - Reduce version-specific details in upgrade pages --- .../upgrade/disable-auto-finalization.md | 20 ++ .../common/upgrade/finalize-cloud.md | 10 + .../common/upgrade/finalize-self-hosted.md | 36 ++ .../upgrade/major-version-upgrade-cloud.md | 21 ++ .../major-version-upgrade-self-hosted.md | 15 + .../_includes/common/upgrade/overview.md | 10 + .../upgrade/patch-rollback-self-hosted.md | 1 + .../upgrade/patch-upgrade-self-hosted.md | 15 + .../upgrade/prepare-to-upgrade-cloud.md | 12 + .../upgrade/prepare-to-upgrade-self-hosted.md | 16 + .../common/upgrade/rollback-cloud.md | 7 + .../common/upgrade/rollback-self-hosted.md | 4 + .../common/upgrade/see-also-cloud.md | 2 + .../common/upgrade/see-also-self-hosted.md | 4 + .../common/upgrade/troubleshooting-cloud.md | 3 + .../upgrade/troubleshooting-self-hosted.md | 3 + .../common/upgrade/upgrade-high-level.md | 8 + .../_includes/v24.2/upgrade_requirements.md | 3 + .../backups/recommend-backups-for-upgrade.md | 10 +- .../_includes/v24.3/upgrade-requirements.md | 5 + .../upgrade-cockroach-version.md | 52 ++- .../v24.3/upgrade-cockroach-version.md | 337 ++---------------- 22 files changed, 250 insertions(+), 344 deletions(-) create mode 100644 src/current/_includes/common/upgrade/disable-auto-finalization.md create mode 100644 src/current/_includes/common/upgrade/finalize-cloud.md create mode 100644 src/current/_includes/common/upgrade/finalize-self-hosted.md create mode 100644 src/current/_includes/common/upgrade/major-version-upgrade-cloud.md create mode 100644 src/current/_includes/common/upgrade/major-version-upgrade-self-hosted.md create mode 100644 src/current/_includes/common/upgrade/overview.md create mode 100644 src/current/_includes/common/upgrade/patch-rollback-self-hosted.md create mode 100644 src/current/_includes/common/upgrade/patch-upgrade-self-hosted.md create mode 100644 src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md create mode 100644 src/current/_includes/common/upgrade/prepare-to-upgrade-self-hosted.md create mode 100644 src/current/_includes/common/upgrade/rollback-cloud.md create mode 100644 src/current/_includes/common/upgrade/rollback-self-hosted.md create mode 100644 src/current/_includes/common/upgrade/see-also-cloud.md create mode 100644 src/current/_includes/common/upgrade/see-also-self-hosted.md create mode 100644 src/current/_includes/common/upgrade/troubleshooting-cloud.md create mode 100644 src/current/_includes/common/upgrade/troubleshooting-self-hosted.md create mode 100644 src/current/_includes/common/upgrade/upgrade-high-level.md create mode 100644 src/current/_includes/v24.2/upgrade_requirements.md create mode 100644 src/current/_includes/v24.3/upgrade-requirements.md diff --git a/src/current/_includes/common/upgrade/disable-auto-finalization.md b/src/current/_includes/common/upgrade/disable-auto-finalization.md new file mode 100644 index 00000000000..c109d63181b --- /dev/null +++ b/src/current/_includes/common/upgrade/disable-auto-finalization.md @@ -0,0 +1,20 @@ +By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version). + +To disable auto-finalization: + +1. Connect to the cluster using the SQL shell: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cockroach sql + ~~~ + +1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.auto_upgrade.enabled` to `false`. + +Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade) or [roll it back]({#roll-back-a-major-version-upgrade). + +{{site.data.alerts.callout_info}} +Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. + +We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization. +{{site.data.alerts.end}} diff --git a/src/current/_includes/common/upgrade/finalize-cloud.md b/src/current/_includes/common/upgrade/finalize-cloud.md new file mode 100644 index 00000000000..161989a803f --- /dev/null +++ b/src/current/_includes/common/upgrade/finalize-cloud.md @@ -0,0 +1,10 @@ +To finalize a major-version upgrade manually, + +1. Click the cluster's name to open the **Cluster Details** page. +1. At the top of the page, click **Finalize**. + + When finalization begins, a series of migration jobs run to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. These temporary limitations are listed in the release notes for the new major version. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will continue to report the previous major version. + + You can monitor the process of finalization in the CockroachDB {{ site.data.products.cloud }} [Jobs page]({% link cockroachcloud/jobs-page.md %}). Migration jobs have names in the format `{major-version}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has a unique set of migration jobs and IDs. + +When finalization is complete, the **Cluster List** and **Cluster Details** page report the new version, and you can no longer roll back to the previous major version. diff --git a/src/current/_includes/common/upgrade/finalize-self-hosted.md b/src/current/_includes/common/upgrade/finalize-self-hosted.md new file mode 100644 index 00000000000..edf61601916 --- /dev/null +++ b/src/current/_includes/common/upgrade/finalize-self-hosted.md @@ -0,0 +1,36 @@ +To finalize a major-version upgrade, run the following command. Replace `{VERSION}` new major version, such as `{{ page.version.version }}: + +1. Connect to the cluster using the SQL shell: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cockroach sql + ~~~ + +1. Run the following command. Replace `{VERSION}` with the new major version, such as `{{ page.version.version }}`. + + {% include_cached copy-clipboard.html %} + ~~~ shell + SET CLUSTER SETTING version '{VERSION}'; + ~~~ + + A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return `{{ previous_version_numeric }}`. + + You can monitor the process of the migration in the DB Console [Jobs page]({% link {{ page.version.version }}/ui-jobs-page.md %}). Migration jobs have names in the format `{{ major_version_numeric }}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs. + + The amount of time required for finalization depends on the amount of data in the cluster, because finalization runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load. + + {{site.data.alerts.callout_info}} + Finalization is not complete until all [schema change]({% link {{ page.version.version }}/online-schema-changes.md %}) jobs reach a terminal state. Finalization can take as long as the longest-running schema change. + {{site.data.alerts.end}} + + When all migration jobs have completed, the upgrade is complete. + +1. To confirm that finalization has completed, check the cluster version: + + {% include_cached copy-clipboard.html %} + ~~~ sql + > SHOW CLUSTER SETTING version; + ~~~ + + If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of [decommissioning nodes]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#status-change) where decommission has stalled. In most cases, issuing the `decommission` command again resolves the issue. If you have trouble upgrading, [contact Support](https://cockroachlabs.com/support/hc/). diff --git a/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md b/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md new file mode 100644 index 00000000000..d46cad338f3 --- /dev/null +++ b/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md @@ -0,0 +1,21 @@ +This section shows how to perform a major-version upgrade for a cluster in CockroachDB {{ site.data.products.cloud }}. Patch upgrades within a cluster's major version are applied automatically, and no action is required. + +1. Verify the cluster's current major version, and which versions it can be upgraded to: + 1. Sign into [CockroachDB {{ site.data.products.cloud }}](https://cockroachlabs.cloud). + 1. From the **Clusters** page, find the cluster by name. If the cluster is in a folder, click the name of the folder to view its descendants. The cluster's major version and patch are shown in the **Version** column. +1. Check which upgrades are available to the cluster, if any. + + Beginning with v24.1, major versions alternate in type between Regular releases, which are required, and Innovation releases, which can be skipped. + + From a Regular release, you can upgrade to the next major release (an Innovation release) or the subsequent major release (another Regular release). From an Innovation release, you must upgrade to the next Regular release. + + These releases also provide different support periods. + + For details, refer the [CockroachDB Cloud Support Policy]({% link cockroachcloud/upgrade-policy.md %}#cockroachdb-cloud-support-policy). + + To check whether major-version upgrades are available, click the three-dot **Action** menu. If upgrades are available, **Upgrade major version** will be enabled. Click it. In the dialog, if only one newer major version upgrade is available, it is selected automatically. If multiple upgrades are available, the latest Regular release is selected by default. If you intend to upgrade the cluster, keep this dialog open while completing the next steps. Otherwise, close the dialog. +1. Before beginning a major-version upgrade, review its [Release notes]({% link releases/index.md %}), as well as the release notes for any skipped Innovation Releases. If any backward-incompatible changes or new features impact the cluster's workloads, you may need to make adjustments before beginning the upgrade. +1. A CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster remains fully available while it is upgraded. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are upgraded one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node {{ site.data.products.advanced }} cluster will be unavailable while the cluster is upgraded and restarted. If necessary, prepare for the upgrade by communicating ahead of time with your users, and plan to begin the upgrade during a time when the impact on the cluster's workload will be minimized. +1. To begin a major-version upgrade, go back to the dialog in the CockroachDB {{ site.data.products.cloud }} Console. If multiple upgrades are available, select the desired version. Review the details, then click **Start upgrade**. The dialog closes, and the **Version** columnin the **Cluster List** page changes to **Upgrading**. When the upgrade has finished but has not yet been finalized, the **Version** column reports that the new version is **pending**. + +The upgrade is not complete until it is [finalized](#finalize-a-major-version-upgrade). After the upgrade is finalized, the cluster can no longer be [rolled back](#roll-back-a-major-version-upgrade) to its previous major version. If you take no action, finalization occurs automatically after approximately 72 hours. diff --git a/src/current/_includes/common/upgrade/major-version-upgrade-self-hosted.md b/src/current/_includes/common/upgrade/major-version-upgrade-self-hosted.md new file mode 100644 index 00000000000..a0914fcff62 --- /dev/null +++ b/src/current/_includes/common/upgrade/major-version-upgrade-self-hosted.md @@ -0,0 +1,15 @@ +To perform a major upgrade: + +1. One node at a time: + 1. Replace the `cockroach` binary on the node with the binary for the new patch release. For containerized workloads, update the deployment to use the image for the new patch release. + 1. Restart CockroachDB on the node. For containerized workloads, use your orchestration framework to restart the `cockroach` process in the node's container. + 1. Ensure that the node is ready to accept a SQL connection. Unless there are tens of thousands of [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-range) on the node, this takes less than a minute. + + To be certain that the node is ready, run the following command to connect to the cluster and run a test query. + + {% include_cached copy-clipboard.html %} + ~~~ shell + cockroach sql -e 'select 1' + ~~~ +1. If auto-finalization is enabled (the default), finalization begins as soon as the last node rejoins the cluster with the new binary. When finalization finishes, the upgrade is complete. +1. If auto-finalization is disabled, follow your organization's testing procedures to decide whether to [finalize the upgrade](#finalize-a-major-version-upgrade) or [roll back](#roll-back-a-major-version-upgrade) the upgrade. After finalization begins, you can no longer roll back to the cluster's previous major version. diff --git a/src/current/_includes/common/upgrade/overview.md b/src/current/_includes/common/upgrade/overview.md new file mode 100644 index 00000000000..ad2da882b64 --- /dev/null +++ b/src/current/_includes/common/upgrade/overview.md @@ -0,0 +1,10 @@ +CockroachDB offers the following types of upgrades: + +- [**Major-version upgrades**]({% link releases/index.md %}#major-releases): A major-version upgrade moves a cluster from one major version of CockroachDB to another, such as from v24.2 to v24.3. A major-version upgrade may include new features, updates to cluster setting defaults, and backward-incompatible changes. Performing a major-version upgrade requires an additional step to finalize the upgrade. + + As of 2024, every second major version is an **Innovation release**. {% if page.path contains "cockroachcloud" %}For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }},innovation{% else %}Innovation{% endif %} releases offer shorter support windows and can be skipped. {% if page.path contains "cockroachcloud" %}Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically.{% endif %} +- [**Patch upgrades**]({% link releases/index.md %}#patch-releases): A patch upgrade moves a cluster from one patch to another within a major version, such as from v24.2.3 to v24.2.4. Patch upgrades do not introduce backward-incompatible changes. {% if page.path contains "cockroachcloud" %}Patch upgrades are automatically applied to CockroachDB {{ site.data.products.advanced }}, {{ site.data.products.standard }}, and {{ site.data.products.basic }} clusters.{% endif %} + + A major version of CockroachDB has two phases of patch releases: a series of **testing releases** followed by a series of **production releases**. A major version’s initial production release is also known as its GA release. In the lead-up to a new major version's GA release, a series of Testing releases may be made available{% if page.path contains "cockroachcloud" %} to CockroachDB {{ site.data.products.advanced }} as Pre-Production Preview releases{% endif %} for testing and validation. Testing releases are intended for testing and experimentation only, and are not qualified for production environments or eligible for support or uptime SLA commitments. {% if page.path contains "cockroachcloud" %}If a cluster is upgraded to a Pre-Production Preview patch release, it will be automatically upgraded to subsequent patch releases within the major version, including newer Pre-Production Preview releases, the initial GA release, and subsequent patch releases.{% else %}{{site.data.alerts.callout_info}}A cluster cannot be upgraded from an alpha binary of a prior release or from a binary built from the `master` branch of the CockroachDB source code.{{site.data.alerts.end}}{% endif %} + +To learn more about CockroachDB major versions and patches, refer to the [Releases Overview]({% link releases/index.md %}#overview). diff --git a/src/current/_includes/common/upgrade/patch-rollback-self-hosted.md b/src/current/_includes/common/upgrade/patch-rollback-self-hosted.md new file mode 100644 index 00000000000..5707a2e814a --- /dev/null +++ b/src/current/_includes/common/upgrade/patch-rollback-self-hosted.md @@ -0,0 +1 @@ +To roll back a patch upgrade, repeat the steps in [Perform a patch upgrade](#perform-a-patch-upgrade), but replace the newer binary with the older version. diff --git a/src/current/_includes/common/upgrade/patch-upgrade-self-hosted.md b/src/current/_includes/common/upgrade/patch-upgrade-self-hosted.md new file mode 100644 index 00000000000..5845e2f1b06 --- /dev/null +++ b/src/current/_includes/common/upgrade/patch-upgrade-self-hosted.md @@ -0,0 +1,15 @@ +To upgrade from one patch release to another within the same major version, perform the following steps on one node at a time: + +1. Replace the `cockroach` binary on the node with the binary for the new patch release. For containerized workloads, update the deployment's image to the image for the new patch release. +1. Restart CockroachDB on the node. +1. Verify that the node has rejoined the cluster. +1. Ensure that the node is ready to accept a SQL connection. + + Unless there are tens of thousands of ranges on the node, it's usually sufficient to wait one minute. To be certain that the node is ready, run the following command: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cockroach sql -e 'select 1' + ~~~ + +When all nodes are running the new patch version, the upgrade is complete. diff --git a/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md b/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md new file mode 100644 index 00000000000..8cdd0ab402d --- /dev/null +++ b/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md @@ -0,0 +1,12 @@ +Before beginning a major-version upgrade: + +1. **CockroachDB {{ site.data.products.advanced }}**: Verify the overall health of your cluster using the [DB Console]({% link cockroachcloud/tools-page#access-the-db-console.md %}): + - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or [decommission]({% link {{ site.current_cloud_version }}/node-shutdown.md %}?filters=decommission#remove-nodes) them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. + + If any node is not fully decommissioned, try the following: + 1. First, reissue the [decommission command]({% link {{ site.current_cloud_version }}/node-shutdown.md %}?filters=decommission#decommission-the-node). The second command typically succeeds within a few minutes. + 1. If the second decommission command does not succeed, [recommission]({% link {{ site.current_cloud_version }}/node-shutdown.md %}?filters=decommission#recommission-nodes) and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`. + - Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade. +1. Review the cluster's [Metrics page]({% link cockroachcloud/metrics.md %}) to ensure that your cluster's compute and storage capacity is within acceptable values. The cluster must be able to tolerate some increase, in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider increasing the cluster's resources before beginning your upgrade. +1. {% include {{ site.current_cloud_version }}/backups/recommend-backups-for-upgrade.md %} +1. Review the [{{ site.current_cloud_version }} Release Notes]({% link releases/{{ site.current_cloud_version }}.md %}), as well as the release notes for any skipped major version. Pay careful attention to the sections for backward-incompatible changes, deprecations, changes to default cluster settings, and features that are not available until the upgrade is finalized. diff --git a/src/current/_includes/common/upgrade/prepare-to-upgrade-self-hosted.md b/src/current/_includes/common/upgrade/prepare-to-upgrade-self-hosted.md new file mode 100644 index 00000000000..5875f314b32 --- /dev/null +++ b/src/current/_includes/common/upgrade/prepare-to-upgrade-self-hosted.md @@ -0,0 +1,16 @@ +Before beginning a major-version or patch upgrade: + +1. Verify the overall health of your cluster using the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}): + - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or [decommission]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#remove-nodes) them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. + + If any node is not fully decommissioned, try the following: + 1. First, reissue the [decommission command]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#decommission-the-node). The second command typically succeeds within a few minutes. + 1. If the second decommission command does not succeed, [recommission]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#recommission-nodes) and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`. + - Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade. + - In the **Node List**, make sure all nodes are on the same version. Upgrade them to the cluster's current version before continuing. If any nodes are behind, this also indicates that the previous major-version upgrade may not be finalized. + - In the **Metrics** dashboards, make sure [CPU]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#cpu-usage), [memory]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#database-memory-usage), and [storage]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#storage-capacity) capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider [adding nodes]({% link {{ page.version.version }}/cockroach-start.md %}) to your cluster before beginning your upgrade. +1. Make sure your cluster is behind a [load balancer]({% link {{ page.version.version }}/recommended-production-settings.md %}#load-balancing), or your clients are configured to talk to multiple nodes. If your application communicates with only a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail. +1. {% include {{page.version.version}}/storage/compaction-concurrency.md %} +1. {% include {{page.version.version}}/backups/recommend-backups-for-upgrade.md %} Refer to [Restoring backups across versions]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version). +1. Review the [{{ page.version.version }} Release Notes]({% link releases/{{ page.version.version }}.md %}), as well as the release notes for any skipped major version. Pay careful attention to the sections for backward-incompatible changes, deprecations, changes to default cluster settings, and features that are not available until the upgrade is finalized. +1. Optionally [disable auto-finalization](#disable-auto-finalization) to preserve the ability to roll back a major-version upgrade instead of finalizing it. If auto-finalization is disabled, a major-version upgrade is not complete until it is finalized. diff --git a/src/current/_includes/common/upgrade/rollback-cloud.md b/src/current/_includes/common/upgrade/rollback-cloud.md new file mode 100644 index 00000000000..5e612b9061c --- /dev/null +++ b/src/current/_includes/common/upgrade/rollback-cloud.md @@ -0,0 +1,7 @@ +To roll back to the previous major version before the upgrade is finalized automatically: + +1. Click the cluster's name to open the **Cluster Details** page. +1. At the top of the page, click **Roll back**. + + A CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster remains fully available while it is rolled back. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are rolled back one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node CockroachDB {{ site.data.products.advanced }} cluster will be unavailable while the cluster is rolled back and restarted. Like an upgrade, a rollback must be finalized. +1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-versino-upgrade). When the rollback is finalized, the rollback is complete. diff --git a/src/current/_includes/common/upgrade/rollback-self-hosted.md b/src/current/_includes/common/upgrade/rollback-self-hosted.md new file mode 100644 index 00000000000..e754d8bd068 --- /dev/null +++ b/src/current/_includes/common/upgrade/rollback-self-hosted.md @@ -0,0 +1,4 @@ +To roll back a major-version upgrade that has not been finalized: + +1. Follow the steps to [perform a major-version upgrade](#perform-a-major-version-upgrade), replacing the `cockroach` binary on each node with binary for the previous major version. +1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade). When the rollback is finalized, the rollback is complete. diff --git a/src/current/_includes/common/upgrade/see-also-cloud.md b/src/current/_includes/common/upgrade/see-also-cloud.md new file mode 100644 index 00000000000..de43e2b731a --- /dev/null +++ b/src/current/_includes/common/upgrade/see-also-cloud.md @@ -0,0 +1,2 @@ +- [Cloud Upgrade Policy]({% link cockroachcloud/upgrade-policy.md %}) +- [{{ site.current_cloud_version }} Release Notes]({% link releases/{{ site.current_cloud_version }}.md %}) diff --git a/src/current/_includes/common/upgrade/see-also-self-hosted.md b/src/current/_includes/common/upgrade/see-also-self-hosted.md new file mode 100644 index 00000000000..dc5c65ee81f --- /dev/null +++ b/src/current/_includes/common/upgrade/see-also-self-hosted.md @@ -0,0 +1,4 @@ +- [Release Support Policy]({% link releases/release-support-policy.md %}) +- [Collect Debug Information]({% link {{ page.version.version }}/cockroach-debug-zip.md %}) +- [View Version Details]({% link {{ page.version.version }}/cockroach-version.md %}) +- [{{ page.version.version }} Release Notes]({% link releases/{{page.version.version}}.md %}) diff --git a/src/current/_includes/common/upgrade/troubleshooting-cloud.md b/src/current/_includes/common/upgrade/troubleshooting-cloud.md new file mode 100644 index 00000000000..1c47ea3a0b4 --- /dev/null +++ b/src/current/_includes/common/upgrade/troubleshooting-cloud.md @@ -0,0 +1,3 @@ +After the upgrade has finalized (whether manually or automatically), it is no longer possible to roll back the upgrade. If you are experiencing problems, we recommend that you [open a support request]({% link {{ site.current_cloud_version }}/support-resources.md %}) for assistance. + +In the event of catastrophic failure or corruption, it may be necessary to [restore]({% link cockroachcloud/managed-backups.md %}) from a backup to a new cluster running the previous major version. diff --git a/src/current/_includes/common/upgrade/troubleshooting-self-hosted.md b/src/current/_includes/common/upgrade/troubleshooting-self-hosted.md new file mode 100644 index 00000000000..91a5a2a3b74 --- /dev/null +++ b/src/current/_includes/common/upgrade/troubleshooting-self-hosted.md @@ -0,0 +1,3 @@ +After the upgrade has finalized (whether manually or automatically), it is no longer possible to roll back the upgrade. If you are experiencing problems, we recommend that you [open a support request]({% link {{ page.version.version }}/support-resources.md %}) for assistance. + +In the event of catastrophic failure or corruption, it may be necessary to [restore]({% link {{ page.version.version }}/restore.md %}) from a backup to a new cluster running the previous major version. diff --git a/src/current/_includes/common/upgrade/upgrade-high-level.md b/src/current/_includes/common/upgrade/upgrade-high-level.md new file mode 100644 index 00000000000..d6dfb480494 --- /dev/null +++ b/src/current/_includes/common/upgrade/upgrade-high-level.md @@ -0,0 +1,8 @@ +At a high level, a cluster upgrade involves the following steps, which are detailed in the following sections: + +1. On one node at a time: + 1. Replace the previous `cockroach` binary or container image with the new binary. + 1. Restart the `cockroach` process and verify that the node has rejoined the cluster. +1. When all nodes have rejoined the cluster: + 1. For a patch upgrade within the same major version, the upgrade is complete. + 1. For a major-version upgrade, the upgrade is not complete until it is [finalized](#finalize-a-major-version-upgrade). Auto-finalization is enabled by default, and begins as soon as all nodes have rejoined the cluster using the new binary. If you need the ability to [roll back a major-version upgrade](#roll-back-a-major-version-upgrade), you can disable auto-finalization and finalize the upgrade manually. Certain features and performance improvements, such as those that require changes to system schemas or objects, are not available until the upgrade is finalized. Refer to the {% if page.path contains 'cockroachcloud' %} [{{ site.current_cloud_version }} Release Notes]({% link releases/{{ page.version.version }}.md %}) {% else %}[{{ page.version.version }} Release Notes]({% link releases/{{ page.version.version }}.md %}).{% endif %}. diff --git a/src/current/_includes/v24.2/upgrade_requirements.md b/src/current/_includes/v24.2/upgrade_requirements.md new file mode 100644 index 00000000000..3e9adfff2ec --- /dev/null +++ b/src/current/_includes/v24.2/upgrade_requirements.md @@ -0,0 +1,3 @@ +CockroachDB v24.2 is an Innovation release. To upgrade to it, you must be running v24.1, the previous Regular release. + +Before continuing, [upgrade to v24.1]({% link v24.1/upgrade-cockroach-version.md %}). diff --git a/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md b/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md index 6cb563feab9..fe093883dad 100644 --- a/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md +++ b/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md @@ -1 +1,9 @@ -Because CockroachDB is designed with high fault tolerance, backups are primarily needed for [disaster recovery]({% link {{ page.version.version }}/disaster-recovery-planning.md %}). However, taking regular backups of your data is an operational best practice. When upgrading to a major release, **we recommend [taking a backup]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) of your cluster**. \ No newline at end of file +{% if page.path contains "cockroachcloud" %} +[Managed backups]({% link cockroachcloud/managed-backups.md %}) are automated backups of CockroachDB {{ site.data.products.cloud }} clusters that are stored by Cockroach Labs in cloud storage. By default, Cockroach Labs takes and retains managed backups in all Cloud clusters. + +When upgrading to a major release, you can optionally [take a self-managed backup]({% link cockroachcloud/take-and-restore-self-managed-backups.md %}) of your cluster to your own cloud storage, as an extra layer of protection in case the upgrade leads to issues. +{% else %} +CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for [disaster recovery]({% link {{ page.version.version }}/disaster-recovery-planning.md %}) planning. + +When upgrading to a new major version, **we recommend [taking a self-managed backup]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) of your cluster** so that you can [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version) if the upgrade leads to issues. +{% endif %} diff --git a/src/current/_includes/v24.3/upgrade-requirements.md b/src/current/_includes/v24.3/upgrade-requirements.md new file mode 100644 index 00000000000..f1111c9393d --- /dev/null +++ b/src/current/_includes/v24.3/upgrade-requirements.md @@ -0,0 +1,5 @@ +CockroachDB v24.3 is a Regular release. To upgrade to it, you must be running either: +- [v24.2]({% link v24.2/upgrade-cockroach-version.md %}), the previous Innovation release. +- [v24.1]({% link v24.1/upgrade-cockroach-version.md %}), the previous Regular release. + +Before continuing, upgrade to an appropriate version. diff --git a/src/current/cockroachcloud/upgrade-cockroach-version.md b/src/current/cockroachcloud/upgrade-cockroach-version.md index cf2790a64ad..483718bef3e 100644 --- a/src/current/cockroachcloud/upgrade-cockroach-version.md +++ b/src/current/cockroachcloud/upgrade-cockroach-version.md @@ -9,20 +9,17 @@ This page describes how major-version and patch upgrades work and shows how to u ## Overview -CockroachDB offers the following types of upgrades: +{% include common/upgrade/overview.md %} -- [**Major-version upgrades**]({% link releases/index.md %}#major-releases): A major-version upgrade moves a cluster from one major version of CockroachDB to another, such as from v24.2 to v24.3. A major-version upgrade may include new features, updates to cluster setting defaults, and backward-incompatible changes. Performing a major-version upgrade requires an additional step to finalize the upgrade. +{% include_cached {{ site.current_cloud_version }}/upgrade_requirements.md %} - As of 2024, every second major version is an **Innovation release**. For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }}, Innovation releases offer shorter support windows and can be skipped. Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically. -- [**Patch upgrades**]({% link releases/index.md %}#patch-releases): A patch upgrade moves a cluster from one patch to another within a major version, such as from v24.2.3 to v24.2.4. Patch upgrades do not introduce backward-incompatible changes. Patch upgrades are automatically applied to CockroachDB {{ site.data.products.advanced }}, CockroachDB {{ site.data.products.standard }}, and CockroachDB {{ site.data.products.basic }} clusters. +{% include common/upgrade/upgrade-high-level.md %} - A major version has two types of patch releases: a series of **testing releases** followed by a series of **production releases**. A major version’s initial production release is also known as its GA release. In the lead-up to a new major version's GA release, a series of Testing releases may be made available to CockroachDB {{ site.data.products.advanced }} as Pre-Production Preview releases for testing and validation. Testing releases are intended for testing and experimentation only, and are not qualified for production environments or eligible for support or uptime SLA commitments. If a cluster is upgraded to a Pre-Production Preview patch release as a major-version upgrade, it will be automatically upgraded to subsequent patch releases within the major version, including newer Pre-Production Preview releases, the initial GA release, and subsequent Production patches. +### Availability during an upgrade -To learn more about CockroachDB major versions and patches, refer to the [Releases Overview]({% link releases/index.md %}#overview). +For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.basic }}, a cluster's resources are not impacted by an ongoing upgrade. -## Upgrades and maintenance windows - -If you have [configured a maintenance window]({% link cockroachcloud/advanced-cluster-management.md %}) for a CockroachDB {{ site.data.products.advanced }} cluster, automatic patch upgrades are applied during the maintenance window. Major-version upgrades must be initiated manually. +For CockroachDB {{ site.data.products.advanced }}, nodes are upgraded one at a time in a rolling fashion, and the cluster's resources are reduced as each node is upgraded. If you have [configured a maintenance window]({% link cockroachcloud/advanced-cluster-management.md %}) for a CockroachDB {{ site.data.products.advanced }} cluster, automatic patch upgrades are applied during the maintenance window. Major-version upgrades must be initiated manually. {{site.data.alerts.callout_info}} Maintenance operations that are critical for cluster security or stability may be applied outside of the maintenance window, and upgrades that begin in a maintenance window may not always be completed by the end of the window. @@ -30,31 +27,26 @@ Maintenance operations that are critical for cluster security or stability may b Refer to [Cloud Upgrade Policy]({% link cockroachcloud/upgrade-policy.md %}). +## Prepare to upgrade + +{% include common/upgrade/prepare-to-upgrade-cloud.md %} + ## Perform a major-version upgrade -This section shows how to perform a major-version upgrade for a cluster in CockroachDB {{ site.data.products.cloud }}. Patch upgrades within a cluster's major version are applied automatically, and no action is required. +{% include common/upgrade/major-version-upgrade-cloud.md %} + +### Finalize a major-version upgrade manually + +{% include_cached finalize-cloud.md %} + +### Roll back a major-version upgrade -1. Verify the cluster's current major version, and which versions it can be upgraded to: - 1. Sign into [CockroachDB {{ site.data.products.cloud }}](https://cockroachlabs.cloud). - 1. From the **Clusters** page, find the cluster by name. If the cluster is in a folder, click the name of the folder to view its descendants. The cluster's major version and patch are shown in the **Version** column. -1. Check which upgrades are available to the cluster, if any. +{% include_cached rollback-cloud.md %} - Beginning with v24.1, major versions alternate in type between Regular releases, which are required, and Innovation releases, which can be skipped. - - From a Regular release, you can upgrade to the next major release (an Innovation release) or the subsequent major release (another Regular release). From an Innovation release, you must upgrade to the next Regular release. - - These releases also provide different support periods. - - For details, refer the [CockroachDB Cloud Support Policy]({% link cockroachcloud/upgrade-policy.md %}#cockroachdb-cloud-support-policy). +## Troubleshooting - To check whether major-version upgrades are available, click the three-dot **Action** menu. If upgrades are available, **Upgrade major version** will be enabled. Click it. In the dialog, if only one newer major version upgrade is available, it is selected automatically. If multiple upgrades are available, the latest Regular release is selected by default. If you intend to upgrade the cluster, keep this dialog open while completing the next steps. Otherwise, close the dialog. -1. Before beginning a major-version upgrade, review its [Release notes]({% link releases/index.md %}), as well as the release notes for any skipped Innovation Releases. If any backward-incompatible changes or new features impact the cluster's workloads, you may need to make adjustments before beginning the upgrade. -1. A CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster remains fully available while it is upgraded. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are upgraded one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node {{ site.data.products.advanced }} cluster will be unavailable while the cluster is upgraded and restarted. If necessary, prepare for the upgrade by communicating ahead of time with your users, and plan to begin the upgrade during a time when the impact on the cluster's workload will be minimized. -1. To begin a major-version upgrade, go back to the dialog in the CockroachDB {{ site.data.products.cloud }} Console. If multiple upgrades are available, select the desired version. Review the details, then click **Start upgrade**. The dialog closes, and the **Version** columnin the **Cluster List** page changes to **Upgrading**. When the upgrade has finished but has not yet been finalized, the **Version** column reports that the new version is **pending**. -1. The upgrade is not complete until it is finalized. After the upgrade is finalized, the cluster can no longer be rolled back to its previous major version. If you take no action, finalization occurs automatically after approximately 72 hours. - - To roll back to the previous major version, click **Roll back**. A CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster remains fully available while it is rolled back. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are rolled back one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node CockroachDB {{ site.data.products.advanced }} cluster will be unavailable while the cluster is rolled back and restarted. Like an upgrade, a rollback must be finalized. - - To finalize the cluster upgrade manually, click the cluster's name to open the **Cluster Details** page. At the top of the page, click **Finalize**. When finalization begins, a series of migration jobs run to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. These temporary limitations are listed in the release notes for the new major version. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will continue to report the previous major version. +{% include common/upgrade/troubleshooting-cloud.md %} - You can monitor the process of finalization in the CockroachDB Cloud [Jobs]({% link cockroachcloud/jobs-page.md %}) page. Migration jobs have names in the format `{major-version}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has a unique set of migration jobs and IDs. +## See also - When finalization is complete, the **Cluster List** and **Cluster Details** page report the new version, and you can no longer roll back to the previous major version. +{% include see-also-cloud.md %} diff --git a/src/current/v24.3/upgrade-cockroach-version.md b/src/current/v24.3/upgrade-cockroach-version.md index 61700be5c29..2b906eb0e68 100644 --- a/src/current/v24.3/upgrade-cockroach-version.md +++ b/src/current/v24.3/upgrade-cockroach-version.md @@ -1,344 +1,55 @@ --- -title: Upgrade to CockroachDB v24.2 -summary: Learn how to upgrade your CockroachDB cluster to v24.2. +title: Upgrade CockroachDB self-hosted +summary: Learn how to upgrade a self-hosted CockroachDB cluster. toc: true docs_area: manage --- -{% assign previous_version = site.data.versions | where_exp: "previous_version", "previous_version.major_version == page.version.version" | map: "previous_version" | first %} -{% assign rd = site.data.versions | where_exp: "rd", "rd.major_version == page.version.version" | first %} +This page describes how major-version and patch upgrades work and shows how to upgrade a self-hosted CockroachDB. To upgrade a cluster in CockroachDB {{ site.data.products.cloud }}, refer to [Upgrade a cluster in CockroachDB {{ site.data.products.cloud }}]({% link cockroachcloud/upgrade-cockroach-version.md %}) instead. -{% assign released = false %} -{% assign skippable = false %} -{% if rd.release_date != "N/A" and rd.maint_supp_exp_date != "N/A" %} - {% assign released = true %} - {% if rd.asst_supp_exp_date == "N/A" %} - {% assign skippable = true %} - {% endif %} -{% endif %} +## Overview -{% assign earliest = site.data.releases | where_exp: "earliest", "earliest.major_version == page.version.version" | sort: "release_date" | first %} -{% assign latest = site.data.releases | where_exp: "latest", "latest.major_version == page.version.version" | sort: "release_date" | last %} -{% assign prior = site.data.releases | where_exp: "prior", "prior.major_version == page.version.version" | sort: "release_date" | pop | last %} -{% assign previous_latest_prod = site.data.releases | where_exp: "previous_latest_prod", "previous_latest_prod.major_version == previous_version" | where: "release_type", "Production" | sort: "release_date" | last %} -{% assign actual_latest_prod = site.data.releases | where: "major_version", site.versions["stable"] | where: "release_type", "Production" | sort: "release_date" | last %} +{% include common/upgrade/overview.md %} -{% capture previous_version_numeric %}{{ previous_version | remove_first: 'v' }}{% endcapture %} -{% capture major_version_numeric %}{{ page.version.version | remove_first: 'v' }}{% endcapture %} +{% include_cached {{ page.version.version }}/upgrade_requirements.md %} -Because of CockroachDB's [multi-active availability]({% link {{ page.version.version }}/multi-active-availability.md %}) design, you can perform a "rolling upgrade" of your CockroachDB cluster. This means that you can upgrade nodes one at a time without interrupting the cluster's overall health and operations. +CockroachDB's [multi-active availability]({% link {{ page.version.version }}/multi-active-availability.md %}) means that your cluster remains available while you upgrade one node at a time in a rolling fashion. While a node is being upgraded, its resources are not available to the cluster. -This page describes how to upgrade to the latest **{{ page.version.version }}** release, **{{ latest.release_name }}**{% if latest.lts == true %} ([LTS]({% link releases/release-support-policy.md %}#support-types)){% endif %}. +{% include common/upgrade/upgrade-high-level.md %} -{% include latest-release-details.md %} +## Prepare to upgrade -To upgrade CockroachDB on Kubernetes, refer to [single-cluster]({% link {{ page.version.version }}/upgrade-cockroachdb-kubernetes.md %}) or [multi-cluster]({% link {{ page.version.version }}/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md %}#upgrade-the-cluster) instead. +{% include_cached common/upgrade/prepare-to-upgrade-self-hosted.md %} -## Step 1. Verify that you can upgrade +## Perform a patch upgrade -{{site.data.alerts.callout_info}} -A cluster cannot be upgraded from an alpha binary of a prior release or from a binary built from the `master` branch of the CockroachDB source code. -{{site.data.alerts.end}} +{% include_cached common/upgrade/patch-upgrade-self-hosted.md %} -{% if skippable == true %} -CockroachDB {{ page.version.version }} is an optional [Innovation release]({% link releases/release-support-policy.md %}#innovation-releases). If you skip it, you must upgrade to the next [Regular release]({% link releases/release-support-policy.md %}#regular-releases) when it is available to maintain support. -{% else %} -CockroachDB {{ page.version.version }} is a required [Regular release]({% link releases/release-support-policy.md %}#regular-releases). -{% endif %}{% comment %}TODO before next Regular, add logic for multiple targets to upgrade from to get to a Regular release{% endcomment %} +### Roll back a patch upgrade -To verify that you can upgrade: +{% include_cached common/upgrade/patch-rollback-self-hosted.md %} -1. Run [`cockroach sql`]({% link {{ page.version.version }}/cockroach-sql.md %}) against any node in the cluster to open the SQL shell. Then check your current cluster version: +## Perform a major-version upgrade - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW CLUSTER SETTING version; - ~~~ +{% include_cached common/upgrade/major-version-upgrade-self-hosted.md %} -To upgrade to {{ latest.release_name }}, you must be running{% if prior.release_name %} either{% endif %}: +### Finalize a major-version upgrade -{% if prior.release_name %} -- **Any earlier {{ page.version.version }} release:** {{ earliest.release_name }} to {{ prior.release_name }}. -{% endif %} -- **A {{ previous_version }} production release:** {{ previous_version }}.0 to {{ previous_latest_prod.release_name }}. +{% include finalize-self-hosted.md %} -If you are running any other version, take the following steps **before** continuing on this page: +### Roll back a major-version upgrade -| Version | Action(s) before upgrading to any {{ page.version.version }} release | -|------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Pre-{{ page.version.version }} testing release | Upgrade to a corresponding production release, then upgrade through each subsequent major release, [ending with a {{ previous_version }} production release]({% link {{ previous_version }}/upgrade-cockroach-version.md %}). | -| Pre-{{ previous_version }} production release | Upgrade through each subsequent major release, [ending with a {{ previous_version }} production release]({% link {{ previous_version }}/upgrade-cockroach-version.md %}). | -| {{ previous_version}} testing release | [Upgrade to a {{ previous_version }} production release]({% link {{ previous_version }}/upgrade-cockroach-version.md %}). | +{% include_cached rollback-self-hosted.md %} -When you are ready to upgrade to {{ latest.release_name }}, continue to [step 2](#step-2-prepare-to-upgrade). +## Disable auto-finalization -## Step 2. Prepare to upgrade - -Before starting the upgrade, complete the following steps. - -### Review breaking changes - - - -Review the [backward-incompatible changes]({% link releases/{{ page.version.version }}.md %}{% unless rd.release_date == "N/A" or rd.release_date > today %}#{{ page.version.version | replace: ".", "-" }}-0-backward-incompatible-changes{% endunless %}), [deprecated features]({% link releases/{{ page.version.version }}.md %}#{% unless rd.release_date == "N/A" or rd.release_date > today %}{{ page.version.version | replace: ".", "-" }}-0-deprecations{% endunless %}), and [key cluster setting changes]({% link releases/{{ page.version.version }}.md %}#{% unless rd.release_date == "N/A" or rd.release_date > today %}{{ page.version.version | replace: ".", "-" }}-0-cluster-settings{% endunless %}) in {{ page.version.version }}. If any affect your deployment, make the necessary changes before starting the rolling upgrade to {{ page.version.version }}. - -### Check load balancing - -Make sure your cluster is behind a [load balancer]({% link {{ page.version.version }}/recommended-production-settings.md %}#load-balancing), or your clients are configured to talk to multiple nodes. If your application communicates with a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail. - -### Check cluster health - -Verify the overall health of your cluster using the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}): - -- Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or [decommission](node-shutdown.html?filters=decommission#remove-nodes) them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually). - -- Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade. - -- In the **Node List**: - - Make sure all nodes are on the same version. If any nodes are behind, upgrade them to the cluster's current version first, and then start this process over. - -- In the **Metrics** dashboards: - - Make sure [CPU]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#cpu-usage), [memory]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#database-memory-usage), and [storage]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#storage-capacity) capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider [adding nodes]({% link {{ page.version.version }}/cockroach-start.md %}) to your cluster before beginning your upgrade. - -### Check decommissioned nodes - -If your cluster contains partially-decommissioned nodes, they will block an upgrade attempt. - -1. To check the status of decommissioned nodes, run the [`cockroach node status --decommission`]({% link {{ page.version.version }}/cockroach-node.md %}#show-the-status-of-all-nodes) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach node status --decommission - ~~~ - - In the output, verify that the value of the `membership` field of each node is `decommissioned`. If any node's `membership` value is `decommissioning`, that node is not fully decommissioned. - -1. If any node is not fully decommissioned, try the following: - - 1. First, reissue the [decommission command](node-shutdown.html?filters=decommission#decommission-the-node). The second command typically succeeds within a few minutes. - 1. If the second decommission command does not succeed, [recommission](node-shutdown.html?filters=decommission#recommission-nodes) and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`. - -### Back up cluster - -{% include {{page.version.version}}/backups/recommend-backups-for-upgrade.md%} -See our [support policy for restoring backups across versions]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version). - -## Step 3. Decide how the upgrade will be finalized - -When upgrading from one major version to another, certain features and performance improvements will be enabled only after finalizing the upgrade. {% if released == false %}CockroachDB {{ page.version.version }} is not yet Generally Available. Refer to the [Release notes]({% link releases/{{ page.version.version }}.md %}) for all Testing releases of {{ page.version.version }}.{% else %}Refer to the [Release notes]({% link releases/{{ page.version.version }}.md %}#features-that-require-upgrade-finalization).{% endif %} Even if no features require finalization, finalization is required. - -Finalization does not occur for patch upgrades within the same major version. - -After a major-version upgrade is finalized, it is no longer possible to roll back to the cluster's previous major version. By default, clusters on CockroachDB Cloud are set to **auto-finalize** a major-version upgrade as soon as all nodes have been upgraded. For production clusters, Cockroach Labs recommends that you disable auto-finalization so that you can roll back the upgrade if necessary. Otherwise, in the event of a catastrophic failure or corruption, the only option will be to start a new cluster using the previous binary and then restore from one of the backups created prior to performing the upgrade. - -Before finalizing the upgrade, monitor the stability and performance of the upgraded cluster. If auto-finalization is disabled, the upgrade is not complete until you have followed all of the instructions in [step 4](#step-4-perform-the-rolling-upgrade) and [step 6](#step-6-finish-the-upgrade). - -To disable auto-finalization: - -1. Start the [`cockroach sql`]({% link {{ page.version.version }}/cockroach-sql.md %}) shell against any node in the cluster. -1. Find the cluster's current version: - - {% include_cached copy-clipboard.html %} - ~~~ sql - SHOW CLUSTER SETTING version; - ~~~ - -1. Set the `cluster.preserve_downgrade_option` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}): - - {% include_cached copy-clipboard.html %} - ~~~ sql - SET CLUSTER SETTING cluster.preserve_downgrade_option = '{CURRENT_VERSION}'; - ~~~ - - Replace `{CURRENT_VERSION}` with the cluster's current version. It is an error to set it to any other value. - -## Step 4. Perform the rolling upgrade - -{{site.data.alerts.callout_success}} -Cockroach Labs recommends creating scripts to perform these steps instead of performing them manually. -{{site.data.alerts.end}} - -Follow these steps to perform the rolling upgrade. To upgrade CockroachDB on Kubernetes, refer to [single-cluster]({% link {{ page.version.version }}/upgrade-cockroachdb-kubernetes.md %}) or [multi-cluster]({% link {{ page.version.version }}/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md %}#upgrade-the-cluster) instead. - -For each node in your cluster, complete the following steps. Be sure to upgrade only one node at a time, and wait at least one minute after a node rejoins the cluster to upgrade the next node. Simultaneously upgrading more than one node increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. - -{{site.data.alerts.callout_danger}} -After beginning a major-version upgrade, Cockroach Labs recommends upgrading all nodes as quickly as possible. In a cluster with nodes running different major versions of CockroachDB, a query that is sent to an upgraded node can be distributed only among other upgraded nodes. Data accesses that would otherwise be local may become remote, and the performance of these queries can suffer. -{{site.data.alerts.end}} - -These steps perform an upgrade to the latest {{ page.version.version }} release, **{{ latest.release_name }}**. - -
- - -
- -1. [Drain and shut down the node]({% link {{ page.version.version }}/node-shutdown.md %}#perform-node-shutdown). - -1. Visit [What's New in {{ page.version.version }}?]({% link releases/{{ page.version.version }}.md %}) and download the **CockroachDB {{ latest.release_name }} full binary** for your architecture. - -1. Extract the archive. In the following instructions, replace `{COCKROACHDB_DIR}` with the path to the extracted archive directory. - -1. If you have a previous version of the `cockroach` binary in your `$PATH`, rename the outdated `cockroach` binary, and then move the new one into its place. - - If you get a permission error because the `cockroach` binary is located in a system directory, add `sudo` before each command. The binary will be owned by the effective user, which is `root` if you use `sudo`. - - {% include_cached copy-clipboard.html %} - ~~~ shell - i="$(which cockroach)"; mv "$i" "$i"_old - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - cp -i {COCKROACHDB_DIR}/cockroach /usr/local/bin/cockroach - ~~~ - -1. If a cluster has corrupt descriptors, a major-version upgrade cannot be finalized. Automatic descriptor repair is enabled by default in {{ page.version.version }}. After restarting each cluster node on {{ page.version.version }}, monitor the [cluster logs]({% link {{ page.version.version }}/logging.md %}) for errors. If a descriptor cannot be repaired automatically, [contact support](https://support.cockroachlabs.com/hc) for assistance completing the upgrade. To disable automatic descriptor repair (not generally recommended), set the environment variable `COCKROACH_RUN_FIRST_UPGRADE_PRECONDITION` to `false`. - -1. Start the node so that it can rejoin the cluster. - -
- - Without a process manager like `systemd`, re-run the [`cockroach start`]({% link {{ page.version.version }}/cockroach-start.md %}) command that you used to start the node initially, for example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach start \ - --certs-dir=certs \ - --advertise-addr={node address} \ - --join={node1 address},{node2 address},{node3 address} - ~~~ - - If you are using `systemd` as the process manager, run this command to start the node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - systemctl start {systemd config filename} - ~~~ - -
- -
- - Re-run the [`cockroach start`]({% link {{ page.version.version }}/cockroach-start.md %}) command that you used to start the node initially, for example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach start \ - --certs-dir=certs \ - --advertise-addr={node address} \ - --join={node1 address},{node2 address},{node3 address} - ~~~ - -
- -1. Verify the node has rejoined the cluster through its output to [`stdout`]({% link {{ page.version.version }}/cockroach-start.md %}#standard-output) or through the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-status). - -1. If you use `cockroach` in your `$PATH`, you can remove the previous binary: - - {% include_cached copy-clipboard.html %} - ~~~ shell - rm /usr/local/bin/cockroach_old - ~~~ - -
- - If you leave versioned binaries on your servers, you do not need to do anything. - -
- -1. After the node has rejoined the cluster, ensure that the node is ready to accept a SQL connection. - - Unless there are tens of thousands of ranges on the node, it's usually sufficient to wait one minute. To be certain that the node is ready, run the following command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach sql -e 'select 1' - ~~~ - - The command will automatically wait to complete until the node is ready. - -1. Repeat these steps for the next node. - -## Step 5. Roll back the upgrade (optional) - -If you decide to roll back to {{previous_version}}, you must do so before the upgrade has been [finalized](#step-6-finish-the-upgrade), as described in the next section. It is always possible to roll back to a previous {{ page.version.version }} version. - -To roll back an upgrade, do the following on each cluster node: - -1. [Perform a rolling upgrade](#step-4-perform-the-rolling-upgrade), as described in the previous section, but replace the upgraded `cockroach` binary on each node with the binary for the previous version. -1. Restart the `cockroach` process on the node and verify that it has rejoined the cluster before rolling back the upgrade on the next node. -1. After all nodes have been rolled back and rejoined the cluster, [finalize the rollback](#step-6-finish-the-upgrade) in the same way as you would finalize an upgrade, as described in the next section. - -## Step 6. Finish the upgrade - -Because a finalized major-version upgrade cannot be rolled back, Cockroach Labs recommends that you monitor the stability and performance of your cluster with the upgraded binary for at least a day before deciding to finalize the upgrade. - -{{site.data.alerts.callout_info}} -Finalization is required only when upgrading from {{ previous_version }}.x to {{ page.version.version }}. For upgrades within the {{ page.version.version }}.x series, skip this step. -{{site.data.alerts.end}} - -1. If you disabled auto-finalization in [step 3](#step-3-decide-how-the-upgrade-will-be-finalized), monitor the stability and performance of your cluster for at least a day. If you decide to roll back the upgrade, repeat the [rolling restart procedure](#step-4-perform-the-rolling-upgrade) with the previous binary. Otherwise, perform the following steps to re-enable upgrade finalization to complete the upgrade to {{ page.version.version }}. Cockroach Labs recommends that you either finalize or roll back a major-version upgrade within a relative short period of time; running in a partially-upgraded state is not recommended. - - {{site.data.alerts.callout_danger}} - A cluster that is not finalized on {{ previous_version }} cannot be upgraded to {{ page.version.version }} until the {{ previous_version }} upgrade is finalized. - {{site.data.alerts.end}} - -1. Once you are satisfied with the new version, run [`cockroach sql`]({% link {{ page.version.version }}/cockroach-sql.md %}) against any node in the cluster to open the SQL shell. - -1. Re-enable auto-finalization: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESET CLUSTER SETTING cluster.preserve_downgrade_option; - ~~~ - - A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return `{{ previous_version_numeric }}`. - - You can monitor the process of the migration in the DB Console [Jobs page]({% link {{ page.version.version }}/ui-jobs-page.md %}). Migration jobs have names in the format `{{ major_version_numeric }}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs. - - {{site.data.alerts.callout_info}} - All [schema change]({% link {{ page.version.version }}/online-schema-changes.md %}) jobs must reach a terminal state before finalization can complete. Finalization can therefore take as long as the longest-running schema change. Otherwise, the amount of time required for finalization depends on the amount of data in the cluster, as the process runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load. - {{site.data.alerts.end}} - - When all migration jobs have completed, the upgrade is complete. - -1. To confirm that finalization has completed, check the cluster version: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW CLUSTER SETTING version; - ~~~ - - If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of [decommissioning nodes]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#status-change) where decommission has stalled. In most cases, issuing the `decommission` command again resolves the issue. If you have trouble upgrading, [contact Support](https://cockroachlabs.com/support/hc/). - -After the upgrade to {{ page.version.version }} is finalized, you may notice an increase in [compaction]({% link {{ page.version.version }}/architecture/storage-layer.md %}#compaction) activity due to a background migration job within the storage engine. To observe the migration's progress, check the **Compactions** section of the [Storage Dashboard]({% link {{ page.version.version }}/ui-storage-dashboard.md %}) in the DB Console or monitor the `storage.marked-for-compaction-files` [time-series metric]({% link {{ page.version.version }}/metrics.md %}). When the metric's value nears or reaches `0`, the migration is complete and compaction activity will return to normal levels. - -{{site.data.alerts.callout_success}} -{% include {{page.version.version}}/storage/compaction-concurrency.md %} -{{site.data.alerts.end}} +{% include_cached common/upgrade/disable-auto-finalization.md %} ## Troubleshooting -After the upgrade has finalized (whether manually or automatically), it is no longer possible to downgrade to the previous release. If you are experiencing problems, we therefore recommend that you run the [`cockroach debug zip`]({% link {{ page.version.version }}/cockroach-debug-zip.md %}) command on any cluster node to capture your cluster's state, then [open a support request]({% link {{ page.version.version }}/support-resources.md %}) and share your debug zip. - -In the event of catastrophic failure or corruption, it may be necessary to [restore]({% link {{ page.version.version }}/restore.md %}) from a backup to a new cluster running {{ previous_version }}. +{% include common/upgrade/troubleshooting-self-hosted.md %} ## See also -- [Release Support Policy]({% link releases/release-support-policy.md %}) -- [View Node Details]({% link {{ page.version.version }}/cockroach-node.md %}) -- [Collect Debug Information]({% link {{ page.version.version }}/cockroach-debug-zip.md %}) -- [View Version Details]({% link {{ page.version.version }}/cockroach-version.md %}) -- [Release notes for our latest version]({% link releases/{{page.version.version}}.md %}) +{% include see-also-self-hosted.md %} -[#102961]: https://github.com/cockroachdb/cockroach/pull/102961 -[#104265]: https://github.com/cockroachdb/cockroach/pull/104265 -[#107474]: https://github.com/cockroachdb/cockroach/pull/107474 -[#106177]: https://github.com/cockroachdb/cockroach/pull/106177 -[#98308]: https://github.com/cockroachdb/cockroach/pull/98308 -[#105477]: https://github.com/cockroachdb/cockroach/pull/105477 -[#107903]: https://github.com/cockroachdb/cockroach/pull/107903 -[#108190]: https://github.com/cockroachdb/cockroach/pull/108190 -[#110649]: https://github.com/cockroachdb/cockroach/pull/110649 -[#114203]: https://github.com/cockroachdb/cockroach/pull/114203 From 49a5ec9ee215ce06768329c35cd747aca5190f64 Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Tue, 5 Nov 2024 15:47:50 -0800 Subject: [PATCH 02/19] Update Kubernetes self-hosted upgrade docs --- .../upgrade/disable-auto-finalization.md | 63 +++++ .../common/upgrade/finalize-kubernetes.md | 40 +++ .../common/upgrade/finalize-self-hosted.md | 10 +- .../major-version-upgrade-kubernetes.md | 264 ++++++++++++++++++ .../upgrade/patch-rollback-kubernetes.md | 1 + .../upgrade/patch-upgrade-kubernetes.md | 125 +++++++++ .../upgrade/patch-upgrade-self-hosted.md | 10 +- .../upgrade/prepare-to-upgrade-kubernetes.md | 16 ++ .../common/upgrade/rollback-kubernetes.md | 30 ++ .../kubernetes-upgrade-cluster-helm.md | 259 ----------------- .../kubernetes-upgrade-cluster-manual.md | 248 ---------------- .../v24.3/upgrade-cockroachdb-kubernetes.md | 122 ++------ 12 files changed, 574 insertions(+), 614 deletions(-) create mode 100644 src/current/_includes/common/upgrade/finalize-kubernetes.md create mode 100644 src/current/_includes/common/upgrade/major-version-upgrade-kubernetes.md create mode 100644 src/current/_includes/common/upgrade/patch-rollback-kubernetes.md create mode 100644 src/current/_includes/common/upgrade/patch-upgrade-kubernetes.md create mode 100644 src/current/_includes/common/upgrade/prepare-to-upgrade-kubernetes.md create mode 100644 src/current/_includes/common/upgrade/rollback-kubernetes.md delete mode 100644 src/current/_includes/v24.3/orchestration/kubernetes-upgrade-cluster-helm.md delete mode 100644 src/current/_includes/v24.3/orchestration/kubernetes-upgrade-cluster-manual.md diff --git a/src/current/_includes/common/upgrade/disable-auto-finalization.md b/src/current/_includes/common/upgrade/disable-auto-finalization.md index c109d63181b..67e26aa4cec 100644 --- a/src/current/_includes/common/upgrade/disable-auto-finalization.md +++ b/src/current/_includes/common/upgrade/disable-auto-finalization.md @@ -1,3 +1,65 @@ +{% if page.path contains 'kubernetes' %} + +
+For clusters managed by the Operator, auto-finalization is disabled and cannot be enabled. A major version upgrade is not complete until it is manually [finalized](#finalize-a-major-version-upgrade). +
+ +
+ +By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version). + +To disable auto-finalization: + +1. Connect to the cluster using the SQL shell: + + ~~~ shell + $ kubectl exec -it cockroachdb-client-secure \ + -- ./cockroach sql \ + --certs-dir=/cockroach-certs \ + --host=cockroachdb-public + ~~~ + +1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version. + +Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade) or [roll it back]({#roll-back-a-major-version-upgrade). + +{{site.data.alerts.callout_info}} +Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. + +We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization. +{{site.data.alerts.end}} + +
+ +
+ +By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version). + +To disable auto-finalization: + +1. Connect to the cluster using the SQL shell: + + ~~~ shell + $ kubectl exec -it cockroachdb-client-secure \ + -- ./cockroach sql \ + --certs-dir=/cockroach-certs \ + --host=cockroachdb-public + ~~~ + +1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version. + +Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade) or [roll it back]({#roll-back-a-major-version-upgrade). + +{{site.data.alerts.callout_info}} +Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. + +We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization. +{{site.data.alerts.end}} + +
+ +{% else %} + By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version). To disable auto-finalization: @@ -18,3 +80,4 @@ Previously, to disable automatic finalization and preserve the ability to roll b We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization. {{site.data.alerts.end}} +{% endif %} diff --git a/src/current/_includes/common/upgrade/finalize-kubernetes.md b/src/current/_includes/common/upgrade/finalize-kubernetes.md new file mode 100644 index 00000000000..77b5bd58429 --- /dev/null +++ b/src/current/_includes/common/upgrade/finalize-kubernetes.md @@ -0,0 +1,40 @@ +{% assign major_version_numeric = page.version.version | remove_first: 'v' %} + +To finalize a major-version upgrade: + +1. Connect to the cluster using the SQL shell: + + ~~~ shell + $ kubectl exec -it cockroachdb-client-secure \ + -- ./cockroach sql \ + --certs-dir=/cockroach-certs \ + --host=cockroachdb-public + ~~~ + +1. Run the following command: + + {% include_cached copy-clipboard.html %} + ~~~ sql + > RESET CLUSTER SETTING cluster.preserve_downgrade_option; + ~~~ + + A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return the previous version`. + + You can monitor the process of the migration in the DB Console [Jobs page]({% link {{ page.version.version }}/ui-jobs-page.md %}). Migration jobs have names in the format `{{ major_version_numeric }}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs. + + The amount of time required for finalization depends on the amount of data in the cluster, because finalization runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load. + + {{site.data.alerts.callout_info}} + Finalization is not complete until all [schema change]({% link {{ page.version.version }}/online-schema-changes.md %}) jobs reach a terminal state. Finalization can take as long as the longest-running schema change. + {{site.data.alerts.end}} + + When all migration jobs have completed, the upgrade is complete. + +1. To confirm that finalization has completed, check the cluster version: + + {% include_cached copy-clipboard.html %} + ~~~ sql + > SHOW CLUSTER SETTING version; + ~~~ + + If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of [decommissioning nodes]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#status-change) where decommission has stalled. In most cases, issuing the `decommission` command again resolves the issue. If you have trouble upgrading, [contact Support](https://cockroachlabs.com/support/hc/). diff --git a/src/current/_includes/common/upgrade/finalize-self-hosted.md b/src/current/_includes/common/upgrade/finalize-self-hosted.md index edf61601916..324752098b2 100644 --- a/src/current/_includes/common/upgrade/finalize-self-hosted.md +++ b/src/current/_includes/common/upgrade/finalize-self-hosted.md @@ -1,4 +1,6 @@ -To finalize a major-version upgrade, run the following command. Replace `{VERSION}` new major version, such as `{{ page.version.version }}: +{% assign major_version_numeric = page.version.version | remove_first: 'v' %} + +To finalize a major-version upgrade: 1. Connect to the cluster using the SQL shell: @@ -7,14 +9,14 @@ To finalize a major-version upgrade, run the following command. Replace `{VERSIO cockroach sql ~~~ -1. Run the following command. Replace `{VERSION}` with the new major version, such as `{{ page.version.version }}`. +1. Run the following command. Replace `{VERSION}` with the new major version, such as `{{ major_version_numeric }}`. {% include_cached copy-clipboard.html %} - ~~~ shell + ~~~ sql SET CLUSTER SETTING version '{VERSION}'; ~~~ - A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return `{{ previous_version_numeric }}`. + A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return the previous version. You can monitor the process of the migration in the DB Console [Jobs page]({% link {{ page.version.version }}/ui-jobs-page.md %}). Migration jobs have names in the format `{{ major_version_numeric }}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs. diff --git a/src/current/_includes/common/upgrade/major-version-upgrade-kubernetes.md b/src/current/_includes/common/upgrade/major-version-upgrade-kubernetes.md new file mode 100644 index 00000000000..d8cbeb4717e --- /dev/null +++ b/src/current/_includes/common/upgrade/major-version-upgrade-kubernetes.md @@ -0,0 +1,264 @@ +To perform a major upgrade: + +
+ +1. Change the container image image in the custom resource: + + ~~~ + image: + name: cockroachdb/cockroach:{{page.release_info.version}} + ~~~ + +1. Apply the new settings to the cluster: + + {% include_cached copy-clipboard.html %} + ~~~ shell + kubectl apply -f example.yaml + ~~~ + + The Operator will perform the staged update. + +1. To check the status of the rolling upgrade, run `kubectl get pods`. +1. Verify that all pods have been upgraded: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl get pods \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' + ~~~ + + You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). + +1. Before beginning a major-version upgrade, the Operator disables auto-finalization by setting the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version. Before finalizing an upgrade, follow your organization's testing procedures to decide whether to [finalize](#finalize-a-major-version-upgrade) or [roll back](#roll-back-a-major-version-upgrade) the upgrade. After finalization begins, you can no longer roll back to the cluster's previous major version. + +
+ +
+ +1. +1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl patch statefulset cockroachdb \ + -p='{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}' + ~~~ + + ~~~ + statefulset.apps/cockroachdb patched + ~~~ + +1. Change the container image in the StatefulSet: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl patch statefulset cockroachdb \ + --type='json' \ + -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:{{page.release_info.version}}"}]' + ~~~ + + ~~~ + statefulset.apps/cockroachdb patched + ~~~ + +1. To check the status of the rolling upgrade, run `kubectl get pods`. +1. After the pod has been restarted with the new image, start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}): + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl exec -it cockroachdb-client-secure \-- ./cockroach sql \ + --certs-dir=/cockroach-certs \ + --host=cockroachdb-public + ~~~ + +1. Run the following SQL query to verify that the number of underreplicated ranges is zero: + + {% include_cached copy-clipboard.html %} + ~~~ sql + SELECT sum((metrics->>'ranges.underreplicated')::DECIMAL)::INT AS ranges_underreplicated FROM crdb_internal.kv_store_status; + ~~~ + + ~~~ + ranges_underreplicated + -------------------------- + 0 + (1 row) + ~~~ + + This indicates that it is safe to proceed to the next pod. + +1. Exit the SQL shell: + + {% include_cached copy-clipboard.html %} + ~~~ sql + > \q + ~~~ + +1. Decrement the partition value by 1 to allow the next pod in the cluster to update: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl patch statefulset cockroachdb \ + -p='{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":1}}}}' + ~~~ + + ~~~ + statefulset.apps/cockroachdb patched + ~~~ + +1. Repeat steps 4-8 until all pods have been restarted and are running the new image (the final partition value should be `0`). + +1. Check the image of each pod to confirm that all have been upgraded: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl get pods \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' + ~~~ + + ~~~ + cockroachdb-0 cockroachdb/cockroach:{{page.release_info.version}} + cockroachdb-1 cockroachdb/cockroach:{{page.release_info.version}} + cockroachdb-2 cockroachdb/cockroach:{{page.release_info.version}} + ... + ~~~ + + You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). +1. If auto-finalization is disabled, the upgrade is not complete until you [finalize the upgrade](#finalize-a-major-version-upgrade). + +
+ +
+ +1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ helm upgrade \ + my-release \ + cockroachdb/cockroachdb \ + --set statefulset.updateStrategy.rollingUpdate.partition=2 + ~~~ + +1. Connect to the cluster using the SQL shell: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl exec -it cockroachdb-client-secure \ + -- ./cockroach sql \ + --certs-dir=/cockroach-certs \ + --host=my-release-cockroachdb-public + ~~~ + +1. Remove the cluster initialization job from when the cluster was created: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl delete job my-release-cockroachdb-init + ~~~ + +1. Change the container image in the StatefulSet: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ helm upgrade \ + my-release \ + cockroachdb/cockroachdb \ + --set image.tag={{page.release_info.version}} \ + --reuse-values + ~~~ + + ~~~ + NAME READY STATUS RESTARTS AGE + my-release-cockroachdb-0 1/1 Running 0 2m + my-release-cockroachdb-1 1/1 Running 0 3m + my-release-cockroachdb-2 0/1 ContainerCreating 0 25s + my-release-cockroachdb-init-nwjkh 0/1 ContainerCreating 0 6s + ... + ~~~ + + {{site.data.alerts.callout_info}} + Ignore the pod for cluster initialization. It is re-created as a byproduct of the StatefulSet configuration but does not impact your existing cluster. + {{site.data.alerts.end}} + +1. After the pod has been restarted with the new image, start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}): + + {% if page.secure == true %} + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl exec -it cockroachdb-client-secure \ + -- ./cockroach sql \ + --certs-dir=/cockroach-certs \ + --host=my-release-cockroachdb-public + ~~~ + + {% else %} + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl run cockroachdb -it \ + --image=cockroachdb/cockroach \ + --rm \ + --restart=Never \ + -- sql \ + --insecure \ + --host=my-release-cockroachdb-public + ~~~ + {% endif %} + +1. Run the following SQL query to verify that the number of underreplicated ranges is zero: + + {% include_cached copy-clipboard.html %} + ~~~ sql + SELECT sum((metrics->>'ranges.underreplicated')::DECIMAL)::INT AS ranges_underreplicated FROM crdb_internal.kv_store_status; + ~~~ + + ~~~ + ranges_underreplicated + -------------------------- + 0 + (1 row) + ~~~ + + This indicates that it is safe to proceed to the next pod. + +1. Exit the SQL shell: + + {% include_cached copy-clipboard.html %} + ~~~ sql + > \q + ~~~ + +1. Decrement the partition value by 1 to allow the next pod in the cluster to update: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ helm upgrade \ + my-release \ + cockroachdb/cockroachdb \ + --set statefulset.updateStrategy.rollingUpdate.partition=1 \ + ~~~ + +1. Repeat steps 4-8 until all pods have been restarted and are running the new image (the final partition value should be `0`). + +1. Check the image of each pod to confirm that all have been upgraded: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl get pods \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' + ~~~ + + ~~~ + my-release-cockroachdb-0 cockroachdb/cockroach:{{page.release_info.version}} + my-release-cockroachdb-1 cockroachdb/cockroach:{{page.release_info.version}} + my-release-cockroachdb-2 cockroachdb/cockroach:{{page.release_info.version}} + ... + ~~~ + + You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). + +1. If auto-finalization is disabled, the upgrade is not complete until you [finalize the upgrade](#finalize-a-major-version-upgrade). + +
diff --git a/src/current/_includes/common/upgrade/patch-rollback-kubernetes.md b/src/current/_includes/common/upgrade/patch-rollback-kubernetes.md new file mode 100644 index 00000000000..c0d0644e57d --- /dev/null +++ b/src/current/_includes/common/upgrade/patch-rollback-kubernetes.md @@ -0,0 +1 @@ +To roll back a patch upgrade, repeat the steps in [Perform a patch upgrade](#perform-a-patch-upgrade), but configure the container image for the pods to the previous major version. diff --git a/src/current/_includes/common/upgrade/patch-upgrade-kubernetes.md b/src/current/_includes/common/upgrade/patch-upgrade-kubernetes.md new file mode 100644 index 00000000000..950fe5cd5ae --- /dev/null +++ b/src/current/_includes/common/upgrade/patch-upgrade-kubernetes.md @@ -0,0 +1,125 @@ +
+ +To upgrade from one patch release to another within the same major version, perform the following steps on one node at a time: + +1. Change the container image in the custom resource: + + ~~~ + image: + name: cockroachdb/cockroach:{{page.release_info.version}} + ~~~ + +1. Apply the new settings to the cluster: + + {% include_cached copy-clipboard.html %} + ~~~ shell + kubectl apply -f example.yaml + ~~~ + + The Operator will perform the staged update. + +1. To check the status of the rolling upgrade, run `kubectl get pods`. +1. Verify that all pods have been upgraded: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl get pods \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' + ~~~ + + You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). + +
+ +
+ +1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl patch statefulset cockroachdb \ + -p='{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}' + ~~~ + + ~~~ + statefulset.apps/cockroachdb patched + ~~~ + +1. Change the container image in the StatefulSet: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl patch statefulset cockroachdb \ + --type='json' \ + -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:{{page.release_info.version}}"}]' + ~~~ + + ~~~ + statefulset.apps/cockroachdb patched + ~~~ + +1. To check the status of the rolling upgrade, run `kubectl get pods`. +1. Verify that all pods have been upgraded: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl get pods \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' + ~~~ + + You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). + +
+ +
+ +1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ helm upgrade \ + my-release \ + cockroachdb/cockroachdb \ + --set statefulset.updateStrategy.rollingUpdate.partition=2 + ~~~ + +1. Connect to the cluster using the SQL shell: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl exec -it cockroachdb-client-secure \ + -- ./cockroach sql \ + --certs-dir=/cockroach-certs \ + --host=my-release-cockroachdb-public + ~~~ + +1. Remove the cluster initialization job from when the cluster was created: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl delete job my-release-cockroachdb-init + ~~~ + +1. Change the container image in the StatefulSet: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ helm upgrade \ + my-release \ + cockroachdb/cockroachdb \ + --set image.tag={{page.release_info.version}} \ + --reuse-values + ~~~ + +1. To check the status of the rolling upgrade, run `kubectl get pods`. +1. Verify that all pods have been upgraded: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl get pods \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' + ~~~ + + You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). + +
diff --git a/src/current/_includes/common/upgrade/patch-upgrade-self-hosted.md b/src/current/_includes/common/upgrade/patch-upgrade-self-hosted.md index 5845e2f1b06..135554599b2 100644 --- a/src/current/_includes/common/upgrade/patch-upgrade-self-hosted.md +++ b/src/current/_includes/common/upgrade/patch-upgrade-self-hosted.md @@ -5,11 +5,11 @@ To upgrade from one patch release to another within the same major version, perf 1. Verify that the node has rejoined the cluster. 1. Ensure that the node is ready to accept a SQL connection. - Unless there are tens of thousands of ranges on the node, it's usually sufficient to wait one minute. To be certain that the node is ready, run the following command: + Unless there are tens of thousands of ranges on the node, it's usually sufficient to wait one minute. To be certain that the node is ready, run the following command: - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach sql -e 'select 1' - ~~~ + {% include_cached copy-clipboard.html %} + ~~~ shell + cockroach sql -e 'select 1' + ~~~ When all nodes are running the new patch version, the upgrade is complete. diff --git a/src/current/_includes/common/upgrade/prepare-to-upgrade-kubernetes.md b/src/current/_includes/common/upgrade/prepare-to-upgrade-kubernetes.md new file mode 100644 index 00000000000..5875f314b32 --- /dev/null +++ b/src/current/_includes/common/upgrade/prepare-to-upgrade-kubernetes.md @@ -0,0 +1,16 @@ +Before beginning a major-version or patch upgrade: + +1. Verify the overall health of your cluster using the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}): + - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or [decommission]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#remove-nodes) them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. + + If any node is not fully decommissioned, try the following: + 1. First, reissue the [decommission command]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#decommission-the-node). The second command typically succeeds within a few minutes. + 1. If the second decommission command does not succeed, [recommission]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#recommission-nodes) and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`. + - Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade. + - In the **Node List**, make sure all nodes are on the same version. Upgrade them to the cluster's current version before continuing. If any nodes are behind, this also indicates that the previous major-version upgrade may not be finalized. + - In the **Metrics** dashboards, make sure [CPU]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#cpu-usage), [memory]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#database-memory-usage), and [storage]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#storage-capacity) capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider [adding nodes]({% link {{ page.version.version }}/cockroach-start.md %}) to your cluster before beginning your upgrade. +1. Make sure your cluster is behind a [load balancer]({% link {{ page.version.version }}/recommended-production-settings.md %}#load-balancing), or your clients are configured to talk to multiple nodes. If your application communicates with only a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail. +1. {% include {{page.version.version}}/storage/compaction-concurrency.md %} +1. {% include {{page.version.version}}/backups/recommend-backups-for-upgrade.md %} Refer to [Restoring backups across versions]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version). +1. Review the [{{ page.version.version }} Release Notes]({% link releases/{{ page.version.version }}.md %}), as well as the release notes for any skipped major version. Pay careful attention to the sections for backward-incompatible changes, deprecations, changes to default cluster settings, and features that are not available until the upgrade is finalized. +1. Optionally [disable auto-finalization](#disable-auto-finalization) to preserve the ability to roll back a major-version upgrade instead of finalizing it. If auto-finalization is disabled, a major-version upgrade is not complete until it is finalized. diff --git a/src/current/_includes/common/upgrade/rollback-kubernetes.md b/src/current/_includes/common/upgrade/rollback-kubernetes.md new file mode 100644 index 00000000000..83efa02140c --- /dev/null +++ b/src/current/_includes/common/upgrade/rollback-kubernetes.md @@ -0,0 +1,30 @@ +To roll back to the previous major version before an upgrade is finalized: + +1. Change the container image in the custom resource to use the previous major version: + + ~~~ + image: + name: cockroachdb/cockroach:{{ page.version.version }} + ~~~ + +1. Apply the new settings to the cluster: + + {% include_cached copy-clipboard.html %} + ~~~ shell + kubectl apply -f example.yaml + ~~~ + + The Operator will perform the staged rollback. + +1. To check the status of the rollback, run `kubectl get pods`. +1. Verify that all pods have been rolled back: + + {% include_cached copy-clipboard.html %} + ~~~ shell + $ kubectl get pods \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' + ~~~ + + You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). + +1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade). When the rollback is finalized, the rollback is complete. diff --git a/src/current/_includes/v24.3/orchestration/kubernetes-upgrade-cluster-helm.md b/src/current/_includes/v24.3/orchestration/kubernetes-upgrade-cluster-helm.md deleted file mode 100644 index 4506ae28c74..00000000000 --- a/src/current/_includes/v24.3/orchestration/kubernetes-upgrade-cluster-helm.md +++ /dev/null @@ -1,259 +0,0 @@ -{% assign previous_version = site.data.versions | where_exp: "previous_version", "previous_version.major_version == page.version.version" | first | map: "previous_version" %} - -1. Verify that you can upgrade. - - To upgrade to a new major version, you must first be on a production release of the previous version. The release does not need to be the latest production release of the previous version, but it must be a production [release]({% link releases/index.md %}) and not a testing release (alpha/beta). - - Therefore, in order to upgrade to {{ page.version.version }}, you must be on a production release of {{ previous_version }}. - - 1. If you are upgrading to {{ page.version.version }} from a production release earlier than {{ previous_version }}, or from a testing release (alpha/beta), first [upgrade to a production release of {{ previous_version }}]({% link {{ previous_version }}/upgrade-cockroachdb-kubernetes.md %}?filters=helm). Be sure to complete all the steps. - - 1. Then return to this page and perform a second upgrade to {{ page.version.version }}. - - 1. If you are upgrading from any production release of {{ previous_version }}, or from any earlier {{ page.version.version }} patch release, you do not have to go through intermediate releases; continue to step 2. - -1. Verify the overall health of your cluster using the [DB Console]({% link {{ page.version.version }}/ui-overview.md %}). On the **Overview**: - - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as suspect or dead, identify why the nodes are offline and either restart them or [decommission](scale-cockroachdb-kubernetes.html?filters=helm#remove-nodes) them before beginning your upgrade. If there are dead and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually). - - Under **Replication Status**, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to [identify and resolve the cause of range under-replication and/or unavailability]({% link {{ page.version.version }}/cluster-setup-troubleshooting.md %}#replication-issues) before beginning your upgrade. - - In the **Node List**: - - Make sure all nodes are on the same version. If not all nodes are on the same version, upgrade them to the cluster's highest current version first, and then start this process over. - - Make sure capacity and memory usage are reasonable for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. Also go to **Metrics > Dashboard: Hardware** and make sure CPU percent is reasonable across the cluster. If there's not enough headroom on any of these metrics, consider [adding nodes](scale-cockroachdb-kubernetes.html?filters=helm#add-nodes) to your cluster before beginning your upgrade. - -{% assign rd = site.data.versions | where_exp: "rd", "rd.major_version == page.version.version" | first %} - -1. Review the [backward-incompatible changes in {{ page.version.version }}]({% link releases/{{ page.version.version }}.md %}{% unless rd.release_date == "N/A" or rd.release_date > today %}#{{ page.version.version | replace: ".", "-" }}-0-backward-incompatible-changes{% endunless %}) and [deprecated features]({% link releases/{{ page.version.version }}.md %}#{% unless rd.release_date == "N/A" or rd.release_date > today %}{{ page.version.version | replace: ".", "-" }}-0-deprecations{% endunless %}). If any affect your deployment, make the necessary changes before starting the rolling upgrade to {{ page.version.version }}. - -1. Decide how the upgrade will be finalized. - - By default, after all nodes are running the new version, the upgrade process will be **auto-finalized**. This will enable certain [features and performance improvements introduced in {{ page.version.version }}. After finalization, however, it will no longer be possible to perform a downgrade to {{ previous_version }}. In the event of a catastrophic failure or corruption, the only option is to start a new cluster using the old binary and then restore from a [backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}) created prior to the upgrade. For this reason, **we recommend disabling auto-finalization** so you can monitor the stability and performance of the upgraded cluster before finalizing the upgrade, but note that you will need to follow all of the subsequent directions, including the manual finalization in a later step. - - {{site.data.alerts.callout_info}} - Finalization only applies when performing a major version upgrade (for example, from {{ previous_version }}.x to {{ page.version.version }}). Patch version upgrades (for example, within the {{ page.version.version }}.x series) can always be downgraded. - {{site.data.alerts.end}} - - {% if page.secure == true %} - - 1. Get a shell into the pod with the `cockroach` binary created earlier and start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure \ - -- ./cockroach sql \ - --certs-dir=/cockroach-certs \ - --host=my-release-cockroachdb-public - ~~~ - - {% else %} - - 1. Launch a temporary interactive pod and start the [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}) inside it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl run cockroachdb -it \ - --image=cockroachdb/cockroach \ - --rm \ - --restart=Never \ - -- sql \ - --insecure \ - --host=my-release-cockroachdb-public - ~~~ - - {% endif %} - - 1. Set the `cluster.preserve_downgrade_option` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) to the version you are upgrading from: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.preserve_downgrade_option = '{{ previous_version | remove_first: "v" }}'; - ~~~ - - 1. Exit the SQL shell and delete the temporary pod: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ helm upgrade \ - my-release \ - cockroachdb/cockroachdb \ - --set statefulset.updateStrategy.rollingUpdate.partition=2 - ~~~ - -1. Kick off the upgrade process by changing the Docker image used in the CockroachDB StatefulSet: - - {{site.data.alerts.callout_info}} - For Helm, you must remove the cluster initialization job from when the cluster was created before the cluster version can be changed. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete job my-release-cockroachdb-init - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ helm upgrade \ - my-release \ - cockroachdb/cockroachdb \ - --set image.tag={{page.release_info.version}} \ - --reuse-values - ~~~ - -1. Check the status of your cluster's pods. You should see one of them being restarted: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - my-release-cockroachdb-0 1/1 Running 0 2m - my-release-cockroachdb-1 1/1 Running 0 3m - my-release-cockroachdb-2 0/1 ContainerCreating 0 25s - my-release-cockroachdb-init-nwjkh 0/1 ContainerCreating 0 6s - ... - ~~~ - - {{site.data.alerts.callout_info}} - Ignore the pod for cluster initialization. It is re-created as a byproduct of the StatefulSet configuration but does not impact your existing cluster. - {{site.data.alerts.end}} - -1. After the pod has been restarted with the new image, start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}): - - {% if page.secure == true %} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure \ - -- ./cockroach sql \ - --certs-dir=/cockroach-certs \ - --host=my-release-cockroachdb-public - ~~~ - - {% else %} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl run cockroachdb -it \ - --image=cockroachdb/cockroach \ - --rm \ - --restart=Never \ - -- sql \ - --insecure \ - --host=my-release-cockroachdb-public - ~~~ - {% endif %} - -1. Run the following SQL query to verify that the number of underreplicated ranges is zero: - - {% include_cached copy-clipboard.html %} - ~~~ sql - SELECT sum((metrics->>'ranges.underreplicated')::DECIMAL)::INT AS ranges_underreplicated FROM crdb_internal.kv_store_status; - ~~~ - - ~~~ - ranges_underreplicated - -------------------------- - 0 - (1 row) - ~~~ - - This indicates that it is safe to proceed to the next pod. - -1. Exit the SQL shell: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -1. Decrement the partition value by 1 to allow the next pod in the cluster to update: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ helm upgrade \ - my-release \ - cockroachdb/cockroachdb \ - --set statefulset.updateStrategy.rollingUpdate.partition=1 \ - ~~~ - -1. Repeat steps 4-8 until all pods have been restarted and are running the new image (the final partition value should be `0`). - -1. Check the image of each pod to confirm that all have been upgraded: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods \ - -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' - ~~~ - - ~~~ - my-release-cockroachdb-0 cockroachdb/cockroach:{{page.release_info.version}} - my-release-cockroachdb-1 cockroachdb/cockroach:{{page.release_info.version}} - my-release-cockroachdb-2 cockroachdb/cockroach:{{page.release_info.version}} - ... - ~~~ - - You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). - - -1. If you disabled auto-finalization earlier, monitor the stability and performance of your cluster until you are comfortable with the upgrade (generally at least a day). - - If you decide to roll back the upgrade, repeat the rolling restart procedure with the old binary. - - {{site.data.alerts.callout_info}} - This is only possible when performing a major version upgrade (for example, from {{ previous_version }}.x to {{ page.version.version }}). Patch version upgrades (for example, within the {{ page.version.version }}.x series) are auto-finalized. - {{site.data.alerts.end}} - - To finalize the upgrade, re-enable auto-finalization: - - {% if page.secure == true %} - - 1. Get a shell into the pod with the `cockroach` binary created earlier and start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure \ - -- ./cockroach sql \ - --certs-dir=/cockroach-certs \ - --host=my-release-cockroachdb-public - ~~~ - - {% else %} - - 1. Launch a temporary interactive pod and start the [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}) inside it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl run cockroachdb -it \ - --image=cockroachdb/cockroach \ - --rm \ - --restart=Never \ - -- sql \ - --insecure \ - --host=my-release-cockroachdb-public - ~~~ - - {% endif %} - - 1. Re-enable auto-finalization: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESET CLUSTER SETTING cluster.preserve_downgrade_option; - ~~~ - - After the upgrade to {{ page.version.version }} is finalized, you may notice an increase in compaction activity due to a background migration within the storage engine. To observe the migration's progress, check the **Compactions** section of the [Storage Dashboard]({% link {{ page.version.version }}/ui-storage-dashboard.md %}) in the DB Console or monitor the `storage.marked-for-compaction-files` [time-series metric]({% link {{ page.version.version }}/metrics.md %}). When the metric's value nears or reaches `0`, the migration is complete and compaction activity will return to normal levels. - - 1. Exit the SQL shell and delete the temporary pod: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ diff --git a/src/current/_includes/v24.3/orchestration/kubernetes-upgrade-cluster-manual.md b/src/current/_includes/v24.3/orchestration/kubernetes-upgrade-cluster-manual.md deleted file mode 100644 index 078858ee083..00000000000 --- a/src/current/_includes/v24.3/orchestration/kubernetes-upgrade-cluster-manual.md +++ /dev/null @@ -1,248 +0,0 @@ -{% assign previous_version = site.data.versions | where_exp: "previous_version", "previous_version.major_version == page.version.version" | first | map: "previous_version" %} - -1. Verify that you can upgrade. - - To upgrade to a new major version, you must first be on a production release of the previous version. The release does not need to be the latest production release of the previous version, but it must be a production [release]({% link releases/index.md %}) and not a testing release (alpha/beta). - - Therefore, in order to upgrade to {{ page.version.version }}, you must be on a production release of {{ previous_version }}. - - 1. If you are upgrading to {{ page.version.version }} from a production release earlier than {{ previous_version }}, or from a testing release (alpha/beta), first [upgrade to a production release of {{ previous_version }}]({% link {{ previous_version }}/upgrade-cockroachdb-kubernetes.md %}?filters=manual). Be sure to complete all the steps. - - 1. Then return to this page and perform a second upgrade to {{ page.version.version }}. - - 1. If you are upgrading from any production release of {{ previous_version }}, or from any earlier {{ page.version.version }} patch release, you do not have to go through intermediate releases; continue to step 2. - -1. Verify the overall health of your cluster using the [DB Console]({% link {{ page.version.version }}/ui-overview.md %}). On the **Overview**: - - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as suspect or dead, identify why the nodes are offline and either restart them or [decommission](scale-cockroachdb-kubernetes.html?filters=manual#remove-nodes) them before beginning your upgrade. If there are dead and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually). - - Under **Replication Status**, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to [identify and resolve the cause of range under-replication and/or unavailability]({% link {{ page.version.version }}/cluster-setup-troubleshooting.md %}#replication-issues) before beginning your upgrade. - - In the **Node List**: - - Make sure all nodes are on the same version. If not all nodes are on the same version, upgrade them to the cluster's highest current version first, and then start this process over. - - Make sure capacity and memory usage are reasonable for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. Also go to **Metrics > Dashboard: Hardware** and make sure CPU percent is reasonable across the cluster. If there's not enough headroom on any of these metrics, consider [adding nodes](scale-cockroachdb-kubernetes.html?filters=manual#add-nodes) to your cluster before beginning your upgrade. - -{% assign rd = site.data.versions | where_exp: "rd", "rd.major_version == page.version.version" | first %} - -1. Review the [backward-incompatible changes in {{ page.version.version }}]({% link releases/{{ page.version.version }}.md %}{% unless rd.release_date == "N/A" or rd.release_date > today %}#{{ page.version.version | replace: ".", "-" }}-0-backward-incompatible-changes{% endunless %}) and [deprecated features]({% link releases/{{ page.version.version }}.md %}#{% unless rd.release_date == "N/A" or rd.release_date > today %}{{ page.version.version | replace: ".", "-" }}-0-deprecations{% endunless %}). If any affect your deployment, make the necessary changes before starting the rolling upgrade to {{ page.version.version }}. - -1. Decide how the upgrade will be finalized. - - By default, after all nodes are running the new version, the upgrade process will be **auto-finalized**. This will enable certain [features and performance improvements introduced in {{ page.version.version }}. After finalization, however, it will no longer be possible to perform a downgrade to {{ previous_version }}. In the event of a catastrophic failure or corruption, the only option is to start a new cluster using the old binary and then restore from a [backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}) created prior to the upgrade. For this reason, **we recommend disabling auto-finalization** so you can monitor the stability and performance of the upgraded cluster before finalizing the upgrade, but note that you will need to follow all of the subsequent directions, including the manual finalization in a later step. - - {{site.data.alerts.callout_info}} - Finalization only applies when performing a major version upgrade (for example, from {{ previous_version }}.x to {{ page.version.version }}). Patch version upgrades (for example, within the {{ page.version.version }}.x series) can always be downgraded. - {{site.data.alerts.end}} - - {% if page.secure == true %} - - 1. Start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}). For example, if you followed the steps in [Deploy CockroachDB with Kubernetes](deploy-cockroachdb-with-kubernetes.html?filters=manual#step-3-use-the-built-in-sql-client) to launch a secure client pod, get a shell into the `cockroachdb-client-secure` pod: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure \-- ./cockroach sql \ - --certs-dir=/cockroach-certs \ - --host=cockroachdb-public - ~~~ - - {% else %} - - 1. Launch a temporary interactive pod and start the [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}) inside it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl run cockroachdb -it \ - --image=cockroachdb/cockroach \ - --rm \ - --restart=Never \ - -- sql \ - --insecure \ - --host=cockroachdb-public - ~~~ - - {% endif %} - - 1. Set the `cluster.preserve_downgrade_option` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) to the version you are upgrading from: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.preserve_downgrade_option = '{{ previous_version | remove_first: "v" }}'; - ~~~ - - 1. Exit the SQL shell and delete the temporary pod: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl patch statefulset cockroachdb \ - -p='{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}' - ~~~ - - ~~~ - statefulset.apps/cockroachdb patched - ~~~ - -1. Kick off the upgrade process by changing the Docker image used in the CockroachDB StatefulSet: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl patch statefulset cockroachdb \ - --type='json' \ - -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:{{page.release_info.version}}"}]' - ~~~ - - ~~~ - statefulset.apps/cockroachdb patched - ~~~ - -1. Check the status of your cluster's pods. You should see one of them being restarted: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - cockroachdb-0 1/1 Running 0 2m - cockroachdb-1 1/1 Running 0 2m - cockroachdb-2 0/1 Terminating 0 1m - ... - ~~~ - -1. After the pod has been restarted with the new image, start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}): - - {% if page.secure == true %} - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure \-- ./cockroach sql \ - --certs-dir=/cockroach-certs \ - --host=cockroachdb-public - ~~~ - - {% else %} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl run cockroachdb -it \ - --image=cockroachdb/cockroach \ - --rm \ - --restart=Never \ - -- sql \ - --insecure \ - --host=cockroachdb-public - ~~~ - - {% endif %} - -1. Run the following SQL query to verify that the number of underreplicated ranges is zero: - - {% include_cached copy-clipboard.html %} - ~~~ sql - SELECT sum((metrics->>'ranges.underreplicated')::DECIMAL)::INT AS ranges_underreplicated FROM crdb_internal.kv_store_status; - ~~~ - - ~~~ - ranges_underreplicated - -------------------------- - 0 - (1 row) - ~~~ - - This indicates that it is safe to proceed to the next pod. - -1. Exit the SQL shell: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -1. Decrement the partition value by 1 to allow the next pod in the cluster to update: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl patch statefulset cockroachdb \ - -p='{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":1}}}}' - ~~~ - - ~~~ - statefulset.apps/cockroachdb patched - ~~~ - -1. Repeat steps 4-8 until all pods have been restarted and are running the new image (the final partition value should be `0`). - -1. Check the image of each pod to confirm that all have been upgraded: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods \ - -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' - ~~~ - - ~~~ - cockroachdb-0 cockroachdb/cockroach:{{page.release_info.version}} - cockroachdb-1 cockroachdb/cockroach:{{page.release_info.version}} - cockroachdb-2 cockroachdb/cockroach:{{page.release_info.version}} - ... - ~~~ - - You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). - -1. If you disabled auto-finalization earlier, monitor the stability and performance of your cluster until you are comfortable with the upgrade (generally at least a day). - - If you decide to roll back the upgrade, repeat the rolling restart procedure with the old binary. - - {{site.data.alerts.callout_info}} - This is only possible when performing a major version upgrade (for example, from {{ previous_version }}.x to {{ page.version.version }}). Patch version upgrades (for example, within the {{ page.version.version }}.x series) are auto-finalized. - {{site.data.alerts.end}} - - To finalize the upgrade, re-enable auto-finalization: - - {% if page.secure == true %} - - 1. Start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure \ - -- ./cockroach sql \ - --certs-dir=/cockroach-certs \ - --host=cockroachdb-public - ~~~ - - {% else %} - - 1. Launch a temporary interactive pod and start the [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}) inside it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl run cockroachdb -it \ - --image=cockroachdb/cockroach \ - --rm \ - --restart=Never \ - -- sql \ - --insecure \ - --host=cockroachdb-public - ~~~ - - {% endif %} - - 1. Re-enable auto-finalization: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESET CLUSTER SETTING cluster.preserve_downgrade_option; - ~~~ - - After the upgrade to {{ page.version.version }} is finalized, you may notice an increase in compaction activity due to a background migration within the storage engine. To observe the migration's progress, check the **Compactions** section of the [Storage Dashboard]({% link {{ page.version.version }}/ui-storage-dashboard.md %}) in the DB Console or monitor the `storage.marked-for-compaction-files` [time-series metric]({% link {{ page.version.version }}/metrics.md %}). When the metric's value nears or reaches `0`, the migration is complete and compaction activity will return to normal levels. - - 1. Exit the SQL shell and delete the temporary pod: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ diff --git a/src/current/v24.3/upgrade-cockroachdb-kubernetes.md b/src/current/v24.3/upgrade-cockroachdb-kubernetes.md index e59a58e6918..cac8b5bc5c4 100644 --- a/src/current/v24.3/upgrade-cockroachdb-kubernetes.md +++ b/src/current/v24.3/upgrade-cockroachdb-kubernetes.md @@ -1,5 +1,5 @@ --- -title: Cluster Upgrades +title: Upgrade a cluster in Kubernetes summary: How to upgrade a secure 3-node CockroachDB cluster with Kubernetes. toc: true toc_not_nested: true @@ -7,15 +7,15 @@ secure: true docs_area: deploy --- -{% assign previous_version = site.data.versions | where_exp: "previous_version", "previous_version.major_version == page.version.version" | first | map: "previous_version" %} +This page shows how to upgrade a CockroachDB cluster that is [deployed on a Kubernetes cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-kubernetes.md %}). -{{site.data.alerts.callout_info}} -This article assumes you have already [deployed CockroachDB on a single Kubernetes cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-kubernetes.md %}). -{{site.data.alerts.end}} +## Overview + +{% include common/upgrade/overview.md %} -We strongly recommend that you regularly upgrade your CockroachDB version in order to pick up bug fixes, performance improvements, and new features. +On Kubernewtes, the upgrade is a [staged update](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) in which each pod's container image for CockroachDB is updated in a rolling fashion. The cluster remains available during the upgrade. -The upgrade process on Kubernetes is a [staged update](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) in which the Docker image is applied to the pods one at a time, with each pod being stopped and restarted in turn. This is to ensure that the cluster remains available during the upgrade. +Select the cluster's deployment method to continue.
@@ -23,113 +23,39 @@ The upgrade process on Kubernetes is a [staged update](https://kubernetes.io/doc
-
-{% include {{ page.version.version }}/orchestration/operator-check-namespace.md %} +## Before you begin {{site.data.alerts.callout_success}} If you [deployed CockroachDB on Red Hat OpenShift]({% link {{ page.version.version }}/deploy-cockroachdb-with-kubernetes-openshift.md %}), substitute `kubectl` with `oc` in the following commands. {{site.data.alerts.end}} -1. Verify that you can upgrade. - - To upgrade to a new major version, you must first be on a production release of the previous version. The release does not need to be the latest production release of the previous version, but it must be a production [release]({% link releases/index.md %}) and not a testing release (alpha/beta). - - Therefore, in order to upgrade to {{ page.version.version }}, you must be on a production release of {{ previous_version }}. - - 1. If you are upgrading to {{ page.version.version }} from a production release earlier than {{ previous_version }}, or from a testing release (alpha/beta), first [upgrade to a production release of {{ previous_version }}]({% link {{ previous_version }}/upgrade-cockroachdb-kubernetes.md %}). Be sure to complete all the steps. - - 1. Then return to this page and perform a second upgrade to {{ page.version.version }}. - - 1. If you are upgrading from a production release of {{ previous_version }}, or from any earlier {{ page.version.version }} patch release, you do not have to go through intermediate releases; continue to step 2. - -1. Verify the overall health of your cluster using the [DB Console]({% link {{ page.version.version }}/ui-overview.md %}). On the **Overview**: - - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as suspect or dead, identify why the nodes are offline and either restart them or [decommission]({% link {{ page.version.version }}/scale-cockroachdb-kubernetes.md %}#remove-nodes) them before beginning your upgrade. If there are dead and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually). - - Under **Replication Status**, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to [identify and resolve the cause of range under-replication and/or unavailability]({% link {{ page.version.version }}/cluster-setup-troubleshooting.md %}#replication-issues) before beginning your upgrade. - - In the **Node List**: - - Make sure all nodes are on the same version. If not all nodes are on the same version, upgrade them to the cluster's highest current version first, and then start this process over. - - Make sure capacity and memory usage are reasonable for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. Also go to **Metrics > Dashboard: Hardware** and make sure CPU percent is reasonable across the cluster. If there's not enough headroom on any of these metrics, consider [adding nodes]({% link {{ page.version.version }}/scale-cockroachdb-kubernetes.md %}#add-nodes) to your cluster before beginning your upgrade. - -{% assign rd = site.data.versions | where_exp: "rd", "rd.major_version == page.version.version" | first %} - -1. Review the [backward-incompatible changes in {{ page.version.version }}]({% link releases/{{ page.version.version }}.md %}{% unless rd.release_date == "N/A" or rd.release_date > today %}#{{ page.version.version | replace: ".", "-" }}-0-backward-incompatible-changes{% endunless %}) and [deprecated features]({% link releases/{{ page.version.version }}.md %}#{% unless rd.release_date == "N/A" or rd.release_date > today %}{{ page.version.version | replace: ".", "-" }}-0-deprecations{% endunless %}). If any affect your deployment, make the necessary changes before starting the rolling upgrade to {{ page.version.version }}. - -1. Change the desired Docker image in the custom resource: - - ~~~ - image: - name: cockroachdb/cockroach:{{page.release_info.version}} - ~~~ - -1. Apply the new settings to the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl apply -f example.yaml - ~~~ - - The Operator will perform the staged update. - - {{site.data.alerts.callout_info}} - The Operator automatically sets the `cluster.preserve_downgrade_option` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) to the version you are upgrading from. This disables auto-finalization of the upgrade so that you can monitor the stability and performance of the upgraded cluster before manually finalizing the upgrade. This will enable certain features and performance improvements introduced in {{ page.version.version }}. - - Note that after finalization, it will no longer be possible to perform a downgrade to {{ previous_version }}. In the event of a catastrophic failure or corruption, the only option will be to start a new cluster using the previous binary and then restore from a [backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}) created prior to performing the upgrade. - - Finalization only applies when performing a major version upgrade (for example, from {{ previous_version }}.x to {{ page.version.version }}). Patch version upgrades (for example, within the {{ page.version.version }}.x series) can always be downgraded. - {{site.data.alerts.end}} - -1. To check the status of the rolling upgrade, run `kubectl get pods`. The pods are restarted one at a time with the new image. - -1. Verify that all pods have been upgraded by running: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods \ - -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' - ~~~ - - You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). +{% include {{ page.version.version }}/orchestration/operator-check-namespace.md %} +{% include common/upgrade/prepare-to-upgrade-self-hosted.md %} -1. Monitor the stability and performance of your cluster until you are comfortable with the upgrade (generally at least a day). +## Perform a patch upgrade - If you decide to roll back the upgrade, revert the image name in the custom resource and apply the new value. +{% include_cached common/upgrade/patch-upgrade-kubernetes.md %} - {{site.data.alerts.callout_info}} - This is only possible when performing a major version upgrade (for example, from {{ previous_version }}.x to {{ page.version.version }}). Patch version upgrades (for example, within the {{ page.version.version }}.x series) are auto-finalized. - {{site.data.alerts.end}} +### Roll back a patch upgrade - To finalize the upgrade, re-enable auto-finalization: +{% include_cached common/upgrade/patch-rollback-kubernetes.md %} - 1. Start the CockroachDB [built-in SQL client]({% link {{ page.version.version }}/cockroach-sql.md %}). For example, if you followed the steps in [Deploy CockroachDB with Kubernetes]({% link {{ page.version.version }}/deploy-cockroachdb-with-kubernetes.md %}#step-3-use-the-built-in-sql-client) to launch a secure client pod, get a shell into the `cockroachdb-client-secure` pod: +## Perform a major-version upgrade - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure \-- ./cockroach sql \ - --certs-dir=/cockroach/cockroach-certs \ - --host={cluster-name}-public - ~~~ +{% include common/upgrade/major-version-upgrade-kubernetes.md %} - 1. Re-enable auto-finalization: +### Finalize a major-version upgrade - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESET CLUSTER SETTING cluster.preserve_downgrade_option; - ~~~ +{% include common/upgrade/finalize-kubernetes.md %} - After the upgrade to {{ page.version.version }} is finalized, you may notice an increase in [compaction]({% link {{ page.version.version }}/architecture/storage-layer.md %}#compaction) activity due to a background migration within the storage engine. To observe the migration's progress, check the **Compactions** section of the [Storage Dashboard]({% link {{ page.version.version }}/ui-storage-dashboard.md %}) in the DB Console or monitor the `storage.marked-for-compaction-files` [time-series metric]({% link {{ page.version.version }}/metrics.md %}). When the metric's value nears or reaches `0`, the migration is complete and compaction activity will return to normal levels. {% include {{page.version.version}}/storage/compaction-concurrency.md %} +### Roll back a major-version upgrade - 1. Exit the SQL shell and pod: +{% include common/upgrade/rollback-kubernetes.md %} - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ +## Disable auto-finalization -
+## Troubleshooting -
-{% include {{ page.version.version }}/orchestration/kubernetes-upgrade-cluster-manual.md %} -
+{% include common/upgrade/troubleshooting-self-hosted.md %} -
-{% include {{ page.version.version }}/orchestration/kubernetes-upgrade-cluster-helm.md %} -
+## See also From 5196b90bcd487e7e6086dbb33e20df0b32e21122 Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Tue, 5 Nov 2024 16:27:53 -0800 Subject: [PATCH 03/19] Fixes --- .../common/upgrade/disable-auto-finalization.md | 8 ++++---- .../common/upgrade/major-version-upgrade-cloud.md | 2 +- .../upgrade/major-version-upgrade-kubernetes.md | 6 +++--- .../upgrade/major-version-upgrade-self-hosted.md | 2 +- .../common/upgrade/prepare-to-upgrade-cloud.md | 2 +- .../_includes/common/upgrade/rollback-cloud.md | 2 +- .../_includes/common/upgrade/rollback-kubernetes.md | 2 +- .../_includes/common/upgrade/rollback-self-hosted.md | 2 +- .../_includes/common/upgrade/upgrade-high-level.md | 2 +- .../_includes/releases/v24.3/upgrade-finalization.md | 4 ++-- .../v24.2/backups/recommend-backups-for-upgrade.md | 10 +++++++++- ...pgrade_requirements.md => upgrade-requirements.md} | 0 .../cockroachcloud/upgrade-cockroach-version.md | 8 ++++---- src/current/releases/v24.3.md | 10 ++++++++++ src/current/v24.2/api-support-policy.md | 2 +- src/current/v24.3/api-support-policy.md | 2 +- src/current/v24.3/cluster-virtualization-overview.md | 6 +++++- src/current/v24.3/install-cockroachdb-mac.md | 2 +- .../v24.3/physical-cluster-replication-overview.md | 10 +++++----- src/current/v24.3/restore.md | 2 +- src/current/v24.3/show-jobs.md | 2 +- src/current/v24.3/upgrade-cockroach-version.md | 11 +++++------ src/current/v24.3/upgrade-cockroachdb-kubernetes.md | 2 +- src/current/v24.3/work-with-virtual-clusters.md | 6 +++--- 24 files changed, 63 insertions(+), 42 deletions(-) rename src/current/_includes/v24.2/{upgrade_requirements.md => upgrade-requirements.md} (100%) diff --git a/src/current/_includes/common/upgrade/disable-auto-finalization.md b/src/current/_includes/common/upgrade/disable-auto-finalization.md index 67e26aa4cec..20d9d82695a 100644 --- a/src/current/_includes/common/upgrade/disable-auto-finalization.md +++ b/src/current/_includes/common/upgrade/disable-auto-finalization.md @@ -1,7 +1,7 @@ {% if page.path contains 'kubernetes' %}
-For clusters managed by the Operator, auto-finalization is disabled and cannot be enabled. A major version upgrade is not complete until it is manually [finalized](#finalize-a-major-version-upgrade). +For clusters managed by the Operator, auto-finalization is disabled and cannot be enabled. A major version upgrade is not complete until it is manually [finalized](#finalize-a-major-version-upgrade-manually).
@@ -21,7 +21,7 @@ To disable auto-finalization: 1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version. -Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade) or [roll it back]({#roll-back-a-major-version-upgrade). +Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade). {{site.data.alerts.callout_info}} Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. @@ -48,7 +48,7 @@ To disable auto-finalization: 1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version. -Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade) or [roll it back]({#roll-back-a-major-version-upgrade). +Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade). {{site.data.alerts.callout_info}} Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. @@ -73,7 +73,7 @@ To disable auto-finalization: 1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.auto_upgrade.enabled` to `false`. -Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade) or [roll it back]({#roll-back-a-major-version-upgrade). +Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade). {{site.data.alerts.callout_info}} Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. diff --git a/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md b/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md index d46cad338f3..8c755d140dc 100644 --- a/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md +++ b/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md @@ -18,4 +18,4 @@ This section shows how to perform a major-version upgrade for a cluster in Cockr 1. A CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster remains fully available while it is upgraded. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are upgraded one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node {{ site.data.products.advanced }} cluster will be unavailable while the cluster is upgraded and restarted. If necessary, prepare for the upgrade by communicating ahead of time with your users, and plan to begin the upgrade during a time when the impact on the cluster's workload will be minimized. 1. To begin a major-version upgrade, go back to the dialog in the CockroachDB {{ site.data.products.cloud }} Console. If multiple upgrades are available, select the desired version. Review the details, then click **Start upgrade**. The dialog closes, and the **Version** columnin the **Cluster List** page changes to **Upgrading**. When the upgrade has finished but has not yet been finalized, the **Version** column reports that the new version is **pending**. -The upgrade is not complete until it is [finalized](#finalize-a-major-version-upgrade). After the upgrade is finalized, the cluster can no longer be [rolled back](#roll-back-a-major-version-upgrade) to its previous major version. If you take no action, finalization occurs automatically after approximately 72 hours. +The upgrade is not complete until it is [finalized](#finalize-a-major-version-upgrade-manually). After the upgrade is finalized, the cluster can no longer be [rolled back](#roll-back-a-major-version-upgrade) to its previous major version. If you take no action, finalization occurs automatically after approximately 72 hours. diff --git a/src/current/_includes/common/upgrade/major-version-upgrade-kubernetes.md b/src/current/_includes/common/upgrade/major-version-upgrade-kubernetes.md index d8cbeb4717e..7b61fbab657 100644 --- a/src/current/_includes/common/upgrade/major-version-upgrade-kubernetes.md +++ b/src/current/_includes/common/upgrade/major-version-upgrade-kubernetes.md @@ -29,7 +29,7 @@ To perform a major upgrade: You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). -1. Before beginning a major-version upgrade, the Operator disables auto-finalization by setting the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version. Before finalizing an upgrade, follow your organization's testing procedures to decide whether to [finalize](#finalize-a-major-version-upgrade) or [roll back](#roll-back-a-major-version-upgrade) the upgrade. After finalization begins, you can no longer roll back to the cluster's previous major version. +1. Before beginning a major-version upgrade, the Operator disables auto-finalization by setting the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version. Before finalizing an upgrade, follow your organization's testing procedures to decide whether to [finalize](#finalize-a-major-version-upgrade-manually) or [roll back](#roll-back-a-major-version-upgrade) the upgrade. After finalization begins, you can no longer roll back to the cluster's previous major version.
@@ -124,7 +124,7 @@ To perform a major upgrade: ~~~ You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). -1. If auto-finalization is disabled, the upgrade is not complete until you [finalize the upgrade](#finalize-a-major-version-upgrade). +1. If auto-finalization is disabled, the upgrade is not complete until you [finalize the upgrade](#finalize-a-major-version-upgrade-manually). @@ -259,6 +259,6 @@ To perform a major upgrade: You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). -1. If auto-finalization is disabled, the upgrade is not complete until you [finalize the upgrade](#finalize-a-major-version-upgrade). +1. If auto-finalization is disabled, the upgrade is not complete until you [finalize the upgrade](#finalize-a-major-version-upgrade-manually). diff --git a/src/current/_includes/common/upgrade/major-version-upgrade-self-hosted.md b/src/current/_includes/common/upgrade/major-version-upgrade-self-hosted.md index a0914fcff62..0a592666068 100644 --- a/src/current/_includes/common/upgrade/major-version-upgrade-self-hosted.md +++ b/src/current/_includes/common/upgrade/major-version-upgrade-self-hosted.md @@ -12,4 +12,4 @@ To perform a major upgrade: cockroach sql -e 'select 1' ~~~ 1. If auto-finalization is enabled (the default), finalization begins as soon as the last node rejoins the cluster with the new binary. When finalization finishes, the upgrade is complete. -1. If auto-finalization is disabled, follow your organization's testing procedures to decide whether to [finalize the upgrade](#finalize-a-major-version-upgrade) or [roll back](#roll-back-a-major-version-upgrade) the upgrade. After finalization begins, you can no longer roll back to the cluster's previous major version. +1. If auto-finalization is disabled, follow your organization's testing procedures to decide whether to [finalize the upgrade](#finalize-a-major-version-upgrade-manually) or [roll back](#roll-back-a-major-version-upgrade) the upgrade. After finalization begins, you can no longer roll back to the cluster's previous major version. diff --git a/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md b/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md index 8cdd0ab402d..ccb74c90ac0 100644 --- a/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md +++ b/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md @@ -1,6 +1,6 @@ Before beginning a major-version upgrade: -1. **CockroachDB {{ site.data.products.advanced }}**: Verify the overall health of your cluster using the [DB Console]({% link cockroachcloud/tools-page#access-the-db-console.md %}): +1. **CockroachDB {{ site.data.products.advanced }}**: Verify the overall health of your cluster using the [DB Console]({% link cockroachcloud/tools-page.md %}#access-the-db-console): - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or [decommission]({% link {{ site.current_cloud_version }}/node-shutdown.md %}?filters=decommission#remove-nodes) them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. If any node is not fully decommissioned, try the following: diff --git a/src/current/_includes/common/upgrade/rollback-cloud.md b/src/current/_includes/common/upgrade/rollback-cloud.md index 5e612b9061c..470b69262cb 100644 --- a/src/current/_includes/common/upgrade/rollback-cloud.md +++ b/src/current/_includes/common/upgrade/rollback-cloud.md @@ -4,4 +4,4 @@ To roll back to the previous major version before the upgrade is finalized autom 1. At the top of the page, click **Roll back**. A CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster remains fully available while it is rolled back. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are rolled back one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node CockroachDB {{ site.data.products.advanced }} cluster will be unavailable while the cluster is rolled back and restarted. Like an upgrade, a rollback must be finalized. -1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-versino-upgrade). When the rollback is finalized, the rollback is complete. +1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade-manually). When the rollback is finalized, the rollback is complete. diff --git a/src/current/_includes/common/upgrade/rollback-kubernetes.md b/src/current/_includes/common/upgrade/rollback-kubernetes.md index 83efa02140c..9955b029030 100644 --- a/src/current/_includes/common/upgrade/rollback-kubernetes.md +++ b/src/current/_includes/common/upgrade/rollback-kubernetes.md @@ -27,4 +27,4 @@ To roll back to the previous major version before an upgrade is finalized: You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). -1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade). When the rollback is finalized, the rollback is complete. +1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade-manually). When the rollback is finalized, the rollback is complete. diff --git a/src/current/_includes/common/upgrade/rollback-self-hosted.md b/src/current/_includes/common/upgrade/rollback-self-hosted.md index e754d8bd068..7049dd86fde 100644 --- a/src/current/_includes/common/upgrade/rollback-self-hosted.md +++ b/src/current/_includes/common/upgrade/rollback-self-hosted.md @@ -1,4 +1,4 @@ To roll back a major-version upgrade that has not been finalized: 1. Follow the steps to [perform a major-version upgrade](#perform-a-major-version-upgrade), replacing the `cockroach` binary on each node with binary for the previous major version. -1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade). When the rollback is finalized, the rollback is complete. +1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade-manually). When the rollback is finalized, the rollback is complete. diff --git a/src/current/_includes/common/upgrade/upgrade-high-level.md b/src/current/_includes/common/upgrade/upgrade-high-level.md index d6dfb480494..93abf11c210 100644 --- a/src/current/_includes/common/upgrade/upgrade-high-level.md +++ b/src/current/_includes/common/upgrade/upgrade-high-level.md @@ -5,4 +5,4 @@ At a high level, a cluster upgrade involves the following steps, which are detai 1. Restart the `cockroach` process and verify that the node has rejoined the cluster. 1. When all nodes have rejoined the cluster: 1. For a patch upgrade within the same major version, the upgrade is complete. - 1. For a major-version upgrade, the upgrade is not complete until it is [finalized](#finalize-a-major-version-upgrade). Auto-finalization is enabled by default, and begins as soon as all nodes have rejoined the cluster using the new binary. If you need the ability to [roll back a major-version upgrade](#roll-back-a-major-version-upgrade), you can disable auto-finalization and finalize the upgrade manually. Certain features and performance improvements, such as those that require changes to system schemas or objects, are not available until the upgrade is finalized. Refer to the {% if page.path contains 'cockroachcloud' %} [{{ site.current_cloud_version }} Release Notes]({% link releases/{{ page.version.version }}.md %}) {% else %}[{{ page.version.version }} Release Notes]({% link releases/{{ page.version.version }}.md %}).{% endif %}. + 1. For a major-version upgrade, the upgrade is not complete until it is [finalized](#finalize-a-major-version-upgrade-manually). Auto-finalization is enabled by default, and begins as soon as all nodes have rejoined the cluster using the new binary. If you need the ability to [roll back a major-version upgrade](#roll-back-a-major-version-upgrade), you can disable auto-finalization and finalize the upgrade manually. Certain features and performance improvements, such as those that require changes to system schemas or objects, are not available until the upgrade is finalized. Refer to the {% if page.path contains 'cockroachcloud' %} [{{ site.current_cloud_version }} Release Notes]({% link releases/{{ site.current_cloud_version }}.md %}){% else %}[{{ page.version.version }} Release Notes]({% link releases/{{ page.version.version }}.md %}).{% endif %}. diff --git a/src/current/_includes/releases/v24.3/upgrade-finalization.md b/src/current/_includes/releases/v24.3/upgrade-finalization.md index 0cdad527595..4cd49187ca0 100644 --- a/src/current/_includes/releases/v24.3/upgrade-finalization.md +++ b/src/current/_includes/releases/v24.3/upgrade-finalization.md @@ -1,4 +1,4 @@ During a major-version upgrade, certain features and performance improvements may not be available until the upgrade is finalized. -- A cluster must have an [Enterprise license]({% link {{ page.version.version }}/licensing-faqs.md %}#set-a-license) or a [trial license]({% link {{ page.version.version }}/licensing-faqs.md %}#obtain-a-license) set before an upgrade to {{ page.version.version }} can be finalized. -- New clusters that are initialized for the first time on {{ page.version.version }}, and clusters that are upgraded to {{ page.version.version }} will now have a [zone config]({% link {{ page.version.version }}/configure-zone.md %}) defined for the `timeseries` range if it does not already exist, which specifies the value for `gc.ttlseconds`, but inherits all other attributes from the zone config for the `default` range. +- A cluster must have an [Enterprise license]({% link v24.3/licensing-faqs.md %}#set-a-license) or a [trial license]({% link v24.3/licensing-faqs.md %}#obtain-a-license) set before an upgrade to v24.3 can be finalized. +- New clusters that are initialized for the first time on v24.3, and clusters that are upgraded to v24.3 will now have a [zone config]({% link v24.3/configure-replication-zones.md %}) defined for the `timeseries` range if it does not already exist, which specifies the value for `gc.ttlseconds`, but inherits all other attributes from the zone config for the `default` range. diff --git a/src/current/_includes/v24.2/backups/recommend-backups-for-upgrade.md b/src/current/_includes/v24.2/backups/recommend-backups-for-upgrade.md index 6cb563feab9..fe093883dad 100644 --- a/src/current/_includes/v24.2/backups/recommend-backups-for-upgrade.md +++ b/src/current/_includes/v24.2/backups/recommend-backups-for-upgrade.md @@ -1 +1,9 @@ -Because CockroachDB is designed with high fault tolerance, backups are primarily needed for [disaster recovery]({% link {{ page.version.version }}/disaster-recovery-planning.md %}). However, taking regular backups of your data is an operational best practice. When upgrading to a major release, **we recommend [taking a backup]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) of your cluster**. \ No newline at end of file +{% if page.path contains "cockroachcloud" %} +[Managed backups]({% link cockroachcloud/managed-backups.md %}) are automated backups of CockroachDB {{ site.data.products.cloud }} clusters that are stored by Cockroach Labs in cloud storage. By default, Cockroach Labs takes and retains managed backups in all Cloud clusters. + +When upgrading to a major release, you can optionally [take a self-managed backup]({% link cockroachcloud/take-and-restore-self-managed-backups.md %}) of your cluster to your own cloud storage, as an extra layer of protection in case the upgrade leads to issues. +{% else %} +CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for [disaster recovery]({% link {{ page.version.version }}/disaster-recovery-planning.md %}) planning. + +When upgrading to a new major version, **we recommend [taking a self-managed backup]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) of your cluster** so that you can [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version) if the upgrade leads to issues. +{% endif %} diff --git a/src/current/_includes/v24.2/upgrade_requirements.md b/src/current/_includes/v24.2/upgrade-requirements.md similarity index 100% rename from src/current/_includes/v24.2/upgrade_requirements.md rename to src/current/_includes/v24.2/upgrade-requirements.md diff --git a/src/current/cockroachcloud/upgrade-cockroach-version.md b/src/current/cockroachcloud/upgrade-cockroach-version.md index 483718bef3e..ff859c030c6 100644 --- a/src/current/cockroachcloud/upgrade-cockroach-version.md +++ b/src/current/cockroachcloud/upgrade-cockroach-version.md @@ -11,7 +11,7 @@ This page describes how major-version and patch upgrades work and shows how to u {% include common/upgrade/overview.md %} -{% include_cached {{ site.current_cloud_version }}/upgrade_requirements.md %} +{% include_cached {{ site.current_cloud_version }}/upgrade-requirements.md %} {% include common/upgrade/upgrade-high-level.md %} @@ -37,11 +37,11 @@ Refer to [Cloud Upgrade Policy]({% link cockroachcloud/upgrade-policy.md %}). ### Finalize a major-version upgrade manually -{% include_cached finalize-cloud.md %} +{% include_cached common/upgrade/finalize-cloud.md %} ### Roll back a major-version upgrade -{% include_cached rollback-cloud.md %} +{% include_cached common/upgrade/rollback-cloud.md %} ## Troubleshooting @@ -49,4 +49,4 @@ Refer to [Cloud Upgrade Policy]({% link cockroachcloud/upgrade-policy.md %}). ## See also -{% include see-also-cloud.md %} +{% include common/upgrade/see-also-cloud.md %} diff --git a/src/current/releases/v24.3.md b/src/current/releases/v24.3.md index 5090e6777e8..6830d5a321d 100644 --- a/src/current/releases/v24.3.md +++ b/src/current/releases/v24.3.md @@ -23,6 +23,16 @@ docs_area: releases {% include releases/whats-new-intro.md major_version=vers %} +{% comment %}Remove this block at GA and add the content to v24.3.0.md instead{% endcomment %} +

Backward-incompatible changes and deprecations

+ +Until v24.3 is Generally Available, review the backward-incompatible changes and deprecations listed in each testing release. + +{% comment %}Remove this block at GA and include the content in v24.3.0.md instead{% endcomment %} +

Features that require upgrade finalization

+ +{% include releases/v24.3/upgrade-finalization.md %} + {% for r in rel %} {% include releases/{{ page.major_version }}/{{ r.release_name }}.md release=r.release_name %} {% endfor %} diff --git a/src/current/v24.2/api-support-policy.md b/src/current/v24.2/api-support-policy.md index 81aca6fad8e..be770f94dfa 100644 --- a/src/current/v24.2/api-support-policy.md +++ b/src/current/v24.2/api-support-policy.md @@ -23,7 +23,7 @@ This page includes the following information: | Unstable | Supported for consumption by humans. Not supported for automation. | [Backward-incompatible changes](#backward-incompatible-changes) may be introduced in new major and patch versions. | | Reserved | Intended for use by CockroachDB developers. Not supported for public use. | N/A | -Backward-incompatible changes to **stable APIs** are highlighted in the [release notes]({% link releases/index.md %}#production-releases) for major CockroachDB versions. Users are asked to [consider backward-incompatible changes before upgrading]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#review-breaking-changes) to a new CockroachDB version. +Backward-incompatible changes to **stable APIs** are highlighted in the [release notes]({% link releases/index.md %}) for major CockroachDB versions. Users are asked to consider backward-incompatible changes before [upgrading]({% link releases/{{ page.version.version}}.md %}) to a new CockroachDB version. ### Backward-incompatible changes diff --git a/src/current/v24.3/api-support-policy.md b/src/current/v24.3/api-support-policy.md index 81aca6fad8e..be770f94dfa 100644 --- a/src/current/v24.3/api-support-policy.md +++ b/src/current/v24.3/api-support-policy.md @@ -23,7 +23,7 @@ This page includes the following information: | Unstable | Supported for consumption by humans. Not supported for automation. | [Backward-incompatible changes](#backward-incompatible-changes) may be introduced in new major and patch versions. | | Reserved | Intended for use by CockroachDB developers. Not supported for public use. | N/A | -Backward-incompatible changes to **stable APIs** are highlighted in the [release notes]({% link releases/index.md %}#production-releases) for major CockroachDB versions. Users are asked to [consider backward-incompatible changes before upgrading]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#review-breaking-changes) to a new CockroachDB version. +Backward-incompatible changes to **stable APIs** are highlighted in the [release notes]({% link releases/index.md %}) for major CockroachDB versions. Users are asked to consider backward-incompatible changes before [upgrading]({% link releases/{{ page.version.version}}.md %}) to a new CockroachDB version. ### Backward-incompatible changes diff --git a/src/current/v24.3/cluster-virtualization-overview.md b/src/current/v24.3/cluster-virtualization-overview.md index bf7a85c3c0f..681a3cc895e 100644 --- a/src/current/v24.3/cluster-virtualization-overview.md +++ b/src/current/v24.3/cluster-virtualization-overview.md @@ -48,7 +48,11 @@ To grant access to the system virtual cluster, you must connect to the system vi ### Upgrades -To upgrade to a new major version when cluster virtualization is enabled, you must first replace the binary on each node and restart the node, [finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade) the upgrade on the system virtual cluster to upgrade it, then finalize the upgrade on a virtual cluster to upgrade it. This allows you to roll back an upgrade of the system virtual cluster without impacting schemas or data in virtual clusters. The system virtual cluster can be at most one major version ahead of virtual clusters. For example, a system virtual cluster on CockroachDB v24.1 can have virtual clusters on CockroachDB v23.2. +To upgrade to a new major version when cluster virtualization is enabled, you must: + +1. Replace the binary on each node and restart the node. +1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade on the system virtual cluster to upgrade it (if auto-finalization is disabled). +1. Finalize the upgrade on a virtual cluster to upgrade it (if auto-finalization is disabled). This allows you to roll back an upgrade of the system virtual cluster without impacting schemas or data in virtual clusters. The system virtual cluster can be at most one major version ahead of virtual clusters. For example, a system virtual cluster on CockroachDB v24.1 can have virtual clusters on CockroachDB v23.2. For details, refer to [Work with virtual clusters]({% link {{ page.version.version }}/work-with-virtual-clusters.md %}#upgrade-a-cluster). diff --git a/src/current/v24.3/install-cockroachdb-mac.md b/src/current/v24.3/install-cockroachdb-mac.md index 181fa386316..ef3894723cc 100644 --- a/src/current/v24.3/install-cockroachdb-mac.md +++ b/src/current/v24.3/install-cockroachdb-mac.md @@ -50,7 +50,7 @@ Use one of the options below to install CockroachDB. To upgrade an existing clus {{site.data.alerts.callout_success}} -If you previously installed CockroachDB via Homebrew, you can [upgrade]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}) the CockroachDB binary to the next major version or to a patch version using HomeBrew. After updating the binary on each node, restart the `cockroach` process on the node. When upgrading to a new major version, you must complete additional steps to [finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade) the upgrade. If you need to upgrade through multiple major versions, you must complete each major-version upgrade separately, including finalizing the upgrade, before beginning the next one. +If you previously installed CockroachDB via Homebrew, you can [upgrade]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}) the CockroachDB binary to the next major version or to a patch version using HomeBrew. After updating the binary on each node, restart the `cockroach` process on the node. When upgrading to a new major version, you must complete additional steps to [finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade. If you need to upgrade through multiple major versions, you must complete each major-version upgrade separately, including finalizing the upgrade, before beginning the next one. Before starting the upgrade, review the [release notes]({% link releases/{{ page.version.version }}.md %}), including temporary limitations during the upgrade. diff --git a/src/current/v24.3/physical-cluster-replication-overview.md b/src/current/v24.3/physical-cluster-replication-overview.md index d710c58e959..5ca202bedf9 100644 --- a/src/current/v24.3/physical-cluster-replication-overview.md +++ b/src/current/v24.3/physical-cluster-replication-overview.md @@ -142,11 +142,11 @@ The standby cluster must be at the same version as, or one version ahead of, the When PCR is enabled, upgrade with the following procedure. This upgrades the standby cluster before the primary cluster. Within the primary and standby CockroachDB clusters, the system virtual cluster must be at a cluster version greater than or equal to the virtual cluster: -1. [Upgrade the binaries]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-4-perform-the-rolling-upgrade) on the primary and standby clusters. Replace the binary on each node of the cluster and restart the node. -1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade) the upgrade on the standby's system virtual cluster. -1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade) the upgrade on the primary's system virtual cluster. -1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade) the upgrade on the standby's virtual cluster. -1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade) the upgrade on the primary's virtual cluster. +1. [Upgrade the binaries]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#perform-a-major-version-upgrade) on the primary and standby clusters. Replace the binary on each node of the cluster and restart the node. +1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade on the standby's system virtual cluster if auto-finalization is disabled. +1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade on the primary's system virtual cluster if auto-finalization is disabled. +1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade on the standby's virtual cluster. +1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade on the primary's virtual cluster. The standby cluster must be at the same version as, or one version ahead of, the primary's virtual cluster at the time of [cutover]({% link {{ page.version.version }}/cutover-replication.md %}). diff --git a/src/current/v24.3/restore.md b/src/current/v24.3/restore.md index e1d3ad489fb..a997bb47605 100644 --- a/src/current/v24.3/restore.md +++ b/src/current/v24.3/restore.md @@ -177,7 +177,7 @@ Also, consider that: - [Temporary tables]({% link {{ page.version.version }}/temporary-tables.md %}) will be restored to their original database during a full cluster restore. - The restore will drop the cluster's `defaultdb` and `postgres` [pre-loaded databases]({% link {{ page.version.version }}/show-databases.md %}#preloaded-databases) before the restore begins. You can only restore `defaultdb` and `postgres` if they are present in the original [backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}). - [Changefeed jobs]({% link {{ page.version.version }}/change-data-capture-overview.md %}) will not resume automatically on the new cluster. It is necessary to manually [create changefeeds]({% link {{ page.version.version }}/create-and-configure-changefeeds.md %}) after a full-cluster restore. -- When the cluster is in a mixed-version state during an [upgrade]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}), a full cluster restore will fail. To perform a full cluster restore, it is necessary to first [finalize the upgrade]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-3-decide-how-the-upgrade-will-be-finalized). +- When the cluster is in a mixed-version state during an [upgrade]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}), a full cluster restore will fail. To perform a full cluster restore, it is necessary to first [finalize the upgrade]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually). {{site.data.alerts.callout_info}} When you restore a full cluster with an {{ site.data.products.enterprise }} license, it will restore the [{{ site.data.products.enterprise }} license]({% link {{ page.version.version }}/licensing-faqs.md %}#types-of-licenses) of the cluster you are restoring from. If you want to use a different license in the new cluster, make sure to [update the license]({% link {{ page.version.version }}/licensing-faqs.md %}#set-a-license) **after** the restore is complete. diff --git a/src/current/v24.3/show-jobs.md b/src/current/v24.3/show-jobs.md index df220793598..35107a77636 100644 --- a/src/current/v24.3/show-jobs.md +++ b/src/current/v24.3/show-jobs.md @@ -61,7 +61,7 @@ The following fields are returned for each job: Field | Description ------|------------ `job_id` | A unique ID to identify each job. This value is used if you want to control jobs (i.e., [pause]({% link {{ page.version.version }}/pause-job.md %}), [resume]({% link {{ page.version.version }}/resume-job.md %}), or [cancel]({% link {{ page.version.version }}/cancel-job.md %}) it). -`job_type` | The type of job (e.g., [`SCHEMA CHANGE`]({% link {{ page.version.version }}/online-schema-changes.md %}), [`NEW SCHEMA CHANGE`]({% link {{ page.version.version }}/online-schema-changes.md %}#declarative-schema-changer), [`KEY VISUALIZER`]({% link {{ page.version.version }}/ui-key-visualizer.md %}), [`MIGRATION`]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade), [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), [`IMPORT`]({% link {{ page.version.version }}/import-into.md %}), [`CHANGEFEED`](#show-changefeed-jobs), [`CREATE STATS`]({% link {{ page.version.version }}/create-statistics.md %}), [`ROW LEVEL TTL`]({% link {{ page.version.version }}/row-level-ttl.md %}), [`REPLICATION STREAM INGESTION`]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %}), or [`REPLICATION STREAM PRODUCER`]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %})).

For job types of automatic jobs, see [Show automatic jobs](#show-automatic-jobs). +`job_type` | The type of job (e.g., [`SCHEMA CHANGE`]({% link {{ page.version.version }}/online-schema-changes.md %}), [`NEW SCHEMA CHANGE`]({% link {{ page.version.version }}/online-schema-changes.md %}#declarative-schema-changer), [`KEY VISUALIZER`]({% link {{ page.version.version }}/ui-key-visualizer.md %}), [`MIGRATION`]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually), [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), [`IMPORT`]({% link {{ page.version.version }}/import-into.md %}), [`CHANGEFEED`](#show-changefeed-jobs), [`CREATE STATS`]({% link {{ page.version.version }}/create-statistics.md %}), [`ROW LEVEL TTL`]({% link {{ page.version.version }}/row-level-ttl.md %}), [`REPLICATION STREAM INGESTION`]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %}), or [`REPLICATION STREAM PRODUCER`]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %})).

For job types of automatic jobs, see [Show automatic jobs](#show-automatic-jobs). `description` | The statement that started the job, or a textual description of the job. When you run `SHOW JOBS`, the `description` field is limited to 100 characters. To view the full description for a job, run `SHOW JOB {job ID}`. `statement` | When `description` is a textual description of the job, the statement that started the job is returned in this column. Currently, this field is populated only for the automatic table statistics jobs. `user_name` | The name of the [user]({% link {{ page.version.version }}/security-reference/authorization.md %}#create-and-manage-users) who started the job. diff --git a/src/current/v24.3/upgrade-cockroach-version.md b/src/current/v24.3/upgrade-cockroach-version.md index 2b906eb0e68..e455af59d80 100644 --- a/src/current/v24.3/upgrade-cockroach-version.md +++ b/src/current/v24.3/upgrade-cockroach-version.md @@ -11,7 +11,7 @@ This page describes how major-version and patch upgrades work and shows how to u {% include common/upgrade/overview.md %} -{% include_cached {{ page.version.version }}/upgrade_requirements.md %} +{% include_cached {{ page.version.version }}/upgrade-requirements.md %} CockroachDB's [multi-active availability]({% link {{ page.version.version }}/multi-active-availability.md %}) means that your cluster remains available while you upgrade one node at a time in a rolling fashion. While a node is being upgraded, its resources are not available to the cluster. @@ -33,13 +33,13 @@ CockroachDB's [multi-active availability]({% link {{ page.version.version }}/mul {% include_cached common/upgrade/major-version-upgrade-self-hosted.md %} -### Finalize a major-version upgrade +### Finalize a major-version upgrade manually -{% include finalize-self-hosted.md %} +{% include common/upgrade/finalize-self-hosted.md %} ### Roll back a major-version upgrade -{% include_cached rollback-self-hosted.md %} +{% include_cached common/upgrade/rollback-self-hosted.md %} ## Disable auto-finalization @@ -51,5 +51,4 @@ CockroachDB's [multi-active availability]({% link {{ page.version.version }}/mul ## See also -{% include see-also-self-hosted.md %} - +{% include common/upgrade/see-also-self-hosted.md %} diff --git a/src/current/v24.3/upgrade-cockroachdb-kubernetes.md b/src/current/v24.3/upgrade-cockroachdb-kubernetes.md index cac8b5bc5c4..6b4d702c1da 100644 --- a/src/current/v24.3/upgrade-cockroachdb-kubernetes.md +++ b/src/current/v24.3/upgrade-cockroachdb-kubernetes.md @@ -44,7 +44,7 @@ If you [deployed CockroachDB on Red Hat OpenShift]({% link {{ page.version.versi {% include common/upgrade/major-version-upgrade-kubernetes.md %} -### Finalize a major-version upgrade +### Finalize a major-version upgrade manually {% include common/upgrade/finalize-kubernetes.md %} diff --git a/src/current/v24.3/work-with-virtual-clusters.md b/src/current/v24.3/work-with-virtual-clusters.md index b7c16035c0b..419cfc89931 100644 --- a/src/current/v24.3/work-with-virtual-clusters.md +++ b/src/current/v24.3/work-with-virtual-clusters.md @@ -166,11 +166,11 @@ For more details, including the scope of each cluster setting, refer to [Cluster ## Upgrade a cluster -To upgrade to a new major version when cluster virtualization is enabled , you must: +To upgrade to a new major version when cluster virtualization is enabled, you must: 1. Replace the binary on each node and restart the node. -1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade) the upgrade on the system virtual cluster to upgrade it, or roll back the upgrade if you decide not to finalize it. Until it is finalized, the cluster still operates in compatibility with the previous major version, and virtual clusters cannot be upgraded. -1. After the system virtual cluster is finalized, finalize the upgrade on a virtual cluster to upgrade it, or roll back the upgrade if you decide not to finalize it. Until it is finalized, a virtual cluster still operates in compatibility with the previous major version, and some features may not be available on the virtual cluster. +1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade on the system virtual cluster to upgrade it (if auto-finalization is disabled) or roll back the upgrade if you decide not to finalize it. Until it is finalized, the cluster still operates in compatibility with the previous major version, and virtual clusters cannot be upgraded. +1. Finalize the upgrade on a virtual cluster to upgrade it, or roll back the upgrade if you decide not to finalize it. Until it is finalized, a virtual cluster still operates in compatibility with the previous major version, and some features may not be available on the virtual cluster. This allows you to roll back an upgrade of the system virtual cluster without impacting schemas or data in virtual clusters. The system virtual cluster can be at most one major version ahead of virtual clusters. For example, a system virtual cluster on CockroachDB v24.1 can have virtual clusters on CockroachDB v23.2. From 1ce087e2d9768292e7c9028ef73fe009c6a33fec Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Wed, 6 Nov 2024 16:40:04 -0800 Subject: [PATCH 04/19] Dan's feedback --- src/current/_includes/common/upgrade/finalize-cloud.md | 10 ++++++++-- .../common/upgrade/patch-rollback-self-hosted.md | 2 +- src/current/_includes/common/upgrade/rollback-cloud.md | 10 +++++++--- .../cockroachcloud/upgrade-cockroach-version.md | 8 ++++++-- 4 files changed, 22 insertions(+), 8 deletions(-) diff --git a/src/current/_includes/common/upgrade/finalize-cloud.md b/src/current/_includes/common/upgrade/finalize-cloud.md index 161989a803f..5cb6dcfbf5f 100644 --- a/src/current/_includes/common/upgrade/finalize-cloud.md +++ b/src/current/_includes/common/upgrade/finalize-cloud.md @@ -1,10 +1,16 @@ -To finalize a major-version upgrade manually, +To finalize a major-version upgrade of CockroachDB {{ site.data.products.advanced }} or CockroachDB {{ site.data.products.standard }} manually, 1. Click the cluster's name to open the **Cluster Details** page. 1. At the top of the page, click **Finalize**. - When finalization begins, a series of migration jobs run to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. These temporary limitations are listed in the release notes for the new major version. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will continue to report the previous major version. + When finalization begins, a series of migration jobs run to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. + + Until the upgrade is finalized, these features and functions will not be available, and the command `SHOW CLUSTER SETTING version` will continue to report the previous major version. Features that are available only after finalization are listed in the release notes for the new major version. You can monitor the process of finalization in the CockroachDB {{ site.data.products.cloud }} [Jobs page]({% link cockroachcloud/jobs-page.md %}). Migration jobs have names in the format `{major-version}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has a unique set of migration jobs and IDs. When finalization is complete, the **Cluster List** and **Cluster Details** page report the new version, and you can no longer roll back to the previous major version. + +{{site.data.alerts.callout_info}} +CockroachDB {{ site.data.products.basic }} clusters are upgraded automatically and finalization is not required. +{{site.data.alerts.end}} diff --git a/src/current/_includes/common/upgrade/patch-rollback-self-hosted.md b/src/current/_includes/common/upgrade/patch-rollback-self-hosted.md index 5707a2e814a..b9281f99f63 100644 --- a/src/current/_includes/common/upgrade/patch-rollback-self-hosted.md +++ b/src/current/_includes/common/upgrade/patch-rollback-self-hosted.md @@ -1 +1 @@ -To roll back a patch upgrade, repeat the steps in [Perform a patch upgrade](#perform-a-patch-upgrade), but replace the newer binary with the older version. +To roll back a patch upgrade, repeat the steps in [Perform a patch upgrade](#perform-a-patch-upgrade), but replace the newer binary with the older binary. diff --git a/src/current/_includes/common/upgrade/rollback-cloud.md b/src/current/_includes/common/upgrade/rollback-cloud.md index 470b69262cb..5081cb87e32 100644 --- a/src/current/_includes/common/upgrade/rollback-cloud.md +++ b/src/current/_includes/common/upgrade/rollback-cloud.md @@ -1,7 +1,11 @@ -To roll back to the previous major version before the upgrade is finalized automatically: +To roll back an CockroachDB {{ site.data.products.advanced }} cluster to the previous major version before the upgrade is finalized automatically: 1. Click the cluster's name to open the **Cluster Details** page. 1. At the top of the page, click **Roll back**. - A CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster remains fully available while it is rolled back. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are rolled back one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node CockroachDB {{ site.data.products.advanced }} cluster will be unavailable while the cluster is rolled back and restarted. Like an upgrade, a rollback must be finalized. -1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade-manually). When the rollback is finalized, the rollback is complete. + A CockroachDB {{ site.data.products.standard }} cluster remains fully available while it is rolled back. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are rolled back one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node CockroachDB {{ site.data.products.advanced }} cluster will be unavailable while the cluster is rolled back and restarted. Like an upgrade, a rollback must be finalized. +1. Finalization of a rollback is not required on CockroachDB {{ site.data.products.advanced }} or CockroachDB {{ site.data.products.standard }} clusters. + +{{site.data.alerts.callout_info}} +Rolling back a major-version upgrade is not supported on {{ site.data.products.basic }}. +{{site.data.alerts.end}} diff --git a/src/current/cockroachcloud/upgrade-cockroach-version.md b/src/current/cockroachcloud/upgrade-cockroach-version.md index ff859c030c6..f6b0125fd8e 100644 --- a/src/current/cockroachcloud/upgrade-cockroach-version.md +++ b/src/current/cockroachcloud/upgrade-cockroach-version.md @@ -17,9 +17,13 @@ This page describes how major-version and patch upgrades work and shows how to u ### Availability during an upgrade -For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.basic }}, a cluster's resources are not impacted by an ongoing upgrade. +For CockroachDB {{ site.data.products.standard }} and {{ site.data.products.basic }}, a cluster remains available during an upgrade. -For CockroachDB {{ site.data.products.advanced }}, nodes are upgraded one at a time in a rolling fashion, and the cluster's resources are reduced as each node is upgraded. If you have [configured a maintenance window]({% link cockroachcloud/advanced-cluster-management.md %}) for a CockroachDB {{ site.data.products.advanced }} cluster, automatic patch upgrades are applied during the maintenance window. Major-version upgrades must be initiated manually. +For CockroachDB {{ site.data.products.advanced }}, nodes are upgraded one at a time in a rolling fashion. Multi-node clusters will remain available during the upgrade, but will have reduced capacity as each node restarts. Single-node clusters will be unavailable while the node restarts. + +### Upgrades and maintenance windows + +If you have [configured a maintenance window]({% link cockroachcloud/advanced-cluster-management.md %}) for a CockroachDB {{ site.data.products.advanced }} cluster, automatic patch upgrades are applied during the maintenance window. Major-version upgrades must be initiated manually. {{site.data.alerts.callout_info}} Maintenance operations that are critical for cluster security or stability may be applied outside of the maintenance window, and upgrades that begin in a maintenance window may not always be completed by the end of the window. From ff8b51f500f75427cd39fd594dde890d3e5ee0b5 Mon Sep 17 00:00:00 2001 From: "Matt Linville (he/him)" Date: Thu, 7 Nov 2024 12:34:48 -0800 Subject: [PATCH 05/19] Fix errors in rollback content --- src/current/_includes/common/upgrade/rollback-cloud.md | 3 +-- src/current/_includes/common/upgrade/rollback-kubernetes.md | 4 +--- src/current/_includes/common/upgrade/rollback-self-hosted.md | 3 ++- 3 files changed, 4 insertions(+), 6 deletions(-) diff --git a/src/current/_includes/common/upgrade/rollback-cloud.md b/src/current/_includes/common/upgrade/rollback-cloud.md index 5081cb87e32..3ee47efa360 100644 --- a/src/current/_includes/common/upgrade/rollback-cloud.md +++ b/src/current/_includes/common/upgrade/rollback-cloud.md @@ -3,8 +3,7 @@ To roll back an CockroachDB {{ site.data.products.advanced }} cluster to the pre 1. Click the cluster's name to open the **Cluster Details** page. 1. At the top of the page, click **Roll back**. - A CockroachDB {{ site.data.products.standard }} cluster remains fully available while it is rolled back. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are rolled back one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node CockroachDB {{ site.data.products.advanced }} cluster will be unavailable while the cluster is rolled back and restarted. Like an upgrade, a rollback must be finalized. -1. Finalization of a rollback is not required on CockroachDB {{ site.data.products.advanced }} or CockroachDB {{ site.data.products.standard }} clusters. + A CockroachDB {{ site.data.products.standard }} cluster remains fully available while it is rolled back. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are rolled back one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node CockroachDB {{ site.data.products.advanced }} cluster will be unavailable while the cluster is rolled back and restarted. {{site.data.alerts.callout_info}} Rolling back a major-version upgrade is not supported on {{ site.data.products.basic }}. diff --git a/src/current/_includes/common/upgrade/rollback-kubernetes.md b/src/current/_includes/common/upgrade/rollback-kubernetes.md index 9955b029030..0ca269a6d9d 100644 --- a/src/current/_includes/common/upgrade/rollback-kubernetes.md +++ b/src/current/_includes/common/upgrade/rollback-kubernetes.md @@ -25,6 +25,4 @@ To roll back to the previous major version before an upgrade is finalized: -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' ~~~ - You can also check the CockroachDB version of each node in the [DB Console]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#node-details). - -1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade-manually). When the rollback is finalized, the rollback is complete. +Rollbacks do not require finalization. diff --git a/src/current/_includes/common/upgrade/rollback-self-hosted.md b/src/current/_includes/common/upgrade/rollback-self-hosted.md index 7049dd86fde..9e438870098 100644 --- a/src/current/_includes/common/upgrade/rollback-self-hosted.md +++ b/src/current/_includes/common/upgrade/rollback-self-hosted.md @@ -1,4 +1,5 @@ To roll back a major-version upgrade that has not been finalized: 1. Follow the steps to [perform a major-version upgrade](#perform-a-major-version-upgrade), replacing the `cockroach` binary on each node with binary for the previous major version. -1. Like a major-version upgrade, a rollback must be [finalized](#finalize-a-major-version-upgrade-manually). When the rollback is finalized, the rollback is complete. + +Rollbacks do not require finalization. From 8fe4434323c1aeddbbde2176a15d01d2371ece21 Mon Sep 17 00:00:00 2001 From: "Matt Linville (he/him)" Date: Fri, 8 Nov 2024 09:53:31 -0800 Subject: [PATCH 06/19] Apply suggestions from code review --- src/current/v24.3/upgrade-cockroachdb-kubernetes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/current/v24.3/upgrade-cockroachdb-kubernetes.md b/src/current/v24.3/upgrade-cockroachdb-kubernetes.md index 6b4d702c1da..b0a9aaae009 100644 --- a/src/current/v24.3/upgrade-cockroachdb-kubernetes.md +++ b/src/current/v24.3/upgrade-cockroachdb-kubernetes.md @@ -13,7 +13,7 @@ This page shows how to upgrade a CockroachDB cluster that is [deployed on a Kube {% include common/upgrade/overview.md %} -On Kubernewtes, the upgrade is a [staged update](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) in which each pod's container image for CockroachDB is updated in a rolling fashion. The cluster remains available during the upgrade. +On Kubernetes, the upgrade is a [staged update](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) in which each pod's container image for CockroachDB is updated in a rolling fashion. The cluster remains available during the upgrade. Select the cluster's deployment method to continue. @@ -29,7 +29,7 @@ Select the cluster's deployment method to continue. If you [deployed CockroachDB on Red Hat OpenShift]({% link {{ page.version.version }}/deploy-cockroachdb-with-kubernetes-openshift.md %}), substitute `kubectl` with `oc` in the following commands. {{site.data.alerts.end}} -{% include {{ page.version.version }}/orchestration/operator-check-namespace.md %} +{% include common/orchestration/operator-check-namespace.md %} {% include common/upgrade/prepare-to-upgrade-self-hosted.md %} ## Perform a patch upgrade From be7024c45dd4b3946f93ce4e446e93a8debe8e3f Mon Sep 17 00:00:00 2001 From: "Matt Linville (he/him)" Date: Fri, 8 Nov 2024 10:27:10 -0800 Subject: [PATCH 07/19] Update src/current/v24.3/upgrade-cockroachdb-kubernetes.md --- src/current/v24.3/upgrade-cockroachdb-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/current/v24.3/upgrade-cockroachdb-kubernetes.md b/src/current/v24.3/upgrade-cockroachdb-kubernetes.md index b0a9aaae009..36b7ef6d9cf 100644 --- a/src/current/v24.3/upgrade-cockroachdb-kubernetes.md +++ b/src/current/v24.3/upgrade-cockroachdb-kubernetes.md @@ -29,7 +29,7 @@ Select the cluster's deployment method to continue. If you [deployed CockroachDB on Red Hat OpenShift]({% link {{ page.version.version }}/deploy-cockroachdb-with-kubernetes-openshift.md %}), substitute `kubectl` with `oc` in the following commands. {{site.data.alerts.end}} -{% include common/orchestration/operator-check-namespace.md %} +{% include {{ page.version.version }}/orchestration/operator-check-namespace.md %} {% include common/upgrade/prepare-to-upgrade-self-hosted.md %} ## Perform a patch upgrade From ef222f41738385b0df57cb364d13f583047bb85d Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Fri, 8 Nov 2024 12:59:50 -0800 Subject: [PATCH 08/19] Remove irrelevant pre-upgrade tasks from Cloud prep steps --- .../_includes/common/upgrade/prepare-to-upgrade-cloud.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md b/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md index ccb74c90ac0..e8a7d0ee480 100644 --- a/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md +++ b/src/current/_includes/common/upgrade/prepare-to-upgrade-cloud.md @@ -1,12 +1,5 @@ Before beginning a major-version upgrade: -1. **CockroachDB {{ site.data.products.advanced }}**: Verify the overall health of your cluster using the [DB Console]({% link cockroachcloud/tools-page.md %}#access-the-db-console): - - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or [decommission]({% link {{ site.current_cloud_version }}/node-shutdown.md %}?filters=decommission#remove-nodes) them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. - - If any node is not fully decommissioned, try the following: - 1. First, reissue the [decommission command]({% link {{ site.current_cloud_version }}/node-shutdown.md %}?filters=decommission#decommission-the-node). The second command typically succeeds within a few minutes. - 1. If the second decommission command does not succeed, [recommission]({% link {{ site.current_cloud_version }}/node-shutdown.md %}?filters=decommission#recommission-nodes) and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`. - - Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade. 1. Review the cluster's [Metrics page]({% link cockroachcloud/metrics.md %}) to ensure that your cluster's compute and storage capacity is within acceptable values. The cluster must be able to tolerate some increase, in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider increasing the cluster's resources before beginning your upgrade. 1. {% include {{ site.current_cloud_version }}/backups/recommend-backups-for-upgrade.md %} 1. Review the [{{ site.current_cloud_version }} Release Notes]({% link releases/{{ site.current_cloud_version }}.md %}), as well as the release notes for any skipped major version. Pay careful attention to the sections for backward-incompatible changes, deprecations, changes to default cluster settings, and features that are not available until the upgrade is finalized. From dbe656534fc8571c0ecc39b29ffd8666afb1494f Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Mon, 11 Nov 2024 16:54:18 -0800 Subject: [PATCH 09/19] Most of David's feedback --- .../upgrade/disable-auto-finalization.md | 29 +++++++++---------- .../common/upgrade/finalize-kubernetes.md | 4 --- 2 files changed, 14 insertions(+), 19 deletions(-) diff --git a/src/current/_includes/common/upgrade/disable-auto-finalization.md b/src/current/_includes/common/upgrade/disable-auto-finalization.md index 20d9d82695a..7ad55225c85 100644 --- a/src/current/_includes/common/upgrade/disable-auto-finalization.md +++ b/src/current/_includes/common/upgrade/disable-auto-finalization.md @@ -1,3 +1,13 @@ +{% capture new_flag %} +{{site.data.alerts.callout_info}} +Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade, and then to unset the setting to finalize the upgrade. + +We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2. The setting does not need to be modified after it is initially set. + +Either of these settings prevents automatic finalization. +{{site.data.alerts.end}} +{% endcapture %} + {% if page.path contains 'kubernetes' %}
@@ -19,15 +29,11 @@ To disable auto-finalization: --host=cockroachdb-public ~~~ -1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version. +1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.auto_upgrade.enabled` to `false`. Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade). -{{site.data.alerts.callout_info}} -Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. - -We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization. -{{site.data.alerts.end}} +{{ new_flag }}
@@ -50,11 +56,7 @@ To disable auto-finalization: Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade). -{{site.data.alerts.callout_info}} -Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. - -We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization. -{{site.data.alerts.end}} +{{ new_flag }} @@ -75,9 +77,6 @@ To disable auto-finalization: Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade). -{{site.data.alerts.callout_info}} -Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade. +{{ new_flag }} -We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization. -{{site.data.alerts.end}} {% endif %} diff --git a/src/current/_includes/common/upgrade/finalize-kubernetes.md b/src/current/_includes/common/upgrade/finalize-kubernetes.md index 77b5bd58429..bd5daa280cb 100644 --- a/src/current/_includes/common/upgrade/finalize-kubernetes.md +++ b/src/current/_includes/common/upgrade/finalize-kubernetes.md @@ -24,10 +24,6 @@ To finalize a major-version upgrade: The amount of time required for finalization depends on the amount of data in the cluster, because finalization runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load. - {{site.data.alerts.callout_info}} - Finalization is not complete until all [schema change]({% link {{ page.version.version }}/online-schema-changes.md %}) jobs reach a terminal state. Finalization can take as long as the longest-running schema change. - {{site.data.alerts.end}} - When all migration jobs have completed, the upgrade is complete. 1. To confirm that finalization has completed, check the cluster version: From f3cc19b95a8ebe1495bb3cbe22f0bf9ce62b5586 Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Tue, 12 Nov 2024 09:29:47 -0800 Subject: [PATCH 10/19] Feedback from Alicia and Yandu --- .../_includes/common/upgrade/disable-auto-finalization.md | 6 ++---- .../v24.2/backups/recommend-backups-for-upgrade.md | 2 +- .../v24.3/backups/recommend-backups-for-upgrade.md | 2 +- 3 files changed, 4 insertions(+), 6 deletions(-) diff --git a/src/current/_includes/common/upgrade/disable-auto-finalization.md b/src/current/_includes/common/upgrade/disable-auto-finalization.md index 7ad55225c85..996d514396b 100644 --- a/src/current/_includes/common/upgrade/disable-auto-finalization.md +++ b/src/current/_includes/common/upgrade/disable-auto-finalization.md @@ -11,7 +11,7 @@ Either of these settings prevents automatic finalization. {% if page.path contains 'kubernetes' %}
-For clusters managed by the Operator, auto-finalization is disabled and cannot be enabled. A major version upgrade is not complete until it is manually [finalized](#finalize-a-major-version-upgrade-manually). +For clusters managed by the Operator, auto-finalization is disabled and cannot be enabled. A major version upgrade is not complete until it is manually [finalized](#finalize-a-major-version-upgrade-manually). The Operator does not yet support the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.auto_upgrade.enabled`.
@@ -52,12 +52,10 @@ To disable auto-finalization: --host=cockroachdb-public ~~~ -1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version. +1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version. The Helm chart does not yet support the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.auto_upgrade.enabled`. Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade). -{{ new_flag }} -
{% else %} diff --git a/src/current/_includes/v24.2/backups/recommend-backups-for-upgrade.md b/src/current/_includes/v24.2/backups/recommend-backups-for-upgrade.md index fe093883dad..375f4489914 100644 --- a/src/current/_includes/v24.2/backups/recommend-backups-for-upgrade.md +++ b/src/current/_includes/v24.2/backups/recommend-backups-for-upgrade.md @@ -5,5 +5,5 @@ When upgrading to a major release, you can optionally [take a self-managed backu {% else %} CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for [disaster recovery]({% link {{ page.version.version }}/disaster-recovery-planning.md %}) planning. -When upgrading to a new major version, **we recommend [taking a self-managed backup]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) of your cluster** so that you can [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version) if the upgrade leads to issues. +We recommend that you enable [managed backups]({% link cockroachcloud/managed-backups.md %}#managed-backup-settings) and confirm that the cluster is backed up before beginning a major-version upgrade. This provides an extra layer of protection in case the upgrade leads to issues. {% endif %} diff --git a/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md b/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md index fe093883dad..375f4489914 100644 --- a/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md +++ b/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md @@ -5,5 +5,5 @@ When upgrading to a major release, you can optionally [take a self-managed backu {% else %} CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for [disaster recovery]({% link {{ page.version.version }}/disaster-recovery-planning.md %}) planning. -When upgrading to a new major version, **we recommend [taking a self-managed backup]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) of your cluster** so that you can [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version) if the upgrade leads to issues. +We recommend that you enable [managed backups]({% link cockroachcloud/managed-backups.md %}#managed-backup-settings) and confirm that the cluster is backed up before beginning a major-version upgrade. This provides an extra layer of protection in case the upgrade leads to issues. {% endif %} From 3d26155b49d50d50ab467831fe187439ce39773f Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Wed, 13 Nov 2024 16:07:13 -0800 Subject: [PATCH 11/19] Update Cluster Virtualization upgrade details for version skipping --- src/current/v24.3/cluster-virtualization-overview.md | 4 +++- src/current/v24.3/work-with-virtual-clusters.md | 9 +++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/src/current/v24.3/cluster-virtualization-overview.md b/src/current/v24.3/cluster-virtualization-overview.md index 681a3cc895e..51332d3dc02 100644 --- a/src/current/v24.3/cluster-virtualization-overview.md +++ b/src/current/v24.3/cluster-virtualization-overview.md @@ -48,11 +48,13 @@ To grant access to the system virtual cluster, you must connect to the system vi ### Upgrades +The system virtual cluster can be at most one [Regular release]({% link releases/index.md %}#release-types) ahead of virtual clusters. For example, a system virtual cluster on CockroachDB v24.3 can have virtual clusters on CockroachDB v24.1 (a Regular release) or v24.2 (an Innovation release). This allows you to roll back an upgrade of the system virtual cluster without impacting schemas or data in virtual clusters. + To upgrade to a new major version when cluster virtualization is enabled, you must: 1. Replace the binary on each node and restart the node. 1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade on the system virtual cluster to upgrade it (if auto-finalization is disabled). -1. Finalize the upgrade on a virtual cluster to upgrade it (if auto-finalization is disabled). This allows you to roll back an upgrade of the system virtual cluster without impacting schemas or data in virtual clusters. The system virtual cluster can be at most one major version ahead of virtual clusters. For example, a system virtual cluster on CockroachDB v24.1 can have virtual clusters on CockroachDB v23.2. +1. Finalize the upgrade on a virtual cluster to upgrade it (if auto-finalization is disabled). For details, refer to [Work with virtual clusters]({% link {{ page.version.version }}/work-with-virtual-clusters.md %}#upgrade-a-cluster). diff --git a/src/current/v24.3/work-with-virtual-clusters.md b/src/current/v24.3/work-with-virtual-clusters.md index 419cfc89931..f2b91363df6 100644 --- a/src/current/v24.3/work-with-virtual-clusters.md +++ b/src/current/v24.3/work-with-virtual-clusters.md @@ -172,10 +172,15 @@ To upgrade to a new major version when cluster virtualization is enabled, you mu 1. [Finalize]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#finalize-a-major-version-upgrade-manually) the upgrade on the system virtual cluster to upgrade it (if auto-finalization is disabled) or roll back the upgrade if you decide not to finalize it. Until it is finalized, the cluster still operates in compatibility with the previous major version, and virtual clusters cannot be upgraded. 1. Finalize the upgrade on a virtual cluster to upgrade it, or roll back the upgrade if you decide not to finalize it. Until it is finalized, a virtual cluster still operates in compatibility with the previous major version, and some features may not be available on the virtual cluster. -This allows you to roll back an upgrade of the system virtual cluster without impacting schemas or data in virtual clusters. The system virtual cluster can be at most one major version ahead of virtual clusters. For example, a system virtual cluster on CockroachDB v24.1 can have virtual clusters on CockroachDB v23.2. +This allows you to roll back an upgrade of the system virtual cluster without impacting schemas or data in virtual clusters. The system virtual cluster can be at most one [Regular release]({% link releases/index.md %}#release-types) ahead of virtual clusters. For example, a system virtual cluster on CockroachDB v24.3 can have virtual clusters on CockroachDB v24.1 (a Regular release) or v24.2 (an Innovation release). {{site.data.alerts.callout_info}} -The `preserve_downgrade_option` cluster setting is scoped to a virtual cluster. To prevent automatic finalization of the upgrade, you must set it to `false` both in the virtual cluster and in the system virtual cluster. +Both the `auto_upgrade.enabled` and `preserve_downgrade_option` [cluster settings]({% link {{ page.version.version }}/cluster-settings.md %}) are scoped to a virtual cluster. To prevent automatic finalization of an upgrade, you must do one of the following: + +- Set `auto_upgrade.enabled` to `false` before beginning an upgrade. This setting persists after an upgrade. To finalize the upgrade manually, refer to [Upgrade a cluster]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}). This is the preferred way to disable auto-upgrade. +- Set `preserve_downgrade_option` to the cluster's current major version. This setting is reset during upgrade finalization and must be set again before the next major-version upgrade. + +Either setting, if enabled, disables auto-finalization. {{site.data.alerts.end}} To apply a patch-version upgrade, you must replace the binary on each node and restart the node. Finalization is not required. From b1a2bb74b983037d497c9ab3aeb4a5f3255b5d0e Mon Sep 17 00:00:00 2001 From: "Matt Linville (he/him)" Date: Fri, 15 Nov 2024 10:14:19 -0800 Subject: [PATCH 12/19] Apply suggestions from code review Co-authored-by: Mike Lewis <76072290+mikeCRL@users.noreply.github.com> --- src/current/_includes/common/upgrade/finalize-cloud.md | 2 +- .../_includes/common/upgrade/major-version-upgrade-cloud.md | 2 +- src/current/cockroachcloud/upgrade-cockroach-version.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/current/_includes/common/upgrade/finalize-cloud.md b/src/current/_includes/common/upgrade/finalize-cloud.md index 5cb6dcfbf5f..5e44b9c9e75 100644 --- a/src/current/_includes/common/upgrade/finalize-cloud.md +++ b/src/current/_includes/common/upgrade/finalize-cloud.md @@ -1,4 +1,4 @@ -To finalize a major-version upgrade of CockroachDB {{ site.data.products.advanced }} or CockroachDB {{ site.data.products.standard }} manually, +To finalize a major-version upgrade of CockroachDB {{ site.data.products.advanced }} or CockroachDB {{ site.data.products.standard }} manually: 1. Click the cluster's name to open the **Cluster Details** page. 1. At the top of the page, click **Finalize**. diff --git a/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md b/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md index 8c755d140dc..21481e212b7 100644 --- a/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md +++ b/src/current/_includes/common/upgrade/major-version-upgrade-cloud.md @@ -16,6 +16,6 @@ This section shows how to perform a major-version upgrade for a cluster in Cockr To check whether major-version upgrades are available, click the three-dot **Action** menu. If upgrades are available, **Upgrade major version** will be enabled. Click it. In the dialog, if only one newer major version upgrade is available, it is selected automatically. If multiple upgrades are available, the latest Regular release is selected by default. If you intend to upgrade the cluster, keep this dialog open while completing the next steps. Otherwise, close the dialog. 1. Before beginning a major-version upgrade, review its [Release notes]({% link releases/index.md %}), as well as the release notes for any skipped Innovation Releases. If any backward-incompatible changes or new features impact the cluster's workloads, you may need to make adjustments before beginning the upgrade. 1. A CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster remains fully available while it is upgraded. For a multi-node CockroachDB {{ site.data.products.advanced }} cluster, nodes are upgraded one at a time in a rolling fashion so the cluster remains available, with one node unavailable at a time. A single-node {{ site.data.products.advanced }} cluster will be unavailable while the cluster is upgraded and restarted. If necessary, prepare for the upgrade by communicating ahead of time with your users, and plan to begin the upgrade during a time when the impact on the cluster's workload will be minimized. -1. To begin a major-version upgrade, go back to the dialog in the CockroachDB {{ site.data.products.cloud }} Console. If multiple upgrades are available, select the desired version. Review the details, then click **Start upgrade**. The dialog closes, and the **Version** columnin the **Cluster List** page changes to **Upgrading**. When the upgrade has finished but has not yet been finalized, the **Version** column reports that the new version is **pending**. +1. To begin a major-version upgrade, go back to the dialog in the CockroachDB {{ site.data.products.cloud }} Console. If multiple upgrades are available, select the desired version. Review the details, then click **Start upgrade**. The dialog closes, and the **Version** column in the **Cluster List** page changes to **Upgrading**. When the upgrade has finished but has not yet been finalized, the **Version** column reports that the new version is **pending**. The upgrade is not complete until it is [finalized](#finalize-a-major-version-upgrade-manually). After the upgrade is finalized, the cluster can no longer be [rolled back](#roll-back-a-major-version-upgrade) to its previous major version. If you take no action, finalization occurs automatically after approximately 72 hours. diff --git a/src/current/cockroachcloud/upgrade-cockroach-version.md b/src/current/cockroachcloud/upgrade-cockroach-version.md index f9223df0d42..e329e5e7813 100644 --- a/src/current/cockroachcloud/upgrade-cockroach-version.md +++ b/src/current/cockroachcloud/upgrade-cockroach-version.md @@ -29,7 +29,7 @@ If you have [configured a maintenance window]({% link cockroachcloud/advanced-cl Maintenance operations that are critical for cluster security or stability may be applied outside of the maintenance window, and upgrades that begin in a maintenance window may not always be completed by the end of the window. {{site.data.alerts.end}} -Refer to [Cloud Upgrade Policy]({% link cockroachcloud/upgrade-policy.md %}). +For more information, refer to the [Cloud Upgrade Policy]({% link cockroachcloud/upgrade-policy.md %}). ## Prepare to upgrade From 271769ae2fefddc7cf1d37a4726d0b6f0dcee937 Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Fri, 15 Nov 2024 10:17:05 -0800 Subject: [PATCH 13/19] Conditionalize the high-level steps intro --- src/current/_includes/common/upgrade/upgrade-high-level.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/current/_includes/common/upgrade/upgrade-high-level.md b/src/current/_includes/common/upgrade/upgrade-high-level.md index 93abf11c210..f67f16dd8ec 100644 --- a/src/current/_includes/common/upgrade/upgrade-high-level.md +++ b/src/current/_includes/common/upgrade/upgrade-high-level.md @@ -1,4 +1,8 @@ -At a high level, a cluster upgrade involves the following steps, which are detailed in the following sections: +{% if page.path contains "cockroachcloud" %} +When you initiate a major-version upgrade, CockroachDB {{ site.data.products.cloud }} performs the following high-level steps, which are described in detail in the following sections. +{% else %} +A major-version upgrade involves the following high-level steps, which are described in detail in the following sections. +{% endif %} 1. On one node at a time: 1. Replace the previous `cockroach` binary or container image with the new binary. From 7d848ff4560078f4b39c3176fee7884ec884d746 Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Fri, 15 Nov 2024 11:13:01 -0800 Subject: [PATCH 14/19] Mike's feedback --- src/current/_includes/common/upgrade/overview.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/src/current/_includes/common/upgrade/overview.md b/src/current/_includes/common/upgrade/overview.md index ad2da882b64..891b117fc7b 100644 --- a/src/current/_includes/common/upgrade/overview.md +++ b/src/current/_includes/common/upgrade/overview.md @@ -1,9 +1,17 @@ CockroachDB offers the following types of upgrades: -- [**Major-version upgrades**]({% link releases/index.md %}#major-releases): A major-version upgrade moves a cluster from one major version of CockroachDB to another, such as from v24.2 to v24.3. A major-version upgrade may include new features, updates to cluster setting defaults, and backward-incompatible changes. Performing a major-version upgrade requires an additional step to finalize the upgrade. +- **Major-version upgrades**: A [major-version upgrade]({% link releases/index.md %}#major-releases) moves a cluster from one major version of CockroachDB to another, such as from v24.2 to v24.3. A major-version upgrade may include new features, updates to cluster setting defaults, and backward-incompatible changes. Performing a major-version upgrade requires an additional step to finalize the upgrade. - As of 2024, every second major version is an **Innovation release**. {% if page.path contains "cockroachcloud" %}For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }},innovation{% else %}Innovation{% endif %} releases offer shorter support windows and can be skipped. {% if page.path contains "cockroachcloud" %}Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically.{% endif %} -- [**Patch upgrades**]({% link releases/index.md %}#patch-releases): A patch upgrade moves a cluster from one patch to another within a major version, such as from v24.2.3 to v24.2.4. Patch upgrades do not introduce backward-incompatible changes. {% if page.path contains "cockroachcloud" %}Patch upgrades are automatically applied to CockroachDB {{ site.data.products.advanced }}, {{ site.data.products.standard }}, and {{ site.data.products.basic }} clusters.{% endif %} +{% if page.path contains "cockroachcloud" %} + As of 2024, every second major version is an[Innovation release]({% link cockroachcloud/upgrade-policy.md %}#innovation-release). For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }},innovation releases offer shorter support windows and can be skipped. Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically. +{% else %} + As of 2024, every second major version is an [Innovation release]({% link releases/release-suppoert-policy.md %}#innovation-release). Innovation releases offer shorter support windows and can be skipped. {% if page.path contains "cockroachcloud" %} +{% endif %} +- **Patch upgrades**: A [patch upgrade]({% link releases/index.md %}#patch-releases) moves a cluster from one patch to another within a major version, such as from v24.2.3 to v24.2.4. Patch upgrades do not introduce backward-incompatible changes. + +{% if page.path contains "cockroachcloud" %} + Patch upgrades are automatically applied to CockroachDB {{ site.data.products.advanced }}, {{ site.data.products.standard }}, and {{ site.data.products.basic }} clusters. +{% endif %} A major version of CockroachDB has two phases of patch releases: a series of **testing releases** followed by a series of **production releases**. A major version’s initial production release is also known as its GA release. In the lead-up to a new major version's GA release, a series of Testing releases may be made available{% if page.path contains "cockroachcloud" %} to CockroachDB {{ site.data.products.advanced }} as Pre-Production Preview releases{% endif %} for testing and validation. Testing releases are intended for testing and experimentation only, and are not qualified for production environments or eligible for support or uptime SLA commitments. {% if page.path contains "cockroachcloud" %}If a cluster is upgraded to a Pre-Production Preview patch release, it will be automatically upgraded to subsequent patch releases within the major version, including newer Pre-Production Preview releases, the initial GA release, and subsequent patch releases.{% else %}{{site.data.alerts.callout_info}}A cluster cannot be upgraded from an alpha binary of a prior release or from a binary built from the `master` branch of the CockroachDB source code.{{site.data.alerts.end}}{% endif %} From 18161dcc89df76ac0d89851b2d1faaec6f010024 Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Fri, 15 Nov 2024 11:26:51 -0800 Subject: [PATCH 15/19] Remove liquid fragment --- src/current/_includes/common/upgrade/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/current/_includes/common/upgrade/overview.md b/src/current/_includes/common/upgrade/overview.md index 891b117fc7b..a461c68e066 100644 --- a/src/current/_includes/common/upgrade/overview.md +++ b/src/current/_includes/common/upgrade/overview.md @@ -5,7 +5,7 @@ CockroachDB offers the following types of upgrades: {% if page.path contains "cockroachcloud" %} As of 2024, every second major version is an[Innovation release]({% link cockroachcloud/upgrade-policy.md %}#innovation-release). For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }},innovation releases offer shorter support windows and can be skipped. Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically. {% else %} - As of 2024, every second major version is an [Innovation release]({% link releases/release-suppoert-policy.md %}#innovation-release). Innovation releases offer shorter support windows and can be skipped. {% if page.path contains "cockroachcloud" %} + As of 2024, every second major version is an [Innovation release]({% link releases/release-suppoert-policy.md %}#innovation-release). Innovation releases offer shorter support windows and can be skipped. {% endif %} - **Patch upgrades**: A [patch upgrade]({% link releases/index.md %}#patch-releases) moves a cluster from one patch to another within a major version, such as from v24.2.3 to v24.2.4. Patch upgrades do not introduce backward-incompatible changes. From 39d279145892962f0f55fd97635d72dc3da2e970 Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Fri, 15 Nov 2024 11:28:54 -0800 Subject: [PATCH 16/19] Fix typo --- src/current/_includes/common/upgrade/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/current/_includes/common/upgrade/overview.md b/src/current/_includes/common/upgrade/overview.md index a461c68e066..38e10f48588 100644 --- a/src/current/_includes/common/upgrade/overview.md +++ b/src/current/_includes/common/upgrade/overview.md @@ -5,7 +5,7 @@ CockroachDB offers the following types of upgrades: {% if page.path contains "cockroachcloud" %} As of 2024, every second major version is an[Innovation release]({% link cockroachcloud/upgrade-policy.md %}#innovation-release). For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }},innovation releases offer shorter support windows and can be skipped. Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically. {% else %} - As of 2024, every second major version is an [Innovation release]({% link releases/release-suppoert-policy.md %}#innovation-release). Innovation releases offer shorter support windows and can be skipped. + As of 2024, every second major version is an [Innovation release]({% link releases/release-support-policy.md %}#innovation-release). Innovation releases offer shorter support windows and can be skipped. {% endif %} - **Patch upgrades**: A [patch upgrade]({% link releases/index.md %}#patch-releases) moves a cluster from one patch to another within a major version, such as from v24.2.3 to v24.2.4. Patch upgrades do not introduce backward-incompatible changes. From d6bcbedf1512c6070ae27e6c5d415eb9acd4f3e1 Mon Sep 17 00:00:00 2001 From: Matt Linville Date: Fri, 15 Nov 2024 11:47:08 -0800 Subject: [PATCH 17/19] Fix links --- src/current/_includes/common/upgrade/overview.md | 4 ++-- src/current/v24.3/show-jobs.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/current/_includes/common/upgrade/overview.md b/src/current/_includes/common/upgrade/overview.md index 38e10f48588..aeb15959146 100644 --- a/src/current/_includes/common/upgrade/overview.md +++ b/src/current/_includes/common/upgrade/overview.md @@ -3,9 +3,9 @@ CockroachDB offers the following types of upgrades: - **Major-version upgrades**: A [major-version upgrade]({% link releases/index.md %}#major-releases) moves a cluster from one major version of CockroachDB to another, such as from v24.2 to v24.3. A major-version upgrade may include new features, updates to cluster setting defaults, and backward-incompatible changes. Performing a major-version upgrade requires an additional step to finalize the upgrade. {% if page.path contains "cockroachcloud" %} - As of 2024, every second major version is an[Innovation release]({% link cockroachcloud/upgrade-policy.md %}#innovation-release). For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }},innovation releases offer shorter support windows and can be skipped. Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically. + As of 2024, every second major version is an[Innovation release]({% link cockroachcloud/upgrade-policy.md %}#innovation-releases). For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }},innovation releases offer shorter support windows and can be skipped. Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically. {% else %} - As of 2024, every second major version is an [Innovation release]({% link releases/release-support-policy.md %}#innovation-release). Innovation releases offer shorter support windows and can be skipped. + As of 2024, every second major version is an [Innovation release]({% link releases/release-support-policy.md %}#innovation-releases). Innovation releases offer shorter support windows and can be skipped. {% endif %} - **Patch upgrades**: A [patch upgrade]({% link releases/index.md %}#patch-releases) moves a cluster from one patch to another within a major version, such as from v24.2.3 to v24.2.4. Patch upgrades do not introduce backward-incompatible changes. diff --git a/src/current/v24.3/show-jobs.md b/src/current/v24.3/show-jobs.md index 975b56b6f5d..c4f6f8ba8c9 100644 --- a/src/current/v24.3/show-jobs.md +++ b/src/current/v24.3/show-jobs.md @@ -61,7 +61,7 @@ The following fields are returned for each job: Field | Description ------|------------ `job_id` | A unique ID to identify each job. This value is used if you want to control jobs (i.e., [pause]({% link {{ page.version.version }}/pause-job.md %}), [resume]({% link {{ page.version.version }}/resume-job.md %}), or [cancel]({% link {{ page.version.version }}/cancel-job.md %}) it). -`job_type` | The type of job: [`SCHEMA CHANGE`]({% link {{ page.version.version }}/online-schema-changes.md %}), [`NEW SCHEMA CHANGE`]({% link {{ page.version.version }}/online-schema-changes.md %}#declarative-schema-changer), [`KEY VISUALIZER`]({% link {{ page.version.version }}/ui-key-visualizer.md %}), [`MIGRATION`]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#step-6-finish-the-upgrade), [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), [`IMPORT`]({% link {{ page.version.version }}/import-into.md %}), [`CHANGEFEED`](#show-changefeed-jobs), [`CREATE STATS`]({% link {{ page.version.version }}/create-statistics.md %}), [`ROW LEVEL TTL`]({% link {{ page.version.version }}/row-level-ttl.md %}), [`REPLICATION STREAM INGESTION`]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %}), `REPLICATION STREAM PRODUCER`([physical cluster replication]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %}) or [logical data replication]({% link {{ page.version.version }}/logical-data-replication-monitoring.md %})), [`LOGICAL REPLICATION`]({% link {{ page.version.version }}/logical-data-replication-monitoring.md %}).

For job types of automatic jobs, see [Show automatic jobs](#show-automatic-jobs). +`job_type` | The type of job: [`SCHEMA CHANGE`]({% link {{ page.version.version }}/online-schema-changes.md %}), [`NEW SCHEMA CHANGE`]({% link {{ page.version.version }}/online-schema-changes.md %}#declarative-schema-changer), [`KEY VISUALIZER`]({% link {{ page.version.version }}/ui-key-visualizer.md %}), [`MIGRATION`]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}#overview), [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), [`IMPORT`]({% link {{ page.version.version }}/import-into.md %}), [`CHANGEFEED`](#show-changefeed-jobs), [`CREATE STATS`]({% link {{ page.version.version }}/create-statistics.md %}), [`ROW LEVEL TTL`]({% link {{ page.version.version }}/row-level-ttl.md %}), [`REPLICATION STREAM INGESTION`]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %}), `REPLICATION STREAM PRODUCER`([physical cluster replication]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %}) or [logical data replication]({% link {{ page.version.version }}/logical-data-replication-monitoring.md %})), [`LOGICAL REPLICATION`]({% link {{ page.version.version }}/logical-data-replication-monitoring.md %}).

For job types of automatic jobs, see [Show automatic jobs](#show-automatic-jobs). `description` | The statement that started the job, or a textual description of the job. When you run `SHOW JOBS`, the `description` field is limited to 100 characters. To view the full description for a job, run `SHOW JOB {job ID}`. `statement` | When `description` is a textual description of the job, the statement that started the job is returned in this column. Currently, this field is populated only for the automatic table statistics jobs. `user_name` | The name of the [user]({% link {{ page.version.version }}/security-reference/authorization.md %}#create-and-manage-users) who started the job. From 379587d5d49f12f322f01406000d5031d379ce53 Mon Sep 17 00:00:00 2001 From: Mike Lewis <76072290+mikeCRL@users.noreply.github.com> Date: Mon, 18 Nov 2024 09:51:05 -0500 Subject: [PATCH 18/19] Add missing spaces --- src/current/_includes/common/upgrade/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/current/_includes/common/upgrade/overview.md b/src/current/_includes/common/upgrade/overview.md index aeb15959146..d0748be0975 100644 --- a/src/current/_includes/common/upgrade/overview.md +++ b/src/current/_includes/common/upgrade/overview.md @@ -3,7 +3,7 @@ CockroachDB offers the following types of upgrades: - **Major-version upgrades**: A [major-version upgrade]({% link releases/index.md %}#major-releases) moves a cluster from one major version of CockroachDB to another, such as from v24.2 to v24.3. A major-version upgrade may include new features, updates to cluster setting defaults, and backward-incompatible changes. Performing a major-version upgrade requires an additional step to finalize the upgrade. {% if page.path contains "cockroachcloud" %} - As of 2024, every second major version is an[Innovation release]({% link cockroachcloud/upgrade-policy.md %}#innovation-releases). For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }},innovation releases offer shorter support windows and can be skipped. Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically. + As of 2024, every second major version is an [Innovation release]({% link cockroachcloud/upgrade-policy.md %}#innovation-releases). For CockroachDB {{ site.data.products.standard }} and CockroachDB {{ site.data.products.advanced }}, innovation releases offer shorter support windows and can be skipped. Innovation releases are required for CockroachDB {{ site.data.products.basic }}, and are applied automatically. {% else %} As of 2024, every second major version is an [Innovation release]({% link releases/release-support-policy.md %}#innovation-releases). Innovation releases offer shorter support windows and can be skipped. {% endif %} From 2fc3ce160898ef46c2e5e73cc077298c33cb1530 Mon Sep 17 00:00:00 2001 From: mikeCRL Date: Mon, 18 Nov 2024 10:03:16 -0500 Subject: [PATCH 19/19] Remove old New In statement from Virtual Clusters page --- src/current/v24.2/work-with-virtual-clusters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/current/v24.2/work-with-virtual-clusters.md b/src/current/v24.2/work-with-virtual-clusters.md index b7c16035c0b..507c01bd8d4 100644 --- a/src/current/v24.2/work-with-virtual-clusters.md +++ b/src/current/v24.2/work-with-virtual-clusters.md @@ -9,7 +9,7 @@ docs_area: deploy {% include feature-phases/preview.md %} {{site.data.alerts.end}} -{% include_cached new-in.html version="v23.2" %} Enable [cluster virtualization]({% link {{ page.version.version }}/cluster-virtualization-overview.md %}) in your CockroachDB cluster to set up a [**physical cluster replication (PCR)**]({% link {{ page.version.version }}/physical-cluster-replication-overview.md %}) stream. This page is a guide to working with virtual clusters. +Enable [cluster virtualization]({% link {{ page.version.version }}/cluster-virtualization-overview.md %}) in your CockroachDB cluster to set up a [**physical cluster replication (PCR)**]({% link {{ page.version.version }}/physical-cluster-replication-overview.md %}) stream. This page is a guide to working with virtual clusters. ## Connect to a virtual cluster