From f3482fec9a29087dc2f3bb8203604b7c99e03a30 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Mon, 2 May 2022 11:39:07 -0700 Subject: [PATCH 01/14] upgrade and patch a cluster draft Signed-off-by: Teresa Killmond --- .../ROOT/pages/sysadmin-cluster-upgrade.adoc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index fa1e8c1d5e..5526673c42 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -4,11 +4,13 @@ :page-aliases: /admin/system-admin/upgrade-a-cluster.adoc :experimental: -Learn how to upgrade a cluster to a new release, or patch a cluster to update services and allow for bug fixes. +Learn how to upgrade a single-node, ThoughtSpot-managed CentOS cluster to a new release, or patch a cluster to update services and allow for bug fixes. -To learn how to upgrade or patching ThoughtSpot, take the following ThoughtSpot U course: https://training.thoughtspot.com/create-upgrade-patch-a-thoughtspot-cluster/431164[Upgrade & Patch a Cluster^]. - -See other training resources at https://training.thoughtspot.com/[ThoughtSpot U^]. +''' +> **ThoughtSpot Training** +> +> * For best results in upgrading or patching ThoughtSpot, take the following ThoughtSpot U course: https://training.thoughtspot.com/create-upgrade-patch-a-thoughtspot-cluster/431164[Upgrade & Patch a Cluster^]. +> * See other training resources at https://training.thoughtspot.com/[ThoughtSpot U^]. ''' @@ -22,7 +24,6 @@ Installation and upgrade use the same process, replacing the older OS image on a ThoughtSpot Support will contact you to schedule an update when a minor or major upgrade becomes available. -//// == View available releases NOTE: This feature is now deprecated. You may not use it starting with the 7.2.1 release. For details, see xref:deprecation.adoc[Deprecation Announcements]. @@ -211,5 +212,4 @@ Run `tscli patch ls` and look for the new patch name. [source,bash] ---- $ tscli patch ls ----- -//// \ No newline at end of file +---- \ No newline at end of file From 55a001421c59343fec0f420bb8a80ee0453cb9ce Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Mon, 2 May 2022 16:23:23 -0700 Subject: [PATCH 02/14] more updates Signed-off-by: Teresa Killmond --- .../ROOT/pages/sysadmin-cluster-upgrade.adoc | 57 ++++++++++++++++--- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 5526673c42..5fccee1267 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -14,9 +14,11 @@ Learn how to upgrade a single-node, ThoughtSpot-managed CentOS cluster to a new ''' +NOTE: This article teaches you how to upgrade a single-node, ThoughtSpot-managed CentOS cluster. To upgrade multi-node clusters, or clusters that use customer-managed xref:rhel.adoc[RHEL], xref:rhel.adoc[OEL], or xref:al2.adoc[Amazon Linux 2], contact {support-url}. + == About cluster upgrades and patches ThoughtSpot is installed or updated from a release tarball that contains the ThoughtSpot application (binaries and configuration), third-party software, and Operating System image. -Third party softwares are licensed software components necessary for operation of the ThoughtSpot application. +Third party software includes licensed software components necessary for operation of the ThoughtSpot application. These include Java, Boost C{pp} libraries, Google protocol buffers, and so on. ThoughtSpot patches the Operating System at the time of upgrades. @@ -44,16 +46,18 @@ To upgrade your cluster, follow these steps: | ❏ | <> |=== +NOTE: If any commands return errors, warnings, or failures while you upgrade, contact {support-url}. + [#upgrade-step-1] === Step 1: Check cluster health . SSH into your cluster. -Run `ssh admin@`. +Run `ssh admin@`. + -Replace `nodeIP` with your specific network information. +Replace `ThoughtSpotnodeIP` with your specific network information. + [source,bash] ---- - $ ssh admin@ + $ ssh admin@ ---- . Run `tscli cluster status`. This tells you what version you are currently on, and ensures that the cluster is running and ready. @@ -62,6 +66,8 @@ Replace `nodeIP` with your specific network information. ---- $ tscli cluster status ---- ++ +Make sure that the output says `Cluster: RUNNING`, `Database: READY`, and `Search Engine: READY`. . Run `tscli cluster check` to ensure there are no component failures. + @@ -69,14 +75,40 @@ Replace `nodeIP` with your specific network information. ---- $ tscli cluster check ---- ++ +Make sure that the output for each component is `SUCCESS`. -. If `tscli cluster status` or `tscli cluster check` return any failures, xref:support-contact.adoc[contact ThoughtSpot Support] before you proceed with the upgrade. +. If `tscli cluster status` or `tscli cluster check` return any errors, warnings, or failures, contact {support-url} before you proceed with the upgrade. [#upgrade-step-2] === Step 2: Download the release -. Use `tscli fileserver download-release` to download the release bundle. +You can download the release and move it to the correct directory in one of two ways: + +* using the <> command +* using the <> + +[#fileserver] +==== Download the release using tscli +Use `tscli fileserver download-release` to download the release bundle. + +// should they cd to the specific export directory here? + +. First, ensure that you already xref:tscli-command-ref.adoc#tscli-fileserver[configured the fileserver] by running `tscli fileserver show-config`. ++ +[source,bash] +---- +tscli fileserver show-config +---- +If the fileserver is already configured, the output is `Fileserver is configured`. + +.. If the fileserver is *not* configured, xref:tscli-command-ref.adoc#tscli-fileserver[configure the fileserver] by running `tscli fileserver configure --user USER --password PASSWORD`. + -You must xref:tscli-command-ref.adoc#tscli-fileserver[configure the fileserver] by running `tscli fileserver configure` before you can download the release. +[source,bash] +---- +tscli fileserver configure --user USER --password PASSWORD +---- + +. Download the release from the fileserver by running `tscli fileserver download-release --user --out `. + [source,bash] ---- @@ -89,27 +121,34 @@ Note the following parameters: `username`:: is the username for the fileserver that you set up earlier, when configuring the fileserver. `release-location`:: is the location path of the release bundle on your local machine. For example, `/export/sdb1/TS_TASKS/install/6.0.tar.gz`. +[#egnyte] +==== Download the release using the Egnyte link +// should they cd to the specific export directory here? +. On a machine that can also access your ThoughtSpot cluster, navigate to [LINK]. +. Select *Download*. [i'm not sure what this actually looks like on egnyte] . Copy the downloaded release bundle to a specific path in the `/export` folder. + You must copy the bundle to a different path in the `/export` folder, depending on your deployment platform: + +[#platform-export-path] -- SMC, Dell, VMware, and GCP:: `/export/sdb1/TS_SRE_TASKS` AWS:: `/export/xvdb1/TS_SRE_TASKS` Azure:: `/export/sdc1/TS_SRE_TASKS` -- + -Run `scp .tar.gz admin@:/export/sdb1/TS_TASKS/install/`. +Run `scp .tar.gz admin@:/export//TS_TASKS/install/`. + [source,bash] ---- -$ scp .tar.gz admin@:/export/sdb1/TS_TASKS/install/ +$ scp .tar.gz admin@:/export//TS_TASKS/install/ ---- + Note the following parameters: + `release-number`:: is the release number of your ThoughtSpot instance, such as 5.3, 5.3.1, 6.0, and so on. `hostname`:: is your specific hostname. +`path`:: is the path within the export folder. This differs based on your deployment platform; refer to <>. `file-name`:: is the name of the tarball file on your local machine. NOTE: You can use another secure copy method, if you prefer a method other than the `scp` command. From 7f1127dea48c3eba24b3a4106404f5ce1f963081 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Tue, 3 May 2022 17:12:39 -0700 Subject: [PATCH 03/14] upgrade article feedback from bimoo Signed-off-by: Teresa Killmond --- .../ROOT/pages/sysadmin-cluster-upgrade.adoc | 200 ++++++++++-------- 1 file changed, 117 insertions(+), 83 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 5fccee1267..bdf7652937 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -26,6 +26,8 @@ Installation and upgrade use the same process, replacing the older OS image on a ThoughtSpot Support will contact you to schedule an update when a minor or major upgrade becomes available. +// is this last sentence still true + == View available releases NOTE: This feature is now deprecated. You may not use it starting with the 7.2.1 release. For details, see xref:deprecation.adoc[Deprecation Announcements]. @@ -40,17 +42,24 @@ The *Upgrade* page shows your cluster's current version, and any new releases th To upgrade your cluster, follow these steps: [cols="5,~",grid=none,frame=none] |=== -| ❏ | <> -| ❏ | <> -| ❏ | <> -| ❏ | <> +| ❏ | <> +| ❏ | <> +| ❏ | <> +| ❏ | <> +| ❏ | <> +| ❏ | <> +| ❏ | <> |=== NOTE: If any commands return errors, warnings, or failures while you upgrade, contact {support-url}. [#upgrade-step-1] -=== Step 1: Check cluster health -. SSH into your cluster. +=== Step 1: Download the release +Download the release, including any patches for the version you are installing, and move it to a folder in your ThoughtSpot node. + +. Navigate to [LINK]. +. Download the release content folder for the version you would like to upgrade to. [i'm not sure what this actually looks like on egnyte] +. In your terminal application, `ssh` into your cluster. Run `ssh admin@`. + Replace `ThoughtSpotnodeIP` with your specific network information. @@ -59,111 +68,90 @@ Replace `ThoughtSpotnodeIP` with your specific network information. ---- $ ssh admin@ ---- - -. Run `tscli cluster status`. This tells you what version you are currently on, and ensures that the cluster is running and ready. +. Copy the downloaded release bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. Use one of two methods: ++ +* Use any secure copy method, such as `winscp`. +* If your ThoughtSpot cluster allows access to Egnyte, you can use the following `wget` command: + [source,bash] ---- - $ tscli cluster status + $ wget -O ---- -+ -Make sure that the output says `Cluster: RUNNING`, `Database: READY`, and `Search Engine: READY`. -. Run `tscli cluster check` to ensure there are no component failures. +[#upgrade-step-2] +=== Step 2: Check disk space +. Check disk space by running `df -h`. `/tmp` needs at least 21 GB free for upgrades. + [source,bash] ---- - $ tscli cluster check + $ df -h ---- + -Make sure that the output for each component is `SUCCESS`. - -. If `tscli cluster status` or `tscli cluster check` return any errors, warnings, or failures, contact {support-url} before you proceed with the upgrade. - -[#upgrade-step-2] -=== Step 2: Download the release -You can download the release and move it to the correct directory in one of two ways: +The output of this command shows all disks in the ThoughtSpot node and their availability, in percentage and in number of GB. -* using the <> command -* using the <> - -[#fileserver] -==== Download the release using tscli -Use `tscli fileserver download-release` to download the release bundle. +[#upgrade-step-3] +=== Step 3: Take manual snapshot and backup +. Take a manual snapshot of the cluster. For information on how to take a manual snapshot, refer to xref:snapshots.adoc#manual-snapshot[Work with snapshots]. -// should they cd to the specific export directory here? +. Take a manual backup of the cluster. For information on how to take a manual backup, refer to xref:backup-manual.adoc[Create a manual backup]. -. First, ensure that you already xref:tscli-command-ref.adoc#tscli-fileserver[configured the fileserver] by running `tscli fileserver show-config`. +[#upgrade-step-4] +=== Step 4: Check cluster health and release integrity +. Run `tscli cluster status`. This tells you what version you are currently on, and ensures that the cluster is running and ready. + [source,bash] ---- -tscli fileserver show-config + $ tscli cluster status ---- -If the fileserver is already configured, the output is `Fileserver is configured`. ++ +Make sure that the output says `Cluster: RUNNING`, `Database: READY`, and `Search Engine: READY`. -.. If the fileserver is *not* configured, xref:tscli-command-ref.adoc#tscli-fileserver[configure the fileserver] by running `tscli fileserver configure --user USER --password PASSWORD`. +. Run `tscli cluster check` to ensure there are no component failures. + [source,bash] ---- -tscli fileserver configure --user USER --password PASSWORD + $ tscli cluster check ---- ++ +Make sure that the output for each component is `SUCCESS`. -. Download the release from the fileserver by running `tscli fileserver download-release --user --out `. +. Run `./capture_cluster_information.sh`. This provides {support-url} with information about your cluster in case of upgrade or post-upgrade failures. + [source,bash] ---- -$ tscli fileserver download-release --user --out + $ ./capture_cluster_information.sh ---- -+ -Note the following parameters: -+ -`release-number`:: is the release number of your ThoughtSpot instance, such as 5.3, 5.3.1, 6.0, and so on. -`username`:: is the username for the fileserver that you set up earlier, when configuring the fileserver. -`release-location`:: is the location path of the release bundle on your local machine. For example, `/export/sdb1/TS_TASKS/install/6.0.tar.gz`. -[#egnyte] -==== Download the release using the Egnyte link -// should they cd to the specific export directory here? -. On a machine that can also access your ThoughtSpot cluster, navigate to [LINK]. -. Select *Download*. [i'm not sure what this actually looks like on egnyte] -. Copy the downloaded release bundle to a specific path in the `/export` folder. -+ -You must copy the bundle to a different path in the `/export` folder, depending on your deployment platform: -+ -[#platform-export-path] --- -SMC, Dell, VMware, and GCP:: `/export/sdb1/TS_SRE_TASKS` -AWS:: `/export/xvdb1/TS_SRE_TASKS` -Azure:: `/export/sdc1/TS_SRE_TASKS` --- -+ -Run `scp .tar.gz admin@:/export//TS_TASKS/install/`. +. Check the integrity of the release by running the `md5sum -c ` command. Replace `MD5checksum file` with the name of the checksum file in your release bundle. + [source,bash] ---- -$ scp .tar.gz admin@:/export//TS_TASKS/install/ + $ md5sum -c ---- + -Note the following parameters: -+ -`release-number`:: is the release number of your ThoughtSpot instance, such as 5.3, 5.3.1, 6.0, and so on. -`hostname`:: is your specific hostname. -`path`:: is the path within the export folder. This differs based on your deployment platform; refer to <>. -`file-name`:: is the name of the tarball file on your local machine. +The output should be a green `OK`. -NOTE: You can use another secure copy method, if you prefer a method other than the `scp` command. +. If `tscli cluster status`, `tscli cluster check`, or the `md5sum` command return any errors, warnings, or failures, contact {support-url} before you proceed with the upgrade. -[#upgrade-step-3] -=== Step 3: Upgrade the cluster -. Run `tscli cluster update .tar.gz`. +[#upgrade-step-5] +=== Step 5: Upgrade the cluster +. Launch a screen session. Use screen to ensure that your installation does not stop if you lose network connectivity. ++ +[source,bash] +---- + $ screen -S upgrade +---- +. Run `tscli cluster update .tar.gz`. This may take about one hour. + Note the following parameters: `release-number`:: is the release number of your ThoughtSpot installation, such as `6.0`, `5.3`, `5.3.1`, and so on. -. The nodes reboot one by one. Wait about 15 minutes before you log back in. +. During the upgrade process, the node reboots. The node reboot logs you out of the node. Wait about 15 minutes before you `ssh` back in. If the `ssh` output says something similar to `Connection refused`, the node is still rebooting. + +. If you run into an error during upgrade, and the upgrade fails, refer to <>. -. To see which step the upgrade is in, run `tscli cluster status --tail`. +. To see which step the upgrade is in, run `tscli cluster status --tail`. When the upgrade is complete, the output of this command says that the upgrade is complete. + [source,bash] ---- @@ -174,8 +162,8 @@ NOTE: During the upgrade process, some services may temporarily be unavailable. . The upgrade takes about 1.5 hours to complete, including the objects upgrader. -[#upgrade-step-4] -=== Finalize installation +[#upgrade-step-6] +=== Step 6: Finalize installation . After the upgrade completes, log out of the shell. + [source,bash] @@ -184,13 +172,13 @@ $ logout ---- . `SSH` back into the cluster. -Run `ssh admin@`. +Run `ssh admin@`. + -Replace `nodeIP` with your specific network information. +Replace `ThoughtSpotnodeIP` with your specific network information. + [source,bash] ---- - $ ssh admin@ + $ ssh admin@ ---- . To check that the cluster is ready, run `tscli cluster status`. @@ -202,8 +190,22 @@ Replace `nodeIP` with your specific network information. + Ensure that the `DATABASE` and `SEARCH ENGINE` fields in the `tscli cluster status` command show `READY`. +. Run `tscli cluster check` to ensure there are no component failures. ++ +[source,bash] +---- + $ tscli cluster check +---- ++ +Make sure that the output for each component is `SUCCESS`. + . Sign into the ThoughtSpot application on your browser. +[#upgrade-step-7] +=== Step 7: Apply patches +Your release bundle may include patches for the ThoughtSpot application. These patches update services and provide bug fixes. If your release bundle included any patches, <>. + +[#patch] == Patch a cluster To patch your cluster, follow these steps: [cols="5,~",grid=none,frame=none] @@ -215,14 +217,33 @@ To patch your cluster, follow these steps: |=== [#patch-step-1] -=== Obtain cluster patch -ThoughtSpot Support must provide you with the correct patch to apply. Do not apply any other patches. xref:support-contact.adoc[] to obtain the correct cluster patch. +=== Step 1: Obtain cluster patch +Download the patch or patches for the version you are running, and move it to a folder in your ThoughtSpot node. If you just upgraded to a new version, the release bundle you downloaded and copied to your ThoughtSpot node should contain the patches you need to apply. -After ThoughtSpot Support supplies you with the patch, copy it to your cluster. +. Navigate to [LINK]. +. Download the release content folder or patch folder for the version you are running. [i'm not sure what this actually looks like on egnyte] +. In your terminal application, `ssh` into your cluster. +Run `ssh admin@`. ++ +Replace `ThoughtSpotnodeIP` with your specific network information. ++ +[source,bash] +---- + $ ssh admin@ +---- +. Copy the downloaded release or patch bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. Use one of two methods: ++ +* Use any secure copy method, such as `winscp`. +* If your ThoughtSpot cluster allows access to Egnyte, you can use the following `wget` command: ++ +[source,bash] +---- + $ wget -O +---- [#patch-step-2] === Step 2: Verify patch integrity -To verify the integrity of the patch file, check the checksum. +To verify the integrity of the patch files, check the checksum for each patch. Run `md5sum -c .tar.gz.MD5checksum`. @@ -235,7 +256,7 @@ Your output says `ok` if you have the correct release. [#patch-step-3] === Step 3: Apply the patch to the cluster -Run `tscli patch apply `. +Run `tscli patch apply ` for each patch. [source,bash] ---- $ tscli patch apply `. + +. Run the following command to resume the upgrade: ++ +[source,bash] +---- + $ tscli cluster resume-update +---- +. If that command does not work, or if the upgrade fails again, contact {support-url}. \ No newline at end of file From 93b9e342f5fb4dfff8ebae611c7397129b34c133 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Mon, 9 May 2022 16:32:22 -0700 Subject: [PATCH 04/14] small changes to the upgrade article per SME review Signed-off-by: Teresa Killmond --- software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index bdf7652937..088c9571b4 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -58,7 +58,7 @@ NOTE: If any commands return errors, warnings, or failures while you upgrade, co Download the release, including any patches for the version you are installing, and move it to a folder in your ThoughtSpot node. . Navigate to [LINK]. -. Download the release content folder for the version you would like to upgrade to. [i'm not sure what this actually looks like on egnyte] +. Download the release content folder for the version you would like to upgrade to. This includes the release, any patches, the `Md5checksum`, and `capture_cluster_information.sh`. [i'm not sure what this actually looks like on egnyte] . In your terminal application, `ssh` into your cluster. Run `ssh admin@`. + @@ -70,7 +70,7 @@ Replace `ThoughtSpotnodeIP` with your specific network information. ---- . Copy the downloaded release bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. Use one of two methods: + -* Use any secure copy method, such as `winscp`. +* Use `winscp` or a similar secure copy method. * If your ThoughtSpot cluster allows access to Egnyte, you can use the following `wget` command: + [source,bash] @@ -115,7 +115,7 @@ Make sure that the output says `Cluster: RUNNING`, `Database: READY`, and `Searc + Make sure that the output for each component is `SUCCESS`. -. Run `./capture_cluster_information.sh`. This provides {support-url} with information about your cluster in case of upgrade or post-upgrade failures. +. Run `./capture_cluster_information.sh`. This captures current information about your cluster in case {support-url} needs it for any purpose after the upgrade. + [source,bash] ---- From 5d48943213b807d9260761a65c6cbf00db0458a0 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Tue, 10 May 2022 15:34:47 -0700 Subject: [PATCH 05/14] upgrade updates Signed-off-by: Teresa Killmond --- software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 088c9571b4..a673728eff 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -203,7 +203,7 @@ Make sure that the output for each component is `SUCCESS`. [#upgrade-step-7] === Step 7: Apply patches -Your release bundle may include patches for the ThoughtSpot application. These patches update services and provide bug fixes. If your release bundle included any patches, <>. +Your release bundle may include patches for the ThoughtSpot application. These patches update services and provide bug fixes. If your release bundle includes any patches, <>. [#patch] == Patch a cluster From 2880fc3bb8b86b9b95858e8e0660182b70e18dc2 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Mon, 16 May 2022 13:56:09 -0700 Subject: [PATCH 06/14] rohit's comments part 1 Signed-off-by: Teresa Killmond --- .../ROOT/pages/sysadmin-cluster-upgrade.adoc | 43 +++++++++---------- 1 file changed, 20 insertions(+), 23 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index a673728eff..54dcbd0409 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -44,8 +44,8 @@ To upgrade your cluster, follow these steps: |=== | ❏ | <> | ❏ | <> -| ❏ | <> -| ❏ | <> +| ❏ | <> +| ❏ | <> | ❏ | <> | ❏ | <> | ❏ | <> @@ -90,13 +90,13 @@ Replace `ThoughtSpotnodeIP` with your specific network information. The output of this command shows all disks in the ThoughtSpot node and their availability, in percentage and in number of GB. [#upgrade-step-3] -=== Step 3: Take manual snapshot and backup +=== Step 3: Create a cluster snapshot and back it up on disk . Take a manual snapshot of the cluster. For information on how to take a manual snapshot, refer to xref:snapshots.adoc#manual-snapshot[Work with snapshots]. . Take a manual backup of the cluster. For information on how to take a manual backup, refer to xref:backup-manual.adoc[Create a manual backup]. [#upgrade-step-4] -=== Step 4: Check cluster health and release integrity +=== Step 4: Check the cluster's health and integrity of provided software . Run `tscli cluster status`. This tells you what version you are currently on, and ensures that the cluster is running and ready. + [source,bash] @@ -104,7 +104,7 @@ The output of this command shows all disks in the ThoughtSpot node and their ava $ tscli cluster status ---- + -Make sure that the output says `Cluster: RUNNING`, `Database: READY`, and `Search Engine: READY`. +Make sure that the output includes `Cluster: RUNNING`, `Database: READY`, and `Search Engine: READY` or `UPDATING`. Make sure the output does NOT include any errors. . Run `tscli cluster check` to ensure there are no component failures. + @@ -129,7 +129,7 @@ Make sure that the output for each component is `SUCCESS`. $ md5sum -c ---- + -The output should be a green `OK`. +The output should be a list of filenames, followed by `OK`. . If `tscli cluster status`, `tscli cluster check`, or the `md5sum` command return any errors, warnings, or failures, contact {support-url} before you proceed with the upgrade. @@ -188,7 +188,7 @@ Replace `ThoughtSpotnodeIP` with your specific network information. $ tscli cluster status ---- + -Ensure that the `DATABASE` and `SEARCH ENGINE` fields in the `tscli cluster status` command show `READY`. +Ensure that the `DATABASE` and `SEARCH ENGINE` fields in the `tscli cluster status` command output show `READY`, and that the output reports no errors. . Run `tscli cluster check` to ensure there are no component failures. + @@ -201,6 +201,11 @@ Make sure that the output for each component is `SUCCESS`. . Sign into the ThoughtSpot application on your browser. +. Verify the release version in the UI matches the version you upgraded to: + +.. Navigate to *Admin > Cluster*. +.. In the *Cluster details* panel, confirm that the *Release* version matches the version you upgraded to. + [#upgrade-step-7] === Step 7: Apply patches Your release bundle may include patches for the ThoughtSpot application. These patches update services and provide bug fixes. If your release bundle includes any patches, <>. @@ -242,26 +247,17 @@ Replace `ThoughtSpotnodeIP` with your specific network information. ---- [#patch-step-2] -=== Step 2: Verify patch integrity -To verify the integrity of the patch files, check the checksum for each patch. - -Run `md5sum -c .tar.gz.MD5checksum`. - -[source,bash] ----- - $ md5sum -c .tar.gz.MD5checksum ----- - -Your output says `ok` if you have the correct release. +=== Step 2: Verify integrity of patch files +To verify the integrity of the patch files, check the checksum for each patch, as explained in step 4 of <>. [#patch-step-3] === Step 3: Apply the patch to the cluster -Run `tscli patch apply ` for each patch. +Run `tscli patch apply ` for each patch, one file at a time. [source,bash] ---- $ tscli patch apply `. +During the upgrade process, after you run `tscli cluster update `, the upgrade may fail with a message similar to `Update failed for cluster `. -. Run the following command to resume the upgrade: +. Try to resume the upgrade with the following command: + [source,bash] ---- $ tscli cluster resume-update ---- -. If that command does not work, or if the upgrade fails again, contact {support-url}. \ No newline at end of file +. If the `resume-update` command does not work, and you see an error you have fixed with the help of {support-url} before, make a note of it and fix the error in the same way that {support-url} did, and run `tscli cluster resume-update` again. Do *NOT* try to fix any errors that you have not fixed with {support-url} in previous upgrades. +. If the upgrade fails again, contact {support-url}. \ No newline at end of file From de81d6a68179b1b5be28c1b64d0a156f0f04b7b1 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Wed, 1 Jun 2022 15:48:59 -0700 Subject: [PATCH 07/14] comments from rohit Signed-off-by: Teresa Killmond --- .../ROOT/pages/sysadmin-cluster-upgrade.adoc | 36 ++++++++----------- 1 file changed, 15 insertions(+), 21 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 54dcbd0409..dabb8c8289 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -42,7 +42,7 @@ The *Upgrade* page shows your cluster's current version, and any new releases th To upgrade your cluster, follow these steps: [cols="5,~",grid=none,frame=none] |=== -| ❏ | <> +| ❏ | <> | ❏ | <> | ❏ | <> | ❏ | <> @@ -54,12 +54,16 @@ To upgrade your cluster, follow these steps: NOTE: If any commands return errors, warnings, or failures while you upgrade, contact {support-url}. [#upgrade-step-1] -=== Step 1: Download the release -Download the release, including any patches for the version you are installing, and move it to a folder in your ThoughtSpot node. +=== Step 1: Download the release files +Download the release, including any patches for the version you are installing, to your computer. Move the release files to a directory with sufficient space in your ThoughtSpot node. + +WARNING: Do not put the release files in the `/home/admin` or `/home/thoughtspot` directories. You should put the release files in one of the bigger hard disks/volumes on one node in the cluster. Typically, this is `/export/sdb1/` for hardware appliances, VMware and GCP, `/export/sdc1/` for Azure, and `/export/xvdb1/` for AWS. + +To download the release files, follow these steps: . Navigate to [LINK]. -. Download the release content folder for the version you would like to upgrade to. This includes the release, any patches, the `Md5checksum`, and `capture_cluster_information.sh`. [i'm not sure what this actually looks like on egnyte] -. In your terminal application, `ssh` into your cluster. +. Download the release content folder for the version you would like to upgrade to. This includes the release, any patches, the `Md5checksum`, and the `capture_cluster_information.sh` script. This script inventories a ThoughtSpot cluster for use in audit or recovery, if necessary. [i'm not sure what this actually looks like on egnyte] +. In your terminal application, connect to the first node in your ThoughtSpot cluster using `ssh`. Run `ssh admin@`. + Replace `ThoughtSpotnodeIP` with your specific network information. @@ -71,11 +75,11 @@ Replace `ThoughtSpotnodeIP` with your specific network information. . Copy the downloaded release bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. Use one of two methods: + * Use `winscp` or a similar secure copy method. -* If your ThoughtSpot cluster allows access to Egnyte, you can use the following `wget` command: +* If your ThoughtSpot cluster allows access to Egnyte, you download the files directly to the node. Use the following `wget` command: + [source,bash] ---- - $ wget -O + $ wget -c -O ---- [#upgrade-step-2] @@ -212,7 +216,7 @@ Your release bundle may include patches for the ThoughtSpot application. These p [#patch] == Patch a cluster -To patch your cluster, follow these steps: +Patching a cluster updates the ThoughtSpot services and allows for bug fixes. The process is similar to upgrading a cluster. To patch your cluster, follow these steps: [cols="5,~",grid=none,frame=none] |=== | ❏ | <> @@ -226,7 +230,7 @@ To patch your cluster, follow these steps: Download the patch or patches for the version you are running, and move it to a folder in your ThoughtSpot node. If you just upgraded to a new version, the release bundle you downloaded and copied to your ThoughtSpot node should contain the patches you need to apply. . Navigate to [LINK]. -. Download the release content folder or patch folder for the version you are running. [i'm not sure what this actually looks like on egnyte] +. Download the required patches for the ThoughtSpot version you are running. [i'm not sure what this actually looks like on egnyte] . In your terminal application, `ssh` into your cluster. Run `ssh admin@`. + @@ -236,15 +240,7 @@ Replace `ThoughtSpotnodeIP` with your specific network information. ---- $ ssh admin@ ---- -. Copy the downloaded release or patch bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. Use one of two methods: -+ -* Use any secure copy method, such as `winscp`. -* If your ThoughtSpot cluster allows access to Egnyte, you can use the following `wget` command: -+ -[source,bash] ----- - $ wget -O ----- +. Copy the downloaded patch bundle from your Downloads folder to the first node in your ThoughtSpot cluster. To copy the patch bundle, refer to step 4 of <>. [#patch-step-2] === Step 2: Verify integrity of patch files @@ -261,9 +257,7 @@ The patch process for each patch takes about 10 minutes. Once a patch has been a [#patch-step-4] === Step 4: Finalize installation -Ensure that ThoughtSpot applied the patches successfully. - -Run `tscli patch ls` and look for the new patch names. +Ensure that ThoughtSpot applied the patches successfully by running the `tscli patch ls` command. Ensure that the new patches you applied appear in the output. [source,bash] ---- From 8b709a029c8364b328f2d315c768f1b6558f187d Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Wed, 1 Jun 2022 16:08:00 -0700 Subject: [PATCH 08/14] typo Signed-off-by: Teresa Killmond --- software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index dabb8c8289..22edd5b064 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -251,7 +251,7 @@ To verify the integrity of the patch files, check the checksum for each patch, a Run `tscli patch apply ` for each patch, one file at a time. [source,bash] ---- - $ tscli patch apply ---- The patch process for each patch takes about 10 minutes. Once a patch has been applied successfully, you can proceed with the next patch immediately, without waiting for any services to restart and without carrying out any extra checks. From 64493fb5538a86848ded65118f1b2b481f107245 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Wed, 8 Jun 2022 14:28:03 -0700 Subject: [PATCH 09/14] feedback from laszlo Signed-off-by: Teresa Killmond --- .../ROOT/pages/sysadmin-cluster-upgrade.adoc | 123 +++++++++++------- 1 file changed, 74 insertions(+), 49 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 22edd5b064..abcb81a730 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -59,32 +59,37 @@ Download the release, including any patches for the version you are installing, WARNING: Do not put the release files in the `/home/admin` or `/home/thoughtspot` directories. You should put the release files in one of the bigger hard disks/volumes on one node in the cluster. Typically, this is `/export/sdb1/` for hardware appliances, VMware and GCP, `/export/sdc1/` for Azure, and `/export/xvdb1/` for AWS. -To download the release files, follow these steps: +To download the release files, use one of two methods: -. Navigate to [LINK]. -. Download the release content folder for the version you would like to upgrade to. This includes the release, any patches, the `Md5checksum`, and the `capture_cluster_information.sh` script. This script inventories a ThoughtSpot cluster for use in audit or recovery, if necessary. [i'm not sure what this actually looks like on egnyte] -. In your terminal application, connect to the first node in your ThoughtSpot cluster using `ssh`. -Run `ssh admin@`. +* Download the release files from Egnyte, and use `winscp` or a similar secure copy method to copy the downloaded release bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. + -Replace `ThoughtSpotnodeIP` with your specific network information. -+ -[source,bash] ----- - $ ssh admin@ ----- -. Copy the downloaded release bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. Use one of two methods: +<>. + +* If your ThoughtSpot cluster has internet access and allows access to Egnyte, you can download the files directly to any path in the `/export` folder of your ThoughtSpot node that has enough space, using a `wget` command. + -* Use `winscp` or a similar secure copy method. -* If your ThoughtSpot cluster allows access to Egnyte, you download the files directly to the node. Use the following `wget` command: +<>. + +[#download-scp] +==== Download using secure copy +. Navigate to [RELEASE LINK]. +. Download the release content folder for the version you would like to upgrade to. This includes the release, any patches, the `Md5checksum`, and the `capture_cluster_information.sh` script. This script inventories a ThoughtSpot cluster for use in audit or recovery, if necessary. +. In your terminal application, log in to your ThoughtSpot node using `ssh`. +. Copy the downloaded release bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space, using winscp or a similar secure copy method. + +[#download-wget] +==== Download using wget +. In your terminal application, log in to your ThoughtSpot node using `ssh`. +. Navigate to any path in the `/export` folder of your ThoughtSpot node that has enough space. +. If your ThoughtSpot cluster has internet access and allows access to Egnyte, you can download the files directly to the node. Use the following `wget` command: + [source,bash] ---- - $ wget -c -O + $ wget -c -O ---- [#upgrade-step-2] === Step 2: Check disk space -. Check disk space by running `df -h`. `/tmp` needs at least 21 GB free for upgrades. +. Check disk space by running `df -h`. `/tmp` needs at least 21 GB free for upgrades, and you need at least 10% availability on every other filesystem, except for `/update`. + [source,bash] ---- @@ -108,7 +113,7 @@ The output of this command shows all disks in the ThoughtSpot node and their ava $ tscli cluster status ---- + -Make sure that the output includes `Cluster: RUNNING`, `Database: READY`, and `Search Engine: READY` or `UPDATING`. Make sure the output does NOT include any errors. +Make sure that the output includes `Cluster: RUNNING`, `Database: READY`, and `Search Engine: READY`, `PENDING`, `INDEXING`, or `UPDATING`. Make sure the output does NOT include any errors. . Run `tscli cluster check` to ensure there are no component failures. + @@ -126,11 +131,11 @@ Make sure that the output for each component is `SUCCESS`. $ ./capture_cluster_information.sh ---- -. Check the integrity of the release by running the `md5sum -c ` command. Replace `MD5checksum file` with the name of the checksum file in your release bundle. +. Check the integrity of the release by running the `md5sum -c ` command. Replace `checksum file name` with the name of the checksum file in your release bundle. It will likely be in the format `\*.MD*.` + [source,bash] ---- - $ md5sum -c + $ md5sum -c ---- + The output should be a list of filenames, followed by `OK`. @@ -149,7 +154,7 @@ The output should be a list of filenames, followed by `OK`. + Note the following parameters: -`release-number`:: is the release number of your ThoughtSpot installation, such as `6.0`, `5.3`, `5.3.1`, and so on. +`release-number`:: is the release number of your ThoughtSpot installation, such as 8.4.1.sw, 7.2.1, and so on. . During the upgrade process, the node reboots. The node reboot logs you out of the node. Wait about 15 minutes before you `ssh` back in. If the `ssh` output says something similar to `Connection refused`, the node is still rebooting. @@ -162,29 +167,12 @@ Note the following parameters: $ tscli cluster status --tail ---- + -NOTE: During the upgrade process, some services may temporarily be unavailable. Their status in the `tscli cluster status --tail` command might be `FAILURE`. The service should return to `SUCCESS` as the upgrade continues. If a service continues to fail, xref:support-contact.adoc[contact ThoughtSpot Support]. +NOTE: During the upgrade process, some services may temporarily be unavailable. The status of an update task in the `tscli cluster status --tail` command might be `FAILURE`. In this case, the installer will run the command again, and the update task should return to `SUCCESS. If an update task continues to fail, xref:support-contact.adoc[contact ThoughtSpot Support]. -. The upgrade takes about 1.5 hours to complete, including the objects upgrader. +. The upgrade takes about 1.5 hours to complete. [#upgrade-step-6] === Step 6: Finalize installation -. After the upgrade completes, log out of the shell. -+ -[source,bash] ----- -$ logout ----- - -. `SSH` back into the cluster. -Run `ssh admin@`. -+ -Replace `ThoughtSpotnodeIP` with your specific network information. -+ -[source,bash] ----- - $ ssh admin@ ----- - . To check that the cluster is ready, run `tscli cluster status`. + [source,bash] @@ -192,7 +180,7 @@ Replace `ThoughtSpotnodeIP` with your specific network information. $ tscli cluster status ---- + -Ensure that the `DATABASE` and `SEARCH ENGINE` fields in the `tscli cluster status` command output show `READY`, and that the output reports no errors. +Ensure that the `DATABASE` and `SEARCH ENGINE` fields in the `tscli cluster status` command output show `READY`, and that the output reports no errors. It may take up to an hour for the `DATABASE` and `SEARCH ENGINE` fields to show `READY`, depending on how much data you have. . Run `tscli cluster check` to ensure there are no component failures. + @@ -203,7 +191,7 @@ Ensure that the `DATABASE` and `SEARCH ENGINE` fields in the `tscli cluster stat + Make sure that the output for each component is `SUCCESS`. -. Sign into the ThoughtSpot application on your browser. +. Sign in to the ThoughtSpot application on your browser. Make sure you sign in to ThoughtSpot in a new tab. . Verify the release version in the UI matches the version you upgraded to: @@ -229,18 +217,35 @@ Patching a cluster updates the ThoughtSpot services and allows for bug fixes. Th === Step 1: Obtain cluster patch Download the patch or patches for the version you are running, and move it to a folder in your ThoughtSpot node. If you just upgraded to a new version, the release bundle you downloaded and copied to your ThoughtSpot node should contain the patches you need to apply. -. Navigate to [LINK]. -. Download the required patches for the ThoughtSpot version you are running. [i'm not sure what this actually looks like on egnyte] -. In your terminal application, `ssh` into your cluster. -Run `ssh admin@`. +WARNING: Do not put the patch files in the `/home/admin` or `/home/thoughtspot` directories. You should put the patch files in one of the bigger hard disks/volumes on one node in the cluster. Typically, this is `/export/sdb1/` for hardware appliances, VMware and GCP, `/export/sdc1/` for Azure, and `/export/xvdb1/` for AWS. + +To download the patch files, use one of two methods: + +* Download the patch files from Egnyte, and use `winscp` or a similar secure copy method to copy the downloaded patch bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. ++ +<>. + +* If your ThoughtSpot cluster has internet access and allows access to Egnyte, you can download the files directly to any path in the `/export` folder of your ThoughtSpot node that has enough space, using a `wget` command. + -Replace `ThoughtSpotnodeIP` with your specific network information. +<>. + +[#download-scp-patch] +==== Download using secure copy +. Navigate to [PATCH LINK]. +. Download the patch folder for the version you would like to upgrade to. This includes the patches, the `Md5checksum`, and the `capture_cluster_information.sh` script. This script inventories a ThoughtSpot cluster for use in audit or recovery, if necessary. +. In your terminal application, log in to your ThoughtSpot node using `ssh`. +. Copy the downloaded patch bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space, using winscp or a similar secure copy method. + +[#download-wget-patch] +==== Download using wget +. In your terminal application, log in to your ThoughtSpot node using `ssh`. +. Navigate to any path in the `/export` folder of your ThoughtSpot node that has enough space. +. If your ThoughtSpot cluster has internet access and allows access to Egnyte, you can download the patch files directly to the node. Use the following `wget` command: + [source,bash] ---- - $ ssh admin@ + $ wget -c -O ---- -. Copy the downloaded patch bundle from your Downloads folder to the first node in your ThoughtSpot cluster. To copy the patch bundle, refer to step 4 of <>. [#patch-step-2] === Step 2: Verify integrity of patch files @@ -257,13 +262,33 @@ The patch process for each patch takes about 10 minutes. Once a patch has been a [#patch-step-4] === Step 4: Finalize installation -Ensure that ThoughtSpot applied the patches successfully by running the `tscli patch ls` command. Ensure that the new patches you applied appear in the output. +Ensure that ThoughtSpot applied the patches successfully by running several commands. +. Ensure that the new patches you applied appear in the `tscli patch ls` output. ++ [source,bash] ---- $ tscli patch ls ---- +. To check that the cluster is ready, run `tscli cluster status`. ++ +[source,bash] +---- + $ tscli cluster status +---- ++ +Ensure that the `DATABASE` and `SEARCH ENGINE` fields in the `tscli cluster status` command output show `READY`, and that the output reports no errors. It may take up to an hour for the `DATABASE` and `SEARCH ENGINE` fields to show `READY`, depending on how much data you have. + +. Run `tscli cluster check` to ensure there are no component failures. ++ +[source,bash] +---- + $ tscli cluster check +---- ++ +Make sure that the output for each component is `SUCCESS`. + [#error-recovery] == Error recovery From 49d7e9b3c6c74aac93c69e776634a133b18e7d15 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Thu, 30 Jun 2022 11:40:21 -0700 Subject: [PATCH 10/14] upgrade comments from raja sudhan Signed-off-by: Teresa Killmond --- software/modules/ROOT/nav.adoc | 2 +- software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/software/modules/ROOT/nav.adoc b/software/modules/ROOT/nav.adoc index 4dddd85cfb..60f3e4eb56 100644 --- a/software/modules/ROOT/nav.adoc +++ b/software/modules/ROOT/nav.adoc @@ -275,7 +275,7 @@ span.badge.badge-new { ** xref:sysadmin-overview.adoc[System administration] *** xref:send-logs-to-administrator.adoc[Send logs when reporting problems] -*** xref:sysadmin-cluster-upgrade.adoc[Upgrade a cluster] +*** xref:sysadmin-cluster-upgrade.adoc[Upgrade or patch a single-node ThoughtSpot CentOS cluster] ** xref:backup-strategy.adoc[] *** xref:backup-schedule.adoc[Understand backup/snapshot schedules] *** xref:snapshots.adoc[Work with snapshots] diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index abcb81a730..09b4fbfabb 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -1,4 +1,4 @@ -= Upgrade or patch a cluster += Upgrade or patch a single-node ThoughtSpot CentOS cluster :last_updated: 01/10/2020 :linkattrs: :page-aliases: /admin/system-admin/upgrade-a-cluster.adoc @@ -167,7 +167,7 @@ Note the following parameters: $ tscli cluster status --tail ---- + -NOTE: During the upgrade process, some services may temporarily be unavailable. The status of an update task in the `tscli cluster status --tail` command might be `FAILURE`. In this case, the installer will run the command again, and the update task should return to `SUCCESS. If an update task continues to fail, xref:support-contact.adoc[contact ThoughtSpot Support]. +NOTE: During the upgrade process, some services may temporarily be unavailable. The status of an update task in the `tscli cluster status --tail` command might be `FAILURE`. In this case, the installer will run the command repeatedly until the update task status is `SUCCESS. If an update task continues to fail, xref:support-contact.adoc[contact ThoughtSpot Support]. . The upgrade takes about 1.5 hours to complete. @@ -262,7 +262,7 @@ The patch process for each patch takes about 10 minutes. Once a patch has been a [#patch-step-4] === Step 4: Finalize installation -Ensure that ThoughtSpot applied the patches successfully by running several commands. +Ensure that ThoughtSpot applied the patches successfully by running the following commands. . Ensure that the new patches you applied appear in the `tscli patch ls` output. + From 80c2394f48d6f79eb911831accc95d850982be55 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Thu, 30 Jun 2022 11:49:16 -0700 Subject: [PATCH 11/14] more feedback from raja Signed-off-by: Teresa Killmond --- software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 09b4fbfabb..99d0fc7873 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -61,7 +61,7 @@ WARNING: Do not put the release files in the `/home/admin` or `/home/thoughtspot To download the release files, use one of two methods: -* Download the release files from Egnyte, and use `winscp` or a similar secure copy method to copy the downloaded release bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space. +* Download the release files from Egnyte, and use `winscp` or a similar secure copy method to copy the downloaded release bundle from your Downloads folder to any path in the `/export` folder of your ThoughtSpot node that has enough space, at least 20 GB. + <>. @@ -89,7 +89,7 @@ To download the release files, use one of two methods: [#upgrade-step-2] === Step 2: Check disk space -. Check disk space by running `df -h`. `/tmp` needs at least 21 GB free for upgrades, and you need at least 10% availability on every other filesystem, except for `/update`. +. Check disk space by running `df -h`. `/tmp` needs at least 21 GB free for upgrades, `/export` needs at least 30 GB free, and you need at least 10% availability on every other filesystem, except for `/update`. + [source,bash] ---- From 587fde1b989c4f3f14493e14d7b2dfa021162670 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Fri, 1 Jul 2022 10:31:38 -0700 Subject: [PATCH 12/14] tmp fix and feedback from yagnika Signed-off-by: Teresa Killmond --- software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 99d0fc7873..77fb6a92b3 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -89,7 +89,7 @@ To download the release files, use one of two methods: [#upgrade-step-2] === Step 2: Check disk space -. Check disk space by running `df -h`. `/tmp` needs at least 21 GB free for upgrades, `/export` needs at least 30 GB free, and you need at least 10% availability on every other filesystem, except for `/update`. +. Check disk space by running `df -h`. `/tmp` needs at least 50 GB free for upgrades, `/export` needs at least 30 GB free, and you need at least 10% availability on every other filesystem, except for `/update`. + [source,bash] ---- @@ -100,9 +100,9 @@ The output of this command shows all disks in the ThoughtSpot node and their ava [#upgrade-step-3] === Step 3: Create a cluster snapshot and back it up on disk -. Take a manual snapshot of the cluster. For information on how to take a manual snapshot, refer to xref:snapshots.adoc#manual-snapshot[Work with snapshots]. +. Take a manual snapshot of the cluster. For information on how to take a manual snapshot, refer to xref:snapshots.adoc#manual-snapshot[Work with snapshots]. Note that snapshot creation may take some time. -. Take a manual backup of the cluster. For information on how to take a manual backup, refer to xref:backup-manual.adoc[Create a manual backup]. +. Take a manual backup of the cluster. For information on how to take a manual backup, refer to xref:backup-manual.adoc[Create a manual backup]. Note that taking a backup may take some time. [#upgrade-step-4] === Step 4: Check the cluster's health and integrity of provided software From 10d25367394cb16f7c9eb00072c3ab21661ac8b0 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Fri, 8 Jul 2022 10:59:25 -0700 Subject: [PATCH 13/14] error recovery from yagnika Signed-off-by: Teresa Killmond --- .../ROOT/pages/sysadmin-cluster-upgrade.adoc | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 77fb6a92b3..14ffa41218 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -158,7 +158,7 @@ Note the following parameters: . During the upgrade process, the node reboots. The node reboot logs you out of the node. Wait about 15 minutes before you `ssh` back in. If the `ssh` output says something similar to `Connection refused`, the node is still rebooting. -. If you run into an error during upgrade, and the upgrade fails, refer to <>. +. If you run into an error during upgrade, and the upgrade fails, refer to <>. If the `HDFS fsimage check` returns `FAILURE`, refer to the <>. . To see which step the upgrade is in, run `tscli cluster status --tail`. When the upgrade is complete, the output of this command says that the upgrade is complete. + @@ -292,6 +292,23 @@ Make sure that the output for each component is `SUCCESS`. [#error-recovery] == Error recovery +[#fsimage] +=== HDFS fsimage check failure +During the upgrade process, after you run `tscli cluster update `, the `HDFS fsimage check` may return `FAILURE`. If this happens, there is an issue with the DNS resolution. + +. Add the cluster hostname to `/etc/hosts`. + +. Try to resume the upgrade with the following command: ++ +[source,bash] +---- + $ tscli cluster resume-update +---- + +. If the `HDFS fsimage check` fails again, contact {support-url}. + +[#tscli-cluster-update] +=== Upgrade failure during tscli cluster update During the upgrade process, after you run `tscli cluster update `, the upgrade may fail with a message similar to `Update failed for cluster `. . Try to resume the upgrade with the following command: From 5117586d6571d22715eeab43302b966c04e7b639 Mon Sep 17 00:00:00 2001 From: Teresa Killmond Date: Tue, 2 Aug 2022 10:24:51 -0700 Subject: [PATCH 14/14] feedback from raj Signed-off-by: Teresa Killmond --- software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc | 4 ---- 1 file changed, 4 deletions(-) diff --git a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc index 14ffa41218..86f3cbb61a 100644 --- a/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc +++ b/software/modules/ROOT/pages/sysadmin-cluster-upgrade.adoc @@ -24,10 +24,6 @@ These include Java, Boost C{pp} libraries, Google protocol buffers, and so on. ThoughtSpot patches the Operating System at the time of upgrades. Installation and upgrade use the same process, replacing the older OS image on a node by the new image delivered in the release tarball. -ThoughtSpot Support will contact you to schedule an update when a minor or major upgrade becomes available. - -// is this last sentence still true - == View available releases NOTE: This feature is now deprecated. You may not use it starting with the 7.2.1 release. For details, see xref:deprecation.adoc[Deprecation Announcements].