Addressing Leonid's comments

Signed-off-by: Lazar Cvetković <[email protected]>

docs/developers_guide.md

@@ -13,7 +13,7 @@ cd vhive
 ./scripts/install_go.sh; source /etc/profile # or install Go manually
 pushd scripts && go build -o setup_tool && popd && mv scripts/setup_tool .
 
-./setup_tool setup_node [stock-only|gvisor|firecracker]
+./setup_tool setup_node REGULAR [stock-only|gvisor|firecracker]
 sudo containerd
 ./setup_tool create_one_node_cluster [stock-only|gvisor|firecracker]
 # wait for the containers to boot up using
@@ -115,7 +115,7 @@ Assuming you rented a node using the vHive CloudLab profile:
 1. Setup the node for the desired sandbox:
 
 ```bash
-./setup_tool setup_node [firecracker|gvisor]
+./setup_tool setup_node REGULAR [firecracker|gvisor]
 ```
 
 2. Setup the CRI test environment for the desired sandbox:
@@ -240,7 +240,7 @@ Knative functions can use GPU although only `stock-only` mode is supported.
 Follow the guide to [setup stock knative](#testing-stock-knative-setup-or-images). 
 
 ``` bash
-./setup_tool setup_node stock-only
+./setup_tool setup_node REGULAR stock-only
 ```
 
 ### Install NVIDIA Driver and NVIDIA Container Toolkit 

docs/logging.md

@@ -41,7 +41,7 @@ We present how to set up a multi-node cluster, however, the same modifications c
 
 3. Run the node setup script:
     ```bash
-    ./setup_tool setup_node
+    ./setup_tool setup_node REGULAR
     ```
     > **BEWARE:**
     >

docs/quickstart_guide.md

@@ -92,7 +92,7 @@ Another option is to install using official instructions: [https://golang.org/do
     > flags as follows:
     >
     > ```bash
-    > ./setup_tool setup_node stock-only use-stargz
+    > ./setup_tool setup_node REGULAR stock-only use-stargz
     > ```
     > **IMPORTANT**
     > Currently `stargz` is only supported in native kubelet contexts without firecracker. 
@@ -103,7 +103,7 @@ Another option is to install using official instructions: [https://golang.org/do
 
     For the standard setup, run the following script:
     ```bash
-    ./setup_tool setup_node firecracker
+    ./setup_tool setup_node REGULAR firecracker
     ```
     > **BEWARE:**
     >
@@ -250,13 +250,13 @@ In essence, you will execute the same commands for master and worker setups but
 Execute the following below **as a non-root user with sudo rights** using **bash**:
 1. Run the node setup script:
     ```bash
-    ./setup_tool setup_node firecracker
+    ./setup_tool setup_node REGULAR firecracker
     ```
     > **Note:**
     > To enable runs with `stargz` images, setup kubelet by adding the `stock-only` and `use-stargz`
     > flags as follows:
     >   ```bash
-    >   ./setup_tool setup_node stock-only use-stargz
+    >   ./setup_tool setup_node REGULAR stock-only use-stargz
     >   ```
     > **IMPORTANT**
     > Currently `stargz` is only supported in native kubelet contexts without firecracker. 

scripts/README.md

@@ -1,87 +1,116 @@
 # vHive Setup Scripts
+
 - [vHive Setup Scripts](#vhive-setup-scripts)
-  - [1. Get Setup Tool](#1-get-setup-tool)
-    - [1.1 Download the Binary Executable Directly](#11-download-the-binary-executable-directly)
-    - [1.2 Build from Source](#12-build-from-source)
-  - [2. Config the Setup Tool](#2-config-the-setup-tool)
-  - [3. Use of Setup Tool](#3-use-of-setup-tool)
-    - [3.1 General Usage](#31-general-usage)
-    - [3.2 Specify Config Files](#32-specify-config-files)
-    - [3.3 Use with Local vHive Repo](#33-use-with-local-vhive-repo)
-    - [3.4 Use with Remote vHive Repo (Standalone Use)](#34-use-with-remote-vhive-repo-standalone-use)
-    - [3.5 Migrate from Legacy Shell Scripts](#35-migrate-from-legacy-shell-scripts)
-  - [4. Logs](#4-logs)
-  - [5. Supported Platform](#5-supported-platform)
+    - [1. Get Setup Tool](#1-get-setup-tool)
+        - [1.1 Download the Binary Executable Directly](#11-download-the-binary-executable-directly)
+        - [1.2 Build from Source](#12-build-from-source)
+    - [2. Config the Setup Tool](#2-config-the-setup-tool)
+    - [3. Use of Setup Tool](#3-use-of-setup-tool)
+        - [3.1 General Usage](#31-general-usage)
+        - [3.2 Specify Config Files](#32-specify-config-files)
+        - [3.3 Use with Local vHive Repo](#33-use-with-local-vhive-repo)
+        - [3.4 Use with Remote vHive Repo (Standalone Use)](#34-use-with-remote-vhive-repo-standalone-use)
+        - [3.5 Migrate from Legacy Shell Scripts](#35-migrate-from-legacy-shell-scripts)
+    - [4. Logs](#4-logs)
+    - [5. Supported Platform](#5-supported-platform)
+
 ## 1. Get Setup Tool
+
 There are basically two ways to get the setup tool
 
 ### 1.1 Download the Binary Executable Directly
-Check [vHive GitHub Repo](https://github.com/vhive-serverless/vHive/releases) for more details and choose the appropriate version to download.
+
+Check [vHive GitHub Repo](https://github.com/vhive-serverless/vHive/releases) for more details and choose the
+appropriate version to download.
 
 ### 1.2 Build from Source
+
 **Building from source requires Go (version 1.19 at least) installed on your system.**
+
 ```bash
 git clone --depth 1 https://github.com/vhive-serverless/vHive
 cd vHive
 pushd scripts && go build -o setup_tool && popd
 ```
+
 Compiled executable file will be in the `scripts` directory.
 
 ## 2. Config the Setup Tool
-**Normally, just skip this section and use the default config files** which are located in the `configs/setup` directory inside the [vHive repo](https://github.com/vhive-serverless/vHive).
 
-- `configs/setup/knative.json`: knative related configs (all the path in the config file should be relative path inside the vHive repo)
+**Normally, just skip this section and use the default config files** which are located in the `configs/setup` directory
+inside the [vHive repo](https://github.com/vhive-serverless/vHive).
+
+- `configs/setup/knative.json`: knative related configs (all the path in the config file should be relative path inside
+  the vHive repo)
 - `configs/setup/kube.json`: Kubernetes related configs
 - `configs/setup/system.json`: system related configs
 - `configs/setup/vhive.json`: vHive related configs
 
 You can modify the config files on your demand and then place all of them in one directory for the later use.
 
 ## 3. Use of Setup Tool
+
 ### 3.1 General Usage
+
 ```bash
 ./setup_tool [options] <subcommand> [parameters]
 ```
+
 use the `-h` or `--help` option to look for the help
+
 ### 3.2 Specify Config Files
-By default, the setup_tool will use the config files in `configs/setup` directory inside the vHive repo. 
+
+By default, the setup_tool will use the config files in `configs/setup` directory inside the vHive repo.
 
 To change the path of config files, use the `--setup-configs-dir` option to specify it.
+
 ```bash
 ./setup_tool --setup-configs-dir <CONFIG PATH> ...
 ```
 
 ### 3.3 Use with Local vHive Repo
-By default, the setup_tool will check the current directory to ensure it is a vHive repo and then use it during the setup process.
+
+By default, the setup_tool will check the current directory to ensure it is a vHive repo and then use it during the
+setup process.
 
 To use other vHive repos locally, provide the `--vhive-repo-dir` option to specify it.
+
 ```bash
 ./setup_tool --vhive-repo-dir <VHIVE REPO PATH> ...
 ```
 
-If the current directory or the provided path is not a valid vHive repo, the setup_tool will [automatically clone the remote vHive repo and use it](#34-use-with-remote-vhive-repo).
+If the current directory or the provided path is not a valid vHive repo, the setup_tool
+will [automatically clone the remote vHive repo and use it](#34-use-with-remote-vhive-repo).
 
 ### 3.4 Use with Remote vHive Repo (Standalone Use)
-When the setup_tool is directly downloaded or targeted for  standalone use, the setup_tool will automatically clone the remote vHive repo to the temporary directory and then use it during the setup process.
 
-To change the URL and branch of the [default remote vHive repo](https://github.com/vhive-serverless/vHive), use `--vhive-repo-url` and `--vhive-repo-branch` options to specify them.
+When the setup_tool is directly downloaded or targeted for standalone use, the setup_tool will automatically clone the
+remote vHive repo to the temporary directory and then use it during the setup process.
+
+To change the URL and branch of the [default remote vHive repo](https://github.com/vhive-serverless/vHive),
+use `--vhive-repo-url` and `--vhive-repo-branch` options to specify them.
+
 ```bash
 ./setup_tool --vhive-repo-url <URL> --vhive-repo-branch <BRANCH> ...
 ``` 
 
-Besides, when the current directory is a vHive repo or the `--vhive-repo-dir` option is valid, **the local repo will be prioritized for use**. **To force the setup_tool to clone and use the remote vHive repo**, provide `--force-remote` option to the setup_tool.
+Besides, when the current directory is a vHive repo or the `--vhive-repo-dir` option is valid, **the local repo will be
+prioritized for use**. **To force the setup_tool to clone and use the remote vHive repo**, provide `--force-remote`
+option to the setup_tool.
+
 ```bash
 ./setup_tool --force-remote ...
 ```
 
-
 ### 3.5 Migrate from Legacy Shell Scripts
+
 Just type the name of the original shell script and append corresponding parameters behind. For example:
+
 ```bash
 # Legacy ==>
 scripts/cloudlab/setup_node.sh stock-only
 # ==> Current
-./setup_tool [options] setup_node stock-only
+./setup_tool [options] setup_node REGULAR stock-only
 
 # Legacy ==>
 scripts/create_devmapper.sh
@@ -94,14 +123,29 @@ scripts/gpu/setup_nvidia_gpu.sh
 ./setup_tool [options] setup_nvidia_gpu
 ```
 
-**NOTICE**: Shell scripts in `scripts/stargz`, `scripts/self-hosted-kind`, and `scripts/github_runner` **are not supported to be invoked in this way at present**.
+**NOTICE**: Shell scripts in `scripts/stargz`, `scripts/self-hosted-kind`, and `scripts/github_runner` **are not
+supported to be invoked in this way at present**.
+
+### 3.6 Kubernetes Control Plane High-Availability Mode
 
+For fault tolerance purposes, Kubernetes control plane components can be replicated. This can be done by combining
+instructions from
+the following links - [#1](https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md)
+and [#2](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/).
+
+While executing these steps manually is complicated, [InVitro Loader](https://github.com/vhive-serverless/invitro)
+provides an automatized way of creating a high-available control plane just by configuring `CONTROL_PLANE_REPLICAS`
+parameter in `scripts/setup/setup.cfg` and then running `scripts/setup/create_multinode.sh`.
 
 ## 4. Logs
-The log files will be named as `<subcommand>_common.log` and `<subcommand>_error.log`. All log files will be stored in the directory where the setup_tool is executed.
+
+The log files will be named as `<subcommand>_common.log` and `<subcommand>_error.log`. All log files will be stored in
+the directory where the setup_tool is executed.
 
 - `<subcommand>_common.log`: all output originally writes to `stdout` will be redirected to this log file.
 - `<subcommand>_error.log`: all output originally writes to `stderr` will be redirected to this log file.
 
 ## 5. Supported Platform
-At present, only `Ubuntu 20.04 (amd64)` is officially tested. Other versions of `Ubuntu` may also work, but not guaranteed. 
+
+At present, only `Ubuntu 20.04 (amd64)` is officially tested. Other versions of `Ubuntu` may also work, but not
+guaranteed. 

scripts/cluster/create_multinode_cluster.go

@@ -196,7 +196,7 @@ func ExtractMasterNodeInfo() error {
 		return err
 	}
 
-	utils.SuccessPrintf("Join cluster from worker nodes as a new control plane node with command: "+
+	utils.SuccessPrintf("Join cluster as a new control plane node with command: "+
 		"sudo kubeadm join %s:%s --token %s --discovery-token-ca-cert-hash %s --control-plane --certificate-key %s\n",
 		configs.Kube.ApiserverAdvertiseAddress,
 		configs.Kube.ApiserverPort,