examples/tutorials/setup_poc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "654a158f-42c6-4706-84c6-4bbbec79be26",
   "metadata": {},
   "source": [
    "# Set Up NVFLARE in POC Mode\n",
    "\n",
    "This tutorial covers in depth the available features of the POC mode. For more basic setup guides, refer to the [Quickstart](https://nvflare.readthedocs.io/en/main/quickstart.html#setting-up-the-application-environment-in-poc-mode) or [POC Mode Documentation](https://nvflare.readthedocs.io/en/main/user_guide/nvflare_cli/poc_command.html).\n",
    "\n",
    "POC mode allows users to test the features of a full FLARE deployment on a single machine, without the overhead of a true distributed deployment.\n",
    "Compared to the FL Simulator, where the job run is automated on a single system, POC mode allows you to establish and connect distinct server and client \"systems\" which can then be orchestrated using the FLARE Console. Users can also experiment with various deployment options (project.yml), making POC mode a useful tool in preparation for a distributed deployment.\n",
    "\n",
    ">It is ideal to start your NVFLARE system in POC mode from a **terminal**, not from a notebook. The terminal's virual env. must match the kernel's virtual env. In our case, we are using 'nvflare_example'.\n",
    "\n",
    "To get started, let's look at the NVFlare CLI usage for the ``poc`` subcommand:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d3c34b6-fa35-466e-b5bf-1b0be07d399f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare poc -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87989e99",
   "metadata": {},
   "source": [
    "We can further look at the help description for each command\n",
    "\n",
    "> We will need ```tree``` command, so we will install ```tree``` in linux, if tree is not available to you, you can replace it with 'ls -al`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb926188-020e-4699-9411-d9970521fe0f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! python -m pip install tree"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f877f82",
   "metadata": {},
   "source": [
    "## Configure the POC Workspace"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0eb71b3-331e-4caa-996b-bf9af5490c62",
   "metadata": {},
   "source": [
    "### Config command\n",
    "\n",
    "The config command can be used to configure the `poc_workspace`, as well as other locations such as the `startup_kit` and `job_template` directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16f06739",
   "metadata": {},
   "outputs": [],
   "source": [
    "! nvflare config -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bc86c64",
   "metadata": {},
   "source": [
    "To specify the POC workspace, use the `--poc_workspace_dir` or `-pw` option:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c668ce86-b4c9-4acb-8932-1653c8f12601",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare config -pw /tmp/nvflare/poc --job_templates_dir ../../job_templates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17f368e9-6c66-4076-b12f-25330d4420c2",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Hidden Config\n",
    "\n",
    "The config command works by writing the POC workspace directory to a hidden configuration file, which can also be edited directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a98e4fbe-43ce-4e30-8017-46d21fe81b31",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat ~/.nvflare/config.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "524a9d03-cbca-467f-8cf3-46f9481f1dca",
   "metadata": {},
   "source": [
    "### Environment Variable\n",
    "\n",
    "You can also use the env. variable NVFLARE_POC_WORKSPACE if you prefer. If both the hidden config file and env. variable NVFLARE_POC_WORKSPACE are set, the env. variable will take the precedence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b761c9e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%env  NVFLARE_POC_WORKSPACE=/tmp/nvflare/poc\n",
    "! echo $NVFLARE_POC_WORKSPACE "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b5f79f8-4146-4222-a991-a2411dc72b79",
   "metadata": {},
   "source": [
    "## Prepare the POC workspace\n",
    "\n",
    "Now that we've configured our POC configuration, we can prepare the POC workspace.  By default, the following command will generate POC packages for a server and two clients. If there is a pre-existing POC workspace, you will be prompted whether to overwrite and create a new one.\n",
    "\n",
    "Below we explore the various prepare options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0507ce83-ff64-4684-a97c-728d495158ad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!  nvflare poc prepare -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86468ce7",
   "metadata": {},
   "source": [
    "### Number of Clients\n",
    "\n",
    "Specify a different number of clients with -n:\n",
    "\n",
    "```\n",
    "nvflare poc prepare -n [NUMBER_OF_CLIENTS]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e205cf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo 'y' | nvflare poc prepare -n 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d900be1",
   "metadata": {},
   "source": [
    "### Named Clients\n",
    "\n",
    "Specifiy the client site names ((instead of use default site-1,2 etc.) with -c:\n",
    "\n",
    "```\n",
    "nvflare poc prepare -c [CLIENTS ...]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26f6ca9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo 'y' | nvflare poc prepare -c hospital-1 hospital-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e091b13c",
   "metadata": {},
   "outputs": [],
   "source": [
    "! tree /tmp/nvflare/poc/example_project/prod_00"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78ac8384",
   "metadata": {},
   "source": [
    "### Docker Image\n",
    "\n",
    "The -d option instructs the provision tool to create a docker.sh use the specified docker_image. The docker.sh will start in detached mode. \n",
    "\n",
    "```\n",
    "nvflare poc prepare -d [DOCKER_IMAGE]\n",
    "```\n",
    "\n",
    "This will create docker.sh for each client and server which will run `docker pull docker_image` from docker hub, then running the docker in detached mode. The docker_image must use nvflare. For example, we use the following docker image \"nvflare/nvflare\" which has nvflare preinstalled.\n",
    "> Note: you should build your own image if you want to include your dependencies such as pytorch etc.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7810bcc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo 'y'| nvflare poc prepare -d 'nvflare/nvflare'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff2755e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "! tree /tmp/nvflare/poc/example_project/prod_00"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d3a3c06",
   "metadata": {},
   "source": [
    "Notice the new \"docker.sh\", now each client will using docker run with we specified \"nvflare/nvflare\" docker image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6b968e9",
   "metadata": {},
   "source": [
    "### Homomorphic Encryption (HE)\n",
    "\n",
    "The -he option enables homomorphic encryption.\n",
    "\n",
    "```\n",
    "nvflare poc prepare -he\n",
    "```\n",
    "\n",
    "To support HE, we need the provision process to generate TenSEAL homomorphic encryption context for the server and client and write them to the server and client participant folders (See [provision context](https://nvflare.readthedocs.io/en/main/programming_guide/provisioning_system.html#provision-context)). This is achieved by the HEBuilder, which is automatically added to the `project.yml` file using the `-he` in the POC command (See the `-i` option to manually customize the `project.yml` file).\n",
    "\n",
    "First we must install nvflare with TenSEAL:\n",
    "\n",
    "> note: TenSEAL is not available in Mac OS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42d15f8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install nvflare[HE]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd704405",
   "metadata": {},
   "source": [
    "For an example, we can prepare a POC workspace with HE enabled with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f3f05ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo 'y' | nvflare poc prepare -he"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39f96290",
   "metadata": {},
   "outputs": [],
   "source": [
    "! cat  /tmp/nvflare/poc/project.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40e0088f",
   "metadata": {},
   "source": [
    "Notice:  ```path: nvflare.lighter.impl.he.HEBuilder``` is added to the YAML file.\n",
    "Also note the importance of the order of the builders, as the `HEBuilder` must be configured before the `SignatureBuilder` to ensure the tenseal context files are signed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37a6edd6",
   "metadata": {},
   "source": [
    "### Project Input: Simulate production deployment setup locally"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a801671",
   "metadata": {},
   "source": [
    "Assuming we have a new custom project.yml, we can test out the project.yml locally in POC mode before we do the real provisionining.\n",
    "\n",
    "```\n",
    "nvflare poc prepare -i [PROJECT_INPUT]\n",
    "```\n",
    "\n",
    "Let's try this out. We have prepared an custom_project.yml file: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad640119",
   "metadata": {},
   "outputs": [],
   "source": [
    "! cat ./custom_project.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04fdf2e5",
   "metadata": {},
   "source": [
    "Notice in this project file, we define three sites: \n",
    "\n",
    "* nonprofit-server, served as Server,belong to nonprofit organizaton \n",
    "* us_hospital, served as client, belong to us-hospital\n",
    "* europe-hospital, served as client, belong to eu-hospital\n",
    "\n",
    "We also define few users: \n",
    "* admin@nonprofit.org, from nonprofit.org, serves as project admin\n",
    "* admin@hospital.org.us, lead@hospital.org.us,  member@hospital.org.us from us_hospital, serves as org admin, lead and member for us_hospital respectively\n",
    "* admin@hospital.org.eu, lead@hospital.org.eu,  member@hospital.org.eu from eu_hospital, serves as org admin, lead and member for eu_hospital respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c668b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo 'y' | nvflare poc prepare -i ./custom_project.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9d143ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "! tree /tmp/nvflare/poc/health_project/prod_00"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8917139",
   "metadata": {},
   "source": [
    "See that we successfully provisioned all participants from our \"custom_project.yml\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b82a4b5-5eed-47a2-a888-0d140a5531bc",
   "metadata": {},
   "source": [
    "## Prepare job example directories\n",
    "\n",
    "By default, the `nvflare poc prepare` command will setup a symbolic link to the NVFlare/examples directory, assuming that you have used `git clone` to download  NVFlare from github and have defined the NVFLARE_HOME env. variable.\n",
    "    \n",
    "Rather than setting the NVFLARE_HOME env. variables, instead link to your desired jobs with the following command to setup to the job directory:\n",
    "    \n",
    "```\n",
    "nvflare poc prepare-jobs-dir -j [JOBS_DIR]\n",
    "```\n",
    "\n",
    "For example we can set the jobs directory to NVFlare/examples: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41053a0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "! nvflare poc prepare-jobs-dir -j .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0efc93e2-fcc8-4789-822f-69e8d2d32116",
   "metadata": {},
   "source": [
    "## Start NVFlare system in POC mode\n",
    "\n",
    "When starting the POC deployment, it's necessary to use a separate terminal since the `nvflare poc start` command will run  in the foreground emitting output from the server and any connected clients.\n",
    "\n",
    "Also note that `nvflare poc start` starts all clients and server as well as project admin console. All other participants (org admin, lead and member) will not started. \n",
    "\n",
    "In many cases, we want to start the clients and server, but without the FLARE console for project admin, as we are going to interact with the system from a notebook or CLI. You first need to find out the user name for the project admin. By default, this is admin@nvidia.com\n",
    "\n",
    "Create a terminal to execute the following command (in JupyterLab or host terminal):\n",
    "\n",
    "```\n",
    "    nvflare poc start -ex admin@nvidia.com\n",
    "```\n",
    ">**Note:**\n",
    "    Using ```%%bash -bg``` to run the above command in a code cell may not always work\n",
    "    \n",
    "### A More Complex Example\n",
    "    \n",
    "In this example, lets see how to start the individual user participants defined in custom_project.yml. Note the project admin is \"admin@nonprofit.org\" instead of \"admin@nvidia.com\"\n",
    "\n",
    "Please use terminal to try these commands:\n",
    "\n",
    "1. We start the system with two hospital clients and server, without the project admin console:\n",
    "\n",
    "```\n",
    "    nvflare poc start -ex admin@nonprofit.org\n",
    "```\n",
    "\n",
    "2. Next we like to start US hospital's lead's FLARE console:\n",
    "\n",
    "```\n",
    "    nvflare poc start -p lead@hospital.org.us\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4568b87-a711-4441-a2af-2008dbc9e278",
   "metadata": {
    "tags": []
   },
   "source": [
    "Once the system is up, we are now ready to go back to the notebook to work on your project.\n",
    "\n",
    "For more details on how to operate the FLARE system and submit jobs, refer to the [POC Mode Documentation](https://nvflare.readthedocs.io/en/main/user_guide/nvflare_cli/poc_command.html#operating-the-system-and-submitting-a-job)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "275fbe95-faf3-468d-a96a-453310e90aee",
   "metadata": {},
   "source": [
    "## Stop NVFlare system in POC mode\n",
    "\n",
    "Once you are done with the NVFlare system, you can shut down the poc system.\n",
    "From terminal, issue the following command to shutdown all participants including server, clients, and admin consoles.\n",
    "```\n",
    "    nvflare poc stop\n",
    "```\n",
    "\n",
    "If you like to shutdown an invididual participant, use the following -p option:\n",
    "\n",
    "```\n",
    "    nvflare poc stop -p [SERVICE]\n",
    "```\n",
    "\n",
    "Some examples of service participant names can be \"server\", \"site-1\", \"admin@nvidia.com\", or \"lead@hospital.org.us\" depending on your prepared POC. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "721dcecf-c855-4e21-a623-830bfaaddd9b",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Project Yaml File\n",
    "\n",
    "Once we experimented with different options, we are ready to move on to the production beyond local host.\n",
    "The same project.yml file can be used for production, and is auto-generated for you and located at:\n",
    "\n",
    "```\n",
    "    <poc_workspace>/project.yml\n",
    "```\n",
    "\n",
    "For example,\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02b0f998-2776-412f-b97d-a6e8a99ff583",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! echo 'y' | nvflare poc prepare -c hospital_1 hospital_2 -d 'nvflare/nvflare'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e32726e-f48e-4b73-ae0c-ccb5d71983bc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! tree /tmp/nvflare/poc/example_project/prod_00"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97fd1ff1-19f6-4e21-b353-684468a12456",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat  /tmp/nvflare/poc/project.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f74a5c2d-d7ad-4995-9940-9b98e9f9a4bf",
   "metadata": {},
   "source": [
    "Copy this file and you are ready to go !\n",
    "\n",
    "Learn more about how to take your project to the real world with the provisioning tool, dashboard, and various deployment options in our [documentation](https://nvflare.readthedocs.io/en/main/real_world_fl/overview.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea8f1878-7e9d-4e17-ba78-5812fc88b580",
   "metadata": {},
   "source": [
    "## Clean up POC workspace\n",
    "\n",
    "Lastly, if you want to clean up the POC workspace and delete the POC workspace directory containing all of the packages you may have prepared, you can use the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c787b6-e4af-473c-89af-71f7acf3fb17",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare poc clean"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b8e11d2",
   "metadata": {},
   "source": [
    "You can check if the nvflare system is shutdown cleanly with:\n",
    "\n",
    "```\n",
    "ps -eaf | grep nvflare\n",
    "```\n",
    "\n",
    "If you see output like the following, then nvflare systems are still running:\n",
    "\n",
    "```\n",
    "510535    1932  1 18:54 pts/1    00:00:03 python3 -u -m nvflare.private.fed.app.client.client_train -m /tmp/workspace/example_project/prod_00/site-1/startup/.. -s fed_client.json --set secure_train=true uid=site-1 org=nvidia config_folder=config\n",
    "510539    1932  1 18:54 pts/1    00:00:03 python3 -u -m nvflare.private.fed.app.client.client_train -m /tmp/workspace/example_project/prod_00/site-2/startup/.. -s fed_client.json --set secure_train=true uid=site-2 org=nvidia config_folder=config\n",
    "510543    1932  1 18:54 pts/1    00:00:04 python3 -u -m nvflare.private.fed.app.server.server_train -m /tmp/workspace/example_project/prod_00/localhost/startup/.. -s fed_server.json --set secure_train=true org=nvidia config_folder=config\n",
    "```\n",
    "\n",
    "If you have already used ``nvflare poc clean`` to delete the workspace, you may need to kill the processes manually.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nvflare_example",
   "language": "python",
   "name": "nvflare_example"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}