2 Steam: Managing AI Engines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Enterprise Steam for manging AI engines\n",
    "\n",
    "This notebook is intended to help you get started managing your AI Engines in the H2O AI Cloud using python. \n",
    "\n",
    "* **Product Documentation:** https://docs.h2o.ai/steam/index.html\n",
    "* **Python Documentation:** https://docs.h2o.ai/enterprise-steam/latest-stable/docs/python-docs/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequistes\n",
    "\n",
    "Update the `h2o_ai_cloud.py` file with the connection parameters for your H2O AI Cloud environemnt:\n",
    "1. Login to your H2O AI Cloud environment\n",
    "1. Click your username or avatar in the H2O AI Cloud navigation bar\n",
    "1. Navigate to `CLI & API Access`\n",
    "1. Use the variables from the `Accessing H2O AI Cloud APIs` section to populate the parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from getpass import getpass\n",
    "\n",
    "from h2o_ai_cloud import steam_client\n",
    "from h2osteam.clients import DriverlessClient, H2oKubernetesClient\n",
    "\n",
    "import h2osteam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Securely connect\n",
    "\n",
    "We first connect to the H2O AI Cloud using our platform token to create a token provider object. We can then use this object to log into Steam and other APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Visit https://cloud.h2o.ai/auth/get-platform-token to get your platform token\n",
      "Enter your platform token: ········\n"
     ]
    }
   ],
   "source": [
    "steam = steam_client()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding your environment\n",
    "\n",
    "* Ensure that your python client and backend versions match. \n",
    "* View what AI Engines and environments are available "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Backend Version: 1.8.12\n",
      "Library Version: 1.8.12\n"
     ]
    }
   ],
   "source": [
    "print(f\"Backend Version: {h2osteam.api().get_config_meta()['version']}\")\n",
    "print(f\"Library Version: {h2osteam.__version__}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'version': '1.8.12',\n",
       " 'build': '1.8.11-14-g45e554b-dirty',\n",
       " 'built': 'Wed May 11 22:14:52 UTC 2022',\n",
       " 'restart_pending': True,\n",
       " 'support_email': 'cloud-feedback@h2o.ai',\n",
       " 'license_valid': True,\n",
       " 'is_hadoop_enabled': False,\n",
       " 'is_kubernetes_enabled': True,\n",
       " 'is_h2o_enabled': True,\n",
       " 'is_h2o_running': True,\n",
       " 'is_sparkling_enabled': False,\n",
       " 'is_sparkling_running': False,\n",
       " 'is_driverless_enabled': True,\n",
       " 'is_driverless_running': True,\n",
       " 'is_h2o_engine_uploaded': False,\n",
       " 'is_h2o_kubernetes_engine_uploaded': True,\n",
       " 'is_sparkling_engine_uploaded': False,\n",
       " 'is_driverless_engine_uploaded': True,\n",
       " 'driverless_backend_type': 'kubernetes',\n",
       " 'is_minio_enabled': False,\n",
       " 'h2o_backend_type': 'kubernetes',\n",
       " 'inside_cluster': True}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2osteam.api().get_config_meta()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Managing Driverless AI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View available versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.10.1.3\n",
      "1.10.2\n",
      "1.10.3\n"
     ]
    }
   ],
   "source": [
    "for dai_version in steam.get_driverless_engines():\n",
    "    print(dai_version[\"version\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View available profiles\n",
    "\n",
    "* List the profiles\n",
    "* View the options for max run time in this profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['visitor-driverless-kubernetes']\n"
     ]
    }
   ],
   "source": [
    "dai_profiles = [profile[\"name\"] for profile in steam.get_profiles() if profile[\"profile_type\"] == \"driverless_kubernetes\"]\n",
    "\n",
    "print(dai_profiles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 1005,\n",
       " 'min': 1,\n",
       " 'is_min_set': True,\n",
       " 'max': 24,\n",
       " 'is_max_set': True,\n",
       " 'initial': 8,\n",
       " 'is_initial_set': True,\n",
       " 'profile_max': -1,\n",
       " 'is_profile_max_set': True}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "steam.get_profile_by_name(dai_profiles[0])[\"driverless_kubernetes\"][\"max_uptime_hours\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an instance\n",
    "See the `3 AutoML` tutorial for what to do once you've connected to the AutoML platform. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Driverless AI instance is submitted, please wait...\n",
      "Driverless AI instance is running\n"
     ]
    }
   ],
   "source": [
    "dai_machine = DriverlessClient(steam).launch_instance(\n",
    "    name=\"my-test-instance\",\n",
    "    version=steam.get_driverless_engines()[-1][\"version\"],  # newest version of the AutoML software,\n",
    "    profile_name=dai_profiles[0],\n",
    "    max_uptime_h=1\n",
    ")\n",
    "client = dai_machine.connect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 3283,\n",
       " 'profile_name': 'default-driverless-kubernetes',\n",
       " 'name': 'my-test-instance',\n",
       " 'status': 'running',\n",
       " 'target_status': 'running',\n",
       " 'version': '1.10.3',\n",
       " 'backend_type': '',\n",
       " 'instance_type': 'singlenode',\n",
       " 'master_id': -1,\n",
       " 'cpu_count': 8,\n",
       " 'gpu_count': 0,\n",
       " 'memory_gb': 30,\n",
       " 'storage_gb': 64,\n",
       " 'max_idle_seconds': 7200,\n",
       " 'max_uptime_seconds': 3600,\n",
       " 'timeout_seconds': 1800,\n",
       " 'address': 'http://10.1.212.19:12345',\n",
       " 'authentication': 'oidc',\n",
       " 'password': '',\n",
       " 'created_at': 1657064256,\n",
       " 'started_at': 1657064304,\n",
       " 'created_by': 'michelle.tanco@h2o.ai',\n",
       " 'current_uptime_seconds': 11,\n",
       " 'current_idle_seconds': 11,\n",
       " 'pod_latest_event': None,\n",
       " 'service_latest_event': None,\n",
       " 'pvc_latest_event': None,\n",
       " 'stop_reason': '',\n",
       " 'config_toml': '',\n",
       " 'volumes': ''}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dai_machine.details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://steam.cloud.h2o.ai:443/oidc-login-start?forward=/proxy/driverless/3283/openid/callback'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dai_machine.openid_login_url()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List my instances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3274 \t visitor-driverless-kubernetes \t stopped \t dai-quickstart\n",
      "3279 \t visitor-driverless-kubernetes \t stopped \t my-test-instance\n"
     ]
    }
   ],
   "source": [
    "for instance in steam.get_driverless_instances():\n",
    "    print(instance[\"id\"], \"\\t\", instance[\"profile_name\"], \"\\t\", instance[\"status\"], \"\\t\", instance[\"name\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download instance logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dai_machine = DriverlessClient(steam).get_instance(name=\"my-test-instance\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Driverless AI instance logs saved to dai_test_instance_logs.zip\n"
     ]
    }
   ],
   "source": [
    "dai_machine.download_logs('dai_test_instance_logs.zip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  dai_test_instance_logs.zip\n",
      "replace dai_test_instance_logs/steam-config.yaml? [y]es, [n]o, [A]ll, [N]one, [r]ename: ^C\n"
     ]
    }
   ],
   "source": [
    "!unzip dai_test_instance_logs.zip -d dai_test_instance_logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pause an instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pause an instance that is currently running. Pausing an instance shuts it down to save resources, it is similar to powering off a server. You will not loose any data and you can start an instance at any time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Driverless AI instance is stopping, please wait...\n",
      "Driverless AI instance is stopped\n"
     ]
    }
   ],
   "source": [
    "dai_machine.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resume the instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can resume a paused instance by running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Driverless AI instance is starting, please wait...\n",
      "Driverless AI instance is running\n"
     ]
    }
   ],
   "source": [
    "dai_machine.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you no longer need an instance, you can terminate it. Once deleted, there is no way to restart the instance or access any data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "dai_machine.terminate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3274 \t visitor-driverless-kubernetes \t stopped \t dai-quickstart\n",
      "3279 \t visitor-driverless-kubernetes \t stopped \t my-test-instance\n"
     ]
    }
   ],
   "source": [
    "# it takes ~20 seconds for the status to update\n",
    "for instance in steam.get_driverless_instances():\n",
    "    print(instance[\"id\"], \"\\t\", instance[\"profile_name\"], \"\\t\", instance[\"status\"], \"\\t\", instance[\"name\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Managing H2O-3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View available versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.34.0.7\n",
      "3.36.0.1\n",
      "3.36.0.4\n"
     ]
    }
   ],
   "source": [
    "for h2o_version in steam.get_h2o_kubernetes_engines():\n",
    "    print(h2o_version[\"version\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View available profiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['visitor-h2o-kubernetes']\n"
     ]
    }
   ],
   "source": [
    "h2o_profiles = [profile[\"name\"] for profile in steam.get_profiles() if profile[\"profile_type\"] == \"h2o_kubernetes\"]\n",
    "\n",
    "print(h2o_profiles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 1026,\n",
       " 'min': 1,\n",
       " 'is_min_set': True,\n",
       " 'max': 24,\n",
       " 'is_max_set': True,\n",
       " 'initial': 12,\n",
       " 'is_initial_set': True,\n",
       " 'profile_max': -1,\n",
       " 'is_profile_max_set': True}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "steam.get_profile_by_name(h2o_profiles[0])[\"h2o_kubernetes\"][\"max_uptime_hours\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a cluster\n",
    "See the H2O-3 Tutorial for what to do once you've connected to the distributed ML platform. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "h2o_machine = H2oKubernetesClient(steam).launch_cluster(\n",
    "    name=\"my-test-instance\",\n",
    "    version=steam.get_h2o_kubernetes_engines()[-1][\"version\"],  # newest version of the AutoML software,\n",
    "    profile_name=h2o_profiles[0],\n",
    "    max_uptime_h=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "while not h2o_machine.is_running():\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connecting to H2O server at https://steam.cloud.h2o.ai:443/proxy/h2o-k8s/945 ... successful.\n",
      "Warning: Version mismatch. H2O is version 3.36.0.4, but the h2o-python package is version 3.36.0.3. Install the matching h2o-Python version from - http://h2o-release.s3.amazonaws.com/h2o/rel-zorn/4/index.html.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>18 secs</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>Etc/GMT</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.36.0.4</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>3 months and 5 days </td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>my-test-instance</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>817 Mb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>https://steam.cloud.h2o.ai:443/proxy/h2o-k8s/945</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>null</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>True</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.8.12 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ------------------------------------------------\n",
       "H2O_cluster_uptime:         18 secs\n",
       "H2O_cluster_timezone:       Etc/GMT\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.36.0.4\n",
       "H2O_cluster_version_age:    3 months and 5 days\n",
       "H2O_cluster_name:           my-test-instance\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    817 Mb\n",
       "H2O_cluster_total_cores:    1\n",
       "H2O_cluster_allowed_cores:  1\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         https://steam.cloud.h2o.ai:443/proxy/h2o-k8s/945\n",
       "H2O_connection_proxy:       null\n",
       "H2O_internal_security:      True\n",
       "Python_version:             3.8.12 final\n",
       "--------------------------  ------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "client = h2o_machine.connect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://steam.cloud.h2o.ai/proxy/h2o-k8s/945/flow/index.html\n"
     ]
    }
   ],
   "source": [
    "print(\"https://\" + h2o_machine.get_connection_config()[\"connect_params\"][\"ip\"] + \"/\" + h2o_machine.get_connection_config()[\"connect_params\"][\"context_path\"] + \"/flow/index.html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List my clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "for instance in steam.get_h2o_kubernetes_clusters():\n",
    "    print(instance[\"id\"], \"\\t\", instance[\"profile_name\"], \"\\t\", instance[\"status\"], \"\\t\", instance[\"name\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download cluster logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "h2o_machine = H2oKubernetesClient().get_cluster(name=\"my-test-instance\", created_by=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H2O cluster logs saved to h2o_test_instance_logs.zip\n"
     ]
    }
   ],
   "source": [
    "h2o_machine.download_logs('h2o_test_instance_logs.zip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  h2o_test_instance_logs.zip\n",
      "  inflating: h2o_test_instance_logs/server.log  \n",
      "  inflating: h2o_test_instance_logs/events.log  \n",
      "  inflating: h2o_test_instance_logs/h2o.log  \n",
      "  inflating: h2o_test_instance_logs/steam-config.yaml  \n"
     ]
    }
   ],
   "source": [
    "!unzip h2o_test_instance_logs.zip -d h2o_test_instance_logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop an instance\n",
    "H2O clusters cannot be resumed after they are stopped. Only turn off your cluster once you are permanately done with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<h2osteam.clients.h2ok8s.h2ok8s.H2oKubernetesCluster at 0x124eb48b0>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2o_machine.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current status: running\n",
      "Target status: stopped\n"
     ]
    }
   ],
   "source": [
    "print(f\"Current status: {h2o_machine.status}\")\n",
    "print(f\"Target status: {h2o_machine.target_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete an instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<h2osteam.clients.h2ok8s.h2ok8s.H2oKubernetesCluster at 0x124eb48b0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2o_machine.terminate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "for instance in steam.get_h2o_kubernetes_clusters():\n",
    "    print(instance[\"id\"], \"\\t\", instance[\"profile_name\"], \"\\t\", instance[\"status\"], \"\\t\", instance[\"name\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm dai_test_instance_logs.zip\n",
    "!rm -rf dai_test_instance_logs\n",
    "\n",
    "!rm h2o_test_instance_logs.zip\n",
    "!rm -rf h2o_test_instance_logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "336px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}