scikit-learn-pipeline/sklearn-project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstrate Local Or Remote Functions And Full Pipelines\n",
    "  --------------------------------------------------------------------\n",
    "\n",
    "Creating a local function, running predefined functions, creating and running a full ML pipeline with local and library functions.\n",
    "\n",
    "#### **notebook how-to's**\n",
    "* Create and test a simple function\n",
    "* Examine data using serverless (containarized) `describe` function\n",
    "* Create an automated ML pipeline from various library functions\n",
    "* Running and tracking the pipeline results and artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create and Test a Local Function (Iris Data Generator)\n",
    "Import nuclio SDK and magics, <b>do not remove the cell and comment !!!</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuclio: ignore\n",
    "import nuclio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Specify function dependencies and configuration<b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%nuclio cmd -c\n",
    "pip install sklearn\n",
    "pip install pyarrow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "%nuclio: setting spec.build.baseImage to 'mlrun/mlrun'\n"
     ]
    }
   ],
   "source": [
    "%nuclio config spec.build.baseImage = \"mlrun/mlrun\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function code\n",
    "Generate the iris dataset and log the dataframe (as csv or parquet file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "from sklearn.metrics import accuracy_score\n",
    "from mlrun.artifacts import TableArtifact, PlotArtifact\n",
    "import pandas as pd\n",
    "\n",
    "def iris_generator(context, format='csv'):\n",
    "    iris = load_iris()\n",
    "    iris_dataset = pd.DataFrame(data=iris.data, columns=iris.feature_names)\n",
    "    iris_labels = pd.DataFrame(data=iris.target, columns=['label'])\n",
    "    iris_dataset = pd.concat([iris_dataset, iris_labels], axis=1)\n",
    "    \n",
    "    context.logger.info('saving iris dataframe to {}'.format(context.artifact_path))\n",
    "    context.log_dataset('iris_dataset', df=iris_dataset, format=format, index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following end-code annotation tells ```nuclio``` to stop parsing the notebook from this cell. _**Please do not remove this cell**_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuclio: end-code\n",
    "# marks the end of a code section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a project to host our functions, jobs and artifacts\n",
    "\n",
    "Projects are used to package multiple functions, workflows, and artifacts. We usually store project code and definitions in a Git archive.\n",
    "\n",
    "The following code creates a new project in a local dir and initialize git tracking on that"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import path\n",
    "from mlrun import run_local, NewTask, mlconf, import_function, mount_v3io\n",
    "mlconf.dbpath = mlconf.dbpath or 'http://mlrun-api:8080'\n",
    "\n",
    "# specify artifacts target location\n",
    "artifact_path = mlconf.artifact_path or path.abspath('./')\n",
    "project_name = 'sk-project'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlrun import new_project, code_to_function\n",
    "project_dir = './project'\n",
    "skproj = new_project(project_name, project_dir, init_git=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='test-locally'></a>\n",
    "### Run the data generator function locally\n",
    "\n",
    "The functions above can be tested locally. Parameters, inputs, and outputs can be specified in the API or the `Task` object.<br>\n",
    "when using `run_local()` the function inputs and outputs are automatically recorded by MLRun experiment and data tracking DB.\n",
    "\n",
    "In each run we can specify the function, inputs, parameters/hyper-parameters, etc... For more details, see the [mlrun_basics notebook](mlrun_basics.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[mlrun] 2020-06-11 16:24:50,053 starting run iris_gen uid=7ba5b1aab9f14abbb393fdfafca8d98c  -> http://mlrun-api:8080\n",
      "[mlrun] 2020-06-11 16:24:50,091 saving iris dataframe to /User/ml3/demos/scikit-learn-pipeline/data\n",
      "[mlrun] 2020-06-11 16:24:50,278 log artifact iris_dataset at /User/ml3/demos/scikit-learn-pipeline/data/iris_dataset.csv, size: 2776, db: Y\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style> \n",
       ".dictlist {\n",
       "  background-color: #b3edff; \n",
       "  text-align: center; \n",
       "  margin: 4px; \n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer; \n",
       "  background-color: #ffe6cc; \n",
       "  text-align: left; \n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "  \n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "  \n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }  \n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "  \n",
       "  \n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>sk-project</td>\n",
       "      <td><div title=\"7ba5b1aab9f14abbb393fdfafca8d98c\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/sk-project/jobs/7ba5b1aab9f14abbb393fdfafca8d98c/info\" target=\"_blank\" >...fca8d98c</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Jun 11 16:24:50</td>\n",
       "      <td>completed</td>\n",
       "      <td>iris_gen</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=handler</div><div class=\"dictlist\">owner=admin</div><div class=\"dictlist\">host=jupyter-65887d7ffb-5jsn2</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result44dc59f4\" title=\"/files/ml3/demos/scikit-learn-pipeline/data/iris_dataset.csv\">iris_dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result44dc59f4-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result44dc59f4-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result44dc59f4\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result44dc59f4-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run 7ba5b1aab9f14abbb393fdfafca8d98c --project sk-project , !mlrun logs 7ba5b1aab9f14abbb393fdfafca8d98c --project sk-project\n",
      "[mlrun] 2020-06-11 16:24:50,320 run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "# run the function locally\n",
    "gen = run_local(name='iris_gen', handler=iris_generator, \n",
    "                project=project_name, artifact_path=path.join(artifact_path, 'data')) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert our local code to a distributed serverless function object "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f08564e54a8>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_func = code_to_function(name='gen_iris', kind='job')\n",
    "skproj.set_function(gen_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and run a library function (visualize dataset features and stats)\n",
    "\n",
    "<b>Step 1:</b> load the function object from the function hub (marketplace)<br>\n",
    "> note: the function marketplace location is configurable, by default it points to `mlrun/functions` git\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f07fe3670f0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skproj.set_function('hub://describe', 'describe')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function: describe\n",
      "describe and visualizes dataset stats\n",
      "default handler: summarize\n",
      "entry points:\n",
      "  summarize: Summarize a table\n",
      "    context(MLClientCtx)  - the function context\n",
      "    table(DataItem)  - MLRun input pointing to pandas dataframe (csv/parquet file path)\n",
      "    label_column(str)  - ground truth column label, default=labels\n",
      "    class_labels(List[str])  - label for each class in tables and plots\n",
      "    plot_hist(bool)  - (True) set this to False for large tables, default=True\n",
      "    plots_dest(str)  - destination folder of summary plots (relative to artifact_path), default=plots\n"
     ]
    }
   ],
   "source": [
    "# read the remote function doc, params, usage\n",
    "skproj.func('describe').doc()\n",
    "#skproj.func('describe').spec.image_pull_policy = 'Always'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Step 2:</b> Run the describe function as a Kubernetes job with specified parameters.\n",
    "\n",
    "> `mount_v3io()` vonnect our function to v3io shared file system and allow us to pass the data and get back the results (plots) directly to our notebook, we can choose other mount options to use NFS or object storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[mlrun] 2020-06-11 16:25:15,465 starting run describe-summarize uid=bfc89df69c0c4ab1851a8201d7887a1a  -> http://mlrun-api:8080\n",
      "[mlrun] 2020-06-11 16:25:15,535 Job is running in the background, pod: describe-summarize-7ztmz\n",
      "[mlrun] 2020-06-11 16:25:21,228 starting local run: main.py # summarize\n",
      "[mlrun] 2020-06-11 16:25:25,273 log artifact histograms at /User/ml3/demos/scikit-learn-pipeline/plots/hist.html, size: 254343, db: Y\n",
      "[mlrun] 2020-06-11 16:25:25,749 log artifact imbalance at /User/ml3/demos/scikit-learn-pipeline/plots/imbalance.html, size: 21657, db: Y\n",
      "[mlrun] 2020-06-11 16:25:25,764 log artifact imbalance-weights-vec at /User/ml3/demos/scikit-learn-pipeline/plots/imbalance-weights-vec.csv, size: 65, db: Y\n",
      "[mlrun] 2020-06-11 16:25:25,778 log artifact correlation-matrix at /User/ml3/demos/scikit-learn-pipeline/plots/correlation-matrix.csv, size: 324, db: Y\n",
      "[mlrun] 2020-06-11 16:25:25,942 log artifact correlation at /User/ml3/demos/scikit-learn-pipeline/plots/corr.html, size: 26385, db: Y\n",
      "\n",
      "[mlrun] 2020-06-11 16:25:26,013 run executed, status=completed\n",
      "final state: succeeded\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style> \n",
       ".dictlist {\n",
       "  background-color: #b3edff; \n",
       "  text-align: center; \n",
       "  margin: 4px; \n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer; \n",
       "  background-color: #ffe6cc; \n",
       "  text-align: left; \n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "  \n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "  \n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }  \n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "  \n",
       "  \n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>sk-project</td>\n",
       "      <td><div title=\"bfc89df69c0c4ab1851a8201d7887a1a\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/sk-project/jobs/bfc89df69c0c4ab1851a8201d7887a1a/info\" target=\"_blank\" >...d7887a1a</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Jun 11 16:25:22</td>\n",
       "      <td>completed</td>\n",
       "      <td>describe-summarize</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=admin</div><div class=\"dictlist\">host=describe-summarize-7ztmz</div></td>\n",
       "      <td><div title=\"store://sk-project/iris_gen_iris_dataset#7ba5b1aab9f14abbb393fdfafca8d98c\">table</div></td>\n",
       "      <td><div class=\"dictlist\">label_column=label</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result227eb2ec\" title=\"/files/ml3/demos/scikit-learn-pipeline/plots/hist.html\">histograms</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result227eb2ec\" title=\"/files/ml3/demos/scikit-learn-pipeline/plots/imbalance.html\">imbalance</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result227eb2ec\" title=\"/files/ml3/demos/scikit-learn-pipeline/plots/imbalance-weights-vec.csv\">imbalance-weights-vec</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result227eb2ec\" title=\"/files/ml3/demos/scikit-learn-pipeline/plots/correlation-matrix.csv\">correlation-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result227eb2ec\" title=\"/files/ml3/demos/scikit-learn-pipeline/plots/corr.html\">correlation</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result227eb2ec-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result227eb2ec-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result227eb2ec\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result227eb2ec-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run bfc89df69c0c4ab1851a8201d7887a1a --project sk-project , !mlrun logs bfc89df69c0c4ab1851a8201d7887a1a --project sk-project\n",
      "[mlrun] 2020-06-11 16:25:27,696 run executed, status=completed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.model.RunObject at 0x7f07fbe59630>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skproj.func('describe').apply(mount_v3io()).run(params={'label_column': 'label'}, \n",
    "                                                inputs={\"table\": gen.outputs['iris_dataset']}, \n",
    "                                                artifact_path=artifact_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Fully Automated ML Pipeline\n",
    "\n",
    "#### Add more functions to our project to be used in our pipeline (from the functions hub/marketplace)\n",
    "\n",
    "AutoML training (classifier), Model validation (test_classifier), Real-time model server, and Model REST API Tester"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f07fa617710>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skproj.set_function('hub://sklearn_classifier', 'train')\n",
    "skproj.set_function('hub://test_classifier', 'test')\n",
    "skproj.set_function('hub://model_server', 'serving')\n",
    "skproj.set_function('hub://model_server_tester', 'live_tester')\n",
    "#print(skproj.to_yaml())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define and save a pipeline \n",
    "\n",
    "The following workflow definition will be written into a file, it describes a Kubeflow execution graph (DAG)<br>\n",
    "and how functions and data are connected  to form an end to end pipeline. \n",
    "\n",
    "* Build the iris generator (ingest) function container \n",
    "* Ingest the iris data\n",
    "* Analyze the dataset (describe)\n",
    "* Train and test the model\n",
    "* Deploy the model as a real-time serverless function\n",
    "* Test the serverless function REST API with test dataset\n",
    "\n",
    "Check the code below to see how functions objects are initialized and used (by name) inside the workflow.<br>\n",
    "The `workflow.py` file has two parts, initialize the function objects and define pipeline dsl (connect the function inputs and outputs).\n",
    "\n",
    "> Note: the pipeline can include CI steps like building container images and deploying models as illustrated  in the following example.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting project/workflow.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile project/workflow.py\n",
    "from kfp import dsl\n",
    "from mlrun import mount_v3io\n",
    "\n",
    "funcs = {}\n",
    "DATASET = 'iris_dataset'\n",
    "LABELS  = \"label\"\n",
    "\n",
    "\n",
    "# init functions is used to configure function resources and local settings\n",
    "def init_functions(functions: dict, project=None, secrets=None):\n",
    "    for f in functions.values():\n",
    "        f.apply(mount_v3io())\n",
    "     \n",
    "    # uncomment this line to collect the inference results into a stream\n",
    "    # and specify a path in V3IO (<datacontainer>/<subpath>)\n",
    "    #functions['serving'].set_env('INFERENCE_STREAM', 'users/admin/model_stream')\n",
    "\n",
    "    \n",
    "@dsl.pipeline(\n",
    "    name=\"Demo training pipeline\",\n",
    "    description=\"Shows how to use mlrun.\"\n",
    ")\n",
    "def kfpipeline():\n",
    "    \n",
    "    # build our ingestion function (container image)\n",
    "    builder = funcs['gen-iris'].deploy_step(skip_deployed=True)\n",
    "    \n",
    "    # run the ingestion function with the new image and params\n",
    "    ingest = funcs['gen-iris'].as_step(\n",
    "        name=\"get-data\",\n",
    "        handler='iris_generator',\n",
    "        image=builder.outputs['image'],\n",
    "        params={'format': 'pq'},\n",
    "        outputs=[DATASET])\n",
    "\n",
    "    # analyze our dataset\n",
    "    describe = funcs[\"describe\"].as_step(\n",
    "        name=\"summary\",\n",
    "        params={\"label_column\": LABELS},\n",
    "        inputs={\"table\": ingest.outputs[DATASET]})\n",
    "    \n",
    "    # train with hyper-paremeters \n",
    "    train = funcs[\"train\"].as_step(\n",
    "        name=\"train-skrf\",\n",
    "        params={\"sample\"          : -1, \n",
    "                \"label_column\"    : LABELS,\n",
    "                \"test_size\"       : 0.10},\n",
    "        hyperparams={'model_pkg_class': [\"sklearn.ensemble.RandomForestClassifier\", \n",
    "                                         \"sklearn.linear_model.LogisticRegression\",\n",
    "                                         \"sklearn.ensemble.AdaBoostClassifier\"]},\n",
    "        selector='max.accuracy',\n",
    "        inputs={\"dataset\"         : ingest.outputs[DATASET]},\n",
    "        outputs=['model', 'test_set'])\n",
    "\n",
    "    # test and visualize our model\n",
    "    test = funcs[\"test\"].as_step(\n",
    "        name=\"test\",\n",
    "        params={\"label_column\": LABELS},\n",
    "        inputs={\"models_path\" : train.outputs['model'],\n",
    "                \"test_set\"    : train.outputs['test_set']})\n",
    "\n",
    "    # deploy our model as a serverless function\n",
    "    deploy = funcs[\"serving\"].deploy_step(models={f\"{DATASET}_v1\": train.outputs['model']}, tag='v2')\n",
    "    \n",
    "    # test out new model server (via REST API calls)\n",
    "    tester = funcs[\"live_tester\"].as_step(name='model-tester',\n",
    "        params={'addr': deploy.outputs['endpoint'], 'model': f\"{DATASET}_v1\"},\n",
    "        inputs={'table': train.outputs['test_set']})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# register the workflow file as \"main\", embed the workflow code into the project YAML\n",
    "skproj.set_workflow('main', 'workflow.py', embed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the project definitions to a file (project.yaml), it is recommended to commit all changes to a Git repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "skproj.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='run-pipeline'></a>\n",
    "## Run a pipeline workflow\n",
    "use the `run` method to execute a workflow, you can provide alternative arguments and specify the default target for workflow artifacts.<br>\n",
    "The workflow ID is returned and can be used to track the progress or you can use the hyperlinks\n",
    "\n",
    "> Note: The same command can be issued through CLI commands:<br>\n",
    "    `mlrun project my-proj/ -r main -p \"v3io:///users/admin/mlrun/kfp/{{workflow.uid}}/\"`\n",
    "\n",
    "The `dirty` flag allow us to run a project with uncommited changes (when the notebook is in the same git dir it will always be dirty)<br>\n",
    "The `watch` flag will wait for the pipeline to complete and print results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Experiment link <a href=\"https://dashboard.default-tenant.app.yh55.iguazio-cd2.com/pipelines/#/experiments/details/bcde4c58-0c45-45b5-b014-26cf653bcf46\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run link <a href=\"https://dashboard.default-tenant.app.yh55.iguazio-cd2.com/pipelines/#/runs/details/ccd706b0-e2be-40a5-b0c7-68058b9d65be\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[mlrun] 2020-06-11 16:28:40,675 Pipeline run id=ccd706b0-e2be-40a5-b0c7-68058b9d65be, check UI or DB for progress\n",
      "[mlrun] 2020-06-11 16:28:40,676 waiting for pipeline run completion\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h2>Run Results</h2>Workflow ccd706b0-e2be-40a5-b0c7-68058b9d65be finished, status=Succeeded<br>click the hyper links below to see detailed results<br><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>uid</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td><div title=\"aefd30f28dcd4fc29d8a9220279dd45a\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/sk-project/jobs/aefd30f28dcd4fc29d8a9220279dd45a/info\" target=\"_blank\" >...279dd45a</a></div></td>\n",
       "      <td>Jun 11 16:29:31</td>\n",
       "      <td>completed</td>\n",
       "      <td>model-tester</td>\n",
       "      <td><div class=\"dictlist\">total_tests=15</div><div class=\"dictlist\">errors=0</div><div class=\"dictlist\">match=14</div><div class=\"dictlist\">avg_latency=6025</div><div class=\"dictlist\">min_latency=5165</div><div class=\"dictlist\">max_latency=8560</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/latency.html\">latency</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"80f51aa74a4d4707a2216fe68bbe4420\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/sk-project/jobs/80f51aa74a4d4707a2216fe68bbe4420/info\" target=\"_blank\" >...8bbe4420</a></div></td>\n",
       "      <td>Jun 11 16:29:21</td>\n",
       "      <td>completed</td>\n",
       "      <td>test</td>\n",
       "      <td><div class=\"dictlist\">accuracy=0.9333333333333333</div><div class=\"dictlist\">test-error=0.06666666666666667</div><div class=\"dictlist\">auc-micro=0.9977777777777778</div><div class=\"dictlist\">auc-weighted=1.0</div><div class=\"dictlist\">f1-score=0.9137254901960784</div><div class=\"dictlist\">precision_score=0.8888888888888888</div><div class=\"dictlist\">recall_score=0.9629629629629629</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/plots/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/plots/precision-recall-multiclass.html\">precision-recall-multiclass</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/plots/roc-multiclass.html\">roc-multiclass</div><div title=\"/User/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/test_set_preds.parquet\">test_set_preds</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"d8b6c782d359482f878c6b846b72eeb3\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/sk-project/jobs/d8b6c782d359482f878c6b846b72eeb3/info\" target=\"_blank\" >...6b72eeb3</a></div></td>\n",
       "      <td>Jun 11 16:29:06</td>\n",
       "      <td>completed</td>\n",
       "      <td>summary</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/plots/hist.html\">histograms</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/plots/imbalance.html\">imbalance</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/plots/imbalance-weights-vec.csv\">imbalance-weights-vec</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/plots/correlation-matrix.csv\">correlation-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/plots/corr.html\">correlation</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"a8b82670347f4ef4b61bd0e92e278a08\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/sk-project/jobs/a8b82670347f4ef4b61bd0e92e278a08/info\" target=\"_blank\" >...2e278a08</a></div></td>\n",
       "      <td>Jun 11 16:29:05</td>\n",
       "      <td>completed</td>\n",
       "      <td>train-skrf</td>\n",
       "      <td><div class=\"dictlist\">best_iteration=2</div><div class=\"dictlist\">accuracy=0.9705882352941176</div><div class=\"dictlist\">test-error=0.029411764705882353</div><div class=\"dictlist\">auc-micro=0.9961072664359862</div><div class=\"dictlist\">auc-weighted=0.9949732620320856</div><div class=\"dictlist\">f1-score=0.9679633867276888</div><div class=\"dictlist\">precision_score=0.9666666666666667</div><div class=\"dictlist\">recall_score=0.9722222222222222</div></td>\n",
       "      <td><div title=\"/User/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/data/2/test_set.parquet\">test_set</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/model/plots/2/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/model/plots/2/precision-recall-multiclass.html\">precision-recall-multiclass</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/model/plots/2/roc-multiclass.html\">roc-multiclass</div><div title=\"/User/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/model/2\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"871860404a094421bd9bd494a60323f5\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/sk-project/jobs/871860404a094421bd9bd494a60323f5/info\" target=\"_blank\" >...a60323f5</a></div></td>\n",
       "      <td>Jun 11 16:28:55</td>\n",
       "      <td>completed</td>\n",
       "      <td>get-data</td>\n",
       "      <td></td>\n",
       "      <td><div title=\"/User/ml3/demos/scikit-learn-pipeline/pipe/ccd706b0-e2be-40a5-b0c7-68058b9d65be/iris_dataset.parquet\">iris_dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "artifact_path = path.abspath('./pipe/{{workflow.uid}}')\n",
    "run_id = skproj.run(\n",
    "    'main',\n",
    "    arguments={}, \n",
    "    artifact_path=artifact_path, \n",
    "    dirty=True, watch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[back to top](#top)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}