tutorials/create-first-ml-experiment/tutorial-1st-experiment-sdk-train.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/tutorial-quickstart-train-model.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Tutorial: Train your first model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This tutorial is **part two of a two-part tutorial series**. In the previous tutorial, you created a workspace and chose a development environment. In this tutorial, you learn the foundational design patterns in Azure Machine Learning service, and train a simple scikit-learn model based on the diabetes data set. After completing this tutorial, you will have the practical knowledge of the SDK to scale up to developing more-complex experiments and workflows. \n",
        "\n",
        "In this tutorial, you learn the following tasks:\n",
        "\n",
        "> * Connect your workspace and create an experiment \n",
        "> * Load data and train a scikit-learn model\n",
        "> * View training results in the studio\n",
        "> * Retrieve the best model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prerequisites\n",
        "\n",
        "The only prerequisite is to run the previous tutorial, Setup environment and workspace."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Connect workspace and create experiment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Import the `Workspace` class, and load your subscription information from the file `config.json` using the function `from_config().` This looks for the JSON file in the current directory by default, but you can also specify a path parameter to point to the file using `from_config(path=\"your/file/path\")`. If you are running this notebook in a cloud notebook server in your workspace, the file is automatically in the root directory.\n",
        "\n",
        "If the following code asks for additional authentication, simply paste the link in a browser and enter the authentication token. In addition, if you have more than one tenant linked to your user, you will need to add the following lines:\n",
        "```\n",
        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
        "interactive_auth = InteractiveLoginAuthentication(tenant_id=\"your-tenant-id\")\n",
        "Additional details on authentication can be found here: https://aka.ms/aml-notebook-auth \n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Workspace\n",
        "ws = Workspace.from_config()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now create an experiment in your workspace. An experiment is another foundational cloud resource that represents a collection of trials (individual model runs). In this tutorial you use the experiment to create runs and track your model training in the Azure Machine Learning studio. Parameters include your workspace reference, and a string name for the experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "experiment = Experiment(workspace=ws, name=\"diabetes-experiment\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load data and prepare for training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For this tutorial, you use the diabetes data set, which uses features like age, gender, and BMI to predict diabetes disease progression. Load the data from the Azure Open Datasets class, and split it into training and test sets using `train_test_split()`. This function segregates the data so the model has unseen data to use for testing following training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.opendatasets import Diabetes\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "x_df = Diabetes.get_tabular_dataset().to_pandas_dataframe().dropna()\n",
        "y_df = x_df.pop(\"Y\")\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.2, random_state=66)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Train a model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Training a simple scikit-learn model can easily be done locally for small-scale training, but when training many iterations with dozens of different feature permutations and hyperparameter settings, it is easy to lose track of what models you've trained and how you trained them. The following design pattern shows how to leverage the SDK to easily keep track of your training in the cloud.\n",
        "\n",
        "Build a script that trains ridge models in a loop through different hyperparameter alpha values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.linear_model import Ridge\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.externals import joblib\n",
        "import math\n",
        "\n",
        "alphas = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]\n",
        "\n",
        "for alpha in alphas:\n",
        "    run = experiment.start_logging()\n",
        "    run.log(\"alpha_value\", alpha)\n",
        "    \n",
        "    model = Ridge(alpha=alpha)\n",
        "    model.fit(X=X_train, y=y_train)\n",
        "    y_pred = model.predict(X=X_test)\n",
        "    rmse = math.sqrt(mean_squared_error(y_true=y_test, y_pred=y_pred))\n",
        "    run.log(\"rmse\", rmse)\n",
        "    \n",
        "    model_name = \"model_alpha_\" + str(alpha) + \".pkl\"\n",
        "    filename = \"outputs/\" + model_name\n",
        "    \n",
        "    joblib.dump(value=model, filename=filename)\n",
        "    run.upload_file(name=model_name, path_or_stream=filename)\n",
        "    run.complete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The above code accomplishes the following:\n",
        "\n",
        "1. For each alpha hyperparameter value in the `alphas` array, a new run is created within the experiment. The alpha value is logged to differentiate between each run.\n",
        "1. In each run, a Ridge model is instantiated, trained, and used to run predictions. The root-mean-squared-error is calculated for the actual versus predicted values, and then logged to the run. At this point the run has metadata attached for both the alpha value and the rmse accuracy.\n",
        "1. Next, the model for each run is serialized and uploaded to the run. This allows you to download the model file from the run in the studio.\n",
        "1. At the end of each iteration the run is completed by calling `run.complete()`.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "After the training has completed, call the `experiment` variable to fetch a link to the experiment in the studio."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "experiment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## View training results in studio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Following the **Link to Azure Machine Learning studio** takes you to the main experiment page. Here you see all the individual runs in the experiment. Any custom-logged values (`alpha_value` and `rmse`, in this case) become fields for each run, and also become available for the charts and tiles at the top of the experiment page. To add a logged metric to a chart or tile, hover over it, click the edit button, and find your custom-logged metric.\n",
        "\n",
        "When training models at scale over hundreds and thousands of runs, this page makes it easy to see every model you trained, specifically how they were trained, and how your unique metrics have changed over time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Main Experiment page in the studio](./imgs/experiment_main.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Select a run number link in the `RUN NUMBER` column to see the page for an individual run. The default tab **Details** shows you more-detailed information on each run. Navigate to the **Outputs + logs** tab, and you see the `.pkl` file for the model that was uploaded to the run during each training iteration. Here you can download the model file, rather than having to retrain it manually."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Run details page in the studio](./imgs/model_download.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Get the best model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In addition to being able to download model files from the experiment in the studio, you can also download them programmatically. The following code iterates through each run in the experiment, and accesses both the logged run metrics and the run details (which contains the run_id). This keeps track of the best run, in this case the run with the lowest root-mean-squared-error."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "minimum_rmse_runid = None\n",
        "minimum_rmse = None\n",
        "\n",
        "for run in experiment.get_runs():\n",
        "    run_metrics = run.get_metrics()\n",
        "    run_details = run.get_details()\n",
        "    # each logged metric becomes a key in this returned dict\n",
        "    run_rmse = run_metrics[\"rmse\"]\n",
        "    run_id = run_details[\"runId\"]\n",
        "    \n",
        "    if minimum_rmse is None:\n",
        "        minimum_rmse = run_rmse\n",
        "        minimum_rmse_runid = run_id\n",
        "    else:\n",
        "        if run_rmse < minimum_rmse:\n",
        "            minimum_rmse = run_rmse\n",
        "            minimum_rmse_runid = run_id\n",
        "\n",
        "print(\"Best run_id: \" + minimum_rmse_runid)\n",
        "print(\"Best run_id rmse: \" + str(minimum_rmse))    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Use the best run id to fetch the individual run using the `Run` constructor along with the experiment object. Then call `get_file_names()` to see all the files available for download from this run. In this case, you only uploaded one file for each run during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Run\n",
        "best_run = Run(experiment=experiment, run_id=minimum_rmse_runid)\n",
        "print(best_run.get_file_names())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Call `download()` on the run object, specifying the model file name to download. By default this function downloads to the current directory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_run.download_file(name=\"model_alpha_0.1.pkl\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Clean up resources\n",
        "\n",
        "Do not complete this section if you plan on running other Azure Machine Learning service tutorials.\n",
        "\n",
        "### Stop the notebook VM\n",
        "\n",
        "If you used a cloud notebook server, stop the VM when you are not using it to reduce cost.\n",
        "\n",
        "1. In your workspace, select **Compute**.\n",
        "\n",
        "1. Select the **Notebook VMs** tab in the compute page.\n",
        "\n",
        "1. From the list, select the VM.\n",
        "\n",
        "1. Select **Stop**.\n",
        "\n",
        "1. When you're ready to use the server again, select **Start**.\n",
        "\n",
        "### Delete everything\n",
        "\n",
        "If you don't plan to use the resources you created, delete them, so you don't incur any charges:\n",
        "\n",
        "1. In the Azure portal, select **Resource groups** on the far left.\n",
        "\n",
        "1. From the list, select the resource group you created.\n",
        "\n",
        "1. Select **Delete resource group**.\n",
        "\n",
        "1. Enter the resource group name. Then select **Delete**.\n",
        "\n",
        "You can also keep the resource group but delete a single workspace. Display the workspace properties and select **Delete**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Next steps\n",
        "\n",
        "In this tutorial, you did the following tasks:\n",
        "\n",
        "> * Connected your workspace and created an experiment\n",
        "> * Loaded data and trained scikit-learn models\n",
        "> * Viewed training results in the studio and retrieved models\n",
        "\n",
        "[Deploy your model](https://docs.microsoft.com/azure/machine-learning/service/tutorial-deploy-models-with-aml) with Azure Machine Learning.\n",
        "Learn how to develop [automated machine learning](https://docs.microsoft.com/azure/machine-learning/service/tutorial-auto-train-models) experiments."
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "trbye"
      }
    ],
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    },
    "msauthor": "trbye",
    "network_required": false
  },
  "nbformat": 4,
  "nbformat_minor": 2
}