t81_558_class_04_2_schedule.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_2_schedule.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "\n",
    "**Module 4: Training for Tabular Data**\n",
    "\n",
    "- Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "- For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "- Part 4.1: Using K-Fold Cross-validation with PyTorch [[Video]](https://www.youtube.com/watch?v=Q8ZQNvZwsNE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_1_kfold.ipynb)\n",
    "- **Part 4.2: Training Schedules for PyTorch**  [[Video]](https://www.youtube.com/watch?v=lMMlbmfvKDQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_2_schedule.ipynb)\n",
    "- Part 4.3: Dropout Regularization [[Video]](https://www.youtube.com/watch?v=4ixjgw6Q42U&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_3_dropout.ipynb)\n",
    "- Part 4.4: Batch Normalization [[Video]](https://www.youtube.com/watch?v=1U5nOKh9OLQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_4_batch_norm.ipynb)\n",
    "- Part 4.5: RAPIDS for Tabular Data [[Video]](https://www.youtube.com/watch?v=KgoXuhG_kfs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_5_rapids.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running and maps Google Drive if needed. We also initialize the PyTorch device to either GPU/MPS (if available) or CPU.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n",
      "Using device: mps\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "import torch\n",
    "\n",
    "try:\n",
    "    import google.colab\n",
    "\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Make use of a GPU or MPS (Apple) if one is available.  (see module 3.2)\n",
    "import torch\n",
    "has_mps = torch.backends.mps.is_built()\n",
    "device = \"mps\" if has_mps else \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(f\"Using device: {device}\")\n",
    "\n",
    "# Early stopping (see module 3.4)\n",
    "class EarlyStopping:\n",
    "    def __init__(self, patience=5, min_delta=0, restore_best_weights=True):\n",
    "        self.patience = patience\n",
    "        self.min_delta = min_delta\n",
    "        self.restore_best_weights = restore_best_weights\n",
    "        self.best_model = None\n",
    "        self.best_loss = None\n",
    "        self.counter = 0\n",
    "        self.status = \"\"\n",
    "\n",
    "    def __call__(self, model, val_loss):\n",
    "        if self.best_loss is None:\n",
    "            self.best_loss = val_loss\n",
    "            self.best_model = copy.deepcopy(model.state_dict())\n",
    "        elif self.best_loss - val_loss >= self.min_delta:\n",
    "            self.best_model = copy.deepcopy(model.state_dict())\n",
    "            self.best_loss = val_loss\n",
    "            self.counter = 0\n",
    "            self.status = f\"Improvement found, counter reset to {self.counter}\"\n",
    "        else:\n",
    "            self.counter += 1\n",
    "            self.status = f\"No improvement in the last {self.counter} epochs\"\n",
    "            if self.counter >= self.patience:\n",
    "                self.status = f\"Early stopping triggered after {self.counter} epochs.\"\n",
    "                if self.restore_best_weights:\n",
    "                    model.load_state_dict(self.best_model)\n",
    "                return True\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.2: Training Schedules for PyTorch\n",
    "\n",
    "Learning rate schedules are mechanisms used during the training of neural networks to adjust the learning rate over time. They're designed to decrease the learning rate as the training progresses, allowing the network to make large adjustments in the initial stages of training, when the weights are likely far from their optimal values, and then make smaller adjustments as the training progresses, to fine-tune the weights. This adjustment helps mitigate the risk of overshooting the minimum point of the loss function and helps to reach convergence more smoothly.\n",
    "\n",
    "In PyTorch, one of the learning rate scheduling tools is the StepLR class, found in the **torch.optim.lr_scheduler** module. **StepLR** is a type of learning rate schedule that decreases the learning rate by a certain factor every few epochs. This allows the learning rate to decrease in a step-wise fashion rather than continuously, which can be beneficial in some cases, as it gives the model time to 'settle' into areas of the loss landscape before the learning rate is reduced further.\n",
    "\n",
    "StepLR takes three parameters:\n",
    "\n",
    "* **optimizer:** The optimizer you're using to train your model (e.g., SGD, Adam).\n",
    "* **step_size:** This is the number of epochs after which you want to reduce the learning rate. For instance, if step_size=10, then the learning rate will be reduced every 10 epochs.\n",
    "* **gamma:** This is the factor by which the learning rate will be reduced at each step. For instance, if gamma=0.1, the learning rate will be multiplied by 0.1 at each step, effectively reducing it by 90%.\n",
    "\n",
    "The **StepLR** scheduler is used during the training loop. After each step of the optimizer (after **optimizer.step()**), you call scheduler.step() to adjust the learning rate according to the schedule.\n",
    "\n",
    "It's worth noting that the choice of **step_size** and gamma can be important, and may need to be tuned based on your specific problem and dataset. Too large a **step_size** and the learning rate may not reduce quickly enough; too small and it may reduce too quickly. Similarly, a gamma too close to 1 may not reduce the learning rate significantly enough, while a gamma too small may reduce it too quickly.\n",
    "\n",
    "We now apply a learning rate to the k-fold cross validation example from the previous section.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\",dtype=int)],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\",dtype=int)],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\",dtype=int)],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the feature vector is created a 5-fold cross-validation can be performed to generate out-of-sample predictions. We will assume 500 epochs and not use early stopping. Later we will see how we can estimate a more optimal epoch count.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/__init__.py:394: UserWarning: Error detected in LinearBackward0. Traceback of forward call that caused the error:\n",
      "  File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 215, in forward\n",
      "    input = module(input)\n",
      " (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1702400227158/work/torch/csrc/autograd/python_anomaly_mode.cpp:119.)\n",
      "  result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass\n",
      "[2024-01-07 07:04:11,083] [0/2] torch._dynamo.exc: [WARNING] Backend compiler failed with a fake tensor exception at \n",
      "[2024-01-07 07:04:11,083] [0/2] torch._dynamo.exc: [WARNING]   File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 216, in forward\n",
      "[2024-01-07 07:04:11,083] [0/2] torch._dynamo.exc: [WARNING]     return input\n",
      "[2024-01-07 07:04:11,083] [0/2] torch._dynamo.exc: [WARNING] Adding a graph break.\n",
      "/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/__init__.py:394: UserWarning: Error detected in LinearBackward0. Traceback of forward call that caused the error:\n",
      "  File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 215, in forward\n",
      "    input = module(input)\n",
      " (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1702400227158/work/torch/csrc/autograd/python_anomaly_mode.cpp:119.)\n",
      "  result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass\n",
      "[2024-01-07 07:04:11,112] [0/2] torch._dynamo.exc: [WARNING] Backend compiler failed with a fake tensor exception at \n",
      "[2024-01-07 07:04:11,112] [0/2] torch._dynamo.exc: [WARNING]   File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 216, in forward\n",
      "[2024-01-07 07:04:11,112] [0/2] torch._dynamo.exc: [WARNING]     return input\n",
      "[2024-01-07 07:04:11,112] [0/2] torch._dynamo.exc: [WARNING] Adding a graph break.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 199/500, Validation Loss: 0.5704283714294434, Early stopping triggered after 5 epochs.\n",
      "Fold #2\n",
      "Epoch 146/500, Validation Loss: 0.45389053225517273, Early stopping triggered after 5 epochs.\n",
      "Fold #3\n",
      "Epoch 165/500, Validation Loss: 0.7377966046333313, Early stopping triggered after 5 epochs.\n",
      "Fold #4\n",
      "Epoch 165/500, Validation Loss: 0.45836910605430603, Early stopping triggered after 5 epochs.\n",
      "Fold #5\n",
      "Epoch 137/500, Validation Loss: 1.2551432847976685, Early stopping triggered after 5 epochs.\n",
      "Fold score (RMSE): 1.1172937154769897\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.optim.lr_scheduler import StepLR\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Convert to PyTorch Tensors\n",
    "x_columns = df.columns.drop(['age', 'id'])\n",
    "x = torch.tensor(df[x_columns].values, dtype=torch.float32, device=device)\n",
    "y = torch.tensor(df['age'].values, dtype=torch.float32, device=device).view(-1, 1)\n",
    "\n",
    "# Set random seed for reproducibility\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# Cross-Validate\n",
    "kf = KFold(n_splits=5, shuffle=True, random_state=42)\n",
    "\n",
    "# Early stopping parameters\n",
    "patience = 10\n",
    "\n",
    "fold = 0\n",
    "for train_idx, test_idx in kf.split(x):\n",
    "    fold += 1\n",
    "    print(f\"Fold #{fold}\")\n",
    "\n",
    "    x_train, x_test = x[train_idx], x[test_idx]\n",
    "    y_train, y_test = y[train_idx], y[test_idx]\n",
    "\n",
    "    # PyTorch DataLoader\n",
    "    train_dataset = TensorDataset(x_train, y_train)\n",
    "    train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)\n",
    "\n",
    "    # Create the model and optimizer\n",
    "    model = nn.Sequential(\n",
    "        nn.Linear(x.shape[1], 20),\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(20, 10),\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(10, 1)\n",
    "    )\n",
    "    model = torch.compile(model,backend=\"aot_eager\").to(device)\n",
    "    \n",
    "    optimizer = optim.Adam(model.parameters())\n",
    "    # adjust learning rate every 50 epochs\n",
    "    scheduler = StepLR(optimizer, step_size=50, gamma=0.90)  \n",
    "    loss_fn = nn.MSELoss()\n",
    "\n",
    "    # Early Stopping variables\n",
    "    best_loss = float('inf')\n",
    "    early_stopping_counter = 0\n",
    "\n",
    "    # Training loop\n",
    "    EPOCHS = 500\n",
    "    epoch = 0\n",
    "    done = False\n",
    "    es = EarlyStopping()\n",
    "\n",
    "    while not done and epoch<EPOCHS:\n",
    "        epoch += 1\n",
    "        model.train()\n",
    "        for x_batch, y_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            output = model(x_batch)\n",
    "            loss = loss_fn(output, y_batch)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        scheduler.step()  # apply learning rate schedule\n",
    "        # Validation\n",
    "        model.eval()\n",
    "        with torch.no_grad():\n",
    "            val_output = model(x_test)\n",
    "            val_loss = loss_fn(val_output, y_test)\n",
    "\n",
    "        if es(model, val_loss):\n",
    "            done = True\n",
    "\n",
    "    print(f\"Epoch {epoch}/{EPOCHS}, Validation Loss: \"\n",
    "      f\"{val_loss.item()}, {es.status}\")\n",
    "\n",
    "# Final evaluation\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    oos_pred = model(x_test)\n",
    "score = torch.sqrt(loss_fn(oos_pred, y_test)).item()\n",
    "print(f\"Fold score (RMSE): {score}\")\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.11 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}