exercise-missing-values.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3399df6",
   "metadata": {
    "papermill": {
     "duration": 0.023609,
     "end_time": "2022-01-03T11:40:50.569807",
     "exception": false,
     "start_time": "2022-01-03T11:40:50.546198",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Intermediate Machine Learning](https://www.kaggle.com/learn/intermediate-machine-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/missing-values).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e7198ed",
   "metadata": {
    "papermill": {
     "duration": 0.022311,
     "end_time": "2022-01-03T11:40:50.614594",
     "exception": false,
     "start_time": "2022-01-03T11:40:50.592283",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now it's your turn to test your new knowledge of **missing values** handling. You'll probably find it makes a big difference.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6ae9ec6e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:50.668470Z",
     "iopub.status.busy": "2022-01-03T11:40:50.667894Z",
     "iopub.status.idle": "2022-01-03T11:40:50.721677Z",
     "shell.execute_reply": "2022-01-03T11:40:50.722111Z",
     "shell.execute_reply.started": "2022-01-03T10:45:07.094067Z"
    },
    "papermill": {
     "duration": 0.08516,
     "end_time": "2022-01-03T11:40:50.722387",
     "exception": false,
     "start_time": "2022-01-03T11:40:50.637227",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "# Set up code checking\n",
    "import os\n",
    "if not os.path.exists(\"../input/train.csv\"):\n",
    "    os.symlink(\"../input/home-data-for-ml-course/train.csv\", \"../input/train.csv\")  \n",
    "    os.symlink(\"../input/home-data-for-ml-course/test.csv\", \"../input/test.csv\") \n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.ml_intermediate.ex2 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d22da5",
   "metadata": {
    "papermill": {
     "duration": 0.025879,
     "end_time": "2022-01-03T11:40:50.781886",
     "exception": false,
     "start_time": "2022-01-03T11:40:50.756007",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you will work with data from the [Housing Prices Competition for Kaggle Learn Users](https://www.kaggle.com/c/home-data-for-ml-course). \n",
    "\n",
    "![Ames Housing dataset image](https://i.imgur.com/lTJVG4e.png)\n",
    "\n",
    "Run the next code cell without changes to load the training and validation sets in `X_train`, `X_valid`, `y_train`, and `y_valid`.  The test set is loaded in `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "630940e2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:50.837119Z",
     "iopub.status.busy": "2022-01-03T11:40:50.836540Z",
     "iopub.status.idle": "2022-01-03T11:40:51.922882Z",
     "shell.execute_reply": "2022-01-03T11:40:51.922318Z",
     "shell.execute_reply.started": "2022-01-03T10:45:45.389674Z"
    },
    "papermill": {
     "duration": 1.115371,
     "end_time": "2022-01-03T11:40:51.923018",
     "exception": false,
     "start_time": "2022-01-03T11:40:50.807647",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data\n",
    "X_full = pd.read_csv('../input/train.csv', index_col='Id')\n",
    "X_test_full = pd.read_csv('../input/test.csv', index_col='Id')\n",
    "\n",
    "# Remove rows with missing target, separate target from predictors\n",
    "X_full.dropna(axis=0, subset=['SalePrice'], inplace=True)\n",
    "y = X_full.SalePrice\n",
    "X_full.drop(['SalePrice'], axis=1, inplace=True)\n",
    "\n",
    "# To keep things simple, we'll use only numerical predictors\n",
    "X = X_full.select_dtypes(exclude=['object'])\n",
    "X_test = X_test_full.select_dtypes(exclude=['object'])\n",
    "\n",
    "# Break off validation set from training data\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2,\n",
    "                                                      random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "432cc5e8",
   "metadata": {
    "papermill": {
     "duration": 0.022176,
     "end_time": "2022-01-03T11:40:51.967961",
     "exception": false,
     "start_time": "2022-01-03T11:40:51.945785",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Use the next code cell to print the first five rows of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "098db2ea",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:52.023326Z",
     "iopub.status.busy": "2022-01-03T11:40:52.022740Z",
     "iopub.status.idle": "2022-01-03T11:40:52.041274Z",
     "shell.execute_reply": "2022-01-03T11:40:52.040736Z",
     "shell.execute_reply.started": "2022-01-03T10:45:47.307386Z"
    },
    "papermill": {
     "duration": 0.048953,
     "end_time": "2022-01-03T11:40:52.041418",
     "exception": false,
     "start_time": "2022-01-03T11:40:51.992465",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>...</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>619</th>\n",
       "      <td>20</td>\n",
       "      <td>90.0</td>\n",
       "      <td>11694</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>2007</td>\n",
       "      <td>452.0</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>774</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>260</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>20</td>\n",
       "      <td>60.0</td>\n",
       "      <td>6600</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1962</td>\n",
       "      <td>1962</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>308</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>30</td>\n",
       "      <td>80.0</td>\n",
       "      <td>13360</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>1921</td>\n",
       "      <td>2006</td>\n",
       "      <td>0.0</td>\n",
       "      <td>713</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>432</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>818</th>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13265</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2002</td>\n",
       "      <td>2002</td>\n",
       "      <td>148.0</td>\n",
       "      <td>1218</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>857</td>\n",
       "      <td>150</td>\n",
       "      <td>59</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>20</td>\n",
       "      <td>118.0</td>\n",
       "      <td>13704</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>843</td>\n",
       "      <td>468</td>\n",
       "      <td>81</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
       "Id                                                                           \n",
       "619          20         90.0    11694            9            5       2007   \n",
       "871          20         60.0     6600            5            5       1962   \n",
       "93           30         80.0    13360            5            7       1921   \n",
       "818          20          NaN    13265            8            5       2002   \n",
       "303          20        118.0    13704            7            5       2001   \n",
       "\n",
       "     YearRemodAdd  MasVnrArea  BsmtFinSF1  BsmtFinSF2  ...  GarageArea  \\\n",
       "Id                                                     ...               \n",
       "619          2007       452.0          48           0  ...         774   \n",
       "871          1962         0.0           0           0  ...         308   \n",
       "93           2006         0.0         713           0  ...         432   \n",
       "818          2002       148.0        1218           0  ...         857   \n",
       "303          2002       150.0           0           0  ...         843   \n",
       "\n",
       "     WoodDeckSF  OpenPorchSF  EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  \\\n",
       "Id                                                                              \n",
       "619           0          108              0          0          260         0   \n",
       "871           0            0              0          0            0         0   \n",
       "93            0            0             44          0            0         0   \n",
       "818         150           59              0          0            0         0   \n",
       "303         468           81              0          0            0         0   \n",
       "\n",
       "     MiscVal  MoSold  YrSold  \n",
       "Id                            \n",
       "619        0       7    2007  \n",
       "871        0       8    2009  \n",
       "93         0       8    2009  \n",
       "818        0       7    2008  \n",
       "303        0       1    2006  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e3688a5",
   "metadata": {
    "papermill": {
     "duration": 0.023557,
     "end_time": "2022-01-03T11:40:52.088556",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.064999",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You can already see a few missing values in the first several rows.  In the next step, you'll obtain a more comprehensive understanding of the missing values in the dataset.\n",
    "\n",
    "# Step 1: Preliminary investigation\n",
    "\n",
    "Run the code cell below without changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f3ef1894",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:52.142538Z",
     "iopub.status.busy": "2022-01-03T11:40:52.141942Z",
     "iopub.status.idle": "2022-01-03T11:40:52.144265Z",
     "shell.execute_reply": "2022-01-03T11:40:52.144827Z",
     "shell.execute_reply.started": "2022-01-03T10:45:56.544017Z"
    },
    "papermill": {
     "duration": 0.033079,
     "end_time": "2022-01-03T11:40:52.144976",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.111897",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1168, 36)\n",
      "LotFrontage    212\n",
      "MasVnrArea       6\n",
      "GarageYrBlt     58\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Shape of training data (num_rows, num_columns)\n",
    "print(X_train.shape)\n",
    "\n",
    "# Number of missing values in each column of training data\n",
    "missing_val_count_by_column = (X_train.isnull().sum())\n",
    "print(missing_val_count_by_column[missing_val_count_by_column > 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c18ade5",
   "metadata": {
    "papermill": {
     "duration": 0.023543,
     "end_time": "2022-01-03T11:40:52.192210",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.168667",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part A\n",
    "\n",
    "Use the above output to answer the questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c297c6d9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:52.241848Z",
     "iopub.status.busy": "2022-01-03T11:40:52.241221Z",
     "iopub.status.idle": "2022-01-03T11:40:52.246985Z",
     "shell.execute_reply": "2022-01-03T11:40:52.247562Z",
     "shell.execute_reply.started": "2022-01-03T10:47:35.544522Z"
    },
    "papermill": {
     "duration": 0.032043,
     "end_time": "2022-01-03T11:40:52.247719",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.215676",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"1.1_InvestigateEquality\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the line below: How many rows are in the training data?\n",
    "num_rows = 1168\n",
    "\n",
    "# Fill in the line below: How many columns in the training data\n",
    "# have missing values?\n",
    "num_cols_with_missing = 3\n",
    "\n",
    "# Fill in the line below: How many missing entries are contained in \n",
    "# all of the training data?\n",
    "tot_missing = 212 + 6 + 58\n",
    "\n",
    "# Check your answers\n",
    "step_1.a.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1ccaf710",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:52.298905Z",
     "iopub.status.busy": "2022-01-03T11:40:52.298338Z",
     "iopub.status.idle": "2022-01-03T11:40:52.300838Z",
     "shell.execute_reply": "2022-01-03T11:40:52.301343Z",
     "shell.execute_reply.started": "2022-01-03T10:47:37.627966Z"
    },
    "papermill": {
     "duration": 0.029484,
     "end_time": "2022-01-03T11:40:52.301514",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.272030",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_1.a.hint()\n",
    "#step_1.a.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e90f2dc2",
   "metadata": {
    "papermill": {
     "duration": 0.024361,
     "end_time": "2022-01-03T11:40:52.349999",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.325638",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part B\n",
    "Considering your answers above, what do you think is likely the best approach to dealing with the missing values?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62e055bc",
   "metadata": {
    "papermill": {
     "duration": 0.03027,
     "end_time": "2022-01-03T11:40:52.415022",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.384752",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "`fillna`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a503d68",
   "metadata": {
    "papermill": {
     "duration": 0.030497,
     "end_time": "2022-01-03T11:40:52.475167",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.444670",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "aka *imputation*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3a6597db",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:52.527150Z",
     "iopub.status.busy": "2022-01-03T11:40:52.526619Z",
     "iopub.status.idle": "2022-01-03T11:40:52.531554Z",
     "shell.execute_reply": "2022-01-03T11:40:52.532059Z",
     "shell.execute_reply.started": "2022-01-03T10:49:40.078638Z"
    },
    "papermill": {
     "duration": 0.031843,
     "end_time": "2022-01-03T11:40:52.532222",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.500379",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"1.2_InvestigateThought\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "Since there are relatively few missing entries in the data (the column with the greatest percentage of missing values is missing less than 20% of its entries), we can expect that dropping columns is unlikely to yield good results.  This is because we'd be throwing away a lot of valuable data, and so imputation will likely perform better."
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "Since there are relatively few missing entries in the data (the column with the greatest percentage of missing values is missing less than 20% of its entries), we can expect that dropping columns is unlikely to yield good results.  This is because we'd be throwing away a lot of valuable data, and so imputation will likely perform better."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "step_1.b.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "afb3721d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:52.584493Z",
     "iopub.status.busy": "2022-01-03T11:40:52.583926Z",
     "iopub.status.idle": "2022-01-03T11:40:52.586734Z",
     "shell.execute_reply": "2022-01-03T11:40:52.587239Z",
     "shell.execute_reply.started": "2022-01-03T10:49:41.093457Z"
    },
    "papermill": {
     "duration": 0.030336,
     "end_time": "2022-01-03T11:40:52.587392",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.557056",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#step_1.b.hint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99e2eba5",
   "metadata": {
    "papermill": {
     "duration": 0.024536,
     "end_time": "2022-01-03T11:40:52.636735",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.612199",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To compare different approaches to dealing with missing values, you'll use the same `score_dataset()` function from the tutorial.  This function reports the [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error) (MAE) from a random forest model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1beec706",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:52.689869Z",
     "iopub.status.busy": "2022-01-03T11:40:52.689306Z",
     "iopub.status.idle": "2022-01-03T11:40:52.873578Z",
     "shell.execute_reply": "2022-01-03T11:40:52.872897Z",
     "shell.execute_reply.started": "2022-01-03T10:50:18.613881Z"
    },
    "papermill": {
     "duration": 0.211257,
     "end_time": "2022-01-03T11:40:52.873718",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.662461",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "# Function for comparing different approaches\n",
    "def score_dataset(X_train, X_valid, y_train, y_valid):\n",
    "    model = RandomForestRegressor(n_estimators=100, random_state=0)\n",
    "    model.fit(X_train, y_train)\n",
    "    preds = model.predict(X_valid)\n",
    "    return mean_absolute_error(y_valid, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81b22c99",
   "metadata": {
    "papermill": {
     "duration": 0.02505,
     "end_time": "2022-01-03T11:40:52.923649",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.898599",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 2: Drop columns with missing values\n",
    "\n",
    "In this step, you'll preprocess the data in `X_train` and `X_valid` to remove columns with missing values.  Set the preprocessed DataFrames to `reduced_X_train` and `reduced_X_valid`, respectively.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c5779f92",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:52.980796Z",
     "iopub.status.busy": "2022-01-03T11:40:52.980156Z",
     "iopub.status.idle": "2022-01-03T11:40:52.984673Z",
     "shell.execute_reply": "2022-01-03T11:40:52.984045Z",
     "shell.execute_reply.started": "2022-01-03T10:54:58.428400Z"
    },
    "papermill": {
     "duration": 0.035862,
     "end_time": "2022-01-03T11:40:52.984850",
     "exception": false,
     "start_time": "2022-01-03T11:40:52.948988",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LotFrontage', 'MasVnrArea', 'GarageYrBlt']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(missing_val_count_by_column[missing_val_count_by_column > 0].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "275ecfa5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:53.039552Z",
     "iopub.status.busy": "2022-01-03T11:40:53.038991Z",
     "iopub.status.idle": "2022-01-03T11:40:53.056617Z",
     "shell.execute_reply": "2022-01-03T11:40:53.057132Z",
     "shell.execute_reply.started": "2022-01-03T10:57:26.223995Z"
    },
    "papermill": {
     "duration": 0.045819,
     "end_time": "2022-01-03T11:40:53.057288",
     "exception": false,
     "start_time": "2022-01-03T11:40:53.011469",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"2_DropMissing\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the line below: get names of columns with missing values\n",
    "missing_columns = list(missing_val_count_by_column[missing_val_count_by_column > 0].index) # Your code here\n",
    "\n",
    "# Fill in the lines below: drop columns in training and validation data\n",
    "reduced_X_train = X_train.drop(axis=1, columns=missing_columns)\n",
    "reduced_X_valid = X_valid.drop(axis=1, columns=missing_columns)\n",
    "\n",
    "# Check your answers\n",
    "step_2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d3303427",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:53.111858Z",
     "iopub.status.busy": "2022-01-03T11:40:53.111262Z",
     "iopub.status.idle": "2022-01-03T11:40:53.113803Z",
     "shell.execute_reply": "2022-01-03T11:40:53.114316Z",
     "shell.execute_reply.started": "2022-01-03T10:57:29.566776Z"
    },
    "papermill": {
     "duration": 0.031307,
     "end_time": "2022-01-03T11:40:53.114489",
     "exception": false,
     "start_time": "2022-01-03T11:40:53.083182",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_2.hint()\n",
    "#step_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45371bac",
   "metadata": {
    "papermill": {
     "duration": 0.025993,
     "end_time": "2022-01-03T11:40:53.166669",
     "exception": false,
     "start_time": "2022-01-03T11:40:53.140676",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell without changes to obtain the MAE for this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "44fddcc8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:53.221599Z",
     "iopub.status.busy": "2022-01-03T11:40:53.221022Z",
     "iopub.status.idle": "2022-01-03T11:40:54.389790Z",
     "shell.execute_reply": "2022-01-03T11:40:54.389001Z",
     "shell.execute_reply.started": "2022-01-03T10:57:31.362805Z"
    },
    "papermill": {
     "duration": 1.197108,
     "end_time": "2022-01-03T11:40:54.389934",
     "exception": false,
     "start_time": "2022-01-03T11:40:53.192826",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (Drop columns with missing values):\n",
      "17837.82570776256\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE (Drop columns with missing values):\")\n",
    "print(score_dataset(reduced_X_train, reduced_X_valid, y_train, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "add39ce4",
   "metadata": {
    "papermill": {
     "duration": 0.026157,
     "end_time": "2022-01-03T11:40:54.443443",
     "exception": false,
     "start_time": "2022-01-03T11:40:54.417286",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 3: Imputation\n",
    "\n",
    "### Part A\n",
    "\n",
    "Use the next code cell to impute missing values with the mean value along each column.  Set the preprocessed DataFrames to `imputed_X_train` and `imputed_X_valid`.  Make sure that the column names match those in `X_train` and `X_valid`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5cee5f73",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:54.501511Z",
     "iopub.status.busy": "2022-01-03T11:40:54.500715Z",
     "iopub.status.idle": "2022-01-03T11:40:54.526899Z",
     "shell.execute_reply": "2022-01-03T11:40:54.527362Z",
     "shell.execute_reply.started": "2022-01-03T11:15:25.399173Z"
    },
    "papermill": {
     "duration": 0.057562,
     "end_time": "2022-01-03T11:40:54.527561",
     "exception": false,
     "start_time": "2022-01-03T11:40:54.469999",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"3.1_ImputeCode\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# Fill in the lines below: imputation\n",
    "imputer_1 = SimpleImputer() # Your code here\n",
    "imputed_X_train = pd.DataFrame(imputer_1.fit_transform(X_train))\n",
    "imputed_X_valid = pd.DataFrame(imputer_1.transform(X_valid))\n",
    "\n",
    "# Fill in the lines below: imputation removed column names; put them back\n",
    "imputed_X_train.columns = X_train.columns\n",
    "imputed_X_valid.columns = X_valid.columns\n",
    "\n",
    "# Check your answers\n",
    "step_3.a.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "321842ca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:54.585423Z",
     "iopub.status.busy": "2022-01-03T11:40:54.584864Z",
     "iopub.status.idle": "2022-01-03T11:40:54.587482Z",
     "shell.execute_reply": "2022-01-03T11:40:54.587867Z",
     "shell.execute_reply.started": "2022-01-03T11:15:28.113850Z"
    },
    "papermill": {
     "duration": 0.033237,
     "end_time": "2022-01-03T11:40:54.588020",
     "exception": false,
     "start_time": "2022-01-03T11:40:54.554783",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_3.a.hint()\n",
    "#step_3.a.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68e6f9f3",
   "metadata": {
    "papermill": {
     "duration": 0.026782,
     "end_time": "2022-01-03T11:40:54.641832",
     "exception": false,
     "start_time": "2022-01-03T11:40:54.615050",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell without changes to obtain the MAE for this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "703aa3e6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:54.699716Z",
     "iopub.status.busy": "2022-01-03T11:40:54.699124Z",
     "iopub.status.idle": "2022-01-03T11:40:55.988315Z",
     "shell.execute_reply": "2022-01-03T11:40:55.987734Z",
     "shell.execute_reply.started": "2022-01-03T11:15:34.373210Z"
    },
    "papermill": {
     "duration": 1.318681,
     "end_time": "2022-01-03T11:40:55.988471",
     "exception": false,
     "start_time": "2022-01-03T11:40:54.669790",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (Imputation):\n",
      "18062.894611872147\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE (Imputation):\")\n",
    "print(score_dataset(imputed_X_train, imputed_X_valid, y_train, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "718a770b",
   "metadata": {
    "papermill": {
     "duration": 0.027476,
     "end_time": "2022-01-03T11:40:56.043940",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.016464",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part B\n",
    "\n",
    "Compare the MAE from each approach.  Does anything surprise you about the results?  Why do you think one approach performed better than the other?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b223045",
   "metadata": {
    "papermill": {
     "duration": 0.027441,
     "end_time": "2022-01-03T11:40:56.100619",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.073178",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Imputation was bad 🤯!!! Either the missing data columns were useless or replacing data with mean was a bad imputation strategy in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a5615887",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:56.159548Z",
     "iopub.status.busy": "2022-01-03T11:40:56.158655Z",
     "iopub.status.idle": "2022-01-03T11:40:56.165120Z",
     "shell.execute_reply": "2022-01-03T11:40:56.165696Z",
     "shell.execute_reply.started": "2022-01-03T11:17:49.844335Z"
    },
    "papermill": {
     "duration": 0.037582,
     "end_time": "2022-01-03T11:40:56.165862",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.128280",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"3.2_ImputeThought\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "Given that thre are so few missing values in the dataset, we'd expect imputation to perform better than dropping columns entirely.  However, we see that dropping columns performs slightly better!  While this can probably partially be attributed to noise in the dataset, another potential explanation is that the imputation method is not a great match to this dataset.  That is, maybe instead of filling in the mean value, it makes more sense to set every missing value to a value of 0, to fill in the most frequently encountered value, or to use some other method.  For instance, consider the `GarageYrBlt` column (which indicates the year that the garage was built).  It's likely that in some cases, a missing value could indicate a house that does not have a garage.  Does it make more sense to fill in the median value along each column in this case?  Or could we get better results by filling in the minimum value along each column?  It's not quite clear what's best in this case, but perhaps we can rule out some options immediately - for instance, setting missing values in this column to 0 is likely to yield horrible results!"
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "Given that thre are so few missing values in the dataset, we'd expect imputation to perform better than dropping columns entirely.  However, we see that dropping columns performs slightly better!  While this can probably partially be attributed to noise in the dataset, another potential explanation is that the imputation method is not a great match to this dataset.  That is, maybe instead of filling in the mean value, it makes more sense to set every missing value to a value of 0, to fill in the most frequently encountered value, or to use some other method.  For instance, consider the `GarageYrBlt` column (which indicates the year that the garage was built).  It's likely that in some cases, a missing value could indicate a house that does not have a garage.  Does it make more sense to fill in the median value along each column in this case?  Or could we get better results by filling in the minimum value along each column?  It's not quite clear what's best in this case, but perhaps we can rule out some options immediately - for instance, setting missing values in this column to 0 is likely to yield horrible results!"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "step_3.b.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "325b16c5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:56.226505Z",
     "iopub.status.busy": "2022-01-03T11:40:56.225619Z",
     "iopub.status.idle": "2022-01-03T11:40:56.228876Z",
     "shell.execute_reply": "2022-01-03T11:40:56.229338Z",
     "shell.execute_reply.started": "2022-01-03T11:18:45.419576Z"
    },
    "papermill": {
     "duration": 0.034942,
     "end_time": "2022-01-03T11:40:56.229513",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.194571",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#step_3.b.hint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3bdf082",
   "metadata": {
    "papermill": {
     "duration": 0.028514,
     "end_time": "2022-01-03T11:40:56.287618",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.259104",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 4: Generate test predictions\n",
    "\n",
    "In this final step, you'll use any approach of your choosing to deal with missing values.  Once you've preprocessed the training and validation features, you'll train and evaluate a random forest model.  Then, you'll preprocess the test data before generating predictions that can be submitted to the competition!\n",
    "\n",
    "### Part A\n",
    "\n",
    "Use the next code cell to preprocess the training and validation data.  Set the preprocessed DataFrames to `final_X_train` and `final_X_valid`.  **You can use any approach of your choosing here!**  in order for this step to be marked as correct, you need only ensure:\n",
    "- the preprocessed DataFrames have the same number of columns,\n",
    "- the preprocessed DataFrames have no missing values, \n",
    "- `final_X_train` and `y_train` have the same number of rows, and\n",
    "- `final_X_valid` and `y_valid` have the same number of rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "cc3e7b50",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:56.348213Z",
     "iopub.status.busy": "2022-01-03T11:40:56.347608Z",
     "iopub.status.idle": "2022-01-03T11:40:56.360628Z",
     "shell.execute_reply": "2022-01-03T11:40:56.361175Z",
     "shell.execute_reply.started": "2022-01-03T11:19:53.327133Z"
    },
    "papermill": {
     "duration": 0.044963,
     "end_time": "2022-01-03T11:40:56.361332",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.316369",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4.1_PredsCodeA\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Preprocessed training and validation features\n",
    "final_X_train = reduced_X_train\n",
    "final_X_valid = reduced_X_valid\n",
    "\n",
    "# Check your answers\n",
    "step_4.a.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bbd1db4e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:56.423269Z",
     "iopub.status.busy": "2022-01-03T11:40:56.422369Z",
     "iopub.status.idle": "2022-01-03T11:40:56.425525Z",
     "shell.execute_reply": "2022-01-03T11:40:56.425958Z",
     "shell.execute_reply.started": "2022-01-03T11:19:54.748601Z"
    },
    "papermill": {
     "duration": 0.035476,
     "end_time": "2022-01-03T11:40:56.426115",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.390639",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_4.a.hint()\n",
    "#step_4.a.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5442554d",
   "metadata": {
    "papermill": {
     "duration": 0.028943,
     "end_time": "2022-01-03T11:40:56.484101",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.455158",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell to train and evaluate a random forest model.  (*Note that we don't use the `score_dataset()` function above, because we will soon use the trained model to generate test predictions!*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f1c8a2f3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:56.545804Z",
     "iopub.status.busy": "2022-01-03T11:40:56.544904Z",
     "iopub.status.idle": "2022-01-03T11:40:58.016442Z",
     "shell.execute_reply": "2022-01-03T11:40:58.016941Z",
     "shell.execute_reply.started": "2022-01-03T11:31:03.603933Z"
    },
    "papermill": {
     "duration": 1.5039,
     "end_time": "2022-01-03T11:40:58.017104",
     "exception": false,
     "start_time": "2022-01-03T11:40:56.513204",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (Your approach):\n",
      "6832.060924657533\n"
     ]
    }
   ],
   "source": [
    "# Define and fit model\n",
    "model = RandomForestRegressor(n_estimators=100, random_state=0)\n",
    "model.fit(X.drop(axis=1, columns=missing_columns), y)\n",
    "\n",
    "# Get validation predictions and MAE\n",
    "preds_valid = model.predict(final_X_valid)\n",
    "print(\"MAE (Your approach):\")\n",
    "print(mean_absolute_error(y_valid, preds_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc1302bf",
   "metadata": {
    "papermill": {
     "duration": 0.030002,
     "end_time": "2022-01-03T11:40:58.079007",
     "exception": false,
     "start_time": "2022-01-03T11:40:58.049005",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part B\n",
    "\n",
    "Use the next code cell to preprocess your test data.  Make sure that you use a method that agrees with how you preprocessed the training and validation data, and set the preprocessed test features to `final_X_test`.\n",
    "\n",
    "Then, use the preprocessed test features and the trained model to generate test predictions in `preds_test`.\n",
    "\n",
    "In order for this step to be marked correct, you need only ensure:\n",
    "- the preprocessed test DataFrame has no missing values, and\n",
    "- `final_X_test` has the same number of rows as `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ee1b423c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:58.147328Z",
     "iopub.status.busy": "2022-01-03T11:40:58.146392Z",
     "iopub.status.idle": "2022-01-03T11:40:58.189382Z",
     "shell.execute_reply": "2022-01-03T11:40:58.189866Z",
     "shell.execute_reply.started": "2022-01-03T11:39:51.240445Z"
    },
    "papermill": {
     "duration": 0.079836,
     "end_time": "2022-01-03T11:40:58.190026",
     "exception": false,
     "start_time": "2022-01-03T11:40:58.110190",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4.2_PredsCodeB\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the line below: preprocess test data\n",
    "final_X_test = pd.DataFrame(imputer_1.transform(X_test))\n",
    "final_X_test.columns = X_test.columns\n",
    "final_X_test = final_X_test.drop(axis=1, columns=missing_columns)\n",
    "\n",
    "# Fill in the line below: get test predictions\n",
    "\n",
    "preds_test = model.predict(final_X_test)\n",
    "\n",
    "# Check your answers\n",
    "step_4.b.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c4d0851f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:58.254021Z",
     "iopub.status.busy": "2022-01-03T11:40:58.253135Z",
     "iopub.status.idle": "2022-01-03T11:40:58.262694Z",
     "shell.execute_reply": "2022-01-03T11:40:58.263117Z",
     "shell.execute_reply.started": "2022-01-03T11:39:58.779247Z"
    },
    "papermill": {
     "duration": 0.042852,
     "end_time": "2022-01-03T11:40:58.263271",
     "exception": false,
     "start_time": "2022-01-03T11:40:58.220419",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 2, \"questionType\": 2, \"questionId\": \"4.2_PredsCodeB\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#3366cc\">Hint:</span> After preprocessing the test data, you can get the model's predictions by using `model.predict()`."
      ],
      "text/plain": [
       "Hint: After preprocessing the test data, you can get the model's predictions by using `model.predict()`."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 2, \"questionId\": \"4.2_PredsCodeB\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "```python\n",
       "# Preprocess test data\n",
       "final_X_test = pd.DataFrame(final_imputer.transform(X_test))\n",
       "\n",
       "# Get test predictions\n",
       "preds_test = model.predict(final_X_test)\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "Solution: \n",
       "```python\n",
       "# Preprocess test data\n",
       "final_X_test = pd.DataFrame(final_imputer.transform(X_test))\n",
       "\n",
       "# Get test predictions\n",
       "preds_test = model.predict(final_X_test)\n",
       "\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "step_4.b.hint()\n",
    "step_4.b.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0573d555",
   "metadata": {
    "papermill": {
     "duration": 0.031704,
     "end_time": "2022-01-03T11:40:58.326868",
     "exception": false,
     "start_time": "2022-01-03T11:40:58.295164",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell without changes to save your results to a CSV file that can be submitted directly to the competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "0fd2f20c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-03T11:40:58.394767Z",
     "iopub.status.busy": "2022-01-03T11:40:58.393888Z",
     "iopub.status.idle": "2022-01-03T11:40:58.403721Z",
     "shell.execute_reply": "2022-01-03T11:40:58.404184Z",
     "shell.execute_reply.started": "2022-01-03T11:40:01.388157Z"
    },
    "papermill": {
     "duration": 0.045255,
     "end_time": "2022-01-03T11:40:58.404344",
     "exception": false,
     "start_time": "2022-01-03T11:40:58.359089",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Save test predictions to file\n",
    "output = pd.DataFrame({'Id': X_test.index,\n",
    "                       'SalePrice': preds_test})\n",
    "output.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3122474f",
   "metadata": {
    "papermill": {
     "duration": 0.031639,
     "end_time": "2022-01-03T11:40:58.467742",
     "exception": false,
     "start_time": "2022-01-03T11:40:58.436103",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Submit your results\n",
    "\n",
    "Once you have successfully completed Step 4, you're ready to submit your results to the leaderboard!  (_You also learned how to do this in the previous exercise.  If you need a reminder of how to do this, please use the instructions below._)  \n",
    "\n",
    "First, you'll need to join the competition if you haven't already.  So open a new window by clicking on [this link](https://www.kaggle.com/c/home-data-for-ml-course).  Then click on the **Join Competition** button.\n",
    "\n",
    "![join competition image](https://i.imgur.com/wLmFtH3.png)\n",
    "\n",
    "Next, follow the instructions below:\n",
    "1. Begin by clicking on the **Save Version** button in the top right corner of the window.  This will generate a pop-up window.  \n",
    "2. Ensure that the **Save and Run All** option is selected, and then click on the **Save** button.\n",
    "3. This generates a window in the bottom left corner of the notebook.  After it has finished running, click on the number to the right of the **Save Version** button.  This pulls up a list of versions on the right of the screen.  Click on the ellipsis **(...)** to the right of the most recent version, and select **Open in Viewer**.  This brings you into view mode of the same page. You will need to scroll down to get back to these instructions.\n",
    "4. Click on the **Output** tab on the right of the screen.  Then, click on the file you would like to submit, and click on the **Submit** button to submit your results to the leaderboard.\n",
    "\n",
    "You have now successfully submitted to the competition!\n",
    "\n",
    "If you want to keep working to improve your performance, select the **Edit** button in the top right of the screen. Then you can change your code and repeat the process. There's a lot of room to improve, and you will climb up the leaderboard as you work.\n",
    "\n",
    "\n",
    "# Keep going\n",
    "\n",
    "Move on to learn what **[categorical variables](https://www.kaggle.com/alexisbcook/categorical-variables)** are, along with how to incorporate them into your machine learning models.  Categorical variables are very common in real-world data, but you'll get an error if you try to plug them into your models without processing them first!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c1a733f",
   "metadata": {
    "papermill": {
     "duration": 0.033128,
     "end_time": "2022-01-03T11:40:58.532691",
     "exception": false,
     "start_time": "2022-01-03T11:40:58.499563",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/intermediate-machine-learning/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 16.996591,
   "end_time": "2022-01-03T11:40:59.174201",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-01-03T11:40:42.177610",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}