05_model_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing machine learning models in scikit-learn\n",
    "*From the video series: [Introduction to machine learning with scikit-learn](https://github.com/justmarkham/scikit-learn-videos)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "- How do I choose **which model to use** for my supervised learning task?\n",
    "- How do I choose the **best tuning parameters** for that model?\n",
    "- How do I estimate the **likely performance of my model** on out-of-sample data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review\n",
    "\n",
    "- Classification task: Predicting the species of an unknown iris\n",
    "- Used three classification models: KNN (K=1), KNN (K=5), logistic regression\n",
    "- Need a way to choose between the models\n",
    "\n",
    "**Solution:** Model evaluation procedures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation procedure #1: Train and test on the entire dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Train the model on the **entire dataset**.\n",
    "2. Test the model on the **same dataset**, and evaluate how well we did by comparing the **predicted** response values with the **true** response values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# read in the iris data\n",
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()\n",
    "\n",
    "# create X (features) and y (response)\n",
    "X = iris.data\n",
    "y = iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1,\n",
       "       1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
       "       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2,\n",
       "       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import the class\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# instantiate the model (using the default parameters)\n",
    "logreg = LogisticRegression()\n",
    "\n",
    "# fit the model with data\n",
    "logreg.fit(X, y)\n",
    "\n",
    "# predict the response values for the observations in X\n",
    "logreg.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# store the predicted response values\n",
    "y_pred = logreg.predict(X)\n",
    "\n",
    "# check how many predictions were generated\n",
    "len(y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification accuracy:\n",
    "\n",
    "- **Proportion** of correct predictions\n",
    "- Common **evaluation metric** for classification problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.96\n"
     ]
    }
   ],
   "source": [
    "# compute classification accuracy for the logistic regression model\n",
    "from sklearn import metrics\n",
    "print(metrics.accuracy_score(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Known as **training accuracy** when you train and test the model on the same data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNN (K=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.966666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn = KNeighborsClassifier(n_neighbors=5)\n",
    "knn.fit(X, y)\n",
    "y_pred = knn.predict(X)\n",
    "print(metrics.accuracy_score(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNN (K=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=1)\n",
    "knn.fit(X, y)\n",
    "y_pred = knn.predict(X)\n",
    "print(metrics.accuracy_score(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problems with training and testing on the same data\n",
    "\n",
    "- Goal is to estimate likely performance of a model on **out-of-sample data**\n",
    "- But, maximizing training accuracy rewards **overly complex models** that won't necessarily generalize\n",
    "- Unnecessarily complex models **overfit** the training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Overfitting](images/05_overfitting.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Image Credit: [Overfitting](http://commons.wikimedia.org/wiki/File:Overfitting.svg#/media/File:Overfitting.svg) by Chabacano. Licensed under GFDL via Wikimedia Commons.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation procedure #2: Train/test split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Split the dataset into two pieces: a **training set** and a **testing set**.\n",
    "2. Train the model on the **training set**.\n",
    "3. Test the model on the **testing set**, and evaluate how well we did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150L, 4L)\n",
      "(150L,)\n"
     ]
    }
   ],
   "source": [
    "# print the shapes of X and y\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# STEP 1: split X and y into training and testing sets\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Train/test split](images/05_train_test_split.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What did this accomplish?\n",
    "\n",
    "- Model can be trained and tested on **different data**\n",
    "- Response values are known for the testing set, and thus **predictions can be evaluated**\n",
    "- **Testing accuracy** is a better estimate than training accuracy of out-of-sample performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(90L, 4L)\n",
      "(60L, 4L)\n"
     ]
    }
   ],
   "source": [
    "# print the shapes of the new X objects\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(90L,)\n",
      "(60L,)\n"
     ]
    }
   ],
   "source": [
    "# print the shapes of the new y objects\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# STEP 2: train the model on the training set\n",
    "logreg = LogisticRegression()\n",
    "logreg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.95\n"
     ]
    }
   ],
   "source": [
    "# STEP 3: make predictions on the testing set\n",
    "y_pred = logreg.predict(X_test)\n",
    "\n",
    "# compare actual response values (y_test) with predicted response values (y_pred)\n",
    "print(metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat for KNN with K=5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.966666666667\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=5)\n",
    "knn.fit(X_train, y_train)\n",
    "y_pred = knn.predict(X_test)\n",
    "print(metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat for KNN with K=1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.95\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=1)\n",
    "knn.fit(X_train, y_train)\n",
    "y_pred = knn.predict(X_test)\n",
    "print(metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we locate an even better value for K?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# try K=1 through K=25 and record testing accuracy\n",
    "k_range = list(range(1, 26))\n",
    "scores = []\n",
    "for k in k_range:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn.fit(X_train, y_train)\n",
    "    y_pred = knn.predict(X_test)\n",
    "    scores.append(metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x8ffad68>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEPCAYAAACHuClZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu4XHV97/H3JzcgAZIIuZDbThBFYrmqKdZSNwUlWhWf\nHKtgrXh50mjFW09PiZy2iR6rQE95ioIVKuWJHpSqRYGqCAhb0ZoSTQgXQ0CT7CTkAoEEEsIlyf6e\nP9aaZDKZvffsPWutmT3zeT3PfjKzrr81LOY7v9/39/stRQRmZmb1GNboApiZ2dDnYGJmZnVzMDEz\ns7o5mJiZWd0cTMzMrG4OJmZmVrfcg4mkuZIekfSopEuqrB8n6WZJKyUtlTS7bN2nJT0k6QFJN0oa\nlS5fJGmjpOXp39y8r8PMzHqXazCRNAy4GjgPeDVwoaRXVWx2KbAiIk4FLgK+lO47Bfg4cEZEnAKM\nAC4o2+/KiDgj/bs9z+swM7O+5V0zmQM8FhHdEbEHuAk4v2Kb2cDdABGxGpgpaUK6bjgwRtIIYDSw\nqWw/5VpyMzOrWd7BZCqwoez9xnRZuZXAPABJc4AZwLSI2AT8E7AeeBzYERF3le13saT7JX1N0ti8\nLsDMzPrXDAn4y4DxkpYDHwNWAPskjSOpxXQAU4AjJb033ecrwPERcRqwBbiy+GKbmVnJiJyP/zhJ\nTaNkWrpsv4jYCXyo9F7SGmANMBdYExFPp8tvBv4A+GZEPFl2iH8Fbqt2ckmeeMzMbBAiYkCphLxr\nJsuAEyR1pD2xLgBuLd9A0lhJI9PX84GfRcQukuatMyUdLknAOcCqdLvJZYeYBzzUWwEiwn8RLFq0\nqOFlaJY/fxb+LPxZ9P03GLnWTCJin6SLgTtIAtf1EbFK0oJkdVwHnAQskdQDPAx8ON33PknfJWn2\n2pP+e1166CsknQb0AOuABXleh5mZ9S3vZi4i6bZ7YsWya8teL61cX7bus8Bnqyx/f8bFNDOzOjRD\nAt4K0NnZ2egiNA1/Fgf4szjAn0V9NNj2saFAUrTy9ZmZ5UES0WQJeDMzawMOJmZmVjcHEzMzq1vu\nvblsaNq3D372M9i7t9ElsVZw2GFw1lmgAmbUW7MGjj8+//PYwRxMrKpf/hLe9S4444xGl8RawS9/\nCStWwCteke951qyB174Wnn463/PYoRxMrKo1a2DuXLjxxkaXxFrBuecm91TeweR3v4Pt22HHDhg3\nLt9z2cGcM7Gq1q2DmTMbXQprFTNnJvdU3krnKOJcdjAHE6tq3TqYNavRpbBWMWuWg0mrczCxqlwz\nsSwVWTMZP97BpBEcTKyqtWsdTCw7M2cm91Te1q6Fzs5izmUHczCxQ+zdC5s2wfTpjS6JtYoim7nO\nPts1k0ZwMLFDPP44TJyYjA0wy8LkyUkPq+efz+8cL7wATz0Fr3+9g0kjOJjYIZwvsawNGwYzZkB3\nd37nWL8+qU2//OXJPew5XovlYGKHcL7E8pB33qR035bGl2zfnt+57FAOJnYI10wsD3nnTUrd2aXi\ncjR2gIOJHcJjTCwPeXcPLv8RVFRXZDvAwcQO4ZqJ5cHBpLU5mNghnDOxPBSVMyniXHYoBxM7yJ49\nsHmzx5hY9orKmRRxLjuUg4kdZONGOO44GDmy0SWxVjNpEuzcCc89l/2xn38+GccyeXLy3s1cxXMw\nsYM4X2J5kaCjI58v+e7uZBzLsPQbrXQejzUpTu7BRNJcSY9IelTSJVXWj5N0s6SVkpZKml227tOS\nHpL0gKQbJY1Kl4+XdIek1ZJ+LGls3tfRLpwvsTzlVWOovG/HjYMRI5IR8VaMXIOJpGHA1cB5wKuB\nCyW9qmKzS4EVEXEqcBHwpXTfKcDHgTMi4hSSB3ldkO6zELgrIk4E7gY+k+d1tBN3C7Y85ZXLqHbf\nOm9SrLxrJnOAxyKiOyL2ADcB51dsM5skIBARq4GZkiak64YDYySNAEYDj6fLzweWpK+XAO/M7xLa\ni5u5LE951Uyq3bfOmxQr72AyFdhQ9n5juqzcSmAegKQ5wAxgWkRsAv4JWE8SRHZExE/SfSZGxFaA\niNgCTMztCtqMg4nlKa8uuw4mjdcMz4C/DLhK0nLgQWAFsE/SOJIaSAfwDPBdSe+NiG9WOUavabbF\nixfvf93Z2UlnZ2d2JW9BzplYnorKmZTOtXp19udqRV1dXXR1ddV1DEWO3R0knQksjoi56fuFQETE\n5X3sswY4BZgLnBcR89Plfw78fkRcLGkV0BkRWyVNBu6JiJOqHCvyvL5W89JLcNRRSdfNEc3wM8Na\nzhNPwOzZsG1btsedOBEeeOBA12CA226Dr34VfvCDbM/VDiQRERrIPnk3cy0DTpDUkfbEugC4tXwD\nSWMljUxfzwd+FhG7SJq3zpR0uCQB5wCr0t1uBT6Qvr4IuCXn62gLGzbAlCkOJJafCRNg9+5kvElW\nnnsuOd6kSQcvdzNXsXINJhGxD7gYuAN4GLgpIlZJWiDpL9LNTgIeSmsb5wGfTPe9D/guSbPXSkDA\ndek+lwNvkrSaJMhclud1tAvnSyxvUvZf8t3dybgSVfyO9liTYuX+GzQibgdOrFh2bdnrpZXry9Z9\nFvhsleVPA+dmW1JzvsSKUOqye/LJ2Ryvt/v26KPh8MPhySeTZjDLl0fA234eY2JFyLpm0td967Em\nxXEwsf3czGVFyCOY9HbfOm9SHAcT28/BxIqQ9VgTB5Pm4GBi+zlnYkXIuumpr/vWzzUpjoOJAfDi\ni0nf/6mV8xOYZcw5k9bkYGIArF8P06bB8OGNLom1umOOSQbIPvNM/cfauTMZtzJhQvX1buYqjoOJ\nAc6XWHGyHGvS3Z0cq3KMSUlHR7KNx5rkz8HEAOdLrFhZNT/1d98eeSSMGQNbt9Z/Luubg4kBHmNi\nxcqqZlLLfeu8STEcTAxwM5cVK6teVrXct86bFMPBxAAHEytWljUTB5Pm4GBigHMmVqyicibgsSZF\ncTAxnn8etm9Ppp83K4JzJq3HwcRYvx6mT4dhvhusIOPHQ09P8iNmsJ55JhmvcswxfW/nZq5i+OvD\nnC+xwmUx1qS/MSYlHR3JD6aensGfy/rnYGLOl1hD1Nv8tHZtbd3ZR4+GsWNhy5bBn8v652BiHmNi\nDVFvzWQgNWo3deXPwcTczGUNUW8vKweT5uJgYm7msoaot5lroMHE3YPz5WBirplYQ9RbW6g1ZwLu\nHlwEB5M2t3s3PPssTJ7c6JJYuykFk8HO6OtmrubiYNLmurthxgyPMbHijRuX3HdPPz3wfXfsSLr6\njh9f2/YOJvnzV0ibc77EGmmwzU+lWkl/Y0xKOjpgwwbYt2/g57La5B5MJM2V9IikRyVdUmX9OEk3\nS1opaamk2enyV0paIWl5+u8zkj6RrlskaWO6brmkuXlfR6tyvsQaabA1hoHkSwAOPxxe9jLYvHng\n57LajMjz4JKGAVcD5wCbgGWSbomIR8o2uxRYERHzJJ0IXAOcGxGPAqeXHWcjcHPZfldGxJV5lr8d\neIyJNdJgg8lgfgSVzjVt2sDPZ/3Lu2YyB3gsIrojYg9wE3B+xTazgbsBImI1MFNS5ROdzwV+FxEb\ny5bVWMG1vrhmYo002C679QQTy0fewWQqsKHs/cZ0WbmVwDwASXOAGUDlb4f3AN+qWHaxpPslfU3S\n2OyK3F6cM7FGqjdnMhAea5KvXJu5anQZcJWk5cCDwApgf5pM0kjgHcDCsn2+AnwuIkLS54ErgQ9X\nO/jixYv3v+7s7KSzszPj4g9trplYIxWVM4Fk+6VLB36udtDV1UVXV1ddx1AMtpN3LQeXzgQWR8Tc\n9P1CICLi8j72WQucHBG70vfvAP6ydIwq23cAt0XEKVXWRZ7XN9Tt2gUTJ8Jzz9XeK8YsS888kzxH\nZ9eu2u/BiGTixvXrk+7FtbrzTrjsMvjJTwZX1nYiiYgY0LdC3s1cy4ATJHVIGgVcANxavoGksWnt\nA0nzgZ+WAknqQiqauCSVD7GbBzyUR+FbXXd30mXSgcQaZexYOOww2Lat9n22b0/GpwwkkIBzJnnL\ntZkrIvZJuhi4gyRwXR8RqyQtSFbHdcBJwBJJPcDDlDVXSRpNknz/i4pDXyHpNKAHWAcsyPM6WpXz\nJdYMSnmTCZXdbnox2KbZGTNg48ZkrMnw4QPf3/qWe84kIm4HTqxYdm3Z66WV68vW7QYOucUi4v0Z\nF7MtOV9izaBUY3jd62rbfjD5EkhqQBMmwOOPJ4HFsuUR8G3MY0ysGQy0l1U9P4Lc1JUfB5M25pqJ\nNYOBfsE7mDSnfoOJpI96HEdrcs7EmsFAx5rUc996rEl+aqmZdADLJX1T0rl5F8iK45qJNYPB1EwG\n2zzr55rkp99gEhELgVcANwIfkfSYpM9Jmplz2SxHzz4LL7xQew8as7wM5LkmEcm2HR31ncuyV1PO\nJCJKXXDXkXTHPQ64RdIXcyuZ5aq7e2BTeJvl5cgjYcwYeOKJ/rd96ikYNSoZnzIYDib5qSVn8jFJ\n9wFXAb8GTomI+SQz+r4n5/JZTpwvsWZS65d8vfft9OmwaRPs3Tv4Y1h1tdRMpgAXRsS5EfGtiHgR\n9tdW3pFr6Sw3zpdYM6k1mNTbnX3UKJg0KRm8aNmqJZh8H9haeiPpKEmvBYgIT2MyRHmMiTWTWntZ\nZfEjyE1d+aglmFwH7C57/xxwbS/b2hDhmok1k1p7WTmYNK9agsmwtEkL2N+8NTK/IlkRnDOxZlJU\nzqR0Lo81yV4twWRtOnBxuKRhkj5G0qvLhjDXTKyZFJUzAY81yUstwWQByTPct6Z/bwTm51koy9eO\nHUlvlmOOaXRJzBIdHUl39Z6e3repd4xJiZu58tHvrMERsRV4VwFlsYJ4jIk1mzFj4OijYetWOO64\n6ts8+SSMHg1HHVXfuRxM8tFvMJF0GPAB4NXA4aXlEVH5jBEbIpwvsWZU+pLvLZhkdd9OmwZbtsCe\nPTDS2d/M1NLM9XVgJvA24L+BlwMv5Fgmy5nzJdaM+kuMZ9WdfeTIJGBt2FD/seyAWoLJKyPiM8Cu\niLgemAvMybdYliePMbFm1F/zU5Y/gtzUlb1agsme9N8dkk4CjgIm5lcky5trJtaM+utl5WDS3GoJ\nJtdLGg8sAn4MPAr831xLZblyzsSaUX9f8Fnetx5rkr0+E/CShgPbImI7cA/gJycPcaXulQ4m1myK\nyplAcpy77srmWJbos2YSEfuASwsqixVgx47k3/HjG1sOs0odHUlSvNpYk4ikS3u9Y0xK3MyVvVqa\nue6Q9ClJx0k6uvSXe8ksF6VaiceYWLM54ojkR87mzYeu27o1GV8yZkw253IwyV6/40yA96X//s+y\nZYGbvIYk50usmZW+5KdOPXh51vft1KnJw7hefBEOOyy747azWh7bO73KX82BRNJcSY9IelTSJVXW\nj5N0s6SVkpZKmp0uf6WkFZKWp/8+I+kT6brxku6QtFrSjyUN8rlr7cfdgq2Z9ZY3yfq+HTEiCSge\na5KdWkbAv7fa8oj4Zg37DgOuJpnbaxOwTNItEfFI2WaXAisiYp6kE4FrgHMj4lGSpzmWjrMRuDnd\nZyFwV0RckQaoz6TLrB/r1sHxxze6FGbV9dY9OI9OI6Va0AknZHvcdlVLzuSssr83AV+k9rm65gCP\nRUR3ROwBbgLOr9hmNnA3QESsBmZKmlCxzbnA7yKi9Hy084El6eslwDtrLE/bc08ua2a95TLyDCaW\njVqauT5a9vdB4DTgiBqPPxUor0huTJeVWwnMA5A0hyQXM61im/cA3yp7PzGdgJKI2IIHUdbMORNr\nZr19wedx33qsSbZqScBX2glk2VByGXCVpOXAg8AKYF9ppaSRJM+a76sZK3pbsXjx4v2vOzs76ezs\nrK+0Q5jHmFizKypnAsnxbr8922MOVV1dXXR1ddV1DEX0+j2cbCB9jwNf1sNIZg++JSL+ut+DS2cC\niyNibvp+IRARcXkf+6wFTo6IXen7dwB/WTpGumwV0BkRWyVNBu6JiJOqHCv6u7528tRTSfvw9u2N\nLolZdS++mExFv3s3DB+eLOvpSaae37496T6clXvvhYUL4Re/yO6YrUISETGgAQS11EyuLnu9F+iO\niHU1Hn8ZcIKkDmAzcAFwYfkGaU+s3RGxR9J84KelQJK6kIObuABuJZkW/3LgIuCWGsvT1lwrsWZ3\n2GFw7LGwaRNMn54s27IFxo3LNpCAcyZZqyWYPAY8EREvAEg6QtL0iOi3U11E7JN0MXAHSa3m+ohY\nJWlBsjquA04ClkjqAR4GPlzaX9JokuR75bNTLge+LelDQDfw7hquo+05X2JDQelLvhRM8rpvp0yB\nbdvghRfg8MP73976VkswuRn4g7L3PcB/UOM09BFxO3BixbJry14vrVxftm43UNmzi4h4miTI2AB4\njIkNBaW8yVlnJe/zum+HD08C1vr18MpXZn/8dlNL1+AREfFS6U1EvAh4zOgQ5GYuGwoqx5rked+6\nqSs7tQSTpyS9tfRG0tuAp/MrkuXFwcSGgsoveAeToaGWYPJR4HOS1qY9rf4eWJBvsSwPzpnYUFDZ\nPTjP+9ZjTbLTb84kndbktZLGpe935F4qy5zHmNhQUa1mkleub9Ys+M//zOfY7abfmomk/yNpXETs\niIgd6SSLny2icJadbduSHitH++EB1uRmzEi6Bu/dC/v2JZMxzshpjnI3c2Wnlmaut5XXRtKnLr49\nvyJZHlwrsaFi1CiYOBEefzx5tskxx+TXddfBJDu1dA0eLmlUqUeXpMOBUfkWy7LmfIkNJaVcxvDh\n+d63xx2XjKx//vnsB0W2m1qCyU3AnZL+LX3/IaDf6eetuXiMiQ0lpRrD8OH53rfDhiVNaN3d8KpX\n5XeedlBLAv4Lkh7gwCDBKyLiB/kWy7K2bh3Mnt3oUpjVpjTWJO+aCRwIXA4m9aklZ0JE/GdEfCoi\nPkUy7uSqnMtlGXPOxIaS0hd8Efet8ybZqCmYSDpZ0hck/Q74R8A9s4cY50xsKCnlTIq4bz3WJBu9\nNnNJOp5kxt4LgV3AvwMjI+KsgspmGYlI2oQdTGyoKG/myjvXN2sWfP/7+Z6jHfSVM/ktcC8wLx24\niKSPF1Iqy9QTT8CYMXDkkY0uiVltpk1LugVLB2YPzoububLRVzPXu4EngbskfUXSG4EBPSzFmoPz\nJTbUjByZdNudMCF5xkmeHEyy0WswiYjvRsS7SJ6suJTksbmTJH1Z0h8XVUCrn/MlNhTNnFnMfTtp\nEjz7LDz3XP7namX9JuAjYmdEfD0i3gLMAFYBi3IvmWXGY0xsKJo1q5j7dtgw6OhI8oo2eLUMWtwv\nIrYBX0n/rA433wzvLuj5kD09cMMNxZzLLCunnprMzVWE3/s9OPnkJEfTrK6+Gj7ykUaXoneKiEaX\nITeSolmvb9Gi5Et+UUF1vBED+tlg1l4iigtcg/Ev/wK/+U3ybxEkEREDCq3+immQtWvh7LP9JW/W\nDKTm/n/x+OPhB00+70hNgxYte85jmFmtKh9l3Iz6jcWStgOVbUXPAL8C/ldErMuhXC3P3XXNrFal\nDgIRzZvXqaVidw2wmQMzBV8IzARWAjcAZ+dSshb20kuwdWsyMMvMrD9jxsBRRyXfG5MnN7o01dXS\nzPX2iLgmIranf18B3hwRNwIvy7l8LWnDhmRAVjO30ZpZc2n2OcRqCSbPS5pXepO+fjF929PfzpLm\nSnpE0qOSLqmyfpykmyWtlLRU0uyydWMlfUfSKkkPS/r9dPkiSRslLU//5tZwHU3D+RIzG6hmz5vU\nEkzeB8yX9LSkp4D5wJ9LGg18qq8dJQ0DrgbOIxlJf6GkyqcGXAqsiIhTgYuAL5Wtuwr4YUScBJxK\nMmCy5MqIOCP9u72G62gazpeY2UA1+7QvtTwc67fAW3pZ/dN+dp8DPBYR3QCSbgLOBx4p22Y28MX0\nXKslzZQ0gaT2c1ZEfCBdtxd4tmy/Jk1D9c/BxMwGauZMWLGi0aXoXb81E0nHSvqbdLLH60p/NR5/\nKrCh7P3GdFm5lcC89FxzSKZsmQbMArZJuiFtyrpOUvlTmi+WdL+kr0kaW2N5moLnyjKzgRryNRPg\nFpKJHn8O5DFG9DLgKknLgQeBFel5RgJnAB+LiF9J+meSySYXkUzn8rmICEmfB64EPlzt4IsXL97/\nurOzk87OzhwuYWCcMzGzgcozZ9LV1UVXV1ddx+h3OhVJ90fEaYM6uHQmsDgi5qbvFwIREZf3sc9a\n4GRgDPDLiDg+Xf6HwCUR8faK7TuA2yLilCrHasrpVKZNg//6L5gxo9ElMbOh4vnnYfx42L07mZwy\nT4OZTqWWIv1I0psHWaZlwAmSOiSNAi4Abi3fIO2xNTJ9PR/4aUTsioitwAZJr0w3PQf4TbpdeU/r\necBDgyxf4V58EZ58EqZMaXRJzGwoOeIIGDcueWhYM6qlmesjwCWSdgMvkSS+IyL6HWMSEfskXQzc\nQRK4ro+IVZIWpMe4DjgJWCKpB3iYg5urPgHcmAabNcAH0+VXSDqNpGvyOmBBDdfRFNavh6lTPcbE\nzAaulDeZWpl5bgK1fKUdW88J0m67J1Ysu7bs9dLK9WXrVgKvq7L8/fWUqZGcLzGzwSrlTd7whkaX\n5FC9BhNJr4iIx0jGh1TzQD5Fam3uFmxmg9XMPbr6qpksJGlyuqbKugD+KJcStTgHEzMbrJkzYdmy\nRpeiul6DSUSUchd/HBF7yteVEuY2cGvXwlt6GwJqZtaHmTPhO99pdCmqq6U313/XuMxq4JyJmQ1W\nM8/P1VfOZCJwHHCEpJM5MH3J0cDoAsrWktzMZWaDNWNGMuv4vn0wfHijS3OwvnImfwJ8iGRqk2s4\nEEx2An+Xc7la0gsvwFNPJdPPm5kN1OGHwzHHwKZNMH16o0tzsL5yJjcAN0h6d0R8u8Aytazu7uQG\naLZfFGY2dJSaupotmNSSM5ko6WgASV+VdJ+kc3IuV0tyvsTM6tWs3YNrCSZ/ERHPplOqHEfyPJMr\n8i1Wa3K+xMzqNZSDSWmmxLcCX09Hpec8zVhrcjAxs3o16+N7awkKKyX9EHgbyaSPR3IgwNgA+Dkm\nZlavZu0eXMvcXB8EXgP8NiJ2SzqWXp4dYn1zzsTM6jVkm7kiYh9wPPDRdNERtexnh3Izl5nVa/p0\nePxx2Lu30SU5WC2P7b0aOBt4X7roOeCreRaqFe3eDTt2wOTJ/W9rZtabww6DCROSgNJMaqlh/EFE\nLABeAIiIp4FRuZaqBXV3J6NX835Cmpm1vmbMm9Ty1bZH0jDSpLukY0geSmUD4HyJmWWlGfMmvQYT\nSaXk/DXAfwATJH0W+DnQ6zPcrTrnS8wsK83YPbiv3lz3AWdExNcl/Ro4l2R+rj+NiCHzzPVm4W7B\nZpaVmTPh3nsbXYqD9RVMShM7EhEPkzyf3QZp3To4/fRGl8LMWsGsWfCNbzS6FAfrK5hMkPRXva2M\niCtzKE/Lcs7EzLLSjDmTvoLJcOBIymooNnjOmZhZVqZNS6ah37MHRjbJc2/7CiabI+JzhZWkhe3a\nBTt3wqRJjS6JmbWCUaOSMWsbNzZPi0dfXYNdI8lIdzd0dID8iZpZRpptrElfwSSTZ5ZImivpEUmP\nSrqkyvpxkm6WtFLSUkmzy9aNlfQdSaskPSzp99Pl4yXdIWm1pB9LGptFWfPifImZZa3Z8ia9BpN0\npHtd0sGOVwPnAa8GLpT0qorNLgVWRMSpwEXAl8rWXQX8MCJOAk4FVqXLFwJ3RcSJwN3AZ+ota56c\nLzGzrDXbWJO8J/eYAzwWEd0RsQe4CTi/YpvZJAGBiFgNzJQ0IX2641np44OJiL0R8Wy6z/nAkvT1\nEuCdOV9HXTzGxMyyNpSaubIwFdhQ9n5juqzcSmAegKQ5wAxgGjAL2CbpBknLJV0n6Yh0n4kRsRUg\nIrYAE3O8hrq5ZmJmWWu2Zq5anmeSt8uAqyQtBx4EVgD7gJHAGcDHIuJXkv6ZpHlrEYd2Duj1YV2L\nFy/e/7qzs5POzs4sy14T50zMLGtZBpOuri66urrqOoYi8ntooqQzgcURMTd9vxCIiOh1bi9Ja4GT\ngTHALyPi+HT5HwKXRMTbJa0COiNiq6TJwD1pXqXyWJHn9dXq2GPhN7+BiU1dfzKzoWTvXhgzJhl2\nMCrjedwlERED6n+adzPXMuAESR2SRgEXALeWb5D22BqZvp4P/DQidqXNWBskvTLd9BzgN+nrW4EP\npK8vAm7J9zIGb+fO5FkmEyY0uiRm1kpGjIApU2DDhv63LUKuzVwRsU/SxcAdJIHr+ohYJWlBsjqu\nA04ClkjqIZn/q/yRwJ8AbkyDzRqSRwhDMmvxtyV9COgG3p3nddSjlC/xGBMzy1qpqevlL290SQrI\nmUTE7cCJFcuuLXu9tHJ92bqVwOuqLH+aZBbjpud8iZnlpZm6B/u5fzlzTy4zy0sz9ehyMMmZx5iY\nWV6aaayJg0nOXDMxs7y4ZtJGnDMxs7w4Z9JGXDMxs7xMnQrbtsGLLza6JA4muXrmGXjpJTjmmEaX\nxMxa0fDhyYOy1q9vdEkcTHLlMSZmlrdmyZs4mOTI+RIzy1uz5E0cTHLkfImZ5c01kzbgMSZmlrdm\nGWviYJIj10zMLG+umbQB50zMLG/NkjPJ9Xkmjdbo55mMGwdr1sDLXtawIphZi+vpgdGjYft2OOKI\n/revRTM+z6Rtbd+e/EceP77RJTGzVjZsGEyf3vixJg4mOfEYEzMrSjPkTRxMcuJ8iZkVpRnyJg4m\nOXFPLjMrSjN0D3YwyYnHmJhZUdzM1cJcMzGzoriZq4U5Z2JmRWmGmonHmeQgAsaOTbrqjRtX+OnN\nrM309MCYMfDUU8mYk3p5nEmTePrppO+3A4mZFWHYMJgxA7q7G1iGxp26dbmJy8yK1ui8Se7BRNJc\nSY9IelTSJVXWj5N0s6SVkpZKml22bl26fIWk+8qWL5K0UdLy9G9u3tcxEE6+m1nRGp03GZHnwSUN\nA64GzgE2Acsk3RIRj5RtdimwIiLmSToRuAY4N13XA3RGxPYqh78yIq7MsfiD5mBiZkVr9FiTvGsm\nc4DHIqKIBZ5rAAAK3ElEQVQ7IvYANwHnV2wzG7gbICJWAzMlTUjXqY8yNu1EJR5jYmZFa3TNJO9g\nMhXYUPZ+Y7qs3EpgHoCkOcAMYFq6LoA7JS2TNL9iv4sl3S/pa5LGZl/0wXPOxMyK1uicSa7NXDW6\nDLhK0nLgQWAFsC9d94aI2JzWVO6UtCoifg58BfhcRISkzwNXAh+udvDFixfvf93Z2UlnZ2duF1Li\nZi4zK1o9NZOuri66urrqOn+u40wknQksjoi56fuFQETE5X3ssxY4OSJ2VSxfBOyszJNI6gBui4hT\nqhyr8HEmEXDUUfD448lYEzOzIkQkY02eeAKOPLK+YzXjOJNlwAmSOiSNAi4Abi3fQNJYSSPT1/OB\nn0bELkmjJR2ZLh8DvBl4KH0/uewQ80rLm8G2bTBqlAOJmRVLgo6Oxo01ybWZKyL2SboYuIMkcF0f\nEaskLUhWx3XAScASST3AwxxorpoEfE9SpOW8MSLuSNddIek0kt5e64AFeV7HQDhfYmaNUsqbvPrV\nxZ8795xJRNwOnFix7Nqy10sr16fL1wKn9XLM92dczMw4X2JmjdLI7sEeAZ8xBxMza5RGdg92MMmY\nx5iYWaM4mLQQ50zMrFEaOdbEwSRjbuYys0ZxzqRFRCT/ITs6Gl0SM2tHxx4LL7wAzz5b/LkdTDL0\nxBPJg2mOOqrRJTGzdiQ1Lm/iYJIh50vMrNEcTFqA8yVm1miNyps4mGTIwcTMGs01kxbgMSZm1miN\n6h7sYJIh50zMrNFcM2kBbuYys0ZzzmSIi0imfvYYEzNrpJe9DPbuhR07ij2vg0lGtmxJxpeMGdPo\nkphZO2vUWBMHk4w4X2JmzcLBZAhzvsTMmkUj8iYOJhlxt2AzaxaumQxhrpmYWbNoxFgTB5OMOGdi\nZs3CzVxDmGsmZtYsSs1cEcWd08EkAz09sH69x5iYWXMYNy75t8ixJg4mGdi8OfmPd8QRjS6JmdmB\nsSZF5k1yDyaS5kp6RNKjki6psn6cpJslrZS0VNLssnXr0uUrJN1Xtny8pDskrZb0Y0lj876Ovjhf\nYmbNpui8Sa7BRNIw4GrgPODVwIWSXlWx2aXAiog4FbgI+FLZuh6gMyJOj4g5ZcsXAndFxInA3cBn\n8rqGWgyFfElXV1eji9A0/Fkc4M/igFb7LIruHpx3zWQO8FhEdEfEHuAm4PyKbWaTBAQiYjUwU9KE\ndJ16KeP5wJL09RLgnVkXfCCGwhiTVvsfpR7+LA7wZ3FAq30WrdbMNRXYUPZ+Y7qs3EpgHoCkOcAM\nYFq6LoA7JS2TNL9sn4kRsRUgIrYAE3Moe82GQs3EzNpL0TWTEcWdqleXAVdJWg48CKwA9qXr3hAR\nm9Oayp2SVkXEz6scI5MOcBs3wkc/OvD97rsP3v3uLEpgZpaN44+He++Ft7+9mPMpcuyILOlMYHFE\nzE3fLwQiIi7vY5+1wMkRsati+SJgZ0RcKWkVSS5lq6TJwD0RcVKVYxXYy9rMrHVEhAayfd41k2XA\nCZI6gM3ABcCF5RukPbF2R8SetCnrpxGxS9JoYFj6egzwZuCz6W63Ah8ALidJ2t9S7eQD/TDMzGxw\ncg0mEbFP0sXAHST5mesjYpWkBcnquA44CVgiqQd4GPhwuvsk4Htp7WIEcGNE3JGuuxz4tqQPAd2A\nG5nMzBoo12YuMzNrDy05Ar6/gZLtprfBn+1A0vWStkp6oGxZUw16LUovn8UiSRslLU//5jayjEWQ\nNE3S3ZIelvSgpE+ky9vuvqjyWXw8XT7g+6LlaibpQMlHgXOATSR5mwsi4pGGFqyBJK0BXhMR2xtd\nlqJJ+kNgF/D1iDglXXY58FREXJH+2BgfEQsbWc4i9PJZ7O/Y0tDCFSjttDM5Iu6XdCTwa5Kxax+k\nze6LPj6L9zDA+6IVaya1DJRsN70N/mx5aVfyyiDaVINei9LLZwHJ/dE2ImJLRNyfvt4FrCIZ29Z2\n90Uvn0VpLOCA7otW/IKpZaBku+lt8Ge7aqpBr03gYkn3S/paOzTtlJM0EzgNWApMauf7ouyz+O90\n0YDui1YMJnaoN0TEGcBbgY+lzR12QGu19Q7MV4DjI+I0YAvQTs1dRwLfBT6Z/iqvvA/a5r6o8lkM\n+L5oxWDyOMmULCXT0mVtKyI2p/8+CXyPpCmwnW2VNAn2txk/0eDyNExEPBkHEqf/CryukeUpiqQR\nJF+e34iI0ji1trwvqn0Wg7kvWjGY7B8oKWkUyUDJWxtcpoaRNDr91UHZ4M+HGluqwomD239Lg16h\nj0GvLeqgzyL90iyZR/vcG/8G/CYiripb1q73xSGfxWDui5brzQVJ12DgKg4MlLyswUVqGEmzSGoj\n5YM/2+bzkPRNoBM4BtgKLAK+D3wHmE466DUiCnwmXWP08lmcTdJO3gOsAxaU8gatStIbgJ+RzAUY\n6d+lwH3At2mj+6KPz+K9DPC+aMlgYmZmxWrFZi4zMyuYg4mZmdXNwcTMzOrmYGJmZnVzMDEzs7o5\nmJiZWd0cTGxIS6fPflPFsk9Kuqaf/XbmXK5jJS2V9Ou0L3/5unsknZG+npU+KuFNVY7xj+m04L0+\n5rqfMrxR0m1l7z8v6YeSRkrqkrSsbN1rJN1Ttl+PpD8pW3+bpD8aTDmsPTiY2FD3TSoeBU0y68E3\n+9kv7wFW5wIPRMRrIuIX1TaQNA34EfDpiLizyibzgVMioqZn8kgaXmVxpOv+Fng98M50Nu0AJkg6\nr3Lb1Ebgf9dyXjNwMLGh7z+At6bzCyGpAzguIn4haYykuyT9Kn042Dsqd67y6/3Lkt6fvj6j9Ate\n0o9K8zZV7N8h6Sfp8e9MHzZ0Ksmjpc9PHyx0WJVyTwF+DHwmIn5Q5bi3AEcCv5b0p2Xnub90nnS7\nGyT9i6Sl6TmrHEp/BZwHvD0iXipb94/A31b9VGEl8Iykc3pZb3YQBxMb0tIHft0HvCVddAHJlBgA\nL5D8En8t8MfAP/V2mMoFaXD6MvA/IuJ1wA3AF6rs+2Xghog4laQ29OWIWAn8PfDvEXFGRLxYZb8l\n6bbf6+W6zgd2p/t/p+w8p5XOU7b51Ig4MyL+usqh3gAsAN4SEbsrrvmXwIuS3litCMA/AH9XrXxm\nlRxMrBXcRBJESP/9VvpawBclrQTuAqZIqvUZFScCv0fyHJgVJE0+U6ps9/qy832D5Mu7FncC75N0\neB/blE9O2dd5vtPHMX6bHufNvRy714CRPkwrKnM+ZtU4mFgruAU4R9LpwBERsSJd/mfAscDpEXE6\nyZTilV/eezn4/4PSegEPpTWD0yPi1Ih4C4cabO7lCpIZrr+bPmq6mujldaXn+li3heQ5Nv8sqfOQ\nE0TcQ3LNZ/ay/xdImsI8iZ/1ycHEhryIeA7oIplK+1tlq8YCT0REj6SzgY6ydaVf5t3A7LSH0zig\nlCNYTZKgPhOSZi9Js6uc/r840AHgfcC9Ayj3p4Fn0nJXU14zqec8vyWZRvz/STqlyib/APxNL/ve\nCYwHqu1ntp+DibWKb5F84ZUHkxuB16XNXO8jeb51SQBExEaSHMtDJM1ly9Ple4B3AZdLuh9YQdLU\nVOkTwAfTbf4M+GQNZS3/lf8BYHIv3X/Lt+vtPDXVGCLiV8AHgVvTxxJE2bofkdTaejvWP5BMy27W\nK09Bb2ZmdXPNxMzM6uZgYmZmdXMwMTOzujmYmJlZ3RxMzMysbg4mZmZWNwcTMzOrm4OJmZnV7f8D\nw6JECwTkHloAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x8872e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# import Matplotlib (scientific plotting library)\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# allow plots to appear within the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "# plot the relationship between K and testing accuracy\n",
    "plt.plot(k_range, scores)\n",
    "plt.xlabel('Value of K for KNN')\n",
    "plt.ylabel('Testing Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Training accuracy** rises as model complexity increases\n",
    "- **Testing accuracy** penalizes models that are too complex or not complex enough\n",
    "- For KNN models, complexity is determined by the **value of K** (lower value = more complex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions on out-of-sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiate the model with the best known parameters\n",
    "knn = KNeighborsClassifier(n_neighbors=11)\n",
    "\n",
    "# train the model with X and y (not X_train and y_train)\n",
    "knn.fit(X, y)\n",
    "\n",
    "# make a prediction for an out-of-sample observation\n",
    "knn.predict([[3, 5, 4, 2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downsides of train/test split?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Provides a **high-variance estimate** of out-of-sample accuracy\n",
    "- **K-fold cross-validation** overcomes this limitation\n",
    "- But, train/test split is still useful because of its **flexibility and speed**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "- Quora: [What is an intuitive explanation of overfitting?](http://www.quora.com/What-is-an-intuitive-explanation-of-overfitting/answer/Jessica-Su)\n",
    "- Video: [Estimating prediction error](https://www.youtube.com/watch?v=_2ij6eaaSl0&t=2m34s) (12 minutes, starting at 2:34) by Hastie and Tibshirani\n",
    "- [Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html)\n",
    "    - [Guiding questions](https://github.com/justmarkham/DAT8/blob/master/homework/09_bias_variance.md) when reading this article\n",
    "- Video: [Visualizing bias and variance](http://work.caltech.edu/library/081.html) (15 minutes) by Abu-Mostafa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comments or Questions?\n",
    "\n",
    "- Email: <kevin@dataschool.io>\n",
    "- Website: http://dataschool.io\n",
    "- Twitter: [@justmarkham](https://twitter.com/justmarkham)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    @font-face {\n",
       "        font-family: \"Computer Modern\";\n",
       "        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "    }\n",
       "    div.cell{\n",
       "        width: 90%;\n",
       "/*        margin-left:auto;*/\n",
       "/*        margin-right:auto;*/\n",
       "    }\n",
       "    ul {\n",
       "        line-height: 145%;\n",
       "        font-size: 90%;\n",
       "    }\n",
       "    li {\n",
       "        margin-bottom: 1em;\n",
       "    }\n",
       "    h1 {\n",
       "        font-family: Helvetica, serif;\n",
       "    }\n",
       "    h4{\n",
       "        margin-top: 12px;\n",
       "        margin-bottom: 3px;\n",
       "       }\n",
       "    div.text_cell_render{\n",
       "        font-family: Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
       "        line-height: 145%;\n",
       "        font-size: 130%;\n",
       "        width: 90%;\n",
       "        margin-left:auto;\n",
       "        margin-right:auto;\n",
       "    }\n",
       "    .CodeMirror{\n",
       "            font-family: \"Source Code Pro\", source-code-pro,Consolas, monospace;\n",
       "    }\n",
       "/*    .prompt{\n",
       "        display: None;\n",
       "    }*/\n",
       "    .text_cell_render h5 {\n",
       "        font-weight: 300;\n",
       "        font-size: 16pt;\n",
       "        color: #4057A1;\n",
       "        font-style: italic;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 0.5em;\n",
       "        display: block;\n",
       "    }\n",
       "\n",
       "    .warning{\n",
       "        color: rgb( 240, 20, 20 )\n",
       "        }\n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"]\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}