6. Decision Tree.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 6: Supervised Learning - Decision Tree\n",
    "Instructor: Md Shahidullah Kawsar\n",
    "<br>Data Scientist, IDARE, Houston, TX, USA\n",
    "\n",
    "#### Objectives:\n",
    "- Decision Tree Classification and Regression\n",
    "\n",
    "#### References:\n",
    "[1] Decision and Classification Trees, Clearly Explained!!! https://www.youtube.com/watch?v=_L39rN6gz7Y&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[2] StatQuest: Decision Trees: https://www.youtube.com/watch?v=7VeUPuFGJHk&t=302s&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[3] StatQuest: Decision Trees, Part 2 - Feature Selection and Missing Data: https://www.youtube.com/watch?v=wpNl-JwwplA&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[4] scikit-learn decision tree classifier: https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html\n",
    "<br>[5] Decision Trees in Python from Start to Finish: https://www.youtube.com/watch?v=q90UDEgYqeI&ab_channel=StatQuestwithJoshStarmer\n",
    "<br> [6] Plotting a tree diagram https://scikit-learn.org/stable/modules/generated/sklearn.tree.plot_tree.html\n",
    "<br>[7] Plotting confusion matrix: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_confusion_matrix.html\n",
    "<br>[8] Regression Trees, Clearly Explained!!! https://www.youtube.com/watch?v=g9c66TUylZ4&ab_channel=StatQuestwithJoshStarmer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "pd.options.display.max_columns = 100\n",
    "pd.options.display.max_rows = 100\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "from sklearn.metrics import accuracy_score, classification_report, confusion_matrix\n",
    "from sklearn.metrics import roc_curve, roc_auc_score, precision_score, recall_score, f1_score\n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load data\n",
    "Dataset Source: https://learn.datacamp.com/courses/marketing-analytics-predicting-customer-churn-in-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Account_Length</th>\n",
       "      <th>Vmail_Message</th>\n",
       "      <th>Day_Mins</th>\n",
       "      <th>Eve_Mins</th>\n",
       "      <th>Night_Mins</th>\n",
       "      <th>Intl_Mins</th>\n",
       "      <th>CustServ_Calls</th>\n",
       "      <th>Churn</th>\n",
       "      <th>Intl_Plan</th>\n",
       "      <th>Vmail_Plan</th>\n",
       "      <th>Day_Calls</th>\n",
       "      <th>Day_Charge</th>\n",
       "      <th>Eve_Calls</th>\n",
       "      <th>Eve_Charge</th>\n",
       "      <th>Night_Calls</th>\n",
       "      <th>Night_Charge</th>\n",
       "      <th>Intl_Calls</th>\n",
       "      <th>Intl_Charge</th>\n",
       "      <th>State</th>\n",
       "      <th>Area_Code</th>\n",
       "      <th>Phone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>128</td>\n",
       "      <td>25</td>\n",
       "      <td>265.1</td>\n",
       "      <td>197.4</td>\n",
       "      <td>244.7</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>110</td>\n",
       "      <td>45.07</td>\n",
       "      <td>99</td>\n",
       "      <td>16.78</td>\n",
       "      <td>91</td>\n",
       "      <td>11.01</td>\n",
       "      <td>3</td>\n",
       "      <td>2.70</td>\n",
       "      <td>KS</td>\n",
       "      <td>415</td>\n",
       "      <td>382-4657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>107</td>\n",
       "      <td>26</td>\n",
       "      <td>161.6</td>\n",
       "      <td>195.5</td>\n",
       "      <td>254.4</td>\n",
       "      <td>13.7</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>123</td>\n",
       "      <td>27.47</td>\n",
       "      <td>103</td>\n",
       "      <td>16.62</td>\n",
       "      <td>103</td>\n",
       "      <td>11.45</td>\n",
       "      <td>3</td>\n",
       "      <td>3.70</td>\n",
       "      <td>OH</td>\n",
       "      <td>415</td>\n",
       "      <td>371-7191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>137</td>\n",
       "      <td>0</td>\n",
       "      <td>243.4</td>\n",
       "      <td>121.2</td>\n",
       "      <td>162.6</td>\n",
       "      <td>12.2</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>114</td>\n",
       "      <td>41.38</td>\n",
       "      <td>110</td>\n",
       "      <td>10.30</td>\n",
       "      <td>104</td>\n",
       "      <td>7.32</td>\n",
       "      <td>5</td>\n",
       "      <td>3.29</td>\n",
       "      <td>NJ</td>\n",
       "      <td>415</td>\n",
       "      <td>358-1921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>299.4</td>\n",
       "      <td>61.9</td>\n",
       "      <td>196.9</td>\n",
       "      <td>6.6</td>\n",
       "      <td>2</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>71</td>\n",
       "      <td>50.90</td>\n",
       "      <td>88</td>\n",
       "      <td>5.26</td>\n",
       "      <td>89</td>\n",
       "      <td>8.86</td>\n",
       "      <td>7</td>\n",
       "      <td>1.78</td>\n",
       "      <td>OH</td>\n",
       "      <td>408</td>\n",
       "      <td>375-9999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>166.7</td>\n",
       "      <td>148.3</td>\n",
       "      <td>186.9</td>\n",
       "      <td>10.1</td>\n",
       "      <td>3</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>113</td>\n",
       "      <td>28.34</td>\n",
       "      <td>122</td>\n",
       "      <td>12.61</td>\n",
       "      <td>121</td>\n",
       "      <td>8.41</td>\n",
       "      <td>3</td>\n",
       "      <td>2.73</td>\n",
       "      <td>OK</td>\n",
       "      <td>415</td>\n",
       "      <td>330-6626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>118</td>\n",
       "      <td>0</td>\n",
       "      <td>223.4</td>\n",
       "      <td>220.6</td>\n",
       "      <td>203.9</td>\n",
       "      <td>6.3</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>98</td>\n",
       "      <td>37.98</td>\n",
       "      <td>101</td>\n",
       "      <td>18.75</td>\n",
       "      <td>118</td>\n",
       "      <td>9.18</td>\n",
       "      <td>6</td>\n",
       "      <td>1.70</td>\n",
       "      <td>AL</td>\n",
       "      <td>510</td>\n",
       "      <td>391-8027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>121</td>\n",
       "      <td>24</td>\n",
       "      <td>218.2</td>\n",
       "      <td>348.5</td>\n",
       "      <td>212.6</td>\n",
       "      <td>7.5</td>\n",
       "      <td>3</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>88</td>\n",
       "      <td>37.09</td>\n",
       "      <td>108</td>\n",
       "      <td>29.62</td>\n",
       "      <td>118</td>\n",
       "      <td>9.57</td>\n",
       "      <td>7</td>\n",
       "      <td>2.03</td>\n",
       "      <td>MA</td>\n",
       "      <td>510</td>\n",
       "      <td>355-9993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>147</td>\n",
       "      <td>0</td>\n",
       "      <td>157.0</td>\n",
       "      <td>103.1</td>\n",
       "      <td>211.8</td>\n",
       "      <td>7.1</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>79</td>\n",
       "      <td>26.69</td>\n",
       "      <td>94</td>\n",
       "      <td>8.76</td>\n",
       "      <td>96</td>\n",
       "      <td>9.53</td>\n",
       "      <td>6</td>\n",
       "      <td>1.92</td>\n",
       "      <td>MO</td>\n",
       "      <td>415</td>\n",
       "      <td>329-9001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>117</td>\n",
       "      <td>0</td>\n",
       "      <td>184.5</td>\n",
       "      <td>351.6</td>\n",
       "      <td>215.8</td>\n",
       "      <td>8.7</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>97</td>\n",
       "      <td>31.37</td>\n",
       "      <td>80</td>\n",
       "      <td>29.89</td>\n",
       "      <td>90</td>\n",
       "      <td>9.71</td>\n",
       "      <td>4</td>\n",
       "      <td>2.35</td>\n",
       "      <td>LA</td>\n",
       "      <td>408</td>\n",
       "      <td>335-4719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>141</td>\n",
       "      <td>37</td>\n",
       "      <td>258.6</td>\n",
       "      <td>222.0</td>\n",
       "      <td>326.4</td>\n",
       "      <td>11.2</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>84</td>\n",
       "      <td>43.96</td>\n",
       "      <td>111</td>\n",
       "      <td>18.87</td>\n",
       "      <td>97</td>\n",
       "      <td>14.69</td>\n",
       "      <td>5</td>\n",
       "      <td>3.02</td>\n",
       "      <td>WV</td>\n",
       "      <td>415</td>\n",
       "      <td>330-8173</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  Intl_Mins  \\\n",
       "0             128             25     265.1     197.4       244.7       10.0   \n",
       "1             107             26     161.6     195.5       254.4       13.7   \n",
       "2             137              0     243.4     121.2       162.6       12.2   \n",
       "3              84              0     299.4      61.9       196.9        6.6   \n",
       "4              75              0     166.7     148.3       186.9       10.1   \n",
       "5             118              0     223.4     220.6       203.9        6.3   \n",
       "6             121             24     218.2     348.5       212.6        7.5   \n",
       "7             147              0     157.0     103.1       211.8        7.1   \n",
       "8             117              0     184.5     351.6       215.8        8.7   \n",
       "9             141             37     258.6     222.0       326.4       11.2   \n",
       "\n",
       "   CustServ_Calls Churn Intl_Plan Vmail_Plan  Day_Calls  Day_Charge  \\\n",
       "0               1    no        no        yes        110       45.07   \n",
       "1               1    no        no        yes        123       27.47   \n",
       "2               0    no        no         no        114       41.38   \n",
       "3               2    no       yes         no         71       50.90   \n",
       "4               3    no       yes         no        113       28.34   \n",
       "5               0    no       yes         no         98       37.98   \n",
       "6               3    no        no        yes         88       37.09   \n",
       "7               0    no       yes         no         79       26.69   \n",
       "8               1    no        no         no         97       31.37   \n",
       "9               0    no       yes        yes         84       43.96   \n",
       "\n",
       "   Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  Intl_Charge  \\\n",
       "0         99       16.78           91         11.01           3         2.70   \n",
       "1        103       16.62          103         11.45           3         3.70   \n",
       "2        110       10.30          104          7.32           5         3.29   \n",
       "3         88        5.26           89          8.86           7         1.78   \n",
       "4        122       12.61          121          8.41           3         2.73   \n",
       "5        101       18.75          118          9.18           6         1.70   \n",
       "6        108       29.62          118          9.57           7         2.03   \n",
       "7         94        8.76           96          9.53           6         1.92   \n",
       "8         80       29.89           90          9.71           4         2.35   \n",
       "9        111       18.87           97         14.69           5         3.02   \n",
       "\n",
       "  State  Area_Code     Phone  \n",
       "0    KS        415  382-4657  \n",
       "1    OH        415  371-7191  \n",
       "2    NJ        415  358-1921  \n",
       "3    OH        408  375-9999  \n",
       "4    OK        415  330-6626  \n",
       "5    AL        510  391-8027  \n",
       "6    MA        510  355-9993  \n",
       "7    MO        415  329-9001  \n",
       "8    LA        408  335-4719  \n",
       "9    WV        415  330-8173  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.read_csv(\"Churn.csv\")\n",
    "\n",
    "display(df.head(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3333, 21)\n"
     ]
    }
   ],
   "source": [
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 3333 entries, 0 to 3332\n",
      "Data columns (total 21 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   Account_Length  3333 non-null   int64  \n",
      " 1   Vmail_Message   3333 non-null   int64  \n",
      " 2   Day_Mins        3333 non-null   float64\n",
      " 3   Eve_Mins        3333 non-null   float64\n",
      " 4   Night_Mins      3333 non-null   float64\n",
      " 5   Intl_Mins       3333 non-null   float64\n",
      " 6   CustServ_Calls  3333 non-null   int64  \n",
      " 7   Churn           3333 non-null   object \n",
      " 8   Intl_Plan       3333 non-null   object \n",
      " 9   Vmail_Plan      3333 non-null   object \n",
      " 10  Day_Calls       3333 non-null   int64  \n",
      " 11  Day_Charge      3333 non-null   float64\n",
      " 12  Eve_Calls       3333 non-null   int64  \n",
      " 13  Eve_Charge      3333 non-null   float64\n",
      " 14  Night_Calls     3333 non-null   int64  \n",
      " 15  Night_Charge    3333 non-null   float64\n",
      " 16  Intl_Calls      3333 non-null   int64  \n",
      " 17  Intl_Charge     3333 non-null   float64\n",
      " 18  State           3333 non-null   object \n",
      " 19  Area_Code       3333 non-null   int64  \n",
      " 20  Phone           3333 non-null   object \n",
      "dtypes: float64(8), int64(8), object(5)\n",
      "memory usage: 546.9+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['no' 'yes']\n",
      "['no' 'yes']\n",
      "['yes' 'no']\n"
     ]
    }
   ],
   "source": [
    "print(df['Churn'].unique())\n",
    "print(df['Intl_Plan'].unique())\n",
    "print(df['Vmail_Plan'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Account_Length</th>\n",
       "      <th>Vmail_Message</th>\n",
       "      <th>Day_Mins</th>\n",
       "      <th>Eve_Mins</th>\n",
       "      <th>Night_Mins</th>\n",
       "      <th>Intl_Mins</th>\n",
       "      <th>CustServ_Calls</th>\n",
       "      <th>Churn</th>\n",
       "      <th>Intl_Plan</th>\n",
       "      <th>Vmail_Plan</th>\n",
       "      <th>Day_Calls</th>\n",
       "      <th>Day_Charge</th>\n",
       "      <th>Eve_Calls</th>\n",
       "      <th>Eve_Charge</th>\n",
       "      <th>Night_Calls</th>\n",
       "      <th>Night_Charge</th>\n",
       "      <th>Intl_Calls</th>\n",
       "      <th>Intl_Charge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>128</td>\n",
       "      <td>25</td>\n",
       "      <td>265.1</td>\n",
       "      <td>197.4</td>\n",
       "      <td>244.7</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>110</td>\n",
       "      <td>45.07</td>\n",
       "      <td>99</td>\n",
       "      <td>16.78</td>\n",
       "      <td>91</td>\n",
       "      <td>11.01</td>\n",
       "      <td>3</td>\n",
       "      <td>2.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>107</td>\n",
       "      <td>26</td>\n",
       "      <td>161.6</td>\n",
       "      <td>195.5</td>\n",
       "      <td>254.4</td>\n",
       "      <td>13.7</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>123</td>\n",
       "      <td>27.47</td>\n",
       "      <td>103</td>\n",
       "      <td>16.62</td>\n",
       "      <td>103</td>\n",
       "      <td>11.45</td>\n",
       "      <td>3</td>\n",
       "      <td>3.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>137</td>\n",
       "      <td>0</td>\n",
       "      <td>243.4</td>\n",
       "      <td>121.2</td>\n",
       "      <td>162.6</td>\n",
       "      <td>12.2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>114</td>\n",
       "      <td>41.38</td>\n",
       "      <td>110</td>\n",
       "      <td>10.30</td>\n",
       "      <td>104</td>\n",
       "      <td>7.32</td>\n",
       "      <td>5</td>\n",
       "      <td>3.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>299.4</td>\n",
       "      <td>61.9</td>\n",
       "      <td>196.9</td>\n",
       "      <td>6.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71</td>\n",
       "      <td>50.90</td>\n",
       "      <td>88</td>\n",
       "      <td>5.26</td>\n",
       "      <td>89</td>\n",
       "      <td>8.86</td>\n",
       "      <td>7</td>\n",
       "      <td>1.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>166.7</td>\n",
       "      <td>148.3</td>\n",
       "      <td>186.9</td>\n",
       "      <td>10.1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113</td>\n",
       "      <td>28.34</td>\n",
       "      <td>122</td>\n",
       "      <td>12.61</td>\n",
       "      <td>121</td>\n",
       "      <td>8.41</td>\n",
       "      <td>3</td>\n",
       "      <td>2.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>118</td>\n",
       "      <td>0</td>\n",
       "      <td>223.4</td>\n",
       "      <td>220.6</td>\n",
       "      <td>203.9</td>\n",
       "      <td>6.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>98</td>\n",
       "      <td>37.98</td>\n",
       "      <td>101</td>\n",
       "      <td>18.75</td>\n",
       "      <td>118</td>\n",
       "      <td>9.18</td>\n",
       "      <td>6</td>\n",
       "      <td>1.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>121</td>\n",
       "      <td>24</td>\n",
       "      <td>218.2</td>\n",
       "      <td>348.5</td>\n",
       "      <td>212.6</td>\n",
       "      <td>7.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>88</td>\n",
       "      <td>37.09</td>\n",
       "      <td>108</td>\n",
       "      <td>29.62</td>\n",
       "      <td>118</td>\n",
       "      <td>9.57</td>\n",
       "      <td>7</td>\n",
       "      <td>2.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>147</td>\n",
       "      <td>0</td>\n",
       "      <td>157.0</td>\n",
       "      <td>103.1</td>\n",
       "      <td>211.8</td>\n",
       "      <td>7.1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>79</td>\n",
       "      <td>26.69</td>\n",
       "      <td>94</td>\n",
       "      <td>8.76</td>\n",
       "      <td>96</td>\n",
       "      <td>9.53</td>\n",
       "      <td>6</td>\n",
       "      <td>1.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>117</td>\n",
       "      <td>0</td>\n",
       "      <td>184.5</td>\n",
       "      <td>351.6</td>\n",
       "      <td>215.8</td>\n",
       "      <td>8.7</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>97</td>\n",
       "      <td>31.37</td>\n",
       "      <td>80</td>\n",
       "      <td>29.89</td>\n",
       "      <td>90</td>\n",
       "      <td>9.71</td>\n",
       "      <td>4</td>\n",
       "      <td>2.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>141</td>\n",
       "      <td>37</td>\n",
       "      <td>258.6</td>\n",
       "      <td>222.0</td>\n",
       "      <td>326.4</td>\n",
       "      <td>11.2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>84</td>\n",
       "      <td>43.96</td>\n",
       "      <td>111</td>\n",
       "      <td>18.87</td>\n",
       "      <td>97</td>\n",
       "      <td>14.69</td>\n",
       "      <td>5</td>\n",
       "      <td>3.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  Intl_Mins  \\\n",
       "0             128             25     265.1     197.4       244.7       10.0   \n",
       "1             107             26     161.6     195.5       254.4       13.7   \n",
       "2             137              0     243.4     121.2       162.6       12.2   \n",
       "3              84              0     299.4      61.9       196.9        6.6   \n",
       "4              75              0     166.7     148.3       186.9       10.1   \n",
       "5             118              0     223.4     220.6       203.9        6.3   \n",
       "6             121             24     218.2     348.5       212.6        7.5   \n",
       "7             147              0     157.0     103.1       211.8        7.1   \n",
       "8             117              0     184.5     351.6       215.8        8.7   \n",
       "9             141             37     258.6     222.0       326.4       11.2   \n",
       "\n",
       "   CustServ_Calls  Churn  Intl_Plan  Vmail_Plan  Day_Calls  Day_Charge  \\\n",
       "0               1      0          0           1        110       45.07   \n",
       "1               1      0          0           1        123       27.47   \n",
       "2               0      0          0           0        114       41.38   \n",
       "3               2      0          1           0         71       50.90   \n",
       "4               3      0          1           0        113       28.34   \n",
       "5               0      0          1           0         98       37.98   \n",
       "6               3      0          0           1         88       37.09   \n",
       "7               0      0          1           0         79       26.69   \n",
       "8               1      0          0           0         97       31.37   \n",
       "9               0      0          1           1         84       43.96   \n",
       "\n",
       "   Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  Intl_Charge  \n",
       "0         99       16.78           91         11.01           3         2.70  \n",
       "1        103       16.62          103         11.45           3         3.70  \n",
       "2        110       10.30          104          7.32           5         3.29  \n",
       "3         88        5.26           89          8.86           7         1.78  \n",
       "4        122       12.61          121          8.41           3         2.73  \n",
       "5        101       18.75          118          9.18           6         1.70  \n",
       "6        108       29.62          118          9.57           7         2.03  \n",
       "7         94        8.76           96          9.53           6         1.92  \n",
       "8         80       29.89           90          9.71           4         2.35  \n",
       "9        111       18.87           97         14.69           5         3.02  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3333, 18)\n"
     ]
    }
   ],
   "source": [
    "df = df.drop(['State', 'Area_Code', 'Phone'], axis=1)\n",
    "\n",
    "df['Churn'] = df['Churn'].replace(({'no':0, 'yes':1}))\n",
    "df['Intl_Plan'] = df['Intl_Plan'].replace(({'no':0, 'yes':1}))\n",
    "df['Vmail_Plan'] = df['Vmail_Plan'].replace(({'no':0, 'yes':1}))\n",
    "\n",
    "display(df.head(10))\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 3333 entries, 0 to 3332\n",
      "Data columns (total 18 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   Account_Length  3333 non-null   int64  \n",
      " 1   Vmail_Message   3333 non-null   int64  \n",
      " 2   Day_Mins        3333 non-null   float64\n",
      " 3   Eve_Mins        3333 non-null   float64\n",
      " 4   Night_Mins      3333 non-null   float64\n",
      " 5   Intl_Mins       3333 non-null   float64\n",
      " 6   CustServ_Calls  3333 non-null   int64  \n",
      " 7   Churn           3333 non-null   int64  \n",
      " 8   Intl_Plan       3333 non-null   int64  \n",
      " 9   Vmail_Plan      3333 non-null   int64  \n",
      " 10  Day_Calls       3333 non-null   int64  \n",
      " 11  Day_Charge      3333 non-null   float64\n",
      " 12  Eve_Calls       3333 non-null   int64  \n",
      " 13  Eve_Charge      3333 non-null   float64\n",
      " 14  Night_Calls     3333 non-null   int64  \n",
      " 15  Night_Charge    3333 non-null   float64\n",
      " 16  Intl_Calls      3333 non-null   int64  \n",
      " 17  Intl_Charge     3333 non-null   float64\n",
      "dtypes: float64(8), int64(10)\n",
      "memory usage: 468.8 KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Target variable "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2850\n",
       "1     483\n",
       "Name: Churn, dtype: int64"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Churn'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature and target variable separation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3333, 17) (3333, 1)\n"
     ]
    }
   ],
   "source": [
    "X = df.drop('Churn', axis=1)\n",
    "y = df[['Churn']]\n",
    "\n",
    "print(X.shape, y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train test separation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2333, 17) (1000, 17) (2333, 1) (1000, 1)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, \n",
    "                                                    y, \n",
    "                                                    test_size=0.3, \n",
    "                                                    random_state=42, stratify=y)\n",
    "\n",
    "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stratification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.16942355889724312\n"
     ]
    }
   ],
   "source": [
    "class_0_train = y_train['Churn'].value_counts().values[0]\n",
    "class_1_train = y_train['Churn'].value_counts().values[1]\n",
    "\n",
    "train_ratio = class_1_train/class_0_train\n",
    "print(train_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1695906432748538\n"
     ]
    }
   ],
   "source": [
    "class_0_test = y_test['Churn'].value_counts().values[0]\n",
    "class_1_test = y_test['Churn'].value_counts().values[1]\n",
    "\n",
    "test_ratio = class_1_test/class_0_test\n",
    "print(test_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training: Decision Tree Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = LogisticRegression()\n",
    "model = DecisionTreeClassifier()\n",
    "# model = RandomForestClassifier()\n",
    "model = model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prediction Comparison with the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Churn</th>\n",
       "      <th>probability</th>\n",
       "      <th>predicted_Churn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1947</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>417</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2991</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1757</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>697</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>882</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2253</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2614</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2096</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Churn  probability  predicted_Churn\n",
       "1384      0          0.0                0\n",
       "1947      0          0.0                0\n",
       "417       0          0.0                0\n",
       "2991      0          0.0                0\n",
       "1757      0          0.0                0\n",
       "697       0          0.0                0\n",
       "882       0          0.0                0\n",
       "2253      0          0.0                0\n",
       "2614      1          0.0                0\n",
       "2096      0          0.0                0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_test['probability'] = np.round(model.predict_proba(X_test)[:,1], 2)\n",
    "y_test['predicted_Churn'] = y_pred\n",
    "\n",
    "display(y_test.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Account_Length</th>\n",
       "      <th>Vmail_Message</th>\n",
       "      <th>Day_Mins</th>\n",
       "      <th>Eve_Mins</th>\n",
       "      <th>Night_Mins</th>\n",
       "      <th>Intl_Mins</th>\n",
       "      <th>CustServ_Calls</th>\n",
       "      <th>Intl_Plan</th>\n",
       "      <th>Vmail_Plan</th>\n",
       "      <th>Day_Calls</th>\n",
       "      <th>Day_Charge</th>\n",
       "      <th>Eve_Calls</th>\n",
       "      <th>Eve_Charge</th>\n",
       "      <th>Night_Calls</th>\n",
       "      <th>Night_Charge</th>\n",
       "      <th>Intl_Calls</th>\n",
       "      <th>Intl_Charge</th>\n",
       "      <th>Churn</th>\n",
       "      <th>probability</th>\n",
       "      <th>predicted_Churn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1843</th>\n",
       "      <td>105</td>\n",
       "      <td>0</td>\n",
       "      <td>166.1</td>\n",
       "      <td>175.9</td>\n",
       "      <td>243.5</td>\n",
       "      <td>16.2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>93</td>\n",
       "      <td>28.24</td>\n",
       "      <td>106</td>\n",
       "      <td>14.95</td>\n",
       "      <td>55</td>\n",
       "      <td>10.96</td>\n",
       "      <td>3</td>\n",
       "      <td>4.37</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288</th>\n",
       "      <td>167</td>\n",
       "      <td>0</td>\n",
       "      <td>207.6</td>\n",
       "      <td>132.4</td>\n",
       "      <td>255.2</td>\n",
       "      <td>14.1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>88</td>\n",
       "      <td>35.29</td>\n",
       "      <td>63</td>\n",
       "      <td>11.25</td>\n",
       "      <td>98</td>\n",
       "      <td>11.48</td>\n",
       "      <td>5</td>\n",
       "      <td>3.81</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3110</th>\n",
       "      <td>31</td>\n",
       "      <td>28</td>\n",
       "      <td>171.8</td>\n",
       "      <td>240.7</td>\n",
       "      <td>245.5</td>\n",
       "      <td>10.6</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>116</td>\n",
       "      <td>29.21</td>\n",
       "      <td>125</td>\n",
       "      <td>20.46</td>\n",
       "      <td>80</td>\n",
       "      <td>11.05</td>\n",
       "      <td>7</td>\n",
       "      <td>2.86</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2855</th>\n",
       "      <td>95</td>\n",
       "      <td>0</td>\n",
       "      <td>149.2</td>\n",
       "      <td>260.7</td>\n",
       "      <td>201.0</td>\n",
       "      <td>8.1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>96</td>\n",
       "      <td>25.36</td>\n",
       "      <td>116</td>\n",
       "      <td>22.16</td>\n",
       "      <td>120</td>\n",
       "      <td>9.05</td>\n",
       "      <td>2</td>\n",
       "      <td>2.19</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3272</th>\n",
       "      <td>133</td>\n",
       "      <td>0</td>\n",
       "      <td>295.0</td>\n",
       "      <td>223.6</td>\n",
       "      <td>229.4</td>\n",
       "      <td>12.9</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>141</td>\n",
       "      <td>50.15</td>\n",
       "      <td>101</td>\n",
       "      <td>19.01</td>\n",
       "      <td>109</td>\n",
       "      <td>10.32</td>\n",
       "      <td>4</td>\n",
       "      <td>3.48</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1336</th>\n",
       "      <td>124</td>\n",
       "      <td>0</td>\n",
       "      <td>251.5</td>\n",
       "      <td>214.2</td>\n",
       "      <td>186.1</td>\n",
       "      <td>11.1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>85</td>\n",
       "      <td>42.76</td>\n",
       "      <td>98</td>\n",
       "      <td>18.21</td>\n",
       "      <td>71</td>\n",
       "      <td>8.37</td>\n",
       "      <td>6</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2539</th>\n",
       "      <td>138</td>\n",
       "      <td>28</td>\n",
       "      <td>211.2</td>\n",
       "      <td>312.5</td>\n",
       "      <td>178.0</td>\n",
       "      <td>10.7</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>117</td>\n",
       "      <td>35.90</td>\n",
       "      <td>98</td>\n",
       "      <td>26.56</td>\n",
       "      <td>118</td>\n",
       "      <td>8.01</td>\n",
       "      <td>2</td>\n",
       "      <td>2.89</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>602</th>\n",
       "      <td>143</td>\n",
       "      <td>0</td>\n",
       "      <td>202.8</td>\n",
       "      <td>165.8</td>\n",
       "      <td>143.9</td>\n",
       "      <td>4.6</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>109</td>\n",
       "      <td>34.48</td>\n",
       "      <td>104</td>\n",
       "      <td>14.09</td>\n",
       "      <td>71</td>\n",
       "      <td>6.48</td>\n",
       "      <td>4</td>\n",
       "      <td>1.24</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>755</th>\n",
       "      <td>69</td>\n",
       "      <td>0</td>\n",
       "      <td>167.5</td>\n",
       "      <td>242.1</td>\n",
       "      <td>101.2</td>\n",
       "      <td>11.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>76</td>\n",
       "      <td>28.48</td>\n",
       "      <td>92</td>\n",
       "      <td>20.58</td>\n",
       "      <td>103</td>\n",
       "      <td>4.55</td>\n",
       "      <td>4</td>\n",
       "      <td>3.08</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>528</th>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "      <td>124.8</td>\n",
       "      <td>157.3</td>\n",
       "      <td>199.3</td>\n",
       "      <td>8.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>133</td>\n",
       "      <td>21.22</td>\n",
       "      <td>143</td>\n",
       "      <td>13.37</td>\n",
       "      <td>72</td>\n",
       "      <td>8.97</td>\n",
       "      <td>4</td>\n",
       "      <td>2.32</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  \\\n",
       "1843             105              0     166.1     175.9       243.5   \n",
       "288              167              0     207.6     132.4       255.2   \n",
       "3110              31             28     171.8     240.7       245.5   \n",
       "2855              95              0     149.2     260.7       201.0   \n",
       "3272             133              0     295.0     223.6       229.4   \n",
       "1336             124              0     251.5     214.2       186.1   \n",
       "2539             138             28     211.2     312.5       178.0   \n",
       "602              143              0     202.8     165.8       143.9   \n",
       "755               69              0     167.5     242.1       101.2   \n",
       "528               46              0     124.8     157.3       199.3   \n",
       "\n",
       "      Intl_Mins  CustServ_Calls  Intl_Plan  Vmail_Plan  Day_Calls  Day_Charge  \\\n",
       "1843       16.2               2          0           0         93       28.24   \n",
       "288        14.1               0          0           0         88       35.29   \n",
       "3110       10.6               1          0           1        116       29.21   \n",
       "2855        8.1               1          0           0         96       25.36   \n",
       "3272       12.9               2          0           0        141       50.15   \n",
       "1336       11.1               3          0           0         85       42.76   \n",
       "2539       10.7               3          1           1        117       35.90   \n",
       "602         4.6               1          0           0        109       34.48   \n",
       "755        11.4               2          0           0         76       28.48   \n",
       "528         8.6               2          0           0        133       21.22   \n",
       "\n",
       "      Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  \\\n",
       "1843        106       14.95           55         10.96           3   \n",
       "288          63       11.25           98         11.48           5   \n",
       "3110        125       20.46           80         11.05           7   \n",
       "2855        116       22.16          120          9.05           2   \n",
       "3272        101       19.01          109         10.32           4   \n",
       "1336         98       18.21           71          8.37           6   \n",
       "2539         98       26.56          118          8.01           2   \n",
       "602         104       14.09           71          6.48           4   \n",
       "755          92       20.58          103          4.55           4   \n",
       "528         143       13.37           72          8.97           4   \n",
       "\n",
       "      Intl_Charge  Churn  probability  predicted_Churn  \n",
       "1843         4.37      0          0.0                0  \n",
       "288          3.81      0          0.0                0  \n",
       "3110         2.86      0          0.0                0  \n",
       "2855         2.19      0          0.0                0  \n",
       "3272         3.48      1          1.0                1  \n",
       "1336         3.00      0          0.0                0  \n",
       "2539         2.89      1          1.0                1  \n",
       "602          1.24      0          0.0                0  \n",
       "755          3.08      0          0.0                0  \n",
       "528          2.32      0          0.0                0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test = pd.concat([X_test, y_test], axis=1)\n",
    "display(test.sample(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "confusion_matrix: \n",
      " [[813  42]\n",
      " [ 44 101]]\n"
     ]
    }
   ],
   "source": [
    "confusion_matrix_ = confusion_matrix(y_test['Churn'], \n",
    "                                     y_test['predicted_Churn'])\n",
    "\n",
    "print('confusion_matrix: \\n', confusion_matrix_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    855\n",
       "1    145\n",
       "Name: Churn, dtype: int64"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test['Churn'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEHCAYAAAC6IG0BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjOklEQVR4nO3de7hVVb3/8fdng4IKGooQXkEDEzUpLpa3MDmJdp6w82himmien2kW2uWYl9/RsoffIU9HK8uKOiUdr6h5JDXUSI545SbKxRsFKkEgiBcQUeD7+2OOrYvd3mvNvc/a7LUmn1fPfNacY845xlh75ZexxhpzDEUEZmZWHA0dXQEzM6suB3Yzs4JxYDczKxgHdjOzgnFgNzMrGAd2M7OC6dzRFdjWqfMOoe27d3Q1rBU+euA+HV0Fa6U5c2aviojd23p/p533jdi4Pte1sf6V+yJiZLlrJH0d+GcggHnAWcCOwK1AX2AJ8PmIWJOuvwQ4G9gEjI2I+8rm73HsHathx17R5YDPd3Q1rBXWzPxJR1fBWmmH7TQ7Ioa09f7W/Hf69tyfli1L0p7Aw8DAiFgvaRJwLzAQeDUixku6GOgREd+WNBC4GRgG7AH8ERgQEZtarG/eN2Zmtu0SqCHflk9nYAdJncla6suAUcDEdH4icGLaHwXcEhEbImIxsIgsyLfIgd3MrBIBDZ3ybRVExF+BHwAvAcuB1yPifqB3RCxP1ywHeqVb9gReLsliaUprkQO7mVkeUr4NekqaVbKds2U26kHWCu9H1rWyk6TTy5XcTFrZPnT/eGpmVpFa082yqkJ//ghgcUS8AiDpd8DhwApJfSJiuaQ+wMp0/VJg75L79yLrummRW+xmZnnkb7FX8hLwcUk7ShJwLPAMMBkYk64ZA9yV9icDoyV1kdQP6A/MKFeAW+xmZpWI1rTYy4qIJyTdDswBNgJPAhOAbsAkSWeTBf+T0/UL0siZhen688uNiAEHdjOzHHK3xnOJiCuAK5okbyBrvTd3/ThgXN78HdjNzPLIMeKlVjiwm5lV1KofTzucA7uZWSWiql0x7c2B3cwsD7fYzcyKxF0xZmbFIqCTfzw1MysW97GbmRWJu2LMzIrHLXYzs4Jxi93MrEDyT/BVExzYzczy8JQCZmZF4h9PzcyKx10xZmYFUsX52LcGB3Yzs4rcFWNmVjz+8dTMrGDcx25mViCqr66Y+qmpmVlHanxIqdJWMRsdIGluyfaGpAsl7SrpAUkvpNceJfdcImmRpOckHVepDAd2M7McJOXaKomI5yJiUEQMAgYDbwF3AhcDUyOiPzA1HSNpIDAaOAgYCVwnqWyHvwO7mVkF2cp41QnsTRwL/DkiXgRGARNT+kTgxLQ/CrglIjZExGJgETCsXKbuYzczq0RCDe3y4+lo4Oa03zsilgNExHJJvVL6nsDjJfcsTWktcovdzCyHVrTYe0qaVbKd00J+2wOfBW6rVHQzaVHuBrfYzcxyaEU3y6qIGJLjuuOBORGxIh2vkNQntdb7ACtT+lJg75L79gKWlcvYLXYzsxzaoY/9VN7vhgGYDIxJ+2OAu0rSR0vqIqkf0B+YUS5jt9jNzCoRzXeItDU7aUfgH4AvlySPByZJOht4CTgZICIWSJoELAQ2AudHxKZy+Tuwm5lVINo04qVFEfEWsFuTtNVko2Sau34cMC5v/g7sZmY5NDTUT8+1A7uZWQ7VbLG3Nwd2M7NKqtzH3t4c2M3McnCL3cysQKr942l7c2A3M8uhnaYUaBcO7GZmlchdMWZmhePAbmZWMA7sZmYF4h9PzcyKqH7iugO7mVlF8pQCZmaF464YK7zzTj2GL554OESwcNEyzr/yBkYedQjfPucEDujbm2PP/AFzn3kJgI8N3JcfXnYqkH2bHf/Le7ln2tMdWHsD2LRpM8eccRV9eu3Crdecx7/+6E7umz6f7bbrRL+9evLTy09nl+47dnQ1a0f9xPWOW2hD0r2SPpD216bXvpLmVyn/C9Ocx3mvP1PST6pRdtH12X0XvnzKJ/nUGVdx+Oj/R0NDA//06cE88+dlnHHRL3n0yT9vcf0zf17GMWdcxdGnjeeksddxzSWn0qlT/XytLaqf3/IgA/r1fu/4mMM+zKO3XMojN1/K/vv04urr7+/A2tWedlrMul102H9dEXFCRLzWjkVcCGy15oakberbT+fOnejaZTs6dWpgx67b87dXXuf5JStY9OLKv7t2/YZ32bRpMwBdumxHRNnlGm0r+OuKNdz/8ALOGHX4e2mf+viBdO7cCYChB/dj2YrXOqh2tSdvUC90YJd0kaSxaf8aSX9K+8dKuiHtL5HUM2d+wyVNk3S7pGcl3aj0F0x5PilpnqRfp+WjxgJ7AA9KerCZ/IZKelTSU5JmSOqeTu0haYqkFyRdVXL92pL9kyRdn/avl3R1KuP76fjHKe+/SDqpDX++mrf8lde59oapzPv993j2D+N4Y916Hnzi2bL3DD5oXx699TIeuflSvjH+lvcCvXWMS6++g++OPZGGFh6Tv2HyY4w4fOBWrlVt2+YDO/AQcFTaHwJ0k7QdcCQwvY15fpSsFT4Q2A84QlJX4HrglIg4hOw3g/Mi4sdki70eExHHlGaSVga/FbggIg4FRgDr0+lBwCnAIcApkkoXkG3JAGBERHwzHfche5//SLbUVeHs0n0HTjj6EAaNuoIDj7+MHbtuz+ePH1r2ntkLXuTwU8Zx7Jir+PqZn6bL9tvUF5yaMmX6PHr26M6gA/dp9vwPfj2Fzp0bKn6m2xo1KNdWC9orsM8GBqeW8AbgMbIAfxRtD+wzImJpRGwG5gJ9gQOAxRHxfLpmInB0hXwOAJZHxEyAiHgjIjamc1Mj4vWIeJtsfcF9c9TrtibrD/53RGyOiIVA7+ZukHSOpFmSZsXG9c1dUtOGD/swLy5bzerX1rJx02Z+/+BTDPtIv1z3Pr9kBW+tf4cD99+jnWtpLXniqb8wZfo8PvLZyzn70t8wfebznPOvEwG4+e7Huf/h+Uz43pk10/qsFfXUYm+XZlNEvCtpCXAW8CjwNHAMsD/wTBuz3VCyv4ms7m35KwpoqZO3uTJocn3XJvesK5NHs/WLiAnABICGHXvVXYfz0r+9ypBD+rFDl+1Yv+FdPjn0AJ5MI2Cas88eu/HXFWvYtGkze3+wBx/atzcvLVu9FWtspa746iiu+OooAB6e/TzX3jCVCd8bwx8fXciPfvtH7v7FBezYdfsOrmWN8SRg73kI+BbwJWAecDUwO6r7y9mzQF9JH4qIRcAXgf9J594EugOrmrlnD0lDI2Jm+lZRqdm8QtKBwHPA51Le26zZC15k8tQnmXbDt9m0aTNPP7eUiXc+wmeGf4Tvf+tkevboxq3XnMu85//KSWN/yicO3Y8Lzvw0GzduYvPm4Fvfv5VXX2/676F1tIv+fRIb3tnI587PBocNOaQv11xyagfXqjYIqGZcTyMCfwUcTNZw/BJZfLmVrDdiCfD5iFiTrr8EOJuswTk2Iu4rl397BvbpwGXAYxGxTtLbtL0bplkR8baks4Db0qiUmcDP0+kJwB8kLS/tZ4+IdySdAlwraQeyoD6iQlEXA3cDLwPzgW7VfB/1aPyEexk/4d4t0u6Z9nSz49Nv/cNMbv3DzK1VNWuFIwcP4MjBAwCYc+d3OrYyNa3q3Sw/AqZExEnpd78dgUvJuoPHS7qYLO58W9JAYDRwENmgkD9KGtCkC3jL2nroWcdq2LFXdDng8x1dDWuFNTP9uEO92WE7zY6IIW29v+sHB8S+Y67Nde3zV40sW5aknYGngP1KezAkPQcMj4jlkvoA0yLigNRaJyL+LV13H/CdiHispTL8lIiZWSXKumLybEDPxsERaTunSW77Aa8Av0lDtX8laSegd0QsB0ivvdL1e5L1FjRamtJa5DFnZmYVCFoc89+MVRW+HXQGPgZ8LSKekPQjsm6XcsU3VbarxS12M7McWtFir2QpsDQinkjHt5MF+hWpC4b0urLk+tJnavYie06nRQ7sZmY5VGsce0T8DXhZ0gEp6Viy52YmA2NS2hjgrrQ/GRidnqrvB/QHZpQrw10xZmaV5G+N5/U14MY0IuYvZM/8NACTJJ0NvAScDBARCyRNIgv+G4Hzy42IAQd2M7OKhKq60EZEzCV7Gr+pY1u4fhwwLm/+DuxmZjnU0YOnDuxmZnl4SgEzsyKpfh97u3JgNzOrIJsrpn4iuwO7mVkOdRTXHdjNzPJoxZOnHc6B3cysEs/HbmZWLNWej729ObCbmVVUO8ve5eHAbmaWQx3FdQd2M7OK5B9PzcwKxePYzcwKyIHdzKxg6iiuO7CbmeXhFruZWZF4EjAzs2LJFtqon8juwG5mlkNDHTXZHdjNzHKoo7hO9RbxMzMrKKVJwPJs+fLTEknzJM2VNCul7SrpAUkvpNceJddfImmRpOckHVcpfwd2M7McGpRva4VjImJQRDQuan0xMDUi+gNT0zGSBgKjgYOAkcB1kjqVy7jFrhhJ1wLR0vmIGNuqt2BmVse2wo+no4DhaX8iMA34dkq/JSI2AIslLQKGAY+1lFG5PvZZ1aipmVm9E9nImCoK4H5JAfwiIiYAvSNiOUBELJfUK127J/B4yb1LU1qLWgzsETGx9FjSThGxrg1vwMys7rWiwd6zsd88mZACd6kjImJZCt4PSHq2TH7NldxibwrkGBUj6RPAfwLdgH0kHQp8OSK+UuleM7NCaMUPo8Cqkn7zZkXEsvS6UtKdZF0rKyT1Sa31PsDKdPlSYO+S2/cClpXLP8+Ppz8EjgNWp4o8BRyd4z4zs8KQ8m2V89FOkro37gOfBuYDk4Ex6bIxwF1pfzIwWlIXSf2A/sCMcmXkGsceES83+ddqU577zMyKQFT1AaXewJ0ppnYGboqIKZJmApMknQ28BJwMEBELJE0CFgIbgfMjomwMzhPYX5Z0OBCStgfGAs+09R2ZmdWjao2KiYi/AIc2k74aOLaFe8YB4/KWkacr5lzgfLJfYf8KDErHZmbbhLzdMLXydGrFFntErAJO2wp1MTOrWfU0V0zFFruk/ST9XtIrklZKukvSflujcmZmtUI5t1qQpyvmJmAS0AfYA7gNuLk9K2VmVmuqOVdMe8sT2BUR/xURG9N2AxUGx5uZFUk2Kqbqc8W0m3Jzxeyadh+UdDFwC1lAPwW4ZyvUzcysNqg4C23MJgvkje/myyXnAvhee1XKzKzW1Eo3Sx7l5orptzUrYmZWqxq7YupFridPJR0MDAS6NqZFxG/bq1JmZrWmEC32RpKuIJsjeCBwL3A88DDgwG5m24z6Cev5RsWcRPaY698i4iyyR2G7tGutzMxqiASdGpRrqwV5umLWR8RmSRsl7Uw2laQfUDKzbUqhumKAWZI+APySbKTMWipMGWlmVjR1FNdzzRXTuKDGzyVNAXaOiKfbt1pmZrVDqK7miin3gNLHyp2LiDntUyUzsxpTQzM35lGuxf4fZc4F8Kkq12Wb9NED9+GRJ37S0dWwVnj9rXc7ugrWAQrRxx4Rx2zNipiZ1SoBnYoQ2M3M7H01MpIxFwd2M7Mc6imw53lAycxsm5Yte1fd+dgldZL0pKS70/Gukh6Q9EJ67VFy7SWSFkl6TtJxlfLOs4KSJJ0u6fJ0vI+kYblrb2ZWAO0wH/sFwDMlxxcDUyOiPzA1HSNpIDAaOAgYCVwnqVPZuuYo/DrgE8Cp6fhN4Ketqb2ZWb2r5mLWkvYCPgP8qiR5FDAx7U8ETixJvyUiNkTEYmARULZxnaeP/bCI+JikJwEiYo2k7fNV38ys/gnoXN1RMT8ELgK6l6T1jojlABGxXFKvlL4n8HjJdUtTWovytNjfTc3+AJC0O7A5V9XNzAqiFS32npJmlWznbJmP/hFYGRGz8xbdTFrZ5UnztNh/DNwJ9JI0jmy2x/+bs0JmZnVPatWUAqsiYkiZ80cAn5V0AtkaFztLugFYIalPaq33IZtwEbIW+t4l9+8FLCtXgYot9oi4kewrw78By4ETI+K2SveZmRVJtfrYI+KSiNgrIvqS/Sj6p4g4HZgMjEmXjQHuSvuTgdGSukjqB/SnwkSMeRba2Ad4C/h9aVpEvFT5LZiZFcNWGMc+Hpgk6WzgJeBkgIhYIGkSsBDYCJwfEZvKZZSnK+Ye3l/UuivQD3iObOiNmVnhCdplEY2ImAZMS/uryRY1au66ccC4vPnmmbb3kNLjNOvjl/MWYGZW91o/Rr1DtXpKgYiYI2loe1TGzKxWqY5WPc3Tx/6NksMG4GPAK+1WIzOzGiOK12IvHUC/kazP/Y72qY6ZWW0qTGBPDyZ1i4h/2Ur1MTOrSYVYaENS54jYWG6JPDOzbYEEnepoLtxyLfYZZP3pcyVNBm4D1jWejIjftXPdzMxqRiEWsy6xK7CabI3TxvHsATiwm9k2oUg/nvZKI2Lm835Ab1R2Ahozs6KpowZ72cDeCehGG2YWMzMrFtFQkHHsyyPiyq1WEzOzGiWK02Kvo7dhZtaOBJ3rqJO9XGBvdjIaM7NtTWFa7BHx6tasiJlZLSvacEczs21eHcV1B3Yzs0pEvgWia4UDu5lZJXJXjJlZoWRPnjqwm5kVSv2E9frqNjIz6zBSvq1yPuoqaYakpyQtkPTdlL6rpAckvZBee5Tcc4mkRZKek3RcpTIc2M3MKhJSvi2HDcCnIuJQYBAwUtLHgYuBqRHRH5iajpE0EBgNHASMBK5La2W0yIHdzKyCxlExebZKIrM2HW6XtgBGARNT+kTgxLQ/CrglIjZExGJgETCsXBkO7GZmOTRIubY8JHWSNBdYCTwQEU8AvSNiOUB67ZUu3xN4ueT2pSmtRf7x1MysErVqabyekmaVHE+IiAmlF0TEJmCQpA8Ad0o6uHzpf6fsDLsO7GZmFbTyAaVVETEkz4UR8ZqkaWR95ysk9YmI5ZL6kLXmIWuh711y217AsnL5uivGzCyHav14Kmn31FJH0g7ACOBZYDIwJl02Brgr7U8GRkvqIqkf0J9s6dIWucVuZpZDFcex9wEmppEtDcCkiLhb0mPAJElnAy8BJwNExAJJk4CFwEbg/NSV0yIHdjOzCgR0qtKTpxHxNPDRZtJX08J06RExDhiXtwwHdjOzHOpoRgEHdjOzyoTqaFIBB3YzsxzcYjczK5BsuGP9RHYHdjOzSnJO8FUrHNjNzHLwfOxmZgWSLbTR0bXIz4HdzCwHj4oxMyuYOuqJ8VwxVh2bNm3m6NPGc8rXf7ZF+rX/9Ud6DP0qq19b28KdtrX8y/ibGTzqX/n0md9/L+21N9Zx+jd+xvAvjOP0b/yM1998C4A1r69j9AU/ZeDIb3P5D+/oqCrXFOX8Xy0oTGCXdG/JxDpr02tfSfNbkcc0SblmZbMt/fyWBxnQr/cWaUv/toZpM55lrw/2aOEu25pOOn4YE//9nC3SfnbjVA4f3J9pN13G4YP7c92NUwHosn1nvnn28Vx63mc7oqo1p7GPPc9WCwoT2CPihIh4rSPrUGm5qqL664o13P/wAs4YdfgW6Zddcwff+dqJrZnH2trRYYfuzy7dd9oi7YFH5nPSyKEAnDRyKA88PA+AHXfowtCP7EeX7bfb6vWsSTkX2aiVkTN1EdglXSRpbNq/RtKf0v6xkm5I+0sk9WxlnvPSgrLjS06dnBaafV7SUenaMyX9pOTeuyUNT/trJV0p6QngE+l4XMr3cUlbNmML6NKr7+C7Y0+koaS5cu//PE2f3T/AIQP26sCaWSWvrHmTXrvtAkCv3XZh1Rp3mbVEObdaUBeBHXgIOCrtDwG6SdoOOBKY3trMJB1Ptp7gYWlB2atKTneOiGHAhcAVObLbCZgfEYdFxMPp+PGU70PA/2mm/HMkzZI065VVr7S2+jVlyvR59OzRnUEH7vNe2ltvv8PVv7mPS879TAfWzKx6sq6Y+mmx18uomNnAYEndyVb4nkMW4I8CxrYhvxHAbyLiLYCIeLXk3O9KyuybI69NQOmvS+8Ad5fk8Q9Nb0jLZE0AGDx4SNklrmrdE0/9hSnT5/HAowvYsOFd3lz3NudePpEXl63mqC/8GwDLVr7GJ0//PlOv/xd699y5g2tspXbv0Z2Vq1+n1267sHL16/Ts0a2jq1SzaiNk51MXgT0i3pW0BDgLeBR4GjgG2B94pg1ZipbXDNyQXjfx/t9nI1t+u+lasv92k0nv342IxrxL8yikK746iiu+OgqAh2c/z7U3TOW3V235JeUjn72cB397Ebt9wEGj1ow44mBunzKTr5w2gtunzOQfjii39OY2ro4iez0FnYeAbwFfAuYBVwOzS4Joa9wPXC7ppoh4S9KuTVrtTS0BviKpgWx18GFtKNOsQ33tu7/l8bmLWPP6Oj5+0nf4+lkjOe8Lx3L+dyYy6Z4n2KN3D6777pj3rj/ilCtZu24D727cyP0Pz+O/fnAu/ft+sAPfQceqlW6WPOopsE8HLgMei4h1kt6mDf3rABExRdIgYJakd4B7gUvL3PIIsJjsH5T5ZF1B1sSRgwdw5OABf5f+9OQrO6A21tS1V5zRbPpN13yl2fRHbr28PatTd+onrNdRYI+IqcB2JccDmpzvW7LfLb0uAZr9bhkR44HxTdKGl+yvIvWxp28Fp7WQT7eWjiPiduD2lt+VmdWNOors9TIqxsysw2RDGavz5KmkvSU9KOkZSQskXZDSd5X0gKQX0muPknsukbRI0nOSjqtUhgO7mVklaT72PFsOG4FvRsSBwMeB8yUNBC4GpkZEf2BqOiadGw0cBIwErqv0MKQDu5lZDtV6QCkilkfEnLT/JtnIvj2BUcDEdNlEsmdtSOm3RMSGiFgMLKLCAA4HdjOzioSUb2tVrlJf4KPAE0DviFgOWfAHeqXL9gReLrltaUprUd38eGpm1pFaEbN7SppVcjwhPZTYJD91I3u48cKIeKPMPwrNnSg7zNuB3cysglbOA7MqIsrOEpumRLkDuDEiGp92XyGpT0Qsl9QHWJnSlwJ7l9y+F7CsXP7uijEzy6NKnezKmub/CTwTEVeXnJoMND4hNga4qyR9tKQukvoB/YEZ5cpwi93MLIcqLqJxBPBFYJ6kuSntUrLnaiZJOht4CTgZICIWSJoELCQbUXN+k2lM/o4Du5lZDtWaUSDNAttSbse2cM84YFzeMhzYzcwqyT9GvSY4sJuZ5VAr65nm4cBuZlaBcIvdzKxw6iiuO7CbmeVSR5Hdgd3MLAcvtGFmVjD1E9Yd2M3M8qmjyO7AbmZWQeNCG/XCgd3MrBI/oGRmVjx1FNcd2M3MKmv9IhodyYHdzCyHOorrDuxmZpW0cqGNDufAbmaWRx1Fdgd2M7McPNzRzKxg3MduZlYkggYHdjOzoqmfyO7AbmZWQb0ttNHQ0RUwM6sHyrlVzEf6taSVkuaXpO0q6QFJL6TXHiXnLpG0SNJzko7LU1cHdjOzHKR8Ww7XAyObpF0MTI2I/sDUdIykgcBo4KB0z3WSOlUqwIHdzCwHSbm2SiLiIeDVJsmjgIlpfyJwYkn6LRGxISIWA4uAYZXKcGA3M8uhFV0xPSXNKtnOyZF974hYDpBee6X0PYGXS65bmtLK8o+nZmYVtKKbBWBVRAypVtHNpEWlm9xiNzPLQTn/10YrJPUBSK8rU/pSYO+S6/YCllXKzIHdzCyPag2Lad5kYEzaHwPcVZI+WlIXSf2A/sCMSpm5K8bMLIdqDWOXdDMwnKwvfilwBTAemCTpbOAl4GSAiFggaRKwENgInB8RmyqV4cBuZlaRaKjSE0oRcWoLp45t4fpxwLjWlOHAbmZWgZ88NTOzDuUWu5lZDvXUYndgNzPLwQttmJkVSeseUOpwDuxmZhXU24+nDuxmZjm4K8bMrGDcYjczK5g6iusO7GZmudRRZHdgNzOrQFC1KQW2BkVUnNrX2pGkV4AXO7oe7aAnsKqjK2GtUuTPbN+I2L2tN0uaQvb3yWNVRDRd+m6rcmC3diFpVhUXG7CtwJ9ZcXiuGDOzgnFgNzMrGAd2ay8TOroC1mr+zArCfexmZgXjFruZWcE4sG+DJN0r6QNpf2167StpfpXyv1DSjq24/kxJP6lG2du6any2kqZJ8uiYOubAvg2KiBMi4rV2LOJCIHdg/9+S5Aftkq3w2VYkqVNHlm8O7IUj6SJJY9P+NZL+lPaPlXRD2l8iKdfDFpKGpxbc7ZKelXSjlD2Cl/J8UtI8Sb+W1CWVvQfwoKQHm8lvqKRHJT0laYak7unUHpKmSHpB0lUl168t2T9J0vVp/3pJV6cyvp+Of5zy/oukk9rw56tp1f5sS/Kclz6P8SWnTk6fz/OSjkrXbvHNStLdkoan/bWSrpT0BPCJdDwu5fu4pN7/2/dv+TmwF89DwFFpfwjQTdJ2wJHA9Dbm+VGyVvhAYD/gCEldgeuBUyLiELLpKc6LiB8Dy4BjIuKY0kwkbQ/cClwQEYcCI4D16fQg4BTgEOAUSXvnqNcAYEREfDMd9yF7n/8IjG/xrvpV1c9W0vHAicBh6fO4quR054gYRva5X5Eju52A+RFxWEQ8nI4fT/k+BPyf1tbP2s6BvXhmA4NTS3gD8BhZEDiKtgf2GRGxNCI2A3OBvsABwOKIeD5dMxE4ukI+BwDLI2ImQES8EREb07mpEfF6RLwNLAT2zVGv2yJiU8nxf0fE5ohYCBSxhVjtz3YE8JuIeAsgIl4tOfe7kjL75shrE3BHyfE7wN2tzMOqxH2TBRMR70paApwFPAo8DRwD7A8808ZsN5TsbyL7/01bZkQS0NL42ubKoMn1XZvcs65MHvUzY1NO7fDZ5vk8Sj+LjWzZGCz9PN5u8o/su/H+WOrSPGwrcIu9mB4CvpVepwPnAnNL/kOrhmeBvpI+lI6/CPxP2n8T6N7CPXtIGgogqXuOHz5XSDpQUgPwuSrUu95V87O9H/hS4wgmSbtWuH4JMEhSQ+oqG9aGMm0rcGAvpulk/c2PRcQK4G3a3g3TrNRlchZwm6R5wGbg5+n0BOAPTX88jYh3yPrRr5X0FPAAf98Kb+pisq/0fwKWV+8d1K2qfbYRMQWYDMySNJfsH4xyHgEWA/OAHwBz2lKutT8/eWpmVjBusZuZFYwDu5lZwTiwm5kVjAO7mVnBOLCbmRWMA7vVPEmbJM2VNF/Sba2ZObKZvK5vnEdG0q8kDSxz7XBJh7ehjGbna8kzj0vp3Dg5y/qOpErDFG0b48Bu9WB9RAyKiIPJHlU/t/RkW2cTjIh/TtMPtGQ40OrAbtbRHNit3kwHPpRa0w9KugmYJ6mTpH+XNFPS05K+DKDMTyQtlHQP0KsxI5XMOy5ppKQ5aTbCqZL6kv0D8vX0beEoSbtLuiOVMVPSEene3STdr2ymy1+QYzoDSf8tabakBZLOaXLuP1JdpkraPaXtr2z2y9mSpkv6cFX+mlZInr/B6kaafuB4YEpKGgYcHBGLU3B8PSKGSuoCPCLpfrKZKQ8gmzWyN9kEY79uku/uwC+Bo1Neu0bEq5J+DqyNiB+k624CromIhyXtA9wHHEg2++HDEXGlpM8AWwTqFnwplbEDMFPSHRGxmmxWxDkR8U1Jl6e8v0r2NO+5EfGCpMOA64BPteHPaNsAB3arBzukR94ha7H/J1kXyYyIWJzSPw18RO/Pw74L0J9sxsmb0wRVy5TmMG/i48BDjXk1meWw1AhgoPReg3znNNPi0cA/pXvvkbQmx3saK6lx7pu9U11Xk03NcGtKvwH4naRu6f3eVlJ2lxxl2DbKgd3qwfqIGFSakAJc6eyOAr4WEfc1ue4EWp7BsPTePHNrNACfiIj1pYmpLrnn5kiLU4xIeb0laRotz5kTqdzXmv4NzFriPnYrivuA89LCE0gaIGknslkQR6c++D5k09w29RjwSUn90r2Nsxw2naXyfrJuEdJ1g9LuQ8BpKe14oEeFuu4CrElB/cNk3xgaNQCN3zq+QNbF8wawWNLJqQxJOrRCGbYNc2C3ovgVWf/5HGULN/+C7BvpncALZDMS/oz3pxZ+T0S8QtYv/rs062RjV8jvgc81/ngKjAWGpB9nF/L+6JzvAkdLmkPWJfRShbpOATpLehr4HvB4ybl1wEGSZpP1oV+Z0k8Dzk71WwCMyvE3sW2UZ3c0MysYt9jNzArGgd3MrGAc2M3MCsaB3cysYBzYzcwKxoHdzKxgHNjNzArGgd3MrGD+PyTx9WaQ3wJDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(model, \n",
    "                      X_test, \n",
    "                      y_test['Churn'], \n",
    "                      display_labels=[\"will not churn\", \"will churn\"],\n",
    "                      cmap='Blues')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True Negatives =  813\n",
      "False Negatives =  44\n",
      "False Positives =  42\n",
      "True Positives =  101\n"
     ]
    }
   ],
   "source": [
    "TN = confusion_matrix_[0,0]\n",
    "print(\"True Negatives = \", TN)\n",
    "\n",
    "FN = confusion_matrix_[1,0]\n",
    "print(\"False Negatives = \", FN)\n",
    "\n",
    "FP = confusion_matrix_[0,1]\n",
    "print(\"False Positives = \", FP)\n",
    "\n",
    "TP = confusion_matrix_[1,1]\n",
    "print(\"True Positives = \", TP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculated __________________________\n",
      "Accuracy:  0.914\n",
      "\n",
      "Scikit-Learn ________________________\n",
      "Accuracy:  0.914\n"
     ]
    }
   ],
   "source": [
    "print(\"Calculated __________________________\")\n",
    "# calculate the accuracy of the model from the confusion matrix\n",
    "accuracy = (TP+TN)/(TP+FP+FN+TN)\n",
    "print('Accuracy: ', accuracy)\n",
    "\n",
    "print(\"\\nScikit-Learn ________________________\")\n",
    "# calculate the accuracy of the model using the sklearn built-in function\n",
    "accuracy_ = accuracy_score(y_test['Churn'], \n",
    "                           y_test['predicted_Churn'])\n",
    "\n",
    "print('Accuracy: ', accuracy_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Precision, Recall/Sensitivity/True Positive Rate, F1 score\n",
    "\n",
    "**precision:**\n",
    "Precision can be seen as a measure of a classifier’s exactness. For each class, it is defined as the ratio of true positives to the sum of true and false positives. Said another way, “for all instances classified positive, what percent was correct?”\n",
    "\n",
    "**recall:**\n",
    "Recall is a measure of the classifier’s completeness; the ability of a classifier to correctly find all positive instances. For each class, it is defined as the ratio of true positives to the sum of true positives and false negatives. Said another way, “for all instances that were actually positive, what percent was classified correctly?”\n",
    "\n",
    "**f1 score:**\n",
    "The F1 score is a weighted harmonic mean of precision and recall such that the best score is 1.0 and the worst is 0.0. Generally speaking, F1 scores are lower than accuracy measures as they embed precision and recall into their computation. As a rule of thumb, the weighted average of F1 should be used to compare classifier models, not global accuracy.\n",
    "\n",
    "**support:**\n",
    "Support is the number of actual occurrences of the class in the specified dataset. Imbalanced support in the training data may indicate structural weaknesses in the reported scores of the classifier and could indicate the need for stratified sampling or rebalancing. Support doesn’t change between models but instead diagnoses the evaluation process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculated __________________________\n",
      "Precision =  0.71\n",
      "Recall =  0.7\n",
      "f1 score =  0.7\n",
      "\n",
      "Scikit-Learn ________________________\n",
      "Precision =  0.71\n",
      "Recall =  0.7\n",
      "f1 score =  0.7\n"
     ]
    }
   ],
   "source": [
    "print(\"Calculated __________________________\")\n",
    "precision = np.round(TP/(TP+FP), 2)\n",
    "recall = np.round(TP/(TP+FN), 2)\n",
    "f1_score_ = np.round((2*precision*recall)/(precision+recall), 2)\n",
    "\n",
    "print(\"Precision = \", precision)\n",
    "print(\"Recall = \", recall)\n",
    "print(\"f1 score = \", f1_score_)\n",
    "\n",
    "print(\"\\nScikit-Learn ________________________\")\n",
    "precision_ = np.round(precision_score(y_test['Churn'], \n",
    "                                      y_test['predicted_Churn']), 2)\n",
    "recall_ = np.round(recall_score(y_test['Churn'], \n",
    "                                y_test['predicted_Churn']), 2)\n",
    "f1_score__ = np.round(f1_score(y_test['Churn'], \n",
    "                               y_test['predicted_Churn']), 2)\n",
    "\n",
    "print(\"Precision = \", precision_)\n",
    "print(\"Recall = \", recall_)\n",
    "print(\"f1 score = \", f1_score__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### True Positive Rate, False Positive Rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True Positive Rate =  0.7\n",
      "False Positive Rate =  0.05\n"
     ]
    }
   ],
   "source": [
    "TPR = np.round(TP/(TP+FN), 2)\n",
    "FPR = np.round(FP/(FP+TN), 2)\n",
    "\n",
    "print(\"True Positive Rate = \", TPR)\n",
    "print(\"False Positive Rate = \", FPR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ROC: Receiver Operating Characterisitcs and AUC: Area Under the Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAHwCAYAAADuJ7gwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABcmklEQVR4nO3dd3zV1f3H8deHkBD2BoEAQURRFBDD3taBq9aF4tZaa61abbVaR21Lh23dv1pH1eIWxV2tqxqWIEsUBFTUBMLeIxCyzu+Pc7nkQiA34+Z77837+XjwIPd+v7n3A1fhzfmcYc45RERERKR21Qu6ABEREZG6SCFMREREJAAKYSIiIiIBUAgTERERCYBCmIiIiEgAFMJEREREAqAQJiIiIhIAhTARiRtmlmNmO81su5mtNrMJZtZkr3uGmNlHZrbNzLaY2VtmdsRe9zQzs/vNbFnotZaGHrep3V+RiMj+KYSJSLw5zTnXBOgLHA38ZvcFMxsMvA+8AXQEugGfA9PN7ODQPWnA/4BewBigGTAE2AAMiFXRZlY/Vq8tIslJIUxE4pJzbjXwHj6M7fY34Gnn3APOuW3OuY3OuduBmcDvQvdcDHQBznDOLXLOlTrn1jrnxjvn3invvcysl5l9YGYbzWyNmd0aen6Cmf2xzH2jzCyvzOMcM7vZzL4A8s3sdjObtNdrP2BmD4a+bm5mT5jZKjNbYWZ/NLOU6v1OiUiiUggTkbhkZhnAScDS0ONG+BGtl8u5/SXg+NDXxwHvOue2R/k+TYEPgXfxo2uH4EfSojUOOAVoATwDnGxmzUKvnQKMBZ4P3fsUUBx6j6OBE4ArKvFeIpJEFMJEJN68bmbbgOXAWuDO0POt8H9mrSrne1YBu+d7td7PPftzKrDaOXePc64gNML2aSW+/0Hn3HLn3E7nXC4wD/hR6NqxwA7n3Ewza48Pldc75/Kdc2uB+4DzKvFeIpJEFMJEJN78yDnXFBgF9GRPuNoElAIdyvmeDsD60Ncb9nPP/nQGvq1Spd7yvR4/jx8dAzifPaNgXYFUYJWZbTazzcCjQLtqvLeIJDCFMBGJS865ycAE4O7Q43xgBnBOObePZU8L8UPgRDNrHOVbLQe67+daPtCozOODyit1r8cvA6NC7dQz2BPClgO7gDbOuRahH82cc72irFNEkoxCmIjEs/uB482sb+jxLcAlZnadmTU1s5ahifODgd+H7nkGH3heMbOeZlbPzFqb2a1mdnI57/Ef4CAzu97MGoRed2Do2nz8HK9WZnYQcH1FBTvn1gHZwL+B751zi0PPr8Kv7LwntIVGPTPrbmYjK/l7IiJJQiFMROJWKNA8DdwRejwNOBE4Ez/vKxc/wX2Yc+6b0D278JPzlwAfAFuBWfi25j5zvZxz2/CT+k8DVgPfAKNDl5/Bb4GRgw9QE6Ms/flQDc/v9fzFQBqwCN9enUTlWqcikkTMub1H0kVEREQk1jQSJiIiIhIAhTARERGRACiEiYiIiARAIUxEREQkAAphIiIiIgGoH3QBldWmTRuXmZkZdBkiIiIiFZo7d+5651zb8q4lXAjLzMxkzpw5QZchIiIiUiEzy93fNbUjRURERAKgECYiIiISAIUwERERkQAohImIiIgEQCFMREREJAAKYSIiIiIBUAgTERERCYBCmIiIiEgAFMJEREREAqAQJiIiIhIAhTARERGRACiEiYiIiARAIUxEREQkAAphIiIiIgGIWQgzsyfNbK2ZLdzPdTOzB81sqZl9YWb9YlWLiIiISLyJ5UjYBGDMAa6fBPQI/bgSeDiGtYiIiIjElZiFMOfcFGDjAW45HXjaeTOBFmbWIVb1iIiIiADsKNrB+9/8j7e/fjvQOuoH+N6dgOVlHueFnlsVTDkiIiKSjPIL85mRN4PsnGw+/nYqM5/6IaUbDubIa37HKYeeElhdQYYwK+c5V+6NZlfiW5Z06dIlljWJiIhIgtteuJ1Pln/C5JzJZOdmM2vFLIpLi2F7O5j0IuSMhnrF9CldRqkrpZ4Fs04xyBCWB3Qu8zgDWFnejc65x4DHALKyssoNaiIiIlI3bS/czvRl08nOyWZy7mRmr5ztQ1dIPavHYTsuY8W/72P7hua0P6iUSS/XZ9iwXwRYdbAh7E3gGjN7ERgIbHHOqRUpIiIiB7Rt1zamL/ehKzsnmzkr51DiSsLX61k9+nfsz8iuIxnZdRSL3j6W28Y3pLgYhg+HiRPr0SEOZqHHLISZ2QvAKKCNmeUBdwKpAM65R4B3gJOBpcAO4LJY1SIiIiKJa+uurUxbNi3cXpy7cm5E6EqxFAZ0GsCorqMYmTmSYV2G0axBMwAWLoTTb4LSUrjhBvjrXyE1NahfSaSYhTDn3LgKrjvg57F6fxEREUlMWwq2+NCVO5nsnGzmrppLqSsNX0+xFAZ2GsiozFGMyhzF0M5DadqgabmvdeSRcO+90KEDjB1bW7+C6ATZjhQRERFhc8Fmpi2bFp7TNW/VvIjQVb9efQZ2GsjIriMZlTmKIZ2H7Dd0Abz+OjRuDMcf7x//ItipX/ulECYiIiK1anPBZqbmTvVzunKzmb96/j6ha1DGoIjQ1SStSYWvW1wMd9wBd90FrVvDl19C+/ax/JVUj0KYiIiIxNTGnRuZmjs13F6cv3o+rsyuVKn1UhmcMZhRmaMY2XUkQzoPoXFa40q9x9q1MG4cfPQRpKTArbdCu3Y1/SupWQphIiIiUqM27tzIlNwp4fbi56s/3yd0lR3pGtx5MI1SG1X5/T79FM4+G/Ly/MjXxIkwcmRN/EpiSyFMREREqmXDjg3h0JWdm82CNQsiQldaSlpE6BqUMahaoausp5+GK66AoiIYMgRefhk6dqyRl445hTARERGplHX565iSOyXcXlywdkHE9QYpDRiUMSjcXhyUMYiGqQ1jUktmpt9+4rrr4O9/h7S0mLxNTCiEiYiIyAGtzV8b0V5cuHZhxPUGKQ0Y0nlIeKRrYMZA0uunx6yeLVugeXP/9YgRsGgRHHpozN4uZhTCREREJMKa7Wsi2ouL1i2KuJ5ePz0idA3oNCCmoaust96CSy6BZ5+Fk0/2zyViAAOFMBERkTpv9fbVTM6ZHG4vLl6/OOJ6w/oNGdJ5SLi9OKDTABrUb1CrNZaUwJ13wp/+5B9PmrQnhCUqhTAREZE6ZtW2VeHANTl3MkvWL4m43ii1EUM7Dw2PdPXv1J+0lOAmW61fD+efDx98APXqwV/+AjfdFFg5NUYhTEREJMmt3LbSn7sYai9+veHriOuNUhsxrMuwcOjK6pgVaOgqa/Zsv/3EsmXQti28+CIce2zQVdUMhTAREZEks2LrivAoV3ZONt9s/CbieuPUxgzrMizcXszqmEVqSpycal1GURGce64PYAMH+hZkRkbQVdUchTAREZEEt3zL8oj24tKNSyOuN0lr4kNXV3/gdb8O/eIydO0tNRWeecaPft19NzSo3WloMacQJiIikmCWbVkW0V78btN3EdebpjVleNfh4fZivw79qF8vMf7K//57eP99+OlP/eOhQ/2PZJQYn4iIiEgdlrs5N6K9+P3m7yOuN2vQjOFdhofbi0d3ODphQldZ77wDF14ImzZBly5w0klBVxRbifcJiYiIJLmczTl+lCsUvHI250Rcb96gOcO7Dg+3F/se1JeUeinBFFsDSkrgD3+A8ePBOTj1VBg0KOiqYk8hTEREJEDOOb7f/L1vL+b64LVsy7KIe1qkt2BE1xHh9mKf9n0SOnSVtXEjXHABvPsumMEf/wi/+Y3fiiLZKYSJiIjUIucc3236LqK9uHzr8oh7Wqa3ZETXEeH2Yu/2vZMmdJX15Zd+1CsnB1q3huefhxNOCLqq2qMQJiIiEkPOOb7d9G1EezFva17EPa0atvKhK9RePKr9UdSz5B8KatsWioshK8tvP9G1a9AV1S6FMBERkRrknGPpxqXhlYvZOdms3LYy4p7WDVszMnNkuL14ZLsj60ToAigogPr1/Y927eB///OT8NNr5+jJuKIQJiIiUg3OOb7e8HVEe3HV9lUR97Rp1CYcuEZ2HUmvdr3qTOgqKzcXzjoLjjsO7rrLP5eoh2/XBIUwERGRSnDO8dWGryLai6u3r464p22jtozMHBluLx7e9vA6GbrKeu89f/7jxo1+C4rbboOmTYOuKlgKYSIiIgfgnGPJ+iXh9uLknMmsyV8TcU+7xu3Co1yjMkdxeJvDMbOAKo4vpaXwpz/BnXf67SdOPhmefVYBDBTCREREIjjnWLRuUXiUa3LuZNbmr424p33j9ozKHBUOXj3b9FToKsemTXDRRfD22377id//Hm6/vW5sPxENhTAREanTSl1pOHRl52QzJXcK63asi7inQ5MOEe3FQ1sfqtAVhZtu8gGsZUu//cSYMUFXFF8UwkREpE4pdaV8ufbLiPbihp0bIu7p2LRjRHuxR6seCl1VcNddsGED3HsvdOsWdDXxRyFMRESSWqkrZcGaBRHtxY07N0bc06lpp4j24iGtDlHoqoJdu+Chh+DaayE1Fdq0gddeC7qq+KUQJiIiSaXUlfLFmi8i2oubCjZF3JPRLMOHrlB78eCWByt0VdOyZXDOOTBrFqxeDX/7W9AVxT+FMBERSWglpSV7QlduNlNzp+4Tujo368zobqPD7cVuLbopdNWgDz+EceNg/Xq/6/3YsUFXlBgUwkREJKGUlJYwf/X8cHtxSu4UtuzaEnFP1+ZdI9qLmS0yFbpioLTUz/u64w7/9YknwnPP+XMgpWIKYSIiEteKS4vDoSs7J5upy6ayddfWiHu6tegWXr04MtOHLomt/Hy/+eqbb/rHd9zh9wJLSb5zxmNGIUxEROJKcWkxn636LNxenLZs2j6h6+CWB4cD18iuI+naoo6d/BwHGjaEoiJo0QKeeQZOPTXoihKPQpiIiASqqKSIeavmhc9dnLZsGtsKt0Xc071l94j2YufmnQOqVgoK/GHb9er5ne83bYLu3YOuKjEphImISK0qKili7qq54fbi9OXT2V64PeKeHq167DnwOnMkGc0yAqpWdisshF/9Cj77DD76CNLSoFUr/0OqRiFMRERiqqikiDkr54Tbi9OXTSe/KD/inkNbHxrRXuzUrFNA1Up5Vqzw20/MmOH3/5o1C4YNC7qqxKcQJiIiNaqwpJDZK2aH24vTl09nR9GOiHsOa31YuL04ousIOjbtGFC1UpGPP4bzzoO1ayEjAyZNgoEDg64qOSiEiYhItewq3sXslbPDW0ZMXzadncU7I+45vM3h4fbiiK4j6NC0Q0DVSrScg7vvhltu8dtP/OAH8MIL0LZt0JUlD4UwERGplF3Fu5i1Yla4vThj+Yx9QtcRbY+IaC+2b9I+oGqlqiZNgl//2n/9m9/A+PHafqKmKYSJiMgBFRQX8Gnep+H24oy8GRQUF0Tc06ttr4j2YrvG7QKqVmrKWWfBhRfC2WfD6acHXU1yUggTEZEIBcUFzMybGW4vzlg+g10luyLuObLdkeFzF0d0HUHbxupRJYOXX4bBg/3cr3r1/P5fEjsKYSIiddzOop3h0JWdm82neZ/uE7p6t+8dMaerTaM2AVUrsVBU5FuP99/vJ91PnepXQUpsKYSJiNQxO4p2MGP5jHB78dMVn1JYUhi+bhh92vcJb4w6ousIWjfSYYDJatUqf+D2tGk+eF10EdRXOqgV+m0WEUly+YX5zMibEW4vfpr3KUWlReHrhtH3oL7h9uLwrsNp1VA7cNYFU6b4ALZmDXTqtKcdKbVDIUxEJMnkF+bzyfJPwu3F2Stm7xO6+nXoF24vDu8ynJYNWwZYsQThgQf8DvglJTB6NLz4IrTTeopapRAmIpLgthduZ/qy6eH24uyVsykuLQ5fr2f1OKbDMeH24vCuw2mR3iK4giUuFBT4AHbzzfDHP6oFGQT9louIJJhtu7Yxffn0cHtxzso5+4SurI5Z4fbisC7DaJ7ePMCKJV4UFe2ZcP/rX/ujh4YODbamukwhTEQkzm3dtZXpy6aH24tzV86lxJWEr6dYCgM6DQi3F4d2HqrQJfuYNAluugmys6FrVzBTAAuaQpiISJzZUrCFacumhduLc1fNpdSVhq+nWAoDOw0MtxeHdhlKswbNAqxY4llxsT966J57/OMJE+DOOwMtSUIUwkREAra5YDPTlk0LtxfnrZoXEbrq16vPgE4Dwu3FIZ2H0LRB0wArlkSxerU/fHvyZD/n65574Nprg65KdlMIExGpZZsLNjM1d2q4vTh/9fx9QtegjEHh9uKQzkNoktYkwIolEU2fDuec4/cBO+ggv/3EsGFBVyVlKYSJiMTYpp2bmJI7JdxenL96Pg4Xvp5aL5XBGYPD7cUhnYfQOK1xgBVLoluzBo47zq+AHD4cJk6EDh2Crkr2phAmIlLDNu7cyJTcKeH24uerP98ndA3MGBhuLw7uPJhGqY0CrFiSTfv2ftuJlSvhrrt0BFG8UggTEammDTs2hENXdm42C9YsiAhdaSlpEe3FQRmDFLqkxn39NaxY4TdeBb8Rq8Q3hTARkUpal78uor24YO2CiOsNUhowKGNQuL04KGMQDVMbBlSt1AWvvQaXXOIn38+dC926BV2RREMhTESkAmvz10a0FxeuXRhxvUFKAwZ3HhxuLw7MGEh6/fSAqpW6pLgYbr8d/vpX//jss6FNm2BrkugphImI7GVt/lom50wOtxcXrVsUcT29fjpDOg8JtxcHdBqg0CW1bu1av/3Exx9DSgr87W9www1+E1ZJDAphIlLnrdm+JtxazM7JZvH6xRHXG9ZvyJDOQ8LtxQGdBtCgfoOAqhWBTz+Fs87yc8Dat4eXXoIRI4KuSipLIUxE6pxV21YxOXeyH+3KzWbJ+iUR1xulNvKhK9Re7N+pP2kpaQFVK7KvoiK/DcXQoT6AdewYdEVSFQphIpL0Vm5bGW4vTs6dzFcbvoq43ii1EcO6DAu3F7M6Zil0SdwpKfFtR/Cbrn74IQwZou0nEplCmIgknRVbV0S0F7/Z+E3E9capjRnWZVi4vZjVMYvUFP1NJvFr6VI/6X78eDjtNP/cyJHB1iTVpxAmIgkvb2ueH+UKtReXblwacb1JWhMfukLtxX4d+il0ScJ48024+GLYssVvwHrqqZp8nywUwkQk4Szbsiyivfjtpm8jrjdNaxoe6doduurX0x93klhKSuC3v4U//9k/PuMMmDBBASyZ6E8lEYl7uZtzI9qL32/+PuJ6swbNGN5leHhO19EdjlbokoS2fj2MG+fnfdWr548euvFGBbBkoz+lRCTu5GzOCY9yZedkk7M5J+J68wbNGd51eLi92PegvqTUSwmmWJEa5hz88IcwYwa0bQsvvgjHHht0VRILCmEiEijnXDh0Zef6eV25W3Ij7mmR3oLhXYaH24t92vdR6JKkZQb33AO33ALPPQcZGUFXJLGiECYitco5x3ebvotoLy7fujzinpbpLRnRdUS4vdi7fW+FLklqO3fC22/7FZAAgwdDdrbaj8lOIUxEYso5x7ebvo1oL+ZtzYu4p1XDVozoOiLcXjyq/VHUs3oBVSxSu777zu9+P38+vPqqn4APCmB1gUKYiNQo5xxLNy6NaC+u2LYi4p7WDVv70BVqLx7Z7kiFLqmT3n4bLrwQNm+GQw6B7t2Drkhqk0KYiFSLc46vN3wd0V5ctX1VxD1tGrVhZNeR4fZir3a9FLqkTispgT/8wf8APxH/qaegRYtAy5JaphAmIpXinOOrDV9FtBdXb18dcU/bRm0ZmTky3F48vO3hCl0iIRs3wgUXwLvv+u0n/vhHuPlm/7XULTENYWY2BngASAEed87dtdf15sCzQJdQLXc75/4dy5pEpHKccyxZvySivbgmf03EPe0atwuPco3KHMXhbQ7HNKFFpFylpbBoEbRpAy+8AMcdF3RFEpSYhTAzSwEeAo4H8oDZZvamc25Rmdt+Dixyzp1mZm2Br8zsOedcYazqEpEDc86xaN2i8CjX5NzJrM1fG3FP+8btw+cujsocRc82PRW6RCpQWupHu9q08UcRtWwJXboEXZUEKZYjYQOApc657wDM7EXgdKBsCHNAU/N/ejcBNgLFMaxJRPZS6kpZtG5ROHBNzpnMuh3rIu45qMlBfpQr1F48tPWhCl0iUSoogGuvhebN4e67/XN9+gRbk8SHWIawTkDZzX/ygIF73fMP4E1gJdAUONc5VxrDmkTqvFJXypdrvwy3F6fkTmH9jvUR93Ro0iHcWhyVOYoerXoodIlUQU6O3/tr7lxIT4frrtPol+wRyxBW3p/Ybq/HJwLzgWOB7sAHZjbVObc14oXMrgSuBOii/3pFKqXUlbJgzYKI9uLGnRsj7unUtFNEe/GQVocodIlU03vvwfnn+4n43br5PcD0V5iUFcsQlgd0LvM4Az/iVdZlwF3OOQcsNbPvgZ7ArLI3OeceAx4DyMrK2jvIiUgZpa6UL9Z8EQ5cU3Kn7BO6MpplhNuLIzNH0r1ld4UukRpSWgp/+hPceac/B/KUU+CZZ/wcMJGyYhnCZgM9zKwbsAI4Dzh/r3uWAT8ApppZe+Aw4LsY1iSSdEpKS8KhKzs3m6m5U9lUsCnins7NOke0F7u16KbQJRIjf/4z/Pa3fsf78ePh1lu1/YSUL2YhzDlXbGbXAO/ht6h40jn3pZldFbr+CDAemGBmC/Dty5udc+v3+6IiQklpCfNXzw+3F6fkTmHLri0R93Rp3oXRmaPD7cXMFpkKXSK15Oqr4fXX/WjYiScGXY3EM/OdwMSRlZXl5syZE3QZIrWmuLSY+avnh9uLU3On7hO6MltkRrQXM1tkBlOsSB311lswZgykpvrHzunsR/HMbK5zLqu8a9oxXyTOFJcW89mqz8LtxWnLprF1V8RaFQ5ueXB4lGtk15F0bdE1oGpF6rZdu+AXv4BHH/U/33+/f14BTKKhECYSsOLSYuaunBtuL05bNo1thdsi7unesns4cI3MHEmX5lpiJRK0Zcv89hOzZ0ODBnDUUUFXJIlGIUyklhWVFDF31dxwe3HasmlsL9wecc8hrQ4Jb4w6MnMkGc0yAqpWRMrz4Ydw3nmwYQNkZsKkSXDMMUFXJYlGIUwkxopKipizck64vTh92XTyi/Ij7jm09aER7cVOzToFVK2IHEhpKdx1F9xxh/96zBh49llo3TroyiQRKYSJ1LDCkkJmr5gdbi9OXz6dHUU7Iu45rPVhEe3Fjk07BlStiFTWjBk+gN15pw9jKSlBVySJSiFMpJp2Fe9i9srZ4fbi9GXT2Vm8M+Kenm16htuLI7qOoEPTDgFVKyJVsXu1Y716fuPVWbPghBOCrkoSnUKYSCXtKt7FrBWzwqHrk+Wf7BO6jmh7RLi9OKLrCA5qclBA1YpIdT37LEyYAO+8A2lp0KKFApjUDIUwkQoUFBfwad6n4fbijLwZFBQXRNzTq22vcHtxRNcRtG/SPqBqRaSmFBbCL38JDz3kH7/0Elx4YbA1SXJRCBPZS0FxATPzZoZHumYsn8Gukl0R9xzZ7siI9mLbxm0DqlZEYiEvD845B2bO9KNfDz4IF1wQdFWSbBTCpM7bWbQzInTNzJu5T+jq3b53RHuxTaM2AVUrIrH20Ud++4l166BzZ3jlFejfP+iqJBkphEmds6NoBzOWzwi3Fz9d8SmFJYUR9/Rp3yeivdi6kdafi9QFn34Kxx/vVz8edxy88AK00b+5JEYUwiTp7SjawSfLPwmPdH2a9ylFpUXh64bR96C+4fbi8K7DadWwVYAVi0hQBgyA00+HI46A3/9e209IbCmESdLJL8yPCF2zVszaJ3T169Av3F4c3mU4LRu2DLBiEQnSl19C06bQpYvfhmLSJL8VhUisKYRJwtteuJ3py6aH24uzV86muLQ4fL2e1eOYDseE24vDuw6nRXqL4AoWkbjx4ovw4x/7ka+pUyE9XQFMao9CmCScbbu2MX35dCbnTCY7N5s5K+fsE7qyOmYxqqs/d3FYl2EKXSISobAQbrrJr3oE6NnTzwMTqU0KYRL3tu7ayvRl08PtxTkr51DiSsLXUyyF/h37MyrTz+ka2nkozdObB1ixiMSzlSth7FiYPh1SU+G+++Dqq30rUqQ2KYRJ3Nm6ayvTlk3zB17nZDNv1bx9QtfATgPD7cWhXYbSrEGzACsWkUQxeTKcey6sWQOdOvn5X4MGBV2V1FUKYRK4LQVbmLpsari9OG/VPErdnr5AiqUwKGNQuL04tPNQmjZoGmDFIpKo5szxAezYY/32E+3aBV2R1GUKYVLrNhdsZmru1HB78bPVn0WErvr16odHukZljmJI5yE0SWsSYMUiksh2H74N/hii9u39Zqz19TegBEz/CUrMbdq5ianLpobbi/NXz8fhwtdT66UyOGNweMuIIZ2H0DitcYAVi0iyWLwYfvITeOYZ6NbNhzGd/yjxQiFMatzGnRuZkjsl3F78fPXn+4SugRkDw+3FwRmDFbpEpMa9/DJcfjls3w533AHPPht0RSKRFMKk2jbs2MCU3Cnh9uIXa76ICF1pKWkR7cVBGYNolNoowIpFJJkVFcEtt8C99/rH48bBo48GW5NIeRTCpNLW71gfDl3ZOdksWLsg4npaSlpEe3FQxiAapjYMqFoRqUtWr/bbT0yd6ud83XMPXHuttp+Q+KQQJhVal7+OybmTw+3FhWsXRlxvkNKAwZ0Hh9uLAzsNVOgSkVq3cycMHAjLlkGHDr4dOXRo0FWJ7J9CmOxjbf5aH7hC7cUv130ZcT29fjqDMwaH24sDOg0gvX56QNWKiHgNG/rVj6+95o8jOuigoCsSOTBzzlV8VxzJyspyc+bMCbqMpLJm+5rwuYvZOdksXr844nrD+g0Z0nlIuL04oNMAGtRvEFC1IiJ7bN/uV0D27+8fOwclJdp+QuKHmc11zmWVd03/mdZBq7atimgvLlm/JOJ6w/oNGdplaLi92L9jf4UuEYk7X30FZ54JK1bA3LnQvbuf+6UAJolC/6nWASu3rYxoL3614auI641SGzG089BwezGrYxZpKWkBVSsiUrFXX4VLL4Vt2+Dww6G4OOiKRCpPISwJrdi6IqK9+M3GbyKuN05tzLAuw8LtxayOWaSmpAZUrYhI9IqL4bbb4G9/84/POQeeeAKa6iQzSUAKYUkgb2ueH+UKtReXblwacb1JWhOGdRkWbi8e0+EYhS4RSThr1/rjhj7+GFJSfBC74QZtPyGJSyEsAS3fsjw8yjU5dzLfbvo24nrTtKY+dIXai/069KN+PX3UIpLYvv8epk3zqx4nToQRI4KuSKR69DdzAsjdnBvRXvx+8/cR15s1aMbwLsPD7cWjOxyt0CUiSWfgQB++Bg3y+4CJJDr9TR2HcjbnhEe5snOyydmcE3G9WYNmjOg6Itxe7HtQX4UuEUk6+flw1VVw1lnwox/55844I9CSRGqU/uYOmHMuHLqyc/28rtwtuRH3NG/Q3IeuUHuxT/s+pNRLCahiEZHYW7rUbz+xYAF89BGMGQPp2hNakoxCWAAKSwp5YcEL/O/7/5Gdk83yrcsjrrdIb8HIriPD7cXe7XsrdIlInfHmm3DxxbBlCxx6qN+OQgFMkpFCWAAemPkAv/7w1+HHLdNbMjJzZLi9eFS7oxS6RKTOKSmB3/4W/vxn//jMM+Hf/4ZmzYKtSyRWFMICsHvfrkv6XMIvB/+SI9sdST2rF3BVIiLBuuIKmDAB6tWDu+6CG2/U9hOS3PQ3fwDW5K8B4IeH/ZDe7XsrgImIAD/7GXTuDP/7H9x0kwKYJD+NhAVgzXYfwg5qclDAlYiIBMc5mDkTBg/2jwcM8BPy03RqmtQRGoIJwO6RsPaN2wdciYhIMHbs8Gc/DhkCkybteV4BTOoSjYTVMudceCSsfROFMBGpe7791u/99fnn0LChDt+WukshrJZtL9zOzuKdNEptRJO0JkGXIyJSq/7zH7joIti8GQ45xG8/cdRRQVclEgy1I2vZ6u2rAbUiRaRu2b39xGmn+QB2+ukwZ44CmNRtCmG1LDwfTK1IEalDtm2Dp5/220/8+c9+BKx586CrEgmW2pG1LDwfTCNhIlKHtGjhg9fGjXDccUFXIxIfFMJq2e6RMG1PISLJ7vHH4Ztv4K9/9Y/79Qu2HpF4oxBWyzQSJiLJbudOuOYaePJJ//jss6F//2BrEolHCmG1THPCRCSZ5eT47SfmzfOHbj/yiAKYyP4ohNUyrY4UkWT17rtwwQV+3tfBB8Mrr0DfvkFXJRK/tDqylmkkTESS0cSJcPLJPoCdcorffkIBTOTANBJWyzQnTESS0Qkn+M1XL74Ybr3Vb0UhIgemEFbLtDpSRJLFwoVw6KH+vMeWLfccQyQi0dG/VWrR9sLt7CjaQcP6DXVkkYgktH//20+4/+Uv9zynACZSOQphtajswd1mFnA1IiKVt2sX/PSncPnlUFAAhYVQWhp0VSKJSe3IWqSVkSKSyJYt83t+zZ4NDRrAP//pw5iIVI1CWC3SykgRSVQffADjxsGGDZCZ6bef0A74ItUTVQgzsyxgONAR2AksBD50zm2MYW1JRysjRSRRPfKID2BjxsBzz0GrVkFXJJL4DjgnzMwuNbN5wG+AhsBXwFpgGPCBmT1lZl1iX2Zy0MpIEUlUTz4J994Lb7+tACZSUyoaCWsMDHXO7Szvopn1BXoAy2q4rqSkkTARSRSffw5/+Qs89ZSf/9W8OdxwQ9BViSSXA4Yw59xD+7tmZo2dc/NrvKIkpjlhIpIInn7ar4AsKICjjoLbbgu6IpHkVOEWFWbWycyyzCwt9Lidmf0Z+Cbm1SUZrY4UkXi2axdcfTVccokPYJddFrkPmIjUrIrmhF0PzAf+D5hpZpcAi/Hzw46JdXHJRiNhIhKv8vJg5Eh4+GG/A/5jj8ETT2gDVpFYqmhO2JXAYc65jaEJ+EuBEc65mbEvLfloTpiIxKPcXL/7/bp10KULTJrkH4tIbFUUwgp2b0PhnFtmZl8rgFVNfmE++UX5pNdPp1mDZkGXIyIS1qULjBgBW7fC889DmzZBVyRSN1QUwjLM7MEyj9uVfeycuy42ZSWfcCuysY4sEpHgbdkC27ZBRgaY+cn4DRpASkrQlYnUHRWFsJv2ejw3VoUku/CkfM0HE5GALVwIZ54JTZvCtGl+3lejRkFXJVL3VLRFxVNm1hboCix1zm2ulaqSkOaDiUg8eOEFuOIK2LEDeveGjRuhU6egqxKpmypaHXkF8CV+deQSM/thrVSVhMq2I0VEalthIVx3HZx/vg9gF10EM2YogIkEqaJ25PVAL+fcOjM7GHgOeDPmVSWh8EiY2pEiUstWroRzzoFPPoHUVHjgAbjqKj8XTESCU1EIK3TOrQNwzn1nZg1qoaakpHMjRSQob7zhA1hGBrz8MgwaFHRFIgKVXx2ZUZnVkWY2BngASAEed87dVc49o4D7gVRgvXNuZFSVJxi1I0UkKFdd5befuOwyaNcu6GpEZLeYrY40sxTgIeB4IA+YbWZvOucWlbmnBfBPYExoH7Kk/eNBqyNFpLZs2wa/+IU/87F7d992vPnmoKsSkb1VFMIOc87dWsXXHoBfUfkdgJm9CJwOLCpzz/nAq865ZQDOubVVfK+4p9WRIlIbFi/2208sWQLffguTJwddkYjsT0UHeI+pxmt3ApaXeZwXeq6sQ4GWZpZtZnPN7OJqvF9c07mRIhJrL73kjxtasgSOPBIefzzoikTkQCoaCUsxs5ZAuWtodh9ptB/lfY8r5/2PAX6APxR8hpnNdM59HfFCZlfiz7GkS5cuFZQcf3YU7WB74XbSUtJo3qB50OWISJIpKvLtxvvu84/PP98fwN24cbB1iciBVRTCeuLnge0vUB18gO/NAzqXeZwBrCznnvXOuXwg38ymAH2AiBDmnHsMeAwgKytr7yAX93a3Ig9qcpCOLBKRGlVaCiedBP/7H9Sv74PYz3+u7SdEEkFFIWyRc+7oKr72bKCHmXUDVgDn4eeAlfUG8A8zqw+kAQOB+6r4fnFLKyNFJFbq1YMzzvBzwV5+GYYMCboiEYlWRXPCqsw5VwxcA7wHLAZecs59aWZXmdlVoXsWA+8CXwCz8NtYLIxVTUHRykgRqUnO+Un3u119NXz5pQKYSKKpaCTsgeq8uHPuHeCdvZ57ZK/Hfwf+Xp33iXdaGSkiNWX7dn/243//C3PmQI8evvXYokXQlYlIZVU0EjbUzI4q74KZNTazy83sghjUlVTUjhSRmrBkCQwcCBMn+rlgZUfDRCTxVDQS9hBwRyiILQTWAelAD6AZ8CT+PEk5AJ0bKSLV9corcOmlfiTs8MPh1VehZ8+gqxKR6jhgCHPOzQfGmlkTIAvoAOwEFjvnvop9eclB50aKSFUVF8NvfgN33+0fjx0LTzwBTZoEW5eIVF9FI2EAOOe2A9mxLSV5qR0pIlW1eDE8+CCkpPgg9otfaPsJkWQRVQiT6tHqSBGpqqOO8jvfZ2bC8OFBVyMiNUkhrBZodaSIRMs5eOgh6NABzjrLP3fRRcHWJCKxUakQZmaNQ7vbS5R2Fu1kW+E20lLSaJHeIuhyRCSO5efDlVfC889D06YwYgS0bRt0VSISK1Ft1mpmQ8xsEX7TVcysj5n9M6aVJYnd88HaNW6nI4tEZL+++QYGDfIBrHFj+Ne/FMBEkl20O+bfB5wIbABwzn0OjIhVUcmk7LmRIiLlef11yMqChQvhsMNg1iw499ygqxKRWIv62CLn3PK9niqp4VqSklZGisiB3HuvP/tx61Y480wfwI44IuiqRKQ2RBvClpvZEMCZWZqZ3UioNSkHFl4ZqRAmIuUYNcq3H//+d5g0CZo1C7oiEakt0U7Mvwp/jmQnIA94H7g6VkUlE+2WLyJ7W7YMunTxX/frB99/r/lfInVRtCNhhznnLnDOtXfOtXPOXQgcHsvCkoXakSKym3PwyCP+0O0XX9zzvAKYSN0UbQj7vyifk72EQ5hGwkTqtB07/NmPP/sZFBbC/PlBVyQiQTtgO9LMBgNDgLZm9ssyl5oBKbEsLFlodaSIfPutn3T/xRfQqJHffuL884OuSkSCVtGcsDSgSei+pmWe3wqcHauikonakSJ121tv+R3vt2zxbchXX4Ujjwy6KhGJBwcMYc65ycBkM5vgnMutpZqSis6NFKm7Cgvhhht8APvRj2DCBGjePOiqRCReRLs6coeZ/R3oBaTvftI5d2xMqkoSBcUFbN21ldR6qbRMbxl0OSJSy9LS/LYT778PN90EOjRDRMqKdmL+c8ASoBvweyAHmB2jmpLG7vlgOrJIpO6YPRvGj9/zuG9f+PWvFcBEZF/RhrDWzrkngCLn3GTn3OXAoBjWlRS0MlKk7nDOT7gfNgx++1v4z3+CrkhE4l207cii0M+rzOwUYCWQEZuSkodWRorUDTt3wjXXwJNP+sdXXw3HHx9sTSIS/6INYX80s+bAr/D7gzUDro9VUclCKyNFkt/338NZZ8Fnn0HDhvDoo341pIhIRaIKYc653QPrW4DRAGY2NFZFJQudGymS3GbNgjFjYNMm6N4dXnkF+vQJuioRSRQVbdaaAozFnxn5rnNuoZmdCtwKNASOjn2JiUvnRookt8MOg9at/Tywp5+GFi2CrkhEEklFI2FPAJ2BWcCDZpYLDAZucc69HuPaEp7akSLJZ+NGaNwYGjTwe35NnQrt2kG9aJc5iYiEVBTCsoDezrlSM0sH1gOHOOdWx760xKfVkSLJZd48P//rxBP9QdwAB2ndjYhUUUX/dit0zpUCOOcKgK8VwKKn1ZEiyePf/4YhQyAnB+bOhfz8oCsSkURX0UhYTzP7IvS1Ad1Djw1wzrneMa0uwakdKZL4Cgrguuv8HmAAV14JDzwA6ekH/j4RkYpUFMIOr5UqklBBcQGbCzZTv159WjbUkUUiiSg3F84+G+bM8aHrn/+Eyy4LuioRSRYVHeCtQ7uraG3+WsAfWVTPNGNXJBGNH+8DWLdufvuJo7UeXERqULSbtUolhbenUCtSJGHde69fBTl+PLRqFXQ1IpJsNEQTI1oZKZJ4Nm+GG2/088AAmjWDhx5SABOR2Ih6JMzMGgJdnHNfxbCepKGVkSKJ5fPP4cwz4bvvoKjIT74XEYmlqEbCzOw0YD7wbuhxXzN7M4Z1JTytjBRJHE8/DYMG+QDWrx9cf33QFYlIXRBtO/J3wABgM4Bzbj6QGYuCkoXOjRSJf7t2wdVXwyWX+Bbk5ZfD9Ol+Ir6ISKxF244sds5tMbOYFpNMNCdMJL5t3QrHH+8P4U5L83O/rrgi6KpEpC6JNoQtNLPzgRQz6wFcB3wSu7ISn1ZHisS3pk0hMxNWr/bbT2RlBV2RiNQ10bYjrwV6AbuA54EtwPUxqikpaCRMJP6UlsKGDf5rM3jiCX8EkQKYiAQh2pGww5xztwG3xbKYZKLVkSLxZcsWP/crJwc++QQaNYImTfwPEZEgRDsSdq+ZLTGz8WbWK6YVJYFdxbvYVLCJFEuhVUNtMCQStAUL/GjXG2/4ELZ4cdAViYhEGcKcc6OBUcA64DEzW2Bmt8eysESmI4tE4sdzz8HAgbB0KfTp49uPxxwTdFUiIpXYMd85t9o59yBwFX7PsN/GqqhEp/lgIsErLIRrr4ULL4SdO+Hii30bsnv3oCsTEfGi3az1cDP7nZktBP6BXxmZEdPKEphWRooE77XX4B//gNRUePhhmDDBzwMTEYkX0U7M/zfwAnCCc25lDOtJChoJEwne2LEwezacc45vR4qIxJuoQphzblCsC0km4ZWRjbUyUqS2OAf33w+nngo9evgtKO6+O+iqRET274AhzMxecs6NNbMFgCt7CXDOud4xrS5BaSRMpHZt3QqXXQavvgr//jfMmwf1ox3nFxEJSEV/TP0i9POpsS4kmejcSJHa8+WXcNZZ8NVX0KwZjB+vACYiieGAE/Odc6tCX17tnMst+wO4OvblJSaNhInUjokT/Xyvr76CI4+EOXPg9NODrkpEJDrRblFxfDnPnVSThSQTrY4Uib1bboHzzoP8fLjgApg5088FExFJFBXNCfsZfsTrYDP7osylpsD0WBaWyDQSJhJ7mZm+7Xj//XD11X4ivohIIqlo5sTzwH+BvwC3lHl+m3NuY8yqSmBFJUVs3LmRFEuhdcPWQZcjklQ2b4YWLfzXP/0pHHssHHpokBWJiFRdRe1I55zLAX4ObCvzAzPToYjl2H1kUdvGbUmplxJwNSLJwTm47z44+GBYssQ/Z6YAJiKJLZqRsFOBufgtKsoO+Dvg4BjVlbC0MlKkZm3bBldcAS+95B+/9x707BlsTSIiNeGAIcw5d2ro5261U07i03wwkZqzZAmceSYsXgxNm/qjh848M+iqRERqRrRnRw41s8ahry80s3vNrEtsS0tMWhkpUjMmTYL+/X0AO+IIfwSRApiIJJNot6h4GNhhZn2AXwO5wDMxqyqBhUfCFMJEqmzlSrjwQti+Hc49Fz79FA47LOiqRERqVrT7Shc755yZnQ484Jx7wswuiWVhiSp8bmQTnRspUlUdO8I//+nng113nbafEJHkFG0I22ZmvwEuAoabWQqQGruyEpfmhIlUzSefwLp1e3a8v/zyYOsREYm1aNuR5wK7gMudc6uBTsDfY1ZVAtPqSJHKcQ7+7/9g5Ei/8/033wRdkYhI7YgqhIWC13NAczM7FShwzj0d08oSlEbCRKK3+8ih666D4mK/AWtmZtBViYjUjmhXR44FZgHnAGOBT83s7FgWlqi0OlIkOl9/DYMGwQsvQOPG/jDue+6BVE10EJE6Ito5YbcB/Z1zawHMrC3wITApVoUloqKSIjbs3EA9q0ebRm2CLkckbr37rl/1uHWr33j11Vfh8MODrkpEpHZFOyes3u4AFrKhEt9bZ6zbsQ6Ato10ZJHIgRx0EBQWwtlnw6xZCmAiUjdFOxL2rpm9B7wQenwu8E5sSkpc4Vak5oOJ7CM/37cdAfr2hTlz/Cas2n5CROqqaCfm3wQ8CvQG+gCPOedujmVhiUgrI0XK9+mnvu343HN7nuvVSwFMROq2A4YwM+thZm+Y2UL8pPx7nHM3OOdeq53yEotWRopEcg4efhiGD4e8PHjqKf+ciIhUPBL2JPAf4CxgLvB/Ma8ogWllpMgeO3bAJZfA1VdDUZHfhuI//9Hol4jIbhXNCWvqnPtX6OuvzGxerAtKZDo3UsRbuhTOOgu++AIaNYLHH4dx44KuSkQkvlQUwtLN7Ghg979dG5Z97JxTKCtjdwjTuZFSlzkH55/vA9ihh8Irr8CRRwZdlYhI/KkohK0C7i3zeHWZxw44NhZFJSqtjhTx7cYnnoC77vKHcDdvHnRFIiLx6YAhzDk3ujovbmZjgAeAFOBx59xd+7mvPzATONc5l7AbwGp1pNRV69fDSy/5+V8ARx0VuRJSRET2Fe0+YZVmZinAQ8DxQB4w28zedM4tKue+vwLvxaqW2qLVkVIXzZ7tN11dtgyaNYMLLwy6IhGRxBDLXe8HAEudc9855wqBF4HTy7nvWuAVYG051xJGcWkxG3ZswDAdWSR1gnPw2GMwbJgPYAMHwqhRQVclIpI4YhnCOgHLyzzOCz0XZmadgDOAR2JYR61Yl78Oh6NNozbUrxezAUaRuLBzJ1x+Ofz0p/74oauvhsmTISMj6MpERBJHVGnBzAy4ADjYOfcHM+sCHOScm3Wgbyvnub23abwfuNk5V2IH2DzIzK4ErgTo0qVLNCXXOq2MlLoiLw9OOw3mz4eGDeHRR+Gii4KuSkQk8UQ7EvZPYDCwe6efbfj5XgeSB3Qu8zgDWLnXPVnAi2aWA5wN/NPMfrT3CznnHnPOZTnnstq2bRtlybVLKyOlrmjWzI+Ede8OM2cqgImIVFW0fbOBzrl+ZvYZgHNuk5mlVfA9s4EeZtYNWAGcB5xf9gbnXLfdX5vZBOA/zrnXo6wprmhlpCSz0lIoLoa0NB/C3n4bWreGFi2CrkxEJHFFOxJWFFrF6ADMrC1QeqBvcM4VA9fgVz0uBl5yzn1pZleZ2VXVqDkuabd8SVYbNsApp8DPf77nue7dFcBERKor2pGwB4HXgHZm9id86/D2ir7JOfcO8M5ez5U7Cd85d2mUtcQltSMlGc2d648fys31I1+//z107Bh0VSIiySGqEOace87M5gI/wE+4/5FzbnFMK0swGgmTZPPEE370a9cuyMryxw8pgImI1Jyo2pGh1ZA7gLeAN4H80HMSotWRkiwKCuAnP4ErrvAB7Kc/hWnTIE4XJouIJKxo25Fv4+eDGZAOdAO+AnrFqK6Eo3akJIs//hEefxzS0+Hhh+HSS4OuSEQkOUXbjjyq7GMz6wf8NCYVJSitjpRkccstfg+w8ePh6KODrkZEJHlVacd859w8oH8N15KwikuLWb9jPYbRtnF87mMmsj+lpfDII37vL4AmTeA//1EAExGJtWh3zP9lmYf1gH7AuphUlIDW71ivI4skIW3a5DdbffttmDULnnwy6IpEROqOaBND0zJfF+PniL1S8+UkpvB8MLUiJYHMn++3n/juO2jZEsaODboiEZG6pcIQFtqktYlz7qZaqCchaWWkJJqnnoKrrvIrIfv189tPZGYGXZWISN1ywDlhZlbfOVeCbz/KfmhlpCSK4mL42c/8iseCAvjxj2H6dAUwEZEgVDQSNgsfwOab2ZvAy0D+7ovOuVdjWFvC0MpISRQpKbB1KzRoAP/4h98LTEREghHtnLBWwAbgWPbsF+YAhTC0W77Ev8JCf/i2GTz2GNx0E/TtG3RVIiJ1W0UhrF1oZeRC9oSv3VzMqkow4RCmdqTEmdJSuOsuePllv+t948b+hwKYiEjwKgphKUATIsPXbgphIVodKfFo82a45BJ4803/+L334MwzAy1JRETKqCiErXLO/aFWKklgWh0p8eaLL/z2E0uXQosW8OyzcMopQVclIiJlVbRjfnkjYLIXrY6UePLsszBokA9gffrA3LkKYCIi8aiiEPaDWqkigZWUlrBuhz88oG0jHVkkwZo61e+Av3Onb0XOmAEHHxx0VSIiUp4DtiOdcxtrq5BEtX7HekpdKa0btiY1JTXocqSOGzYMrrzSb8B65ZV+NaSIiMQnHXRYTVoZKUH7+GPo2BEOO8yHrkcfDboiERGJRkXtSKmAVkZKUJyDv/0NjjvOr3rcvj3oikREpDI0ElZNWhkpQdi6FS67DF4NbZf8ox9Bw4aBliQiIpWkEFZNGgmT2vbll37k6+uvoVkzePppOP30oKsSEZHKUjuymsLnRmpOmNSCl16CAQN8ADvqKL/9hAKYiEhiUgirJp0bKbVp61bYsQMuvBBmzoRDDgm6IhERqSq1I6tJqyMl1oqLoX7o/9QrroBu3eDYY7X9hIhIotNIWDVpTpjE0pQpcPjhsGjRnud+8AMFMBGRZKAQVk1aHSmx4Bzce68f8Vq6FO67L+iKRESkpqkdWQ0lpSWsy/dHFrVr3C7gaiRZbNsGP/4xvPyyf/zrX8Of/hRsTSIiUvMUwqphw84NlLgSWjVspSOLpEYsXgxnneV/btoUJkzw21GIiEjyUQirBs0Hk5qUnw8jR8K6ddCrF7zyij+KSEREkpPmhFWDVkZKTWrc2B9DdN55fvsJBTARkeSmEFYNGgmT6lq9Gt5/f8/jSy+F55+HJk0CK0lERGqJQlg1aGWkVMe0adCvH5xxBixYsOd5bT8hIlI3KIRVg0bCpCqcgwcegNGjYdUqyMqCtm2DrkpERGqbQlg1rM7XuZFSOdu3w/nnw/XX+53wf/Ur+PBDOEiDqSIidY5WR1aDRsKkMr7+2m838eWXfs7Xk0/COecEXZWIiARFIawatDpSKqOgAL77Dnr2hFdf9ccRiYhI3aUQVg0aCZOKlJZCvVDTv3dv+O9//WT8pk2DrUtERIKnOWFVVOpKWZu/FtCRRVK+tWvhuOPgqaf2PDdypAKYiIh4Ggmroo07N1LiSmiZ3pIG9RsEXY7EmRkz/HyvFSsgJwfGjYO0tKCrEhGReKKRsCpavV0rI2VfzsFDD/kRrxUrYOhQvx+YApiIiOxNIayKNB9M9rZjB1x8MVxzDRQVwS9+AR9/DB07Bl2ZiIjEI7Ujq0grI2Vvl1wCkyZBo0bw+OO+BSkiIrI/CmFVpJEw2dvvfue3oHj6aejVK+hqREQk3qkdWUU6N1JKSuC11/Y87tUL5sxRABMRkegohFVRuB2pkbA6af16GDPG74D/5JN7ntfh2yIiEi21I6tIqyPrrlmz4OyzYflyaNcOunULuiIREUlEGgmrIs0Jq3ucg0cfheHDfQAbPBjmzYPRo4OuTEREEpFCWBVpdWTdsnMnXH45XHUVFBb6bSiys6FTp6ArExGRRKV2ZBXoyKK6p6QEPv0UGjaEf/0LLrgg6IpERCTRKYRVwaadmyguLaZFegvS66cHXY7EkHN+sn2TJvDqq34UrHfvoKsSEZFkoBBWBVoZmfxKSuAPf4C8PL/xqhn07Bl0VSIikkwUwqpAKyOT24YNvt343ntQrx5cey307Rt0VSIikmw0Mb8KtDIyec2dC8cc4wNYmzb+ZwUwERGJBYWwKlA7Mjk9/jgMHQq5uTBggN9+4rjjgq5KRESSlUJYFYRHwtSOTBpPPAE/+Qns2uW3oZgyBTp3DroqERFJZgphVaBzI5PPeedB//4wYQI8/DA0aBB0RSIikuw0Mb8KwhPz1Y5MaNnZvu3YqBE0bgwzZ/qJ+CIiIrVBf+VUgXbLT2ylpTB+PBx7LPz0p34vMFAAExGR2qWRsCrQ6sjEtWkTXHQRvP223/vr0EODrkhEROoqhbBKcs6FjyzSSFhi+ewzOOss+P57aNUKnn8eTjwx6KpERKSuUgOmkjYVbKKotIhmDZrpyKIEMmECDBniA9gxx/j9wBTAREQkSAphlbS7FamVkYnl44+hoACuuAKmTYPMzKArEhGRuk7tyErSysjEsfvwbfDbTpxyCowdG2xNIiIiu2kkrJK0MjIxfPghjB4N27f7x40aKYCJiEh8UQirJK2MjG+lpfDnP/v5XpMn+xEwERGReKR2ZCXp3Mj4tXkzXHwxvPWWb0PeeSf88pdBVyUiIlI+hbBK0rmR8enzz/32E99+Cy1awHPPwcknB12ViIjI/imEVZLOjYw/S5fC4MGwcyf07QuvvAIHHxx0VSIiIgemEFZJWh0Zf7p3h3Hj/Hywf/4TGjYMuiIREZGKKYRVklZHxoe8PNixwx87ZAaPPgopKXu2pBAREYl3MV0daWZjzOwrM1tqZreUc/0CM/si9OMTM+sTy3qqK+LIIo2EBeajj6BfPzj9dNi2zT9Xv74CmIiIJJaYhTAzSwEeAk4CjgDGmdkRe932PTDSOdcbGA88Fqt6asLmgs0UlhTSNK0pDVPV86ptzsFf/wrHHw/r1kFGBhQWBl2ViIhI1cRyJGwAsNQ5951zrhB4ETi97A3OuU+cc5tCD2cCGTGsp9rUigzOli1+9eMtt/i5X7fdBu++C61bB12ZiIhI1cRyTlgnYHmZx3nAwAPc/2PgvzGsp9p0bmQwFi6EM8+Eb76B5s3hmWfgtNOCrkpERKR6YhnCypuh48q90Ww0PoQN28/1K4ErAbp06VJT9VWaVkYGY+5cH8B69/bbTxxySNAViYiIVF8sQ1ge0LnM4wxg5d43mVlv4HHgJOfchvJeyDn3GKH5YllZWeUGudqg3fKDcckl/udzzvFnQIqIiCSDWM4Jmw30MLNuZpYGnAe8WfYGM+sCvApc5Jz7Ooa11Ajtll87Vq70Zz8uXLjnuUsuUQATEZHkErMQ5pwrBq4B3gMWAy855740s6vM7KrQbb8FWgP/NLP5ZjYnVvXUBI2Exd7kyX77ifffh+uuC7oaERGR2InpZq3OuXeAd/Z67pEyX18BXBHLGmqSVkfGjnNw771w881QUgKjR8OLLwZdlYiISOzEdLPWZKPVkbGxbRuMHQs33ugD2M03+5Gwdu2CrkxERCR2dGxRJWh1ZM0rLYVRo2DePGjaFJ56Cs44I+iqREREYk8jYVFyzqkdGQP16sENN0CvXjBnjgKYiIjUHQphUdqyawuFJYU0SWtCo1Qt06uOoiKYPXvP4wsv9CNhhx4aXE0iIiK1TSEsSuHtKdSKrJbVq+EHP4ARI+Dzz/c8n5YWXE0iIiJBUAiLklqR1Tdtmt9+YupUaNUKCgqCrkhERCQ4CmFR0srIqnMOHnjAbzuxahWMHOmPIhp4oJNERUREkpxCWJS0MrJqtm+HcePg+uuhuNhvQ/Hhh3CQsqyIiNRx2qIiStotv2qWLYO33oImTeDf/4azzw66IhERkfigEBYlnRtZNUccARMnwiGHQM+eQVcjIiISP9SOjJJGwqJTXAy//jU8+eSe5049VQFMRERkbxoJi5JWR1ZszRo47zzIzvbtx9NPh9atg65KREQkPmkkLEpaHXlgM2b47Seys/2k+7ffVgATERE5EIWwKDjntDpyP5yDf/zDbzuxciUMG+Z3vx8xIujKRERE4ptCWBS27trKrpJdNE5tTOO0xkGXE1fuuAOuvdYfRXTDDfDRR9ChQ9BViYiIxD+FsChoPtj+XXCBbz+++CLcey+kpgZdkYiISGLQxPwo6NzISPPmwdFHgxkcfjh89x00bBh0VSIiIolFI2FR0EiYV1ICt94KxxwD//rXnucVwERERCpPI2FR0EgYrFsH55/vjxyqVw/y84OuSEREJLEphEVh98rIuro9xaxZ/rih5cuhXTs//2v06KCrEhERSWxqR0ahru6W7xw88ggMH+4D2ODBfj6YApiIiEj1KYRFoa7OCdu1y+8BVljot6HIzoZOnYKuSkREJDmoHRmFujonLD0dXn0V5szx88FERESk5mgkLAp1aSTsP/+Ba67xrUiAQw9VABMREYkFhbAKOOfqxEhYSYnf/f600+Chh/zZjyIiIhI7akdWYFvhNnYW76RRaiOapDUJupyY2LDBj3a9/77ffuJPf4KTTw66KhERkeSmEFaBsqNgZhZwNTVvzhw46yxYtgzatPHbT/zgB0FXJSIikvwUwiqQzPPBPv4Yxozxqx8HDIBJk6Bz56CrEhERqRsUwiqQzPPBBg6Enj1h6FC47z5o0CDoikREROoOhbAKJNtGrTk5vu3YpAk0agTTp/uvRUREpHZpdWQFwiNhSdCO/O9/oV8/+MlP9mxBoQAmIiISDIWwCiTDuZGlpfD738Mpp8CmTbB9OxQUBF2ViIhI3aZ2ZAUSvR25cSNcdBG88w6YwfjxcOutfisKERERCY5CWAUSeXXkZ5/BmWf6eWCtWsHzz8OJJwZdlYiIiIBCWIUSeXXko4/6AJaV5bef6No16IpERERkN4WwCiTySNh990FGBtx4oz+MW0REROKHZgYdwPbC7ewo2kF6/XSapjUNupwK5ebCBRfAtm3+ccOGcPvtCmAiIiLxSCNhB1B2ZWS8H1n0/vswbpyfiN+qFfzf/wVdkYiIiByIRsIOIBHmg5WW+gO3x4zxAWzMGL8dhYiIiMQ3hbADiPf5YJs3w49+5FuOAHfeCW+/7UfCREREJL6pHXkA8TwStmkT9O8P334LLVvCc8/BSScFXZWIiIhESyHsAOJ5o9aWLeHYY6FZM3jlFejWLeiKREREpDIUwg4g3s6NLCyE1auhSxf/+MEH/RmQDRsGW5eIiIhUnuaEHcDq/Pg5NzIvD0aMgOOOgy1b/HPp6QpgIiIiiUoh7ADiZU7YRx9Bv37w6aewaxesXBloOSIiIlIDFMIOIOjVkc7BXXfB8cfDunVwwgkwdy4cfngg5YiIiEgNUgg7gCBHwrZs8Ydv/+Y3fi+w22+Hd96BNm1qvRQRERGJAU3M34/8wnzyi/JpkNKAZg2a1fr7f/ABvP46NG8OzzwDp51W6yWIiIhIDCmE7UfZVmQQRxadfTb87W9+NKx791p/exEREYkxtSP3o+y5kbWhsBB+9Sv4/PM9z910kwKYiIhIstJI2H7U5nywFStg7Fj45BP4739hwQJISYn524qIiEiANBK2H7W1W352tt9+4pNPICMDnnxSAUxERKQuUAjbj1jvlu8c3H2333x17Vp/BNHcuTBoUEzeTkREROKMQth+xHok7LLL/JyvkhK45RZ47z1o1y4mbyUiIiJxSCFsP2K9UeuJJ/rDt197Df7yF6iv2XkiIiJ1ikLYfsRideSyZXu+HjcOvv0WfvSjGnt5ERERSSAKYftRk6sji4rgl7+Eww6DefP2PK/d70VEROouhbD9qKl25KpV8IMfwH33QXExLFxYE9WJiIhIotNMpHLsKNrB9sLtpKWk0bxB8yq/ztSpfv+v1auhY0d4+WUYMqQGCxUREZGEpZGwcpRtRVblyCLn4P77YfRoH8BGjvRtSAUwERER2U0hrBzVbUXm5cEdd/jtJ266CT78ENrHfuN9ERERSSBqR5ajuisjO3eGp57yI2JnnVWTlYmIiEiyUAgrR1VWRr7yCmzbBpde6h+feWYMChMREZGkoRBWjsrsll9cDL/5jT+CKC0Nhg6FHj1iXaGIiIgkOoWwckR7buSaNXDuuTB5st/x/q9/hUMOqY0KRUREJNEphJUjmpGwTz6Bc86BlSvhoIPgpZdg+PDaqlBEREQSnVZHlqOi1ZHPP++3nVi50gevefMUwERERKRyFMLKUdHqyKwsaNQIbrgB/vc/6NChNqsTERGRZKB2ZDnKWx25apVvO5rBoYfCkiUKXyIiIlJ1Ggnby86inWwr3EZaShot0lsA8Prr0LMnPPLInvsUwERERKQ6YhrCzGyMmX1lZkvN7JZyrpuZPRi6/oWZ9YtlPdHYPR+sXeN2lJQYt9wCZ5wBW7f6syCdC7hAERERSQoxa0eaWQrwEHA8kAfMNrM3nXOLytx2EtAj9GMg8HDo58DsbkW2KunJiSfCRx9BSgrcdRf86le+HSkiIlKeoqIi8vLyKCgoCLoUqWXp6elkZGSQmpoa9ffEck7YAGCpc+47ADN7ETgdKBvCTgeeds45YKaZtTCzDs65VTGs64DW5K+BvAF8/erzfLER2rWDiRNh1KigKhIRkUSRl5dH06ZNyczMxPSv9jrDOceGDRvIy8ujW7duUX9fLNuRnYDlZR7nhZ6r7D21atW21fDO/1GwsS1DhvjtJxTAREQkGgUFBbRu3VoBrI4xM1q3bl3pEdBYhrDy/gvce0ZVNPdgZlea2Rwzm7Nu3boaKW5/NhdsgrPHMXDsFD7+GDoFGglFRCTRKIDVTVX53GMZwvKAzmUeZwArq3APzrnHnHNZzrmstm3b1nihZd087GZ23vMlHz7Tj7S0mL6ViIhIjVu9ejXnnXce3bt354gjjuDkk0/m66+/rvU6+vTpw7hx42r9ffdn7ty5HHXUURxyyCFcd911uHJW2hUVFXHJJZdw1FFHcfjhh/OXv/wFgB07dnDKKafQs2dPevXqxS237LPWsEpiGcJmAz3MrJuZpQHnAW/udc+bwMWhVZKDgC1BzgfbLb1+Ok3SmgRdhoiISKU45zjjjDMYNWoU3377LYsWLeLPf/4za9asifo1SkpKql3H4sWLKS0tZcqUKeTn55d7T3FxcbXfpzJ+9rOf8dhjj/HNN9/wzTff8O677+5zz8svv8yuXbtYsGABc+fO5dFHHyUnJweAG2+8kSVLlvDZZ58xffp0/vvf/1a7ppiFMOdcMXAN8B6wGHjJOfelmV1lZleFbnsH+A5YCvwLuDpW9YiIiCS7jz/+mNTUVK666qrwc3379mX48OFkZ2dz6qmnhp+/5pprmDBhAgCZmZn84Q9/YNiwYfztb39jwIAB4ftycnLo3bs34EeTRo4cyTHHHMOJJ57IqlXlj5s8//zzXHTRRZxwwgm8+eae8ZdRo0Zx6623MnLkSB544IH9vt6//vUv+vfvT58+fTjrrLPYsWNHtX5fVq1axdatWxk8eDBmxsUXX8zrr7++z31mRn5+PsXFxezcuZO0tDSaNWtGo0aNGD16NABpaWn069ePvLy8atUEMd4x3zn3Dj5olX3ukTJfO+DnsaxBREQkCPb72MwNc3fuf8PKhQsXcswxx1TpddPT05k2bRoAEydO5LvvvuPggw9m4sSJjB07lqKiIq699lreeOMN2rZty8SJE7ntttt48skn93mtiRMn8sEHH/DVV1/xj3/8I6ItuXnzZiZPnkxRUREjR44s9/XOPPNMfvKTnwBw++2388QTT3DttddGvMfHH3/MDTfcsM97N2rUiE8++STiuRUrVpCRkRF+nJGRwYoVK/b53rPPPps33niDDh06sGPHDu677z5atWoVcc/mzZt56623+MUvflHRb2mFdGyRiIiIcO6554a/Hjt2LC+99BK33HILEydOZOLEiXz11VcsXLiQ448/HvBtyw7lHB8ze/Zs2rZtS9euXcnIyODyyy9n06ZNtGzZMuJ9DvR6Cxcu5Pbbb2fz5s1s376dE088cZ/3GT16NPPnz4/q11be/K/yJtLPmjWLlJQUVq5cyaZNmxg+fDjHHXccBx98MOBbqOPGjeO6664LP1cdCmEiIiIxcKARq1jp1asXkyZNKvda/fr1KS0tDT/eezuFxo0bh78+99xzOeecczjzzDMxM3r06MGCBQvo1asXM2bMOGANL7zwAkuWLCEzMxOArVu38sorr3DFFVdEvI9zbr+vd+mll/L666/Tp08fJkyYQHZ29j73VGYkLCMjI6J9mJeXR8eOHff53ueff54xY8aQmppKu3btGDp0KHPmzAkHriuvvJIePXpw/fXXH/D3IFo6O1JERCRJHHvssezatYt//etf4edmz57N5MmT6dq1K4sWLWLXrl1s2bKF//3vf/t9ne7du5OSksL48ePDI1eHHXYY69atC4emoqIivvzyy4jvKy0t5eWXX+aLL74gJyeHnJwc3njjDV544YV93uNAr7dt2zY6dOhAUVERzz33XLk17h4J2/vH3gEMoEOHDjRt2pSZM2finOPpp5/m9NNP3+e+Ll268NFHH+GcIz8/n5kzZ9KzZ0/At0W3bNnC/fffv9/ft8pSCBMREUkSZsZrr73GBx98QPfu3enVqxe/+93v6NixI507d2bs2LH07t2bCy64gKOPPvqAr3Xuuefy7LPPMnbsWMBPSJ80aRI333wzffr0oW/fvvsEnilTptCpUyc6ldlkc8SIESxatGifSfwHer3x48czcOBAjj/++HAIqq6HH36YK664gkMOOYTu3btz0kknAfDmm2/y29/+FoCf//znbN++nSOPPJL+/ftz2WWX0bt3b/Ly8vjTn/7EokWL6NevH3379uXxxx+vdk1WXp80nmVlZbk5c+YEXYaIiMg+Fi9ezOGHHx50GRKQ8j5/M5vrnMsq736NhImIiIgEQCFMREREJAAKYSIiIiIBUAgTERGpQYk211pqRlU+d4UwERGRGpKens6GDRsUxOoY5xwbNmwgPT29Ut+nzVpFRERqyO5NQdetWxd0KVLL0tPTI45GioZCmIiISA1JTU2lW7duQZchCULtSBEREZEAKISJiIiIBEAhTERERCQACXdskZmtA3Jr4a3aAOtr4X0kevpM4o8+k/ikzyX+6DOJT7XxuXR1zrUt70LChbDaYmZz9nfWkwRDn0n80WcSn/S5xB99JvEp6M9F7UgRERGRACiEiYiIiARAIWz/Hgu6ANmHPpP4o88kPulziT/6TOJToJ+L5oSJiIiIBEAjYSIiIiIBqNMhzMzGmNlXZrbUzG4p57qZ2YOh61+YWb8g6qxrovhcLgh9Hl+Y2Sdm1ieIOuuSij6TMvf1N7MSMzu7Nuurq6L5XMxslJnNN7MvzWxybddY10Tx51dzM3vLzD4PfSaXBVFnXWJmT5rZWjNbuJ/rgf1dX2dDmJmlAA8BJwFHAOPM7Ii9bjsJ6BH6cSXwcK0WWQdF+bl8D4x0zvUGxqO5FjEV5Wey+76/Au/VboV1UzSfi5m1AP4J/NA51ws4p7brrEui/H/l58Ai51wfYBRwj5ml1Wqhdc8EYMwBrgf2d32dDWHAAGCpc+4751wh8CJw+l73nA487byZQAsz61DbhdYxFX4uzrlPnHObQg9nApU7tl4qK5r/VwCuBV4B1tZmcXVYNJ/L+cCrzrllAM45fTaxFc1n4oCmZmZAE2AjUFy7ZdYtzrkp+N/n/Qns7/q6HMI6AcvLPM4LPVfZe6RmVfb3/MfAf2NakVT4mZhZJ+AM4JFarKuui+b/lUOBlmaWbWZzzeziWquubormM/kHcDiwElgA/MI5V1o75cl+BPZ3ff3aeJM4ZeU8t/dS0WjukZoV9e+5mY3Gh7BhMa1IovlM7gduds6V+H/gSy2I5nOpDxwD/ABoCMwws5nOua9jXVwdFc1nciIwHzgW6A58YGZTnXNbY1yb7F9gf9fX5RCWB3Qu8zgD/y+Tyt4jNSuq33Mz6w08DpzknNtQS7XVVdF8JlnAi6EA1gY42cyKnXOv10qFdVO0f4atd87lA/lmNgXoAyiExUY0n8llwF3O7w+11My+B3oCs2qnRClHYH/X1+V25Gygh5l1C02KPA94c6973gQuDq2cGARscc6tqu1C65gKPxcz6wK8Clykf9HXigo/E+dcN+dcpnMuE5gEXK0AFnPR/Bn2BjDczOqbWSNgILC4luusS6L5TJbhRyYxs/bAYcB3tVql7C2wv+vr7EiYc67YzK7Br+RKAZ50zn1pZleFrj8CvAOcDCwFduD/BSMxFOXn8lugNfDP0MhLsQ7GjZ0oPxOpZdF8Ls65xWb2LvAFUAo87pwrd5m+VF+U/6+MByaY2QJ8G+xm59z6wIquA8zsBfxK1DZmlgfcCaRC8H/Xa8d8ERERkQDU5XakiIiISGAUwkREREQCoBAmIiIiEgCFMBEREZEAKISJiIiIBEAhTERERCQACmEiUi4zKzGz+WV+ZB7g3u018H4TzOz70HvNM7PBVXiNx83siNDXt+517ZPq1hh6nd2/LwvN7C0za1HB/X3N7OQqvE8HM/tP6OtRZralzGfxYej535nZijL1/LCc5xeZ2bgyr3u3mR1b2XpEpOZpnzARKZeZbXfONanpew/wGhOA/zjnJpnZCcDdzrne1Xi9atdU0eua2VPA1865Px3g/kuBLOfcNZV8n78D05xzb5jZKOBG59ype93zO2C7c+5uMzscmAq0w29ovPv5HsBcoLVzrsjMugL/cs6dUJl6RKTmaSRMRKJiZk3M7H+hUaoFZnZ6Ofd0MLMpZUZmhoeeP8HMZoS+92UzqygcTQEOCX3vL0OvtdDMrg8919jM3jazz0PPnxt6PtvMsszsLqBhqI7nQte2h36eWHZkKjQCd5aZpZjZ381stpl9YWY/jeK3ZQbQKfQ6A8zsEzP7LPTzYaGja/4AnBuq5dxQ7U+G3uez8n4fQ84C3o2iBgCcc4uBYvzZnWWf/wa/C3jL0ONcoLWZHRTta4tIbCiEicj+7A4x883sNaAAOMM51w8YDdxjoXOjyjgfeM851xd/UPR8M2sD3A4cF/reOcAvK3jv04AFZnYM/giRgcAg4CdmdjQwBljpnOvjnDuSvcKKc+4WYKdzrq9z7oK9XvtFYHdoS8Of4/cO8GP8mXH9gf6h9+q2vwLNLCX0vbvPBlwCjHDOHY0fifqzc64w9PXEUC0TgduAj0LvMxr4u5k13uu1uwGbnHO7yjw9vMzncVs59QzEH020bq/n+wHfOOfWlnl6HjB0f782EakddfbsSBGp0M5QmALAzFKBP5vZCPxf9p2A9sDqMt8zG3gydO/rzrn5ZjYSOAKYHspsafgRpPL83cxuxweJH+NDzmvOufxQDa8Cw/Gh624z+yu+hTm1Er+u/wIPmlkDfJib4pzbGWqB9jazs0P3NQd6AN/v9f0NzWw+kIlv831Q5v6nQu0/R+hsunKcAPzQzG4MPU4HuhB5sHYH9gpTwNS925EhN5jZhcA24FznnAv9Pt9gZj8BDg79OstaC3TcT30iUksUwkQkWhcAbYFjQnOLcvABIsw5NyUU0k4BngnNa9oEfOCcG7f3C5bjJufcpN0PzOy48m5yzn0dGiU7GfiLmb3vnPtDNL8I51yBmWUDJ+JHxF7Y/XbAtc659yp4iZ3Oub5m1hz4D/Bz4EH8wcwfO+fOML+IIXs/32/AWc65rw70Huz1e3sA9znn7t7f82Z2JvC0mXV3zhWErqWH3kNEAqR2pIhEqzmwNhTARgNd974hNOl7rXPuX8ATQD9gJjDUzHbP8WpkZodG+Z5TgB+FvqcxcAYw1cw6Ajucc88Cd4feZ29FoRG58ryIb3MOB3aHrveAn+3+HjM7dO82YVnOuS3AdcCNoe9pDqwIXb60zK3bgKZlHr8HXLu7lRtqr+7ta/xIW7U5517Ft4AvKfP0ocDCmnh9Eak6hTARidZzQJaZzcGPii0p555R+Hlgn+Enlj/gnFuHDyUvmNkX+FDWM5o3dM7NAyYAs4BPgcedc58BRwGzQm3B24A/lvPtjwFf7J6Yv5f3gRHAh6F5WwCPA4uAeWa2EHiUCroFoVo+B84D/oYflZsOpJS57WPgiN0T8/EjZqmh2haGHu/9uvnAt7uDaw34A/BLM6sXCoyH4IOZiARIW1SIiMQhMzsD3/q9PQav2885d0dNvq6IVJ7mhImIxCHn3Gtm1joGL10fuCcGrysilaSRMBEREZEAaE6YiIiISAAUwkREREQCoBAmIiIiEgCFMBEREZEAKISJiIiIBOD/AQ3dmCvWK8GTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fpr, tpr, thr = roc_curve(y_test['Churn'], \n",
    "                          y_test['probability'])\n",
    "auc = np.round(roc_auc_score(y_test['Churn'], \n",
    "                             y_test['predicted_Churn']), 2)\n",
    "\n",
    "plt.figure(figsize=(10, 8))\n",
    "plt.plot(fpr, \n",
    "         tpr, \n",
    "         color='green', \n",
    "         lw=2, \n",
    "         label=\"Curve Area = \" +str(auc))\n",
    "\n",
    "plt.plot([0, 1], [0, 1], color='blue', lw=2, linestyle='--')\n",
    "plt.xlabel('False Positive Rate (FPR)')\n",
    "plt.ylabel('True Positive Rate (TPR)')\n",
    "plt.title('ROC curve')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>threshold</th>\n",
       "      <th>FPR</th>\n",
       "      <th>TPR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.049123</td>\n",
       "      <td>0.696552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   threshold       FPR       TPR\n",
       "0        2.0  0.000000  0.000000\n",
       "1        1.0  0.049123  0.696552\n",
       "2        0.0  1.000000  1.000000"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "roc_data = pd.DataFrame({\"threshold\": thr, \"FPR\": fpr, \"TPR\": tpr})\n",
    "display(roc_data.tail(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Report: \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.95      0.95       855\n",
      "           1       0.71      0.70      0.70       145\n",
      "\n",
      "    accuracy                           0.91      1000\n",
      "   macro avg       0.83      0.82      0.83      1000\n",
      "weighted avg       0.91      0.91      0.91      1000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "class_report = classification_report(y_test['Churn'], \n",
    "                                     y_test['predicted_Churn'])\n",
    "print(\"Classification Report: \\n\", class_report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}