income_prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "Author: Sajjad Azami\n",
    "\n",
    "January 2nd, 2021"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('./us_census_data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing\n",
    "\n",
    "First, we want to start by checking for missing values.\n",
    "After a quick look at the data, turns out missing values are marked by '?'. So we will replace them with NaN to be able to use the standard pandas methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               False\n",
       "workclass         False\n",
       "fnlwgt            False\n",
       "education         False\n",
       "marital-status    False\n",
       "occupation        False\n",
       "relationship      False\n",
       "race              False\n",
       "sex               False\n",
       "hours-per-week    False\n",
       "native-country    False\n",
       "capital           False\n",
       "income            False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               False\n",
       "workclass          True\n",
       "fnlwgt            False\n",
       "education         False\n",
       "marital-status    False\n",
       "occupation         True\n",
       "relationship      False\n",
       "race              False\n",
       "sex               False\n",
       "hours-per-week    False\n",
       "native-country     True\n",
       "capital           False\n",
       "income            False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.replace('?', np.NaN)\n",
    "df.isna().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False    46012\n",
      "True      2830\n",
      "Name: workclass, dtype: int64\n",
      "False    46033\n",
      "True      2809\n",
      "Name: occupation, dtype: int64\n",
      "False    47985\n",
      "True       857\n",
      "Name: native-country, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df['workclass'].isna().value_counts())\n",
    "print(df['occupation'].isna().value_counts())\n",
    "print(df['native-country'].isna().value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Filling missing values with the most common values of that feature. Although there are more complex methods of dealing with this issue, we will try to focus on the encoding methods in what follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df['workclass'] = df['workclass'].fillna(df['workclass'].value_counts().index[0])\n",
    "df['occupation'] = df['occupation'].fillna(df['occupation'].value_counts().index[0])\n",
    "df['native-country'] = df['native-country'].fillna(df['native-country'].value_counts().index[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               False\n",
       "workclass         False\n",
       "fnlwgt            False\n",
       "education         False\n",
       "marital-status    False\n",
       "occupation        False\n",
       "relationship      False\n",
       "race              False\n",
       "sex               False\n",
       "hours-per-week    False\n",
       "native-country    False\n",
       "capital           False\n",
       "income            False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age                int64\n",
       "workclass         object\n",
       "fnlwgt             int64\n",
       "education         object\n",
       "marital-status    object\n",
       "occupation        object\n",
       "relationship      object\n",
       "race              object\n",
       "sex               object\n",
       "hours-per-week     int64\n",
       "native-country    object\n",
       "capital            int64\n",
       "income            object\n",
       "dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes\n",
    "# so we will leave ['age', 'fnlwgt', 'hours-per-week', 'capital'] since they are already numerical\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df['income_binary'] = (df['income'] == '<=50K').astype(int)\n",
    "# target variable is binary, so we'll replace it with 0,1 values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "workclass          6\n",
       "education         16\n",
       "marital-status     7\n",
       "occupation        14\n",
       "relationship       6\n",
       "race               5\n",
       "sex                2\n",
       "native-country    41\n",
       "dtype: int64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_columns = ['workclass', 'education', 'marital-status',\n",
    "                  'occupation', 'relationship','race',\n",
    "                  'sex','native-country' ]\n",
    "\n",
    "df[categorical_columns].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Comparison\n",
    "We want to compare two models: \n",
    "* GradientBoostingClassifier\n",
    "* CatBoost\n",
    "\n",
    "We also want to try three different encoding methods:\n",
    "* One-Hot Encoding\n",
    "* Label Encoding\n",
    "* Target Encoding\n",
    "\n",
    "First, we fix the encoding method to One-Hot as this is the most natural way of dealing with categorical data. The biggest downside of this method is sparsification of the data, but our goal is to compare the models first. We later will try other encoding methods. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>capital</th>\n",
       "      <th>income</th>\n",
       "      <th>income_binary</th>\n",
       "      <th>workclass_Federal-gov</th>\n",
       "      <th>workclass_Local-gov</th>\n",
       "      <th>workclass_Private</th>\n",
       "      <th>workclass_Self-emp-inc</th>\n",
       "      <th>...</th>\n",
       "      <th>native-country_Portugal</th>\n",
       "      <th>native-country_Puerto-Rico</th>\n",
       "      <th>native-country_Scotland</th>\n",
       "      <th>native-country_South</th>\n",
       "      <th>native-country_Taiwan</th>\n",
       "      <th>native-country_Thailand</th>\n",
       "      <th>native-country_Trinadad&amp;Tobago</th>\n",
       "      <th>native-country_United-States</th>\n",
       "      <th>native-country_Vietnam</th>\n",
       "      <th>native-country_Yugoslavia</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>77516</td>\n",
       "      <td>40</td>\n",
       "      <td>2174</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>83311</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>215646</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>234721</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>338409</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48837</th>\n",
       "      <td>39</td>\n",
       "      <td>215419</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48838</th>\n",
       "      <td>64</td>\n",
       "      <td>321403</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48839</th>\n",
       "      <td>38</td>\n",
       "      <td>374983</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48840</th>\n",
       "      <td>44</td>\n",
       "      <td>83891</td>\n",
       "      <td>40</td>\n",
       "      <td>5455</td>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48841</th>\n",
       "      <td>35</td>\n",
       "      <td>182148</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>&gt;50K</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>48842 rows × 103 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       age  fnlwgt  hours-per-week  capital income  income_binary  \\\n",
       "0       39   77516              40     2174  <=50K              1   \n",
       "1       50   83311              13        0  <=50K              1   \n",
       "2       38  215646              40        0  <=50K              1   \n",
       "3       53  234721              40        0  <=50K              1   \n",
       "4       28  338409              40        0  <=50K              1   \n",
       "...    ...     ...             ...      ...    ...            ...   \n",
       "48837   39  215419              36        0  <=50K              1   \n",
       "48838   64  321403              40        0  <=50K              1   \n",
       "48839   38  374983              50        0  <=50K              1   \n",
       "48840   44   83891              40     5455  <=50K              1   \n",
       "48841   35  182148              60        0   >50K              0   \n",
       "\n",
       "       workclass_Federal-gov  workclass_Local-gov  workclass_Private  \\\n",
       "0                          0                    0                  0   \n",
       "1                          0                    0                  0   \n",
       "2                          0                    0                  1   \n",
       "3                          0                    0                  1   \n",
       "4                          0                    0                  1   \n",
       "...                      ...                  ...                ...   \n",
       "48837                      0                    0                  1   \n",
       "48838                      0                    0                  1   \n",
       "48839                      0                    0                  1   \n",
       "48840                      0                    0                  1   \n",
       "48841                      0                    0                  0   \n",
       "\n",
       "       workclass_Self-emp-inc  ...  native-country_Portugal  \\\n",
       "0                           0  ...                        0   \n",
       "1                           0  ...                        0   \n",
       "2                           0  ...                        0   \n",
       "3                           0  ...                        0   \n",
       "4                           0  ...                        0   \n",
       "...                       ...  ...                      ...   \n",
       "48837                       0  ...                        0   \n",
       "48838                       0  ...                        0   \n",
       "48839                       0  ...                        0   \n",
       "48840                       0  ...                        0   \n",
       "48841                       1  ...                        0   \n",
       "\n",
       "       native-country_Puerto-Rico  native-country_Scotland  \\\n",
       "0                               0                        0   \n",
       "1                               0                        0   \n",
       "2                               0                        0   \n",
       "3                               0                        0   \n",
       "4                               0                        0   \n",
       "...                           ...                      ...   \n",
       "48837                           0                        0   \n",
       "48838                           0                        0   \n",
       "48839                           0                        0   \n",
       "48840                           0                        0   \n",
       "48841                           0                        0   \n",
       "\n",
       "       native-country_South  native-country_Taiwan  native-country_Thailand  \\\n",
       "0                         0                      0                        0   \n",
       "1                         0                      0                        0   \n",
       "2                         0                      0                        0   \n",
       "3                         0                      0                        0   \n",
       "4                         0                      0                        0   \n",
       "...                     ...                    ...                      ...   \n",
       "48837                     0                      0                        0   \n",
       "48838                     0                      0                        0   \n",
       "48839                     0                      0                        0   \n",
       "48840                     0                      0                        0   \n",
       "48841                     0                      0                        0   \n",
       "\n",
       "       native-country_Trinadad&Tobago  native-country_United-States  \\\n",
       "0                                   0                             1   \n",
       "1                                   0                             1   \n",
       "2                                   0                             1   \n",
       "3                                   0                             1   \n",
       "4                                   0                             0   \n",
       "...                               ...                           ...   \n",
       "48837                               0                             1   \n",
       "48838                               0                             1   \n",
       "48839                               0                             1   \n",
       "48840                               0                             1   \n",
       "48841                               0                             1   \n",
       "\n",
       "       native-country_Vietnam  native-country_Yugoslavia  \n",
       "0                           0                          0  \n",
       "1                           0                          0  \n",
       "2                           0                          0  \n",
       "3                           0                          0  \n",
       "4                           0                          0  \n",
       "...                       ...                        ...  \n",
       "48837                       0                          0  \n",
       "48838                       0                          0  \n",
       "48839                       0                          0  \n",
       "48840                       0                          0  \n",
       "48841                       0                          0  \n",
       "\n",
       "[48842 rows x 103 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# one-hot encoding for categorical columns\n",
    "categorical_df = df[categorical_columns]\n",
    "one_hot_df = pd.get_dummies(categorical_df, columns = categorical_columns)\n",
    "df_one_hot_drop = df.drop(categorical_columns, axis = 1)\n",
    "df = pd.concat([df_one_hot_drop, one_hot_df], axis = 1)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train, Validation and Test Set\n",
    "We use %50 of the data for training, %25 percent for validation and %25 percent for test. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size:  (24542, 101)\n",
      "Validation size:  (12089, 101)\n",
      "Test size:  (12211, 101)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X = df.drop(['income', 'income_binary'], axis=1)\n",
    "y = df['income_binary']\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.25, random_state=42)\n",
    "X_train, X_val, y_train, y_val = train_test_split(\n",
    "    X_train, y_train, test_size=0.33, random_state=42)\n",
    "print(\"Train size: \", X_train.shape)\n",
    "print(\"Validation size: \", X_val.shape)\n",
    "print(\"Test size: \", X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GradientBoosting\n",
    "n_estimators=50, learning_rate=1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time: 2.393467903137207s\n",
      "0.8698817106460418\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "import time\n",
    "start = time.time()\n",
    "xgb_clf = GradientBoostingClassifier(n_estimators=50, learning_rate=1.0,\n",
    "    max_depth=3, random_state=0).fit(X_train, y_train)\n",
    "stop = time.time()\n",
    "print(f\"Training time: {stop - start}s\")\n",
    "print(xgb_clf.score(X_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "# CatBoost\n",
    "iterations=100, learning_rate=0.1, loss_function='CrossEntropy'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time: 0.9012150764465332s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8688890727107288"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from catboost import CatBoostClassifier\n",
    "start = time.time()\n",
    "cat_clf = CatBoostClassifier(\n",
    "    iterations=100,\n",
    "    learning_rate=0.1,\n",
    "    loss_function='CrossEntropy', verbose=False).fit(X_train, y_train)\n",
    "stop = time.time()\n",
    "print(f\"Training time: {stop - start}s\")\n",
    "cat_clf.score(X_val, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "### So here is the initial comparison:\n",
    "| Model | Training Time | Validation Accuracy |\n",
    "| --- | --- | --- |\n",
    "| GradientBoosting | 2.393 | 86.98 |\n",
    "| CatBoost | 0.901 | 86.88 |\n",
    "\n",
    "This shows that with the current parameters, CatBoost is far better in terms of training time, but almost the same accuracy. So to make a comparison with equal resources (training time here), we increase the number of iterations for CatBoost to match the training time of ~2.3 seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time: 2.806924819946289s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.876664736537348"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start = time.time()\n",
    "cat_clf = CatBoostClassifier(\n",
    "    iterations=300,\n",
    "    learning_rate=0.1,\n",
    "    loss_function='CrossEntropy', verbose=False).fit(X_train, y_train)\n",
    "stop = time.time()\n",
    "print(f\"Training time: {stop - start}s\")\n",
    "cat_clf.score(X_val, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameter Gridsearch for GradientBoosting\n",
    "Here we perform a quick search to improve the accuracy of the GradientBoosting as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score=nan,\n",
       "             estimator=GradientBoostingClassifier(ccp_alpha=0.0,\n",
       "                                                  criterion='friedman_mse',\n",
       "                                                  init=None, learning_rate=0.1,\n",
       "                                                  loss='deviance', max_depth=3,\n",
       "                                                  max_features=None,\n",
       "                                                  max_leaf_nodes=None,\n",
       "                                                  min_impurity_decrease=0.0,\n",
       "                                                  min_impurity_split=None,\n",
       "                                                  min_samples_leaf=1,\n",
       "                                                  min_samples_split=2,\n",
       "                                                  min_weight_fraction_leaf=0.0,\n",
       "                                                  n_estimators=100,\n",
       "                                                  n_iter_no_change=None,\n",
       "                                                  presort='deprecated',\n",
       "                                                  random_state=None,\n",
       "                                                  subsample=1.0, tol=0.0001,\n",
       "                                                  validation_fraction=0.1,\n",
       "                                                  verbose=0, warm_start=False),\n",
       "             iid='deprecated', n_jobs=None,\n",
       "             param_grid={'max_depth': [3, 10, 20], 'n_estimators': [10, 100]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "parameters = {'n_estimators':[10,100], 'max_depth':[3, 10, 20]}\n",
    "xgb_clf = GradientBoostingClassifier()\n",
    "clf = GridSearchCV(xgb_clf, parameters)\n",
    "clf.fit(X_train, y_train.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'max_depth': 10, 'n_estimators': 100}\n",
      "0.8688890727107288\n"
     ]
    }
   ],
   "source": [
    "print(clf.best_params_)\n",
    "print(clf.score(X_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8663500122840062\n",
      "0.8711817213987388\n"
     ]
    }
   ],
   "source": [
    "print(clf.score(X_test, y_test))\n",
    "print(cat_clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final comparison (with One-hot encoding):\n",
    "| Model | Training Time | Validation Accuracy | Test Accuracy |\n",
    "| --- | --- | --- | --- |\n",
    "| GradientBoosting | 2.281 | 86.88 | 86.63 |\n",
    "| CatBoost | 2.806 | 87.66 | 87.11 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Encodings\n",
    "Now we'll try using other encoding methods and see the possible impact on the performance. Since this part is focused on encoding methods, we will fix the model parameters to best parameters of previous part.\n",
    "* CatBoost {'iterations' : 300, 'learning_rate' : 0.1, 'loss_function' : 'CrossEntropy'}\n",
    "* GradientBoosting: {'max_depth': 3, 'n_estimators': 100}\n",
    "\n",
    "We need to reload the dataset again and redo the necessary preprocessings.\n",
    "\n",
    "## Label Encoder\n",
    "We start with Label Encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "df = pd.read_csv('./us_census_data.csv')\n",
    "df = df.replace('?', np.NaN)\n",
    "df['workclass'] = df['workclass'].fillna(df['workclass'].value_counts().index[0])\n",
    "df['occupation'] = df['occupation'].fillna(df['occupation'].value_counts().index[0])\n",
    "df['native-country'] = df['native-country'].fillna(df['native-country'].value_counts().index[0])\n",
    "df['income_binary'] = (df['income'] == '<=50K').astype(int)\n",
    "X = df.drop(['income', 'income_binary'], axis=1)\n",
    "y = df['income_binary']\n",
    "\n",
    "# Here we encode categorical columns\n",
    "X[categorical_columns] = X[categorical_columns].apply(LabelEncoder().fit_transform)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.25, random_state=42)\n",
    "X_train, X_val, y_train, y_val = train_test_split(\n",
    "    X_train, y_train, test_size=0.33, random_state=42)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "      <th>capital</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21486</th>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>129882</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41634</th>\n",
       "      <td>17</td>\n",
       "      <td>2</td>\n",
       "      <td>141445</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26548</th>\n",
       "      <td>43</td>\n",
       "      <td>2</td>\n",
       "      <td>326379</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>40</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15346</th>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "      <td>199472</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38310</th>\n",
       "      <td>47</td>\n",
       "      <td>4</td>\n",
       "      <td>114222</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27929</th>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>148429</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>40</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6770</th>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>378723</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>937</th>\n",
       "      <td>41</td>\n",
       "      <td>2</td>\n",
       "      <td>132222</td>\n",
       "      <td>14</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>40</td>\n",
       "      <td>38</td>\n",
       "      <td>-2415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44886</th>\n",
       "      <td>39</td>\n",
       "      <td>2</td>\n",
       "      <td>312271</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41860</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>237546</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>99</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>24542 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       age  workclass  fnlwgt  education  marital-status  occupation  \\\n",
       "21486   28          2  129882         15               4           0   \n",
       "41634   17          2  141445          0               4           7   \n",
       "26548   43          2  326379         11               0           2   \n",
       "15346   25          2  199472          8               4           0   \n",
       "38310   47          4  114222          7               2           3   \n",
       "...    ...        ...     ...        ...             ...         ...   \n",
       "27929   28          2  148429         11               2           6   \n",
       "6770    30          2  378723         15               0           0   \n",
       "937     41          2  132222         14               2           9   \n",
       "44886   39          2  312271         15               2          11   \n",
       "41860   56          1  237546         15               2           3   \n",
       "\n",
       "       relationship  race  sex  hours-per-week  native-country  capital  \n",
       "21486             4     2    0              40              38        0  \n",
       "41634             3     4    1              30              38        0  \n",
       "26548             1     4    1              40              38        0  \n",
       "15346             1     4    0              38              38        0  \n",
       "38310             0     4    1              25              38        0  \n",
       "...             ...   ...  ...             ...             ...      ...  \n",
       "27929             0     4    1              40              38        0  \n",
       "6770              3     4    0              55              38        0  \n",
       "937               0     4    1              40              38    -2415  \n",
       "44886             0     4    1              60              38        0  \n",
       "41860             0     4    1              99              38        0  \n",
       "\n",
       "[24542 rows x 12 columns]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time: 2.547102928161621s\n",
      "0.8752584994623211\n",
      "0.8710179346490868\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "cat_clf = CatBoostClassifier(\n",
    "    iterations=300,\n",
    "    learning_rate=0.1,\n",
    "    loss_function='CrossEntropy', verbose=False).fit(X_train, y_train)\n",
    "stop = time.time()\n",
    "print(f\"Training time: {stop - start}s\")\n",
    "print(cat_clf.score(X_val, y_val))\n",
    "print(cat_clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time: 2.1548879146575928s\n",
      "0.8687236330548432\n",
      "0.8611088362951437\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "xgb_clf = GradientBoostingClassifier(n_estimators=100, learning_rate=1.0,\n",
    "    max_depth=3, random_state=0).fit(X_train, y_train)\n",
    "stop = time.time()\n",
    "print(f\"Training time: {stop - start}s\")\n",
    "print(xgb_clf.score(X_val, y_val))\n",
    "print(xgb_clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison (with label encoding):\n",
    "| Model | Training Time | Validation Accuracy | Test Accuracy |\n",
    "| --- | --- | --- | --- |\n",
    "| GradientBoosting | 2.154 | 86.87 | 86.11 |\n",
    "| CatBoost | 2.547 | 87.52 | 87.10 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note:\n",
    "For label encoding and target encoding, unlike one-hot method, we don't have many new features generated. So it makes sense to check feature importances as well to get insight about the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.05020323, 0.00909001, 0.04043135, 0.09410626, 0.00777151,\n",
       "       0.02218894, 0.40350881, 0.00238707, 0.00227762, 0.05240448,\n",
       "       0.00688437, 0.30874634])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb_clf.feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAATQ0lEQVR4nO3df2xe133f8fen8uSsSdM5Mzes+hEpKdNFaTZrYOVswbyitWMFXi3/4aDKkEIdPAgZrCWbW6zKGtidggCuO6QdULWJ0Ggr2rqqa7co0TDVvPxYW7R2RMdeUikTTCuexcmb2cpLhiWzQ+e7P3idPWYeiVfkQ3E8er8Agvece87l90LEh1f3PvfeVBWSpHZ9x1oXIElaXQa9JDXOoJekxhn0ktQ4g16SGnfVWhew2LXXXlvbtm1b6zIkaV157LHH/ryqxoat+/8u6Ldt28b09PRalyFJ60qS/3qhdZ66kaTGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcb2CPsnuJKeTzCQ5eJFxtyepJBMDfR/o5p1OcvMoipYk9bfkDVNJNgCHgZuAWeBEksmqOrVo3HcB7wMeHejbAewF3gJ8D/Afk7ypql4a3S5Iki6mz52xu4CZqjoDkOQYsAc4tWjch4D7gJ8c6NsDHKuqF4AvJ5nptvenKy1cWkvbDn5i5Nt8+t5bRr5NCfqdutkEnB1oz3Z935JkJ7Clqn7/UudKklZXn6DPkL5vvX8wyXcAPw/8xKXOHdjG/iTTSabn5uZ6lCRJ6qtP0M8CWwbam4FzA+3vAr4f+GySp4G3AZPdBdml5gJQVUeqaqKqJsbGhj58TZK0TH2C/gQwnmR7ko0sXFydfHllVX2lqq6tqm1VtQ14BLi1qqa7cXuTXJ1kOzAOfG7keyFJuqAlL8ZW1XySA8BxYANwtKpOJjkETFfV5EXmnkzyAAsXbueBO/3EjSRdXr2eR19VU8DUor67LzD2Bxe1Pwx8eJn1SZJWyDtjJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXG9gj7J7iSnk8wkOThk/XuTfDHJE0n+OMmOrn9bkq93/U8k+eiod0CSdHFLvkowyQbgMHATMAucSDJZVacGht1fVR/txt8KfATY3a17qqquG23ZkqS++hzR7wJmqupMVb0IHAP2DA6oqq8ONF8N1OhKlCStRJ+g3wScHWjPdn2vkOTOJE8B9wHvG1i1PcnjSf5Tkr8/7Ack2Z9kOsn03NzcJZQvSVpKn6DPkL5vO2KvqsNV9Ubgp4APdt3PAluraidwF3B/ktcOmXukqiaqamJsbKx/9ZKkJfUJ+llgy0B7M3DuIuOPAbcBVNULVfUX3fJjwFPAm5ZXqiRpOfoE/QlgPMn2JBuBvcDk4IAk4wPNW4Anu/6x7mIuSd4AjANnRlG4JKmfJT91U1XzSQ4Ax4ENwNGqOpnkEDBdVZPAgSQ3At8Angf2ddNvAA4lmQdeAt5bVedXY0ckScMtGfQAVTUFTC3qu3tg+f0XmPcQ8NBKCpQkrYx3xkpS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4Xs+jlyS90raDnxj5Np++95aRbxM8opek5vUK+iS7k5xOMpPk4JD1703yxSRPJPnjJDsG1n2gm3c6yc2jLF6StLQlg757ufdh4J3ADuDdg0Heub+q3lpV1wH3AR/p5u5g4WXibwF2A7/08svCJUmXR58j+l3ATFWdqaoXgWPAnsEBVfXVgeargeqW9wDHquqFqvoyMNNtT5J0mfS5GLsJODvQngWuXzwoyZ3AXcBG4IcG5j6yaO6mIXP3A/sBtm7d2qduSVJPfY7oM6Svvq2j6nBVvRH4KeCDlzj3SFVNVNXE2NhYj5IkSX31CfpZYMtAezNw7iLjjwG3LXOuJGnE+gT9CWA8yfYkG1m4uDo5OCDJ+EDzFuDJbnkS2Jvk6iTbgXHgcysvW5LU15Ln6KtqPskB4DiwAThaVSeTHAKmq2oSOJDkRuAbwPPAvm7uySQPAKeAeeDOqnpplfZFkjRErztjq2oKmFrUd/fA8vsvMvfDwIeXW6AkaWW8M1aSGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LheQZ9kd5LTSWaSHByy/q4kp5J8Icmnkrx+YN1LSZ7oviYXz5Ukra4l3zCVZANwGLiJhZd9n0gyWVWnBoY9DkxU1deS/FPgPuBHu3Vfr6rrRly3JKmnPkf0u4CZqjpTVS8Cx4A9gwOq6jNV9bWu+QiwebRlSpKWq0/QbwLODrRnu74LuQP45ED7VUmmkzyS5LZhE5Ls78ZMz83N9ShJktRXn5eDZ0hfDR2YvAeYAP7BQPfWqjqX5A3Ap5N8saqeesXGqo4ARwAmJiaGbluStDx9juhngS0D7c3AucWDktwI/DRwa1W98HJ/VZ3rvp8BPgvsXEG9kqRL1CfoTwDjSbYn2QjsBV7x6ZkkO4GPsRDyzw30X5Pk6m75WuDtwOBFXEnSKlvy1E1VzSc5ABwHNgBHq+pkkkPAdFVNAj8HvAb47SQAz1TVrcCbgY8l+SYLf1TuXfRpHUnSKutzjp6qmgKmFvXdPbB84wXm/Qnw1pUUKElaGe+MlaTGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMb1Cvoku5OcTjKT5OCQ9XclOZXkC0k+leT1A+v2JXmy+9o3yuIlSUtbMuiTbAAOA+8EdgDvTrJj0bDHgYmq+lvAg8B93dzXAfcA1wO7gHuSXDO68iVJS+lzRL8LmKmqM1X1InAM2DM4oKo+U1Vf65qPAJu75ZuBh6vqfFU9DzwM7B5N6ZKkPvoE/Sbg7EB7tuu7kDuAT17K3CT7k0wnmZ6bm+tRkiSprz5BnyF9NXRg8h5gAvi5S5lbVUeqaqKqJsbGxnqUJEnqq0/QzwJbBtqbgXOLByW5Efhp4NaqeuFS5kqSVk+foD8BjCfZnmQjsBeYHByQZCfwMRZC/rmBVceBdyS5prsI+46uT5J0mVy11ICqmk9ygIWA3gAcraqTSQ4B01U1ycKpmtcAv50E4JmqurWqzif5EAt/LAAOVdX5VdkTSdJQSwY9QFVNAVOL+u4eWL7xInOPAkeXW6AkaWW8M1aSGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIa1yvok+xOcjrJTJKDQ9bfkOTzSeaT3L5o3UtJnui+JhfPlSStriVfJZhkA3AYuAmYBU4kmayqUwPDngF+HPjJIZv4elVdN4JaJUnL0OedsbuAmao6A5DkGLAH+FbQV9XT3bpvrkKNkqQV6HPqZhNwdqA92/X19aok00keSXLbsAFJ9ndjpufm5i5h05KkpfQJ+gzpq0v4GVuragL4R8AvJHnjt22s6khVTVTVxNjY2CVsWpK0lD5BPwtsGWhvBs71/QFVda77fgb4LLDzEuqTJK1Qn6A/AYwn2Z5kI7AX6PXpmSTXJLm6W74WeDsD5/YlSatvyaCvqnngAHAc+BLwQFWdTHIoya0ASX4gySzwLuBjSU52098MTCf5z8BngHsXfVpHkrTK+nzqhqqaAqYW9d09sHyChVM6i+f9CfDWFdYoSVoB74yVpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWuV9An2Z3kdJKZJAeHrL8hyeeTzCe5fdG6fUme7L72japwSVI/SwZ9kg3AYeCdwA7g3Ul2LBr2DPDjwP2L5r4OuAe4HtgF3JPkmpWXLUnqq88R/S5gpqrOVNWLwDFgz+CAqnq6qr4AfHPR3JuBh6vqfFU9DzwM7B5B3ZKknvoE/Sbg7EB7tuvro9fcJPuTTCeZnpub67lpSVIffYI+Q/qq5/Z7za2qI1U1UVUTY2NjPTctSeqjT9DPAlsG2puBcz23v5K5kqQR6BP0J4DxJNuTbAT2ApM9t38ceEeSa7qLsO/o+iRJl8lVSw2oqvkkB1gI6A3A0ao6meQQMF1Vk0l+APhd4BrgR5L866p6S1WdT/IhFv5YAByqqvOrtC9apm0HPzHybT597y0j36ak5Vky6AGqagqYWtR398DyCRZOywybexQ4uoIaJUkr4J2xktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LheQZ9kd5LTSWaSHByy/uokv9WtfzTJtq5/W5KvJ3mi+/roaMuXJC1lyVcJJtkAHAZuAmaBE0kmq+rUwLA7gOer6nuT7AV+FvjRbt1TVXXdiOuWJPXU552xu4CZqjoDkOQYsAcYDPo9wM90yw8Cv5gkI6xTUgNG/SJ6X0LfT5+g3wScHWjPAtdfaExVzSf5CvBXu3XbkzwOfBX4YFX90eIfkGQ/sB9g69atl7QDi/mLJEmv1Occ/bAj8+o55llga1XtBO4C7k/y2m8bWHWkqiaqamJsbKxHSZKkvvoE/SywZaC9GTh3oTFJrgK+GzhfVS9U1V8AVNVjwFPAm1ZatCSpvz5BfwIYT7I9yUZgLzC5aMwksK9bvh34dFVVkrHuYi5J3gCMA2dGU7okqY8lz9F359wPAMeBDcDRqjqZ5BAwXVWTwMeBX0syA5xn4Y8BwA3AoSTzwEvAe6vq/GrsiCRpuD4XY6mqKWBqUd/dA8v/B3jXkHkPAQ+tsMYr1qgvLIMXl6UrkXfGSlLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrX64YpaT3xCabSK3lEL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrnxyuXwefES1pPDHpdNn6+XZeLv2uv1OvUTZLdSU4nmUlycMj6q5P8Vrf+0STbBtZ9oOs/neTm0ZUuSepjyaDvXu59GHgnsAN4d5Idi4bdATxfVd8L/Dzws93cHSy8P/YtwG7gl15+Wbgk6fLoc+pmFzBTVWcAkhwD9gCnBsbsAX6mW34Q+MUk6fqPVdULwJe7l4fvAv50NOVLbbtc14O87tS2VNXFByS3A7ur6p907R8Drq+qAwNj/qwbM9u1nwKuZyH8H6mqX+/6Pw58sqoeXPQz9gP7u+b3AadXvmtLuhb488vwcy6XlvanpX2BtvanpX2Btvbn9VU1NmxFnyP6DOlb/NfhQmP6zKWqjgBHetQyMkmmq2ricv7M1dTS/rS0L9DW/rS0L9De/lxIn4uxs8CWgfZm4NyFxiS5Cvhu4HzPuZKkVdQn6E8A40m2J9nIwsXVyUVjJoF93fLtwKdr4ZzQJLC3+1TOdmAc+NxoSpck9bHkqZuqmk9yADgObACOVtXJJIeA6aqaBD4O/Fp3sfU8C38M6MY9wMKF23ngzqp6aZX25VJd1lNFl0FL+9PSvkBb+9PSvkB7+zPUkhdjJUnrm8+6kaTGGfSS1LgrLuiXepzDepJkS5LPJPlSkpNJ3r/WNa1Ukg1JHk/y+2tdy0ol+StJHkzyX7p/o7+71jWtRJJ/0f2e/VmS30zyqrWu6VIkOZrkue6+n5f7Xpfk4SRPdt+vWcsaV8sVFfQ9H+ewnswDP1FVbwbeBty5zvcH4P3Al9a6iBH5t8AfVNXfBP4263i/kmwC3gdMVNX3s/DBjL1rW9Ul+/csPIpl0EHgU1U1Dnyqazfnigp6Bh7nUFUvAi8/zmFdqqpnq+rz3fL/YiFINq1tVcuXZDNwC/Ara13LSiV5LXADC59Io6perKr/ubZVrdhVwF/u7pX5TtbZPTFV9YcsfCpw0B7gV7vlXwVuu6xFXSZXWtBvAs4OtGdZx8E4qHti6E7g0bWtZEV+AfiXwDfXupAReAMwB/y77lTUryR59VoXtVxV9d+AfwM8AzwLfKWq/sPaVjUSf72qnoWFAyfgr61xPaviSgv6Xo9kWG+SvAZ4CPjnVfXVta5nOZL8Q+C5qnpsrWsZkauAvwP8clXtBP436/i0QHfueg+wHfge4NVJ3rO2VamvKy3om3skQ5K/xELI/0ZV/c5a17MCbwduTfI0C6fUfijJr69tSSsyC8xW1cv/w3qQheBfr24EvlxVc1X1DeB3gL+3xjWNwv9I8jcAuu/PrXE9q+JKC/o+j3NYN7pHQX8c+FJVfWSt61mJqvpAVW2uqm0s/Lt8uqrW7RFjVf134GyS7+u6fphXPtp7vXkGeFuS7+x+736YdXxxecDg41v2Ab+3hrWsmivqVYIXepzDGpe1Em8Hfgz4YpInur5/VVVTa1iT/p9/BvxGd1BxBvjHa1zPslXVo0keBD7Pwqe9HmedPT4gyW8CPwhcm2QWuAe4F3ggyR0s/DF719pVuHp8BIIkNe5KO3UjSVccg16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ17v8CQ4anJ2e38CUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "\n",
    "pyplot.bar(range(len(xgb_clf.feature_importances_)), xgb_clf.feature_importances_)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Target Encoder\n",
    "Now we try target encoder. In this method, we replace the categorical value with the average value of the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('./us_census_data.csv')\n",
    "df = df.replace('?', np.NaN)\n",
    "df['workclass'] = df['workclass'].fillna(df['workclass'].value_counts().index[0])\n",
    "df['occupation'] = df['occupation'].fillna(df['occupation'].value_counts().index[0])\n",
    "df['native-country'] = df['native-country'].fillna(df['native-country'].value_counts().index[0])\n",
    "df['income_binary'] = (df['income'] == '<=50K').astype(int)\n",
    "\n",
    "X = df.drop(['income', 'income_binary'], axis=1)\n",
    "y = df['income_binary']\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n",
    "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.33, random_state=42)\n",
    "\n",
    "merged_train = X_train.merge(y_train, left_index=True, right_index=True, how='inner')\n",
    "for column in categorical_columns:\n",
    "    encodings = merged_train.groupby(column)['income_binary'].mean().reset_index()\n",
    "    X_train = X_train.merge(encodings, how='left', on=column)\n",
    "    X_train[column] = X_train['income_binary']\n",
    "    X_train = X_train.drop(['income_binary'], axis=1)\n",
    "    X_val = X_val.merge(encodings, how='left', on=column)\n",
    "    X_val[column] = X_val['income_binary']\n",
    "    X_val = X_val.drop(['income_binary'], axis=1)\n",
    "    X_test = X_test.merge(encodings, how='left', on=column)\n",
    "    X_test[column] = X_test['income_binary']\n",
    "    X_test = X_test.drop(['income_binary'], axis=1)\n",
    "X_test['native-country'] = X_test['native-country'].fillna(X_test['native-country'].value_counts().index[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time: 2.165092945098877s\n",
      "0.8666556373562743\n",
      "0.86037179592171\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "xgb_clf = GradientBoostingClassifier(n_estimators=100, learning_rate=1.0,\n",
    "    max_depth=3, random_state=0).fit(X_train, y_train)\n",
    "stop = time.time()\n",
    "print(f\"Training time: {stop - start}s\")\n",
    "print(xgb_clf.score(X_val, y_val))\n",
    "print(xgb_clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAATWklEQVR4nO3dbYwd133f8e8vVCk3dpzK1fYhfDBph05Nx61YbKi0RtQgliwaKki/kGG6cMAUKggXYu1WCRq6CaSChgFFKdwUKBuLiNkaSRRGkYJ2EdNlFT+kCFLJu7JUO6RKaEWr4pZqtQlVu6gd0Sv/+2JH7tX1kjvcvcv1Hn4/wGLnnDln7n/Axe8O586dSVUhSWrX9612AZKklWXQS1LjDHpJapxBL0mNM+glqXHXrHYBw66//vrasmXLapchSWvK448//idVNbbQuu+5oN+yZQtTU1OrXYYkrSlJ/vvF1nnqRpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWuV9An2ZXkdJLpJAcvMe72JJVkfKDvI92800luHUXRkqT+Fr2OPsk64DBwCzADTCaZqKpTQ+N+APgQ8NhA33ZgL/A24IeA30/ylqp6eXS7IEm6lD5H9DuB6ao6U1UXgGPAngXGfRS4D/izgb49wLGqeqmqvgpMd9uTJF0hfb4ZuwE4O9CeAW4cHJBkB7Cpqn4vyc8NzX10aO6G4RdIsh/YD7B58+Z+lUuraMvBT498m8/ee9vItylBvyP6LND3ncdSJfk+4F8CP3u5c7/TUXWkqsaranxsbMFbNUiSlqjPEf0MsGmgvRE4N9D+AeBHgS8kAfgrwESS3T3mSpJWWJ8j+klgW5KtSdYz/+HqxCsrq+prVXV9VW2pqi3Mn6rZXVVT3bi9Sa5NshXYBnxx5HshSbqoRY/oq2ouyQHgBLAOOFpVJ5McAqaqauISc08meRA4BcwBd3rFjSRdWb1uU1xVx4HjQ313X2TsTw61PwZ8bIn1SZKWyW/GSlLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuN6BX2SXUlOJ5lOcnCB9R9M8pUkTyb5wyTbu/4tSb7Z9T+Z5BOj3gFJ0qUt+ijBJOuAw8AtwAwwmWSiqk4NDHugqj7Rjd8NfBzY1a17pqpuGG3ZkqS++hzR7wSmq+pMVV0AjgF7BgdU1dcHmq8FanQlSpKWo0/QbwDODrRnur5XSXJnkmeA+4APDazamuSJJH+Q5CcWeoEk+5NMJZmanZ29jPIlSYvpE/RZoO+7jtir6nBVvRn4eeAXu+7ngc1VtQO4C3ggyesXmHukqsaranxsbKx/9ZKkRfUJ+hlg00B7I3DuEuOPAe8BqKqXqupPu+XHgWeAtyytVEnSUvQJ+klgW5KtSdYDe4GJwQFJtg00bwOe7vrHug9zSfImYBtwZhSFS5L6WfSqm6qaS3IAOAGsA45W1ckkh4CpqpoADiS5GfgW8CKwr5t+E3AoyRzwMvDBqjq/EjsiSVrYokEPUFXHgeNDfXcPLH/4IvMeBh5eToGSpOXxm7GS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMb1Cvoku5KcTjKd5OAC6z+Y5CtJnkzyh0m2D6z7SDfvdJJbR1m8JGlxiwZ998zXw8C7ge3A+weDvPNAVb29qm4A7gM+3s3dzvwzZt8G7AL+zSvPkJUkXRl9juh3AtNVdaaqLgDHgD2DA6rq6wPN1wLVLe8BjlXVS1X1VWC6254k6Qrp88zYDcDZgfYMcOPwoCR3AncB64GfGpj76NDcDUuqVJK0JH2O6LNAX31XR9Xhqnoz8PPAL17O3CT7k0wlmZqdne1RkiSprz5BPwNsGmhvBM5dYvwx4D2XM7eqjlTVeFWNj42N9ShJktRXn6CfBLYl2ZpkPfMfrk4MDkiybaB5G/B0tzwB7E1ybZKtwDbgi8svW5LU16Ln6KtqLskB4ASwDjhaVSeTHAKmqmoCOJDkZuBbwIvAvm7uySQPAqeAOeDOqnp5hfZFkrSAPh/GUlXHgeNDfXcPLH/4EnM/BnxsqQVKkpbHb8ZKUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS43oFfZJdSU4nmU5ycIH1dyU5leTLST6b5I0D615O8mT3MzE8V5K0shZ9lGCSdcBh4BZgBphMMlFVpwaGPQGMV9U3kvxD4D7gfd26b1bVDSOuW5LUU58j+p3AdFWdqaoLwDFgz+CAqvp8VX2jaz4KbBxtmZKkperzcPANwNmB9gxw4yXG3wF8ZqD9miRTwBxwb1X9+8uuUpK+x2w5+OmRb/PZe28b+TahX9Bngb5acGDyAWAc+DsD3Zur6lySNwGfS/KVqnpmaN5+YD/A5s2bexUuSeqnz6mbGWDTQHsjcG54UJKbgV8AdlfVS6/0V9W57vcZ4AvAjuG5VXWkqsaranxsbOyydkCSdGl9gn4S2JZka5L1wF7gVVfPJNkB3M98yL8w0H9dkmu75euBdwCDH+JKklbYoqduqmouyQHgBLAOOFpVJ5McAqaqagL4ZeB1wO8kAXiuqnYDbwXuT/Jt5t9U7h26WkeStML6nKOnqo4Dx4f67h5Yvvki8/4IePtyCpQkLY/fjJWkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TG9Qr6JLuSnE4yneTgAuvvSnIqyZeTfDbJGwfW7UvydPezb5TFS5IWt2jQJ1kHHAbeDWwH3p9k+9CwJ4DxqvrrwEPAfd3cNwD3ADcCO4F7klw3uvIlSYvpc0S/E5iuqjNVdQE4BuwZHFBVn6+qb3TNR4GN3fKtwCNVdb6qXgQeAXaNpnRJUh99gn4DcHagPdP1XcwdwGcuZ26S/UmmkkzNzs72KEmS1FefoM8CfbXgwOQDwDjwy5czt6qOVNV4VY2PjY31KEmS1FefoJ8BNg20NwLnhgcluRn4BWB3Vb10OXMlSSunT9BPAtuSbE2yHtgLTAwOSLIDuJ/5kH9hYNUJ4F1Jrus+hH1X1ydJukKuWWxAVc0lOcB8QK8DjlbVySSHgKmqmmD+VM3rgN9JAvBcVe2uqvNJPsr8mwXAoao6vyJ7Ikla0KJBD1BVx4HjQ313DyzffIm5R4GjSy1QkrQ8fjNWkhrX64hebdty8NMj3+az99428m1KWhqP6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWpcr6BPsivJ6STTSQ4usP6mJF9KMpfk9qF1Lyd5svuZGJ4rSVpZiz54JMk64DBwCzADTCaZqKpTA8OeA34G+LkFNvHNqrphBLVKkpagzxOmdgLTVXUGIMkxYA/wnaCvqme7dd9egRolScvQ59TNBuDsQHum6+vrNUmmkjya5D0LDUiyvxszNTs7exmbliQtpk/QZ4G+uozX2FxV48DfA34lyZu/a2NVR6pqvKrGx8bGLmPTkqTF9An6GWDTQHsjcK7vC1TVue73GeALwI7LqE+StEx9gn4S2JZka5L1wF6g19UzSa5Lcm23fD3wDgbO7UuSVt6iQV9Vc8AB4ATwFPBgVZ1McijJboAkP5ZkBngvcH+Sk930twJTSf4r8Hng3qGrdSRJK6zPVTdU1XHg+FDf3QPLk8yf0hme90fA25dZoyRpGfxmrCQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWuV9An2ZXkdJLpJAcXWH9Tki8lmUty+9C6fUme7n72japwSVI/iwZ9knXAYeDdwHbg/Um2Dw17DvgZ4IGhuW8A7gFuBHYC9yS5bvllS5L66nNEvxOYrqozVXUBOAbsGRxQVc9W1ZeBbw/NvRV4pKrOV9WLwCPArhHULUnqqU/QbwDODrRnur4+es1Nsj/JVJKp2dnZnpuWJPXRJ+izQF/13H6vuVV1pKrGq2p8bGys56YlSX30CfoZYNNAeyNwruf2lzNXkjQCfYJ+EtiWZGuS9cBeYKLn9k8A70pyXfch7Lu6PknSFbJo0FfVHHCA+YB+Cniwqk4mOZRkN0CSH0syA7wXuD/JyW7ueeCjzL9ZTAKHuj5J0hVyTZ9BVXUcOD7Ud/fA8iTzp2UWmnsUOLqMGiVJy+A3YyWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxvYI+ya4kp5NMJzm4wPprk/x2t/6xJFu6/i1Jvpnkye7nE6MtX5K0mEUfJZhkHXAYuAWYASaTTFTVqYFhdwAvVtUPJ9kL/BLwvm7dM1V1w4jrliT11OeIficwXVVnquoCcAzYMzRmD/Cpbvkh4J1JMroyJUlL1SfoNwBnB9ozXd+CY6pqDvga8Be7dVuTPJHkD5L8xEIvkGR/kqkkU7Ozs5e1A5KkS+sT9AsdmVfPMc8Dm6tqB3AX8ECS13/XwKojVTVeVeNjY2M9SpIk9dUn6GeATQPtjcC5i41Jcg3wg8D5qnqpqv4UoKoeB54B3rLcoiVJ/fUJ+klgW5KtSdYDe4GJoTETwL5u+Xbgc1VVSca6D3NJ8iZgG3BmNKVLkvpY9KqbqppLcgA4AawDjlbVySSHgKmqmgA+Cfx6kmngPPNvBgA3AYeSzAEvAx+sqvMrsSOSpIUtGvQAVXUcOD7Ud/fA8p8B711g3sPAw8usUZK0DH4zVpIa1+uIXqtjy8FPj3ybz95728i3KfU16r9p/577aS7o/UOSpFfz1I0kNc6gl6TGGfSS1DiDXpIaZ9BLUuOau+pG8sor6dU8opekxhn0ktQ4g16SGmfQS1LjDHpJapxX3eiK8WoYaXV4RC9JjfOIfgm8fbD0vc3/Pb5aryP6JLuSnE4yneTgAuuvTfLb3frHkmwZWPeRrv90kltHV7okqY9Fj+i7h3sfBm4BZoDJJBNVdWpg2B3Ai1X1w0n2Ar8EvC/JduafH/s24IeA30/ylqp6edQ7Imnp/F9q2/qcutkJTFfVGYAkx4A9wGDQ7wH+ebf8EPCvk6TrP1ZVLwFf7R4evhP4L6MpX2qbAaxRSFVdekByO7Crqv5B1/5p4MaqOjAw5o+7MTNd+xngRubD/9Gq+o2u/5PAZ6rqoaHX2A/s75o/Apxe/q4t6nrgT67A61wpLe1PS/sCbe1PS/sCbe3PG6tqbKEVfY7os0Df8LvDxcb0mUtVHQGO9KhlZJJMVdX4lXzNldTS/rS0L9DW/rS0L9De/lxMnw9jZ4BNA+2NwLmLjUlyDfCDwPmecyVJK6hP0E8C25JsTbKe+Q9XJ4bGTAD7uuXbgc/V/DmhCWBvd1XOVmAb8MXRlC5J6mPRUzdVNZfkAHACWAccraqTSQ4BU1U1AXwS+PXuw9bzzL8Z0I17kPkPbueAO7+Hrri5oqeKroCW9qelfYG29qelfYH29mdBi34YK0la27wFgiQ1zqCXpMZddUG/2O0c1pIkm5J8PslTSU4m+fBq17RcSdYleSLJ7612LcuV5C8keSjJf+v+jf7Wate0HEn+Sfd39sdJfivJa1a7psuR5GiSF7rv/bzS94YkjyR5uvt93WrWuFKuqqAfuJ3Du4HtwPu72zSsVXPAz1bVW4EfB+5c4/sD8GHgqdUuYkT+FfAfq+qvAX+DNbxfSTYAHwLGq+pHmb8wY+/qVnXZ/h2wa6jvIPDZqtoGfLZrN+eqCnoGbudQVReAV27nsCZV1fNV9aVu+f8wHyQbVreqpUuyEbgN+LXVrmW5krweuIn5K9KoqgtV9b9Xt6pluwb48913Zb6fNfadmKr6z8xfFThoD/CpbvlTwHuuaFFXyNUW9BuAswPtGdZwMA7q7hi6A3hsdStZll8B/inw7dUuZATeBMwC/7Y7FfVrSV672kUtVVX9D+BfAM8BzwNfq6r/tLpVjcRfrqrnYf7ACfhLq1zPirjagr7XLRnWmiSvAx4G/nFVfX2161mKJH8XeKGqHl/tWkbkGuBvAr9aVTuA/8saPi3QnbveA2xl/k60r03ygdWtSn1dbUHf3C0Zkvw55kP+N6vqd1e7nmV4B7A7ybPMn1L7qSS/sbolLcsMMFNVr/wP6yHmg3+tuhn4alXNVtW3gN8F/vYq1zQK/yvJXwXofr+wyvWsiKst6PvczmHN6G4F/Ungqar6+GrXsxxV9ZGq2lhVW5j/d/lcVa3ZI8aq+p/A2SQ/0nW9k1ff2nuteQ748STf3/3dvZM1/OHygMHbt+wD/sMq1rJirqpHCV7sdg6rXNZyvAP4aeArSZ7s+v5ZVR1fxZr0//0j4De7g4ozwN9f5XqWrKoeS/IQ8CXmr/Z6gjV2+4AkvwX8JHB9khngHuBe4MEkdzD/Zvbe1atw5XgLBElq3NV26kaSrjoGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWrc/wMEibj8/QhnngAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.bar(range(len(xgb_clf.feature_importances_)), xgb_clf.feature_importances_)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['age', 'workclass', 'fnlwgt', 'education', 'marital-status',\n",
       "       'occupation', 'relationship', 'race', 'sex', 'hours-per-week',\n",
       "       'native-country', 'capital'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time: 2.5396552085876465s\n",
      "0.8757548184299777\n",
      "0.8717549750225206\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "cat_clf = CatBoostClassifier(\n",
    "    iterations=300,\n",
    "    learning_rate=0.1,\n",
    "    loss_function='CrossEntropy', verbose=False).fit(X_train, y_train)\n",
    "stop = time.time()\n",
    "print(f\"Training time: {stop - start}s\")\n",
    "print(cat_clf.score(X_val, y_val))\n",
    "print(cat_clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAMF0lEQVR4nO3dX4hmhXnH8e+vrqWNpnRlR9mqdNIgNlKIhsHaCsHWpphYqoEGIlSkWDYX2moRytab5HIvEtNeFGETrQu1lqAGpUoa2QoSKNJZI3HtNhjs1qzZuiPSKr1J1acXc7ZZJzPO7Ptn3n1evx8Y3vc975/zHHb5ztmz58ykqpAk9fMzsx5AkjQaAy5JTRlwSWrKgEtSUwZckprasZ0r27VrVy0uLm7nKiWpvUOHDr1eVQtrl29rwBcXF1leXt7OVUpSe0n+Y73lHkKRpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekprb1SkxJ6mBx7xMT/8yj+66f+Ge6By5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJamrTgCe5OMnTSY4keTHJHcPyLyV5Ncnzw9dnpj+uJOmkrfwslLeBu6rquSQfBg4leWp47qtV9eXpjSdJ2simAa+q48Dx4f5bSY4AF057MEnS+zutY+BJFoErgGeHRbcn+V6S+5Ps3OA9e5IsJ1leWVkZa1hJ0k9sOeBJzgUeAe6sqjeBe4GPApezuof+lfXeV1X7q2qpqpYWFhYmMLIkCbYY8CRnsxrvB6vqUYCqeq2q3qmqd4GvAVdOb0xJ0lpbOQslwH3Akaq655Tlu0952WeBw5MfT5K0ka2chXI1cDPwQpLnh2V3AzcluRwo4CjwhalMKEla11bOQvkOkHWeenLy40iStsorMSWpKQMuSU0ZcElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDVlwCWpKQMuSU0ZcElqyoBLUlMGXJKaMuCS1NSmAU9ycZKnkxxJ8mKSO4bl5yV5KslLw+3O6Y8rSTppK3vgbwN3VdXHgKuA25JcBuwFDlbVJcDB4bEkaZtsGvCqOl5Vzw333wKOABcCNwAHhpcdAG6c1pCSpJ92WsfAkywCVwDPAhdU1XFYjTxw/gbv2ZNkOcnyysrKeNNKkv7flgOe5FzgEeDOqnpzq++rqv1VtVRVSwsLC6PMKElax5YCnuRsVuP9YFU9Oix+Lcnu4fndwInpjChJWs9WzkIJcB9wpKruOeWpx4Fbhvu3AI9NfjxJ0kZ2bOE1VwM3Ay8keX5YdjewD/hGkluBV4DPTWdESdJ6Ng14VX0HyAZPXzvZcSRJW+WVmJLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDVlwCWpKQMuSU0ZcElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNbRrwJPcnOZHk8CnLvpTk1STPD1+fme6YkqS1trIH/gBw3TrLv1pVlw9fT052LEnSZjYNeFU9A7yxDbNIkk7DOMfAb0/yveEQy86NXpRkT5LlJMsrKytjrE6SdKpRA34v8FHgcuA48JWNXlhV+6tqqaqWFhYWRlydJGmtkQJeVa9V1TtV9S7wNeDKyY4lSdrMjlHelGR3VR0fHn4WOPx+r9d8W9z7xMQ/8+i+6yf+mdK82TTgSR4CrgF2JTkGfBG4JsnlQAFHgS9McUZJ0jo2DXhV3bTO4vumMIsk6TR4JaYkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKZGupR+FrxcW5Leyz1wSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDVlwCWpKQMuSU0ZcElqyoBLUlMGXJKaMuCS1JQBl6SmNg14kvuTnEhy+JRl5yV5KslLw+3O6Y4pSVprK3vgDwDXrVm2FzhYVZcAB4fHkqRttGnAq+oZ4I01i28ADgz3DwA3TnguSdImRj0GfkFVHQcYbs/f6IVJ9iRZTrK8srIy4uokSWtN/T8xq2p/VS1V1dLCwsK0VydJHxijBvy1JLsBhtsTkxtJkrQVO0Z83+PALcC+4faxiU2kiVrc+8REP+/ovusn+nmSRreV0wgfAv4ZuDTJsSS3shruTyV5CfjU8FiStI023QOvqps2eOraCc8iSToNXokpSU0ZcElqyoBLUlMGXJKaGvU0QmluTfrUS/D0S02HAZc0Nr/pzYaHUCSpKQMuSU0ZcElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNWXAJakpr8RUG17tJ72Xe+CS1JQBl6SmDLgkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNjfXTCJMcBd4C3gHerqqlSQwlSdrcJH6c7G9V1esT+BxJ0mnwEIokNTVuwAv4dpJDSfas94Ike5IsJ1leWVkZc3WSpJPGDfjVVfUJ4NPAbUk+ufYFVbW/qpaqamlhYWHM1UmSThrrGHhV/Wi4PZHkm8CVwDOTGGzeTfrXg/mrwaQPnpH3wJOck+TDJ+8DvwscntRgkqT3N84e+AXAN5Oc/Jy/q6pvTWQqSdKmRg54Vb0MfHyCs0iSToOnEUpSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDVlwCWpKQMuSU0ZcElqaqzfiTmPJv27KsHfVylpOtwDl6SmDLgkNeUhFGnOTfqwoIcEzxzugUtSU+6BS2rDkwzeyz1wSWrKgEtSUwZckpryGLg0I54donG5By5JTRlwSWrKgEtSUwZckpoaK+BJrkvy/SQ/SLJ3UkNJkjY3csCTnAX8NfBp4DLgpiSXTWowSdL7G2cP/ErgB1X1clX9GPh74IbJjCVJ2kyqarQ3Jn8AXFdVfzw8vhn49aq6fc3r9gB7hoeXAt8ffdwt2QW8PuV1bCe358w1T9sC87U987QtAL9cVQtrF45zIU/WWfZT3w2qaj+wf4z1nJYky1W1tF3rmza358w1T9sC87U987Qt72ecQyjHgItPeXwR8KPxxpEkbdU4Af8X4JIkH0nys8DngccnM5YkaTMjH0KpqreT3A78I3AWcH9VvTixyUa3bYdrtonbc+aap22B+dqeedqWDY38n5iSpNnySkxJasqAS1JTcxXwebq0P8nFSZ5OciTJi0numPVM40pyVpLvJvmHWc8yriS/mOThJP82/Bn9xqxnGlWSPxv+jh1O8lCSn5v1TKcjyf1JTiQ5fMqy85I8leSl4XbnLGeclrkJ+Bxe2v82cFdVfQy4Crit+fYA3AEcmfUQE/JXwLeq6leBj9N0u5JcCPwpsFRVv8bqCQmfn+1Up+0B4Lo1y/YCB6vqEuDg8HjuzE3AmbNL+6vqeFU9N9x/i9VAXDjbqUaX5CLgeuDrs55lXEl+AfgkcB9AVf24qv5rtlONZQfw80l2AB+i2fUcVfUM8MaaxTcAB4b7B4Abt3WobTJPAb8Q+OEpj4/ROHinSrIIXAE8O9tJxvKXwJ8D7856kAn4FWAF+JvhkNDXk5wz66FGUVWvAl8GXgGOA/9dVd+e7VQTcUFVHYfVnSHg/BnPMxXzFPAtXdrfTZJzgUeAO6vqzVnPM4okvwecqKpDs55lQnYAnwDuraorgP+h6T/Rh2PDNwAfAX4JOCfJH852Km3VPAV87i7tT3I2q/F+sKoenfU8Y7ga+P0kR1k9tPXbSf52tiON5RhwrKpO/ovoYVaD3tHvAP9eVStV9b/Ao8BvznimSXgtyW6A4fbEjOeZinkK+Fxd2p8krB5jPVJV98x6nnFU1V9U1UVVtcjqn8s/VVXbvbyq+k/gh0kuHRZdC/zrDEcaxyvAVUk+NPydu5am/yG7xuPALcP9W4DHZjjL1Izz0wjPKGfwpf2juhq4GXghyfPDsrur6skZzqSf+BPgwWFn4WXgj2Y8z0iq6tkkDwPPsXrm03dpdhl6koeAa4BdSY4BXwT2Ad9Iciur36Q+N7sJp8dL6SWpqXk6hCJJHygGXJKaMuCS1JQBl6SmDLgkNWXAJakpAy5JTf0fqmrBmOVt8/MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.bar(range(len(cat_clf.feature_importances_)), cat_clf.feature_importances_)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison (with target encoding):\n",
    "| Model | Training Time | Validation Accuracy | Test Accuracy |\n",
    "| --- | --- | --- | --- |\n",
    "| GradientBoosting | 2.165 | 86.66 | 86.03 |\n",
    "| CatBoost | 2.539 | 87.57 | 87.17 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion and Questions\n",
    "\n",
    "## (1)\n",
    "The table below summarizes the results for different models and encoding methods\n",
    "\n",
    "| Model | Encoding Method | Training Time | Validation Accuracy | Test Accuracy |\n",
    "| --- | --- | --- | --- | --- |\n",
    "| GradientBoosting | One-hot | 2.281 | 86.88 | 86.63 |\n",
    "| CatBoost | One-hot | 2.806 | (87.66) | 87.11 |\n",
    "| GradientBoosting | Label Encoding | 2.154 | 86.87 | 86.11 |\n",
    "| CatBoost | Label Encoding | 2.547 | 87.52 | 87.10 |\n",
    "| GradientBoosting | Target Encoding | 2.165 | 86.66 | 86.03 |\n",
    "| CatBoost | Target Encoding | 2.539 | 87.57 | (87.17) |\n",
    "\n",
    "It is evident from the results above that CatBoost is performing slightly better in all cases. \n",
    "## (2)\n",
    "I tested three different encoding methods. I expected label encoding to perform the worst since this method is too simple and usually sub-optimal, especially for models using distance metrics. It turns out for this dataset, and with these two methods, different encoding methods don't have a significant impact on the performance. In my opinion, this might be due to existing powerful numerical features like \"capital\" or \"age\" (refer to plots above). Another evidence that this might be the case is that, for a feature like \"native-country\" with 41 (and the most) unique values, the feature importance factor is really low. In other words, the feature does not predict the outcome very well, so the encoding method does not alter the performance as much.\n",
    "\n",
    "For the final model, I would suggest using Target encoding since it performs almost the same as One-hot, but it does not have the sparsification problem, which could be a bigger issue if the data had more unique categories or more categorical features.\n",
    "\n",
    "## (3)\n",
    "I want to briefly mention two ideas that may improve the results:\n",
    "\n",
    "First, I used a very simple method to deal with missing values. Although I think it may not have a significant impact on performance, there are many more imputation methods worth experimenting with.\n",
    "\n",
    "Second, I would like to use cross-validation to obtain a more reliable guarantee (not necessarily a higher score) as well as being able to tune hyperparameters even more and search in a wider space.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}