Models_new.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import modules\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the training set\n",
    "Trainset = pd.read_csv('Kickstarter_Train.csv')\n",
    "Seed = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing of the data\n",
    "\n",
    "Before we can start modelling we need to chose the features which will be used. Secondly we need to hot encode out 'category' variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# There are only a few useful parameters we can pass to our prediction algorithms\n",
    "Trainset = Trainset[['blurb_length', 'category', 'converted_goal_amount',\n",
    "                     'name_length', 'days_total', 'days_until_launch', 'state']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hot-one-encoding of the category parameter\n",
    "category_dummies = pd.get_dummies(Trainset['category'], drop_first=True)\n",
    "\n",
    "# Put them together and drop the original category column\n",
    "Trainset = pd.concat([Trainset, category_dummies], axis=1)\n",
    "Trainset.drop('category', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "There is a lot of data, so another split into train and test can be performed.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the set into X and y\n",
    "from sklearn.model_selection import train_test_split\n",
    "X = Trainset.drop('state', axis=1)\n",
    "y = Trainset.state\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "For scaling we use the RobustScaler as it is more sensitive towards outliers, which are present in the data.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import RobustScaler\n",
    "from sklearn.preprocessing import RobustScaler\n",
    "\n",
    "# Apply Scaling\n",
    "features = ['blurb_length', 'converted_goal_amount', 'name_length', 'days_total', 'days_until_launch']\n",
    "transformer = RobustScaler().fit(X[features]) # Fit the scaler on the complete Trainset. Later one we need to do the same with the validation set.\n",
    "X[features] = transformer.transform(X[features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Next up we import everything we need for the whole modelling process.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import models and other needed modules\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.tree import DecisionTreeClassifier \n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.metrics import precision_recall_curve\n",
    "from sklearn.metrics import plot_precision_recall_curve\n",
    "from sklearn.metrics import average_precision_score\n",
    "from sklearn.metrics import auc\n",
    "from sklearn.metrics import precision_recall_curve\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.metrics import average_precision_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The focus is on recall, so a scorer is needed for later use in cross-validation\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define scorer\n",
    "#scorer = make_scorer(recall_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "For evaluation we are going to use a few different parameters:    \n",
    "<ul>\n",
    "<li>Confusion matrix</li>\n",
    "<li>Classifciation report, which inclued recall, precision, f1-score etc.</li>\n",
    "<li>The singular recall score</li>\n",
    "<li>The Precision-Recall curve and the auc (area under curve)</li>\n",
    "</ul>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Empty df for the metrics we want to compare in the end\n",
    "df_metrics = pd.DataFrame({'model' : ['Dummy', 'KNN', 'DT', 'LR', 'RF'], 'recall' : [1,2,3,4,5], 'auc' : [1,2,3,4,5]})\n",
    "df_metrics.set_index('model', inplace=True)\n",
    "\n",
    "# Print out all evaluation metrics we want\n",
    "def evaluation_metrics (y, y_pred, y_proba, model='default'):\n",
    "    pos_probs = y_proba[:, 1]\n",
    "    precision, recall, _ = precision_recall_curve(y, pos_probs)\n",
    "    recall_real = recall_score(y, y_pred).round(2)\n",
    "    auc_real = auc(recall, precision).round(2)\n",
    "    if model != 'default':\n",
    "        df_metrics.loc[model, 'recall'], df_metrics.loc[model, 'auc'] = recall_real, auc_real\n",
    "    print('---------------------------------------------------------')\n",
    "    print('Confusion Matrix:')\n",
    "    print(confusion_matrix(y, y_pred))\n",
    "    print('---------------------------------------------------------')\n",
    "    print(classification_report(y, y_pred))\n",
    "    print('---------------------------------------------------------')\n",
    "    print(f'Recall score: {recall_score(y, y_pred).round(2)}')\n",
    "    print('---------------------------------------------------------')\n",
    "    print(f'PR-AUC score: {auc(recall, precision).round(2)}')\n",
    "    print('---------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Best parameters from Grid-RandomizedSearch\n",
    "\n",
    "def best_parameters(model_name):\n",
    "    print('---------------')\n",
    "    print('Best parameters')\n",
    "    print('---------------')\n",
    "    for pair in model_name.best_params_.items():\n",
    "        print(f'{pair[0].capitalize()} : {pair[1]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline\n",
    "\n",
    "The dummy classifier can be used in multiple ways. One of the most common would be to simply let the classifier predict everything accoding to the most frequent class, which means it would in this case only predict positives, since 58% of all outcomes are positive. This would lead to a recall score of 1, as there are no false negatives.\n",
    "This is not what we want as baseline, since our goal is to achieve a high recall with a good area under curve score. The baseline would not offer good comparables.\n",
    "Therefore we chose to use another approach and let the classifier chose the outcome somewhat randomly, while still keeping the imbalance of the two classes in mind.\n",
    "This actually gives us a baseline which we can improve upon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[19570 27576]\n",
      " [27580 38494]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.42      0.42      0.42     47146\n",
      "           1       0.58      0.58      0.58     66074\n",
      "\n",
      "    accuracy                           0.51    113220\n",
      "   macro avg       0.50      0.50      0.50    113220\n",
      "weighted avg       0.51      0.51      0.51    113220\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.58\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.7\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Define model\n",
    "dummy_clf = DummyClassifier(strategy=\"stratified\", random_state=Seed) # 'stratified' generates predictions by respecting the training set’s class distribution.\n",
    "\n",
    "# Fit on the data\n",
    "dummy_clf.fit(X_train, y_train)\n",
    "\n",
    "# Predict the target value\n",
    "y_pred_dummy = dummy_clf.predict(X_train)\n",
    "\n",
    "# Calculate probability and precision/recall-auc-score\n",
    "y_proba_dummy = dummy_clf.predict_proba(X_train)\n",
    "\n",
    "evaluation_metrics(y_train, y_pred_dummy, y_proba_dummy, 'Dummy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Our baseline scores are 0.58 for Recall and 0.7 for AUC.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KNN\n",
    "### Trainset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.0min\n",
      "[Parallel(n_jobs=-1)]: Done  50 out of  50 | elapsed:  1.1min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------\n",
      "Best parameters\n",
      "---------------\n",
      "N_neighbors : 35\n",
      "Metric : manhattan\n"
     ]
    }
   ],
   "source": [
    "# Define model and parameters for RandomizedSearchCV\n",
    "knn = KNeighborsClassifier()\n",
    "param_dist_knn = {'n_neighbors' : np.linspace(5, 100, num = 20, dtype='int64').tolist(),\n",
    "                  'metric' : ['manhattan', 'euclidean']\n",
    "                 }\n",
    "\n",
    "# RandomizedSearchCV\n",
    "knn_search = RandomizedSearchCV(knn, param_dist_knn, verbose=1, n_jobs=-1, random_state=Seed, cv=5)\n",
    "knn_search.fit(X_train, y_train)\n",
    "\n",
    "# Print bester parameters\n",
    "best_parameters(knn_search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[23065 24081]\n",
      " [10096 55978]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.70      0.49      0.57     47146\n",
      "           1       0.70      0.85      0.77     66074\n",
      "\n",
      "    accuracy                           0.70    113220\n",
      "   macro avg       0.70      0.67      0.67    113220\n",
      "weighted avg       0.70      0.70      0.69    113220\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.85\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.78\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Use the best parameters \n",
    "knn_best = knn_search.best_estimator_\n",
    "\n",
    "# Fit on the traindata\n",
    "knn_best.fit(X_train, y_train)\n",
    "\n",
    "# Predict the target value\n",
    "y_pred_knn = knn_best.predict(X_train)\n",
    "\n",
    "y_proba_knn = knn_best.predict_proba(X_train)\n",
    "\n",
    "evaluation_metrics (y_train, y_pred_knn, y_proba_knn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "A Recall score of 0.85, while maintaining a AUC of 0.78 looks promising. Remember though, that this is done only with the trainingset.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[ 7307  8408]\n",
      " [ 3691 18334]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.66      0.46      0.55     15715\n",
      "           1       0.69      0.83      0.75     22025\n",
      "\n",
      "    accuracy                           0.68     37740\n",
      "   macro avg       0.67      0.65      0.65     37740\n",
      "weighted avg       0.68      0.68      0.67     37740\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.83\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.74\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Predict the Testset\n",
    "y_pred_knn_test = knn_best.predict(X_test)\n",
    "\n",
    "y_proba_knn_test = knn_best.predict_proba(X_test)\n",
    "\n",
    "evaluation_metrics (y_test, y_pred_knn_test, y_proba_knn_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Recall is still good. The AUC gets close to the baseline of 0.7 though.\n",
    "We have to look at the validation set later to see how it holds up.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree Classifier\n",
    "### Trainset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 100 candidates, totalling 500 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:    2.7s\n",
      "[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:   14.1s\n",
      "[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:   30.3s\n",
      "[Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   34.9s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------\n",
      "Best parameters\n",
      "---------------\n",
      "Splitter : best\n",
      "Min_samples_split : 16\n",
      "Max_leaf_nodes : 89\n",
      "Max_features : None\n",
      "Max_depth : 90\n",
      "Criterion : gini\n"
     ]
    }
   ],
   "source": [
    "# Define model and parameters for GridSearch\n",
    "tree = DecisionTreeClassifier(random_state=Seed, class_weight='balanced')\n",
    "param_dist_tree = {'criterion' : ['gini', 'entropy'],\n",
    "                   'splitter' : ['best', 'random'],\n",
    "                   'max_depth' : np.linspace(5, 100, num = 20, dtype='int64').tolist(),\n",
    "                   'max_features' : ['sqrt', 'log2', None],\n",
    "                   'min_samples_split' : np.linspace(5, 20, num = 5, dtype='int64').tolist(),\n",
    "                   'max_leaf_nodes' : np.linspace(1, 100, num = 50, dtype='int64').tolist() + [None]\n",
    "                  }\n",
    "\n",
    "# Run GridSearch\n",
    "tree_search = RandomizedSearchCV(tree, param_dist_tree, cv=5, n_jobs=-1, verbose=1, n_iter=100)\n",
    "tree_search.fit(X_train, y_train)\n",
    "\n",
    "# Print best parameters\n",
    "best_parameters(tree_search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[30231 16915]\n",
      " [17732 48342]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.63      0.64      0.64     47146\n",
      "           1       0.74      0.73      0.74     66074\n",
      "\n",
      "    accuracy                           0.69    113220\n",
      "   macro avg       0.69      0.69      0.69    113220\n",
      "weighted avg       0.69      0.69      0.69    113220\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.73\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.78\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Use the best parameters on the model\n",
    "tree_best = tree_search.best_estimator_\n",
    "\n",
    "# Fit on the data\n",
    "tree_best.fit(X_train, y_train)\n",
    "\n",
    "# Predict the target value\n",
    "y_pred_tree = tree_best.predict(X_train)\n",
    "y_proba_tree = tree_best.predict_proba(X_train)\n",
    "\n",
    "evaluation_metrics(y_train, y_pred_tree, y_proba_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The Recall score on the trainset is 0.77, which is worse than KNN.  \n",
    "The Curve score is better with 0.79.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[10156  5559]\n",
      " [ 5949 16076]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.63      0.65      0.64     15715\n",
      "           1       0.74      0.73      0.74     22025\n",
      "\n",
      "    accuracy                           0.70     37740\n",
      "   macro avg       0.69      0.69      0.69     37740\n",
      "weighted avg       0.70      0.70      0.70     37740\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.73\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.78\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Testset\n",
    "y_pred_tree_test = tree_best.predict(X_test)\n",
    "y_proba_tree_test = tree_best.predict_proba(X_test)\n",
    "\n",
    "evaluation_metrics(y_test, y_pred_tree_test, y_proba_tree_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Recall and Curve score of the testset are nearly identical to the trainset, which at least speaks for the models consistency.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "### Trainset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 7 candidates, totalling 35 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  35 out of  35 | elapsed:    8.2s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------\n",
      "Best parameters\n",
      "---------------\n",
      "C : 0.05\n"
     ]
    }
   ],
   "source": [
    "# Define model and parameters for GridSearch\n",
    "lr = LogisticRegression(random_state=Seed, class_weight='balanced')\n",
    "param_dist_lr = {'C' : [0.01, 0.025, 0.05, 0.1, 0.5, 1, 2]\n",
    "                }\n",
    "\n",
    "# Run GridSearch\n",
    "lr_search = GridSearchCV(lr, param_dist_lr, cv=5, n_jobs=-1, verbose=1)\n",
    "lr_search.fit(X_train, y_train)\n",
    "\n",
    "# Print the best parameters\n",
    "best_parameters(lr_search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[29998 17148]\n",
      " [24087 41987]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.55      0.64      0.59     47146\n",
      "           1       0.71      0.64      0.67     66074\n",
      "\n",
      "    accuracy                           0.64    113220\n",
      "   macro avg       0.63      0.64      0.63    113220\n",
      "weighted avg       0.65      0.64      0.64    113220\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.64\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.72\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Use the best parameters on the model\n",
    "lr_best = lr_search.best_estimator_\n",
    "\n",
    "# Fit on the data\n",
    "lr_best.fit(X_train, y_train)\n",
    "\n",
    "# Predict the target value\n",
    "y_pred_lr = lr_best.predict(X_train)\n",
    "y_proba_lr = lr_best.predict_proba(X_train)\n",
    "\n",
    "evaluation_metrics(y_train, y_pred_lr, y_proba_lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Recall 0.64 and AUC 0.72.\n",
    "Both are only slightly better than to chose the outcome at random.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[10052  5663]\n",
      " [ 7882 14143]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.56      0.64      0.60     15715\n",
      "           1       0.71      0.64      0.68     22025\n",
      "\n",
      "    accuracy                           0.64     37740\n",
      "   macro avg       0.64      0.64      0.64     37740\n",
      "weighted avg       0.65      0.64      0.64     37740\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.64\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.73\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Testset\n",
    "y_pred_lr_test = lr_best.predict(X_test)\n",
    "y_proba_lr_test = lr_best.predict_proba(X_test)\n",
    "\n",
    "# Print evaluation metrics\n",
    "evaluation_metrics(y_test, y_pred_lr_test, y_proba_lr_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The scores stay pretty much the same. We'd have to do some further analysis to find out if we could improve upon that.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest\n",
    "### Trainset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  2.6min\n",
      "[Parallel(n_jobs=-1)]: Done  50 out of  50 | elapsed:  2.9min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------\n",
      "Best parameters\n",
      "---------------\n",
      "Min_samples_split : 15\n",
      "Min_samples_leaf : 15\n",
      "Max_leaf_nodes : 100\n",
      "Max_features : log2\n",
      "Max_depth : None\n",
      "Class_weight : balanced_subsample\n",
      "Bootstrap : True\n"
     ]
    }
   ],
   "source": [
    "# Define model and parameters for RandomizedSearchCV\n",
    "rf = RandomForestClassifier(random_state=Seed, class_weight='balanced')\n",
    "param_dist_rf = {'max_features' : ['sqrt', 'log2'],\n",
    "                 'max_depth' : np.linspace(10, 110, num = 11).tolist() + [None],\n",
    "                 'min_samples_split' : [5, 10, 15],\n",
    "                 'min_samples_leaf' : [10,15,20],\n",
    "                 'bootstrap' : [True, False],\n",
    "                 'max_leaf_nodes' : [50,100],\n",
    "                 'class_weight' : ['balanced', 'balanced_subsample']\n",
    "                }\n",
    "\n",
    "# Run RandomizedSearchCV\n",
    "rf_search = RandomizedSearchCV(rf, param_dist_rf, cv = 5, n_iter=10,\n",
    "                               verbose=1, random_state=Seed, n_jobs = -1)\n",
    "rf_search.fit(X_train, y_train)\n",
    "\n",
    "# Print the best parameters\n",
    "best_parameters(rf_search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[31438 15708]\n",
      " [18379 47695]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.63      0.67      0.65     47146\n",
      "           1       0.75      0.72      0.74     66074\n",
      "\n",
      "    accuracy                           0.70    113220\n",
      "   macro avg       0.69      0.69      0.69    113220\n",
      "weighted avg       0.70      0.70      0.70    113220\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.72\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.8\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Use the best parameters on the model\n",
    "rf_best = rf_search.best_estimator_\n",
    "\n",
    "# Fit on the data\n",
    "rf_best.fit(X_train, y_train)\n",
    "\n",
    "# Predict the target value\n",
    "y_pred_rf = rf_best.predict(X_train)\n",
    "y_proba_rf = rf_best.predict_proba(X_train)\n",
    "\n",
    "# Print evaluation metrics\n",
    "evaluation_metrics(y_train, y_pred_rf, y_proba_rf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The recall score is quite low with 0.72, but we have the highest curve score yet, which suggests a good balance between precision and recall.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[10529  5186]\n",
      " [ 6152 15873]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.63      0.67      0.65     15715\n",
      "           1       0.75      0.72      0.74     22025\n",
      "\n",
      "    accuracy                           0.70     37740\n",
      "   macro avg       0.69      0.70      0.69     37740\n",
      "weighted avg       0.70      0.70      0.70     37740\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.72\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.8\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Predict testset\n",
    "y_pred_rf_test = rf_best.predict(X_test)\n",
    "y_proba_rf_test = rf_best.predict_proba(X_test)\n",
    "\n",
    "# Print evaluation metrics\n",
    "evaluation_metrics(y_test, y_pred_rf_test, y_proba_rf_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Pretty much the same as the trainset.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Validation on unused data\n",
    "\n",
    "We now important the validation set and transform it the same way as the trainset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "Val_data = pd.read_csv('Kickstarter_Validation.csv')\n",
    "Val_data = Val_data[['blurb_length', 'category', 'converted_goal_amount',\n",
    "            'name_length', 'days_total', 'days_until_launch', 'state']]\n",
    "\n",
    "# Hot-one-encoding of the category parameter\n",
    "category_dummies = pd.get_dummies(Val_data['category'], drop_first=True)\n",
    "\n",
    "# Put them together and drop the original category column\n",
    "Val_data = pd.concat([Val_data, category_dummies], axis=1)\n",
    "Val_data.drop('category', axis=1, inplace=True)\n",
    "\n",
    "# Split the set into X and y\n",
    "X_val = Val_data.drop('state', axis=1)\n",
    "y_val = Val_data.state\n",
    "\n",
    "# Apply scaling from trainset\n",
    "features = ['blurb_length', 'converted_goal_amount', 'name_length', 'days_total', 'days_until_launch']\n",
    "X_val[features] = transformer.transform(X_val[features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[    7 20947]\n",
      " [    0 29366]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.00      0.00     20954\n",
      "           1       0.58      1.00      0.74     29366\n",
      "\n",
      "    accuracy                           0.58     50320\n",
      "   macro avg       0.79      0.50      0.37     50320\n",
      "weighted avg       0.76      0.58      0.43     50320\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 1.0\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.59\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Predict the target value\n",
    "y_pred_knn_val = knn_best.predict(X_val)\n",
    "y_proba_knn_val = knn_best.predict_proba(X_val)\n",
    "\n",
    "# Print the evaluation metrics\n",
    "evaluation_metrics(y_val, y_pred_knn_val, y_proba_knn_val, 'KNN')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "KNN very obviously has a problem here. The Recall score is at 1 because the algorithm set nearly all predictions as positive. The curve score dropped below 0.6, which is worse than the baseline.  \n",
    "It is clear that the KNN in this iteration doesn't seem to be an appropriate algorithm for the problem at hand.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[  488 20466]\n",
      " [  359 29007]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.58      0.02      0.04     20954\n",
      "           1       0.59      0.99      0.74     29366\n",
      "\n",
      "    accuracy                           0.59     50320\n",
      "   macro avg       0.58      0.51      0.39     50320\n",
      "weighted avg       0.58      0.59      0.45     50320\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.99\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.68\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Predict the target value\n",
    "y_pred_tree_val = tree_best.predict(X_val)\n",
    "y_proba_tree_val = tree_best.predict_proba(X_val)\n",
    "\n",
    "# Print the confusion matrix and classification report\n",
    "evaluation_metrics(y_val, y_pred_tree_val, y_proba_tree_val, 'DT')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Just like KNN, the decision tree predicts nearly all outcomes as positive, which gives it a high recall, at the price of precision.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[ 7332 13622]\n",
      " [ 4351 25015]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.63      0.35      0.45     20954\n",
      "           1       0.65      0.85      0.74     29366\n",
      "\n",
      "    accuracy                           0.64     50320\n",
      "   macro avg       0.64      0.60      0.59     50320\n",
      "weighted avg       0.64      0.64      0.62     50320\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.85\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.69\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Predict the target value\n",
    "y_pred_lr_val = lr_best.predict(X_val)\n",
    "y_proba_lr_val = lr_best.predict_proba(X_val)\n",
    "\n",
    "# Print evaluation metrics\n",
    "evaluation_metrics(y_val, y_pred_lr_val, y_proba_lr_val, 'LR')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The logistic regression does far better than expected, witch good recall and an ok AUC.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Confusion Matrix:\n",
      "[[ 8075 12879]\n",
      " [ 6669 22697]]\n",
      "---------------------------------------------------------\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.55      0.39      0.45     20954\n",
      "           1       0.64      0.77      0.70     29366\n",
      "\n",
      "    accuracy                           0.61     50320\n",
      "   macro avg       0.59      0.58      0.58     50320\n",
      "weighted avg       0.60      0.61      0.60     50320\n",
      "\n",
      "---------------------------------------------------------\n",
      "Recall score: 0.77\n",
      "---------------------------------------------------------\n",
      "PR-AUC score: 0.69\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Predict the target value\n",
    "y_pred_rf_val = rf_best.predict(X_val)\n",
    "y_proba_rf_val = rf_best.predict_proba(X_val)\n",
    "\n",
    "# print evaluation metrics\n",
    "evaluation_metrics(y_val, y_pred_rf_val, y_proba_rf_val, 'RF')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The Random forest does ok with a recall of 0.77 and an auc of 0.69. \n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot the results for easy comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_metrics.reset_index(level=0, inplace=True) # we used the modelnames as row indeces, now we need them as a column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAAF1CAYAAAD4E9OzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAd8klEQVR4nO3df7QcZZ3n8feXJBoQDBAiAgGT0SgCEjABZY+MGRENCiILKODIj9WNqIjMOjugHh0c1x3U0QMskZgRjMiP7CBG+Y0CIjoIQ9iJKL80QDBXFJNAgiARknz3j6oLbXNvbiekbufpvF/n9LldVU9XfbvSJ59+nqquisxEkiSVZ7NuFyBJktaPIS5JUqEMcUmSCmWIS5JUKENckqRCGeKSJBXKEJcKEhHTIqKv23Wsr4g4PSIurJ9PiIiMiJHDuV2plxji2uRFxJsi4paIWBERj0bEv0fEPt2uq2kRMScino6IJ+r3/cOI2LXbdUnqnCGuTVpEvBS4Evg/wLbATsDngD9v4O2M2JDr24C+lJlbUr3v3wLndbkeSevAENem7tUAmXlJZq7OzKcy8weZeWd/g4j47xFxT0T8MSLujojX1/NfGxE3RcTyiLgrIt7V8po5EXFuRFwdEU8CfxMRO0bEZRGxJCIejIiTW9rvGxHzI+LxiHgkIr66tqIj4lMRsTQiFkXE++p5+9SvHdnS7vCIWDDUTsjMp4B/A/Zqee3a6h1R13B/vV/uiIid62VnRcTi+r3cERH7D7X9Qd7jaS3rvzsiDmtZdnxE/DQi/iUiHqvrO6hl+cSI+HH92h8C261lO9tExJX1+3ysfj6+ZfmiiHhry/RfDM23jOQsr9/38evzfqX1YYhrU/crYHVEfCsiDoqIbVoXRsSRwOnAscBLgXcByyJiFHAF8APgZcDHgIsi4jUtLz8G+AKwFXBL3f7nVL3eA4BTIuLtdduzgLMy86XAK6kCdTAvpwqlnYDjgNkR8ZrMvB1YBhzY0vZvgW8PtRMi4iXA0cDCenqzIer9H3X7d9T75b8Bf6qX3U71ZWBb4GLg0ogYPVQNA7gf2B8YQzU6cmFE7NCy/A3AfVT74kvAeRER9bKLgTvqZZ+n2k+D2Qz4JvAKYBfgKeCcTgqMiF2Aa6hGcsZRve8FnbxW2iAy04ePTfoBvBaYA/QBq4DLge3rZdcBHx/gNfsDvwc2a5l3CXB6/XwOcEHLsjcAv2lbxyeBb9bPb6YKqu2GqHVaXeNLWub9G/CZ+vmpwEX1822pgnWHQdY1B1gJLAfWAA8Ce3ZY733AoR3u38eAyfXz04EL6+cTgARGdrieBf3bBI4HFrYs26Je18upgrh9H13cv90OtrMX8FjL9CLgrS3Tre/hk8C8bn+GfWy6D3vi2uRl5j2ZeXxmjgf2AHYEzqwX70zVI2y3I7A4M9e0zHuIqtfab3HL81cAO9ZDrssjYjnwKWD7evkHqIb2742I2yPi4LWU/FhmPtm23R3r5xcCh0TElsB7gJ9k5u/Wsq5/ycytqQL1KaB/JGGoegfbL0TEJ+rDDyvq141hLcPZg4mIYyNiQcv292hbz+/7n2Rm/yjAllT7YqB9NNh2toiIr0fEQxHxONUXqq07PI9h0P0gDQdDXGqRmfdS9VD3qGctphrebvcwsHM97NxvF6qTw55dXcvzxcCDmbl1y2OrzHxHvd1fZ+bRVEPzXwS+Uw9xD2SbtmW71PWQmb8FfgYcBryfDobS69f9Bvg4cFZEbD5UvQyyX+rj36dSfYHYpv6CsAKI9rZrExGvAP4VOAkYW6/nlx2u53cMvI8G8wmqLy9vyOpwxl/3l1H/fZKqp9/v5S3PB/t8SMPCENcmLSJ2rXuO4+vpnamO9d5aN/kG8PcRMSUqr6oD5jaq/9z/ISJGRcQ04BBg7iCb+g/g8Yg4NSI2r08M2yPqn7JFxN9GxLi6Z7+8fs3qtZT+uYh4UR2aBwOXtiy7APgH4HXAvE73RWb+kOrLwIyh6q33y+cjYlK9X/aMiLFUx/9XAUuAkRHxWapj5uvqJVRfgpYARMQJPPfFaqj38RAwn+f20Zuo/m0GsxXVKMTyiNgW+Me25QuAo+p/56nAES3LLgLeGhHviYiRETE2IvbqpE5pQzDEtan7I9Xx39uiOov8Vqoe3ycAMvNSqpPTLq7bfg/YNjOfpjrJ7SBgKfA14Ni6J/88mbmaKkj2ojr2vJQqCMfUTaYDd0XEE1QnuR2VmSsHqfn3VMeZH6YKkRPbtjuPajh8XtuQcie+TPUFYOQQ9X6V6lj8D4DHqX6atjnVOQTXUJ0w+BDVMffWwwodycy7ga9QjSo8QvWF5N/XYRXHUP27PkoVyhespe2ZVLUvpfr3v7Zt+WeoetuPUZ23cHFLnb+hOrnvE/W2FgCT16FO6QWJzBy6laSiRMT9wIcy8/pu1yKpOfbEpR4TEYdTDUXf2O1aJDWr8WsWSxo+EXETsBvw/rYz5yX1IIfTJUkqlMPpkiQVyhCXJKlQxR0T32677XLChAndLkOSpGFzxx13LM3Mce3ziwvxCRMmMH/+/G6XIUnSsImIAS8d7HC6JEmFMsQlSSqUIS5JUqGKOyYuSepNzzzzDH19faxcOdhtA3rf6NGjGT9+PKNGjeqovSEuSdoo9PX1sdVWWzFhwgQi1unutT0hM1m2bBl9fX1MnDixo9c4nC5J2iisXLmSsWPHbpIBDhARjB07dp1GIgxxSdJGY1MN8H7r+v4NcUmSaiNGjGCvvfZijz324JBDDmH58uUbdP0TJkxg6dKlAGy55ZYveH0eE5ckbZQmnHbVBl3fojPeOWSbzTffnAULFgBw3HHHMXPmTD796U9v0Do2JHvikiQNYL/99uO3v/0tAPfffz/Tp09nypQp7L///tx7770APPLIIxx22GFMnjyZyZMnc8sttwDw7ne/mylTprD77rsze/bsxmpsrCceEecDBwN/yMw9BlgewFnAO4A/Acdn5v9rqh5Jkjq1evVqbrjhBj7wgQ8AMGPGDGbNmsWkSZO47bbb+MhHPsKNN97IySefzJvf/GbmzZvH6tWreeKJJwA4//zz2XbbbXnqqafYZ599OPzwwxk7duwGr7PJ4fQ5wDnABYMsPwiYVD/eAJxb/5UkqSueeuop9tprLxYtWsSUKVM48MADeeKJJ7jllls48sgjn2335z//GYAbb7yRCy6oYm7EiBGMGTMGgLPPPpt58+YBsHjxYn796183EuKNDadn5s3Ao2tpcihwQVZuBbaOiB2aqkeSpKH0HxN/6KGHePrpp5k5cyZr1qxh6623ZsGCBc8+7rnnnkHXcdNNN3H99dfzs5/9jJ///OfsvffejV3Appsntu0ELG6Z7qvn/a69YUTMAGYA7LLLLsNSnF64DX1SSqc6OXlFktZmzJgxnH322Rx66KF8+MMfZuLEiVx66aUceeSRZCZ33nknkydP5oADDuDcc8/llFNOYfXq1Tz55JOsWLGCbbbZhi222IJ7772XW2+9tbE6u3li20A/hsuBGmbm7MycmplTx4173u1UJUna4Pbee28mT57M3LlzueiiizjvvPOYPHkyu+++O9///vcBOOuss/jRj37E6173OqZMmcJdd93F9OnTWbVqFXvuuSef+cxneOMb39hYjd3sifcBO7dMjwce7lItkqSNTDdG1fpPTOt3xRVXPPv82muvfV777bff/tlAb3XNNdcMuP5FixYNuq310c0Qvxw4KSLmUp3QtiIznzeULukveZhCUr8mf2J2CTAN2C4i+oB/BEYBZOYs4Gqqn5ctpPqJ2QlN1SJJUi9qLMQz8+ghlifw0aa2L0lSr/OKbZIkFcoQlySpUIa4JEmFMsQlSSqUtyKVJG2cTh+zgde3YsOubyNgT1ySpNpAtxDdcsstn13+ne98h+OPPx4Y/Dakw8meuCRJtYFuITqYwW5DOpwMcUmSagPdQnQwg92GdDgZ4pIk8Ze3EN1iiy2YNm0aK1euJOK5+3U1dUvR9eUxcUmSYNBbiG6//fbcc889rFmz5tleOvDsbUgBVq9ezeOPPz7sNRvikiTBoLcQPeOMMzj44IN5y1vewg477PBs+4FuQzrcHE6XJG2chvknYS9+8YsHvYXoEUcc8bx5g92GdDjZE5ckqVCGuCRJhTLEJUkqlCEuSdpoZGa3S+iqdX3/hrgkaaMwevRoli1btskGeWaybNkyRo8e3fFrPDtdkrRRGD9+PH19fSxZsqTbpXTN6NGjGT9+fMftDXFJ0kZh1KhRTJw4sdtlFMXhdEmSCmWIS5JUKENckqRCGeKSJBXKEJckqVCGuCRJhTLEJUkqlCEuSVKhDHFJkgpliEuSVChDXJKkQhnikiQVyhCXJKlQhrgkSYUyxCVJKpQhLklSoQxxSZIKZYhLklQoQ1ySpEIZ4pIkFcoQlySpUIa4JEmFGtntAiRpYzHhtKu6st1FZ7yzK9tV+eyJS5JUKENckqRCGeKSJBXKEJckqVCGuCRJhTLEJUkqlCEuSVKhDHFJkgpliEuSVChDXJKkQhnikiQVqtEQj4jpEXFfRCyMiNMGWD4mIq6IiJ9HxF0RcUKT9UiS1EsaC/GIGAHMBA4CdgOOjojd2pp9FLg7MycD04CvRMSLmqpJkqRe0mRPfF9gYWY+kJlPA3OBQ9vaJLBVRASwJfAosKrBmiRJ6hlNhvhOwOKW6b56XqtzgNcCDwO/AD6emWvaVxQRMyJifkTMX7JkSVP1SpJUlCZDPAaYl23TbwcWADsCewHnRMRLn/eizNmZOTUzp44bN25D1ylJUpGaDPE+YOeW6fFUPe5WJwDfzcpC4EFg1wZrkiSpZzQZ4rcDkyJiYn2y2lHA5W1tfgMcABAR2wOvAR5osCZJknrGyKZWnJmrIuIk4DpgBHB+Zt4VESfWy2cBnwfmRMQvqIbfT83MpU3VJElSL2ksxAEy82rg6rZ5s1qePwy8rckaJEnqVV6xTZKkQhnikiQVyhCXJKlQhrgkSYVq9MQ2SZImnHZVV7a76Ix3dmW7w8meuCRJhTLEJUkqlCEuSVKhDHFJkgpliEuSVChDXJKkQhnikiQVyhCXJKlQhrgkSYUyxCVJKpQhLklSoQxxSZIKZYhLklQoQ1ySpEIZ4pIkFcoQlySpUIa4JEmFMsQlSSqUIS5JUqEMcUmSCjWy2wVs0k4f06XtrujOdiVJG5Q9cUmSCmWIS5JUKIfTJanbPLTWjE1gv9oTlySpUIa4JEmFMsQlSSqUx8QldaZbxxeh94/dSuvJnrgkSYUyxCVJKpQhLklSoQxxSZIKZYhLklQoQ1ySpEIZ4pIkFcoQlySpUIa4JEmFMsQlSSqUIS5JUqEMcUmSCmWIS5JUKENckqRCGeKSJBXKEJckqVCGuCRJhTLEJUkqlCEuSVKhDHFJkgrVaIhHxPSIuC8iFkbEaYO0mRYRCyLiroj4cZP1SJLUS0Y2teKIGAHMBA4E+oDbI+LyzLy7pc3WwNeA6Zn5m4h4WVP1SJLUa5rsie8LLMzMBzLzaWAucGhbm2OA72bmbwAy8w8N1iNJUk9pMsR3Aha3TPfV81q9GtgmIm6KiDsi4tiBVhQRMyJifkTMX7JkSUPlSpJUliZDPAaYl23TI4EpwDuBtwOfiYhXP+9FmbMzc2pmTh03btyGr1SSpAI1dkycque9c8v0eODhAdoszcwngScj4mZgMvCrBuuSJKknNNkTvx2YFBETI+JFwFHA5W1tvg/sHxEjI2IL4A3APQ3WJElSz2isJ56ZqyLiJOA6YARwfmbeFREn1stnZeY9EXEtcCewBvhGZv6yqZq0iTh9TJe2u6I725W0yWpyOJ3MvBq4um3erLbpLwNfbrIOSZJ6kVdskySpUI32xEsw4bSrurbtRaO7tmlJUg+wJy5JUqEMcUmSCtVRiEfEKyPixfXzaRFxcn3dc0mS1CWd9sQvA1ZHxKuA84CJwMWNVSVJkobUaYivycxVwGHAmZn5d8AOzZUlSZKG0mmIPxMRRwPHAVfW80Y1U5IkSepEpyF+ArAf8IXMfDAiJgIXNleWJEkaSke/E8/MuyPiVGCXevpB4IwmC5MkSWvX6dnphwALgGvr6b0iov1mJpIkaRh1Opx+OrAvsBwgMxdQnaEuSZK6pNMQX5WZ7bdoyg1djCRJ6lyn107/ZUQcA4yIiEnAycAtzZUlSZKG0mlP/GPA7sCfqS7ysgI4paGaJElSB4bsiUfECODyzHwr8OnmS5IkSZ0YsieemauBP0XEmGGoR5IkdajTY+IrgV9ExA+BJ/tnZubJjVQlSZKG1GmIX1U/JEnSRqLTK7Z9KyJeBLy6nnVfZj7TXFmSJGkoHYV4REwDvgUsAgLYOSKOy8ybG6tMkiStVafD6V8B3paZ9wFExKuBS4ApTRUmSZLWrtPfiY/qD3CAzPwV3opUkqSu6rQnPj8izgO+XU+/D7ijmZIkSVInOg3xDwMfpbrcagA3A19rqihJkjS0TkN8JHBWZn4Vnr2K24sbq0qSJA2p02PiNwCbt0xvDly/4cuRJEmd6jTER2fmE/0T9fMtmilJkiR1otMQfzIiXt8/ERFTgaeaKUmSJHWi02PipwCXRsTDQAI7Au9tqihJkjS0tfbEI2KfiHh5Zt4O7Ar8X2AVcC3w4DDUJ0mSBjHUcPrXgafr5/sBnwJmAo8BsxusS5IkDWGo4fQRmflo/fy9wOzMvAy4LCIWNFqZJElaq6F64iMioj/oDwBubFnW6fF0SZLUgKGC+BLgxxGxlOps9J8ARMSrgBUN1yZJktZirSGemV+IiBuAHYAfZGbWizYDPtZ0cZIkaXBDDoln5q0DzPtVM+VIkqROdXqxF0mStJExxCVJKpQhLklSoQxxSZIKZYhLklQoQ1ySpEIZ4pIkFcoQlySpUIa4JEmFMsQlSSqUIS5JUqEMcUmSCmWIS5JUKENckqRCGeKSJBXKEJckqVCNhnhETI+I+yJiYUSctpZ2+0TE6og4osl6JEnqJY2FeESMAGYCBwG7AUdHxG6DtPsicF1TtUiS1Iua7InvCyzMzAcy82lgLnDoAO0+BlwG/KHBWiRJ6jlNhvhOwOKW6b563rMiYifgMGBWg3VIktSTmgzxGGBetk2fCZyamavXuqKIGRExPyLmL1myZEPVJ0lS0UY2uO4+YOeW6fHAw21tpgJzIwJgO+AdEbEqM7/X2igzZwOzAaZOndr+RUCSpE1SkyF+OzApIiYCvwWOAo5pbZCZE/ufR8Qc4Mr2AJckSQNrLMQzc1VEnER11vkI4PzMvCsiTqyXexxckqQXoMmeOJl5NXB127wBwzszj2+yFkmSeo1XbJMkqVCGuCRJhTLEJUkqlCEuSVKhDHFJkgpliEuSVChDXJKkQhnikiQVyhCXJKlQhrgkSYUyxCVJKpQhLklSoQxxSZIKZYhLklQoQ1ySpEIZ4pIkFcoQlySpUIa4JEmFMsQlSSqUIS5JUqEMcUmSCmWIS5JUKENckqRCGeKSJBXKEJckqVCGuCRJhTLEJUkqlCEuSVKhDHFJkgpliEuSVChDXJKkQhnikiQVyhCXJKlQhrgkSYUyxCVJKpQhLklSoQxxSZIKZYhLklQoQ1ySpEIZ4pIkFcoQlySpUIa4JEmFMsQlSSqUIS5JUqEMcUmSCmWIS5JUKENckqRCGeKSJBXKEJckqVCGuCRJhTLEJUkqlCEuSVKhDHFJkgrVaIhHxPSIuC8iFkbEaQMsf19E3Fk/bomIyU3WI0lSL2ksxCNiBDATOAjYDTg6InZra/Yg8ObM3BP4PDC7qXokSeo1TfbE9wUWZuYDmfk0MBc4tLVBZt6SmY/Vk7cC4xusR5KkntJkiO8ELG6Z7qvnDeYDwDUN1iNJUk8Z2eC6Y4B5OWDDiL+hCvE3DbJ8BjADYJdddtlQ9UmSVLQme+J9wM4t0+OBh9sbRcSewDeAQzNz2UAryszZmTk1M6eOGzeukWIlSSpNkyF+OzApIiZGxIuAo4DLWxtExC7Ad4H3Z+avGqxFkqSe09hwemauioiTgOuAEcD5mXlXRJxYL58FfBYYC3wtIgBWZebUpmqSJKmXNHlMnMy8Gri6bd6slucfBD7YZA2SJPUqr9gmSVKhDHFJkgpliEuSVChDXJKkQhnikiQVyhCXJKlQhrgkSYUyxCVJKpQhLklSoQxxSZIKZYhLklQoQ1ySpEIZ4pIkFcoQlySpUIa4JEmFMsQlSSqUIS5JUqEMcUmSCmWIS5JUKENckqRCGeKSJBXKEJckqVCGuCRJhTLEJUkqlCEuSVKhDHFJkgpliEuSVChDXJKkQhnikiQVyhCXJKlQhrgkSYUyxCVJKpQhLklSoQxxSZIKZYhLklQoQ1ySpEIZ4pIkFcoQlySpUIa4JEmFMsQlSSqUIS5JUqEMcUmSCmWIS5JUKENckqRCGeKSJBXKEJckqVCGuCRJhTLEJUkqlCEuSVKhDHFJkgpliEuSVChDXJKkQhnikiQVqtEQj4jpEXFfRCyMiNMGWB4RcXa9/M6IeH2T9UiS1EsaC/GIGAHMBA4CdgOOjojd2podBEyqHzOAc5uqR5KkXtNkT3xfYGFmPpCZTwNzgUPb2hwKXJCVW4GtI2KHBmuSJKlnNBniOwGLW6b76nnr2kaSJA1gZIPrjgHm5Xq0ISJmUA23AzwREfe9wNo2CgHbAUuHfcOfG2i39w73azO6tl/BfdsU92szmtmvrxhoZpMh3gfs3DI9Hnh4PdqQmbOB2Ru6wG6LiPmZObXbdfQa92sz3K/Ncd82Y1PYr00Op98OTIqIiRHxIuAo4PK2NpcDx9Znqb8RWJGZv2uwJkmSekZjPfHMXBURJwHXASOA8zPzrog4sV4+C7gaeAewEPgTcEJT9UiS1GuaHE4nM6+mCurWebNanifw0SZr2Mj13CGCjYT7tRnu1+a4b5vR8/s1qhyVJEml8bKrkiQVyhDvUERMiIisHysjYnFEXBQRE7td26agZf9fWU9PiohHImJ5RHyvXrawvlIgEXFTPW+7tn+7afXy4+vpv+/eu9o4re2zHhFzWpa1PyZ0u/aNXfvnuGX+TS378Y8R8eOI2LVbdZam7TO7pv6/4eyI2CwiTh/gs3pmt2veUAzxdfefwInAjcAxwC0R8bLulrRpiYidgB8AWwEHAwvqRa8E3jvEyz/VXGU953mfdapLIx8NfLpuc3M9fTSwpAs19poZwCzgr4GvdLmWEv0ncCxwP/Ax4PCWZWfy3Gf1W8NeWUMM8XX3cGbOyczjgH8FXg58qP5290uAiDiinj69nu7/lv3PEbEsIn4SEf8lIu6NiEcj4oS63bS63byIuLXuZb4/Ir4SEU9ExM0RsXVEHFy3O6V+3evr6TO6skeG17ZUAb4TcERm/rRl2ePAaREx2JUWHgcOjIie/t3oBjTQZ/1tmTmX6t8A4MHMnFs/nuxapb1jXmb+T2AV8FfdLqZAD2fmhVT37YC/3Id3AtfXj7uHu7CmGOIvzDX138kdtn8l1dn6bwKuBM6h+vndV9vavQX4NtUV7eZQXRDne8D+wPH1Ohbx3E/y/mv994J1qr5M+1HdUOdD9a8fWp0LvA44ZJDX/hT4JfbG18e6fta1fraNiLdR/XLoP7pdTIFGRcT2wLR6+vaWZedTjRYtoeqN9wRD/IXp7/F1eor/p3guaK/KzHOovh1uHRHbtLS7IjNnUgXOZsAngW/WyyZm5hqqn07sWd++9XDgjszsmW+Xa7Gm/vueiBjVtmw+1bfsTw7y2gTOAN4NvLaR6nrXun7WtX7uo7q2xgPA33W5lhK9Dfg98EHgzMy8sWXZPwEH1o/rulBbIwzxF+bt9d87qcKl/3f3Ww/SfjnwTP18Rf13df13RFs72tq2t/sG8DTwRWBXNo1eOFTDuPOA6cCcAYbO/zfwRmDvQV4/F3iQ6livOtf6WVdzjqL6jP4VcGqXaynRbVS97EXARyOideToF5l5ff3omSuDNnqxlx61Y0QcD7wZOI7qW99sqmHuiRFxDHBS00Vk5pKIuIzqA/sMcEnT29xIrKZ6z9dRnWy1DHi0f2Fm/igifkY17P48mbk6Ir5EdfKQ1m6wz7peuF3bzmEZXf+9Afg+1YltJ0fEmb0UOMNgaWbOjYg/Ue3Hf6I62a1n2RNfd3sDXwfeClwM7JeZj1B9a34c+F9Uw7rD4dz67zWZucmcGZyZfwbeRXVW+seAz7Y1+echVjGHAW60o+cZ7LOuF+6VVP9n9D/6Q5zMXEl12Gc04E8g10NmXg7cQXV+zJ5dLqdRXrGtUBExiapH+jngXZl5RZdLkiQNM0O8UBExhyrELwI+kP5DStImxxCXJKlQHhOXJKlQhrgkSYUyxCVJKpQhLklSoQxxSZIKZYhLklSo/w+zTqdyT24mdgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "labels = df_metrics['model'].tolist()\n",
    "recall_val = df_metrics['recall'].tolist()\n",
    "auc_val = df_metrics['auc'].tolist()\n",
    "\n",
    "x = np.arange(len(labels))  # the label locations\n",
    "width = 0.35  # the width of the bars\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "fig.set_size_inches(8, 6)\n",
    "rects1 = ax.bar(x - width/2, recall_val, width, label='Recall')\n",
    "rects2 = ax.bar(x + width/2, auc_val, width, label='auc')\n",
    "\n",
    "# Add some text for labels, title and custom x-axis tick labels, etc.\n",
    "ax.set_ylabel('Scores')\n",
    "ax.set_title('Scores by Recall and auc')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "plt.xticks(weight = 'bold')\n",
    "ax.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "We can easily see that KNN and the DecisionTree are not suitable in their current form, as they both have extremely high recall, but the auc is low in comparison to their recall.\n",
    "A more balanced outcome can be found for the LogisticRegression and the RandomForest.\n",
    "The LogisticRegression is actually better than the RandomForest, which is unexpected.\n",
    "While both have a higher recall than the baseline, their auc still is quite poor.\n",
    "It is clear that all of the models need to be improved.\n",
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:nf] *",
   "language": "python",
   "name": "conda-env-nf-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}