Applied Machine Learning in Python W3 Assignment+3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.2** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-machine-learning/resources/bANLa) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3 - Evaluation\n",
    "\n",
    "In this assignment you will train several models and evaluate how effectively they predict instances of fraud using data based on [this dataset from Kaggle](https://www.kaggle.com/dalpozz/creditcardfraud).\n",
    " \n",
    "Each row in `fraud_data.csv` corresponds to a credit card transaction. Features include confidential variables `V1` through `V28` as well as `Amount` which is the amount of the transaction. \n",
    " \n",
    "The target is stored in the `class` column, where a value of 1 corresponds to an instance of fraud and 0 corresponds to an instance of not fraud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "Import the data from `fraud_data.csv`. What percentage of the observations in the dataset are instances of fraud?\n",
    "\n",
    "*This function should return a float between 0 and 1.* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_one():\n",
    "    \n",
    "    df = pd.read_csv('fraud_data.csv')\n",
    "    \n",
    "    return np.divide(sum(df['Class']),df['Class'].count())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use X_train, X_test, y_train, y_test for all of the following questions\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df = pd.read_csv('fraud_data.csv')\n",
    "\n",
    "X = df.iloc[:,:-1]\n",
    "y = df.iloc[:,-1]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "Using `X_train`, `X_test`, `y_train`, and `y_test` (as defined above), train a dummy classifier that classifies everything as the majority class of the training data. What is the accuracy of this classifier? What is the recall?\n",
    "\n",
    "*This function should a return a tuple with two floats, i.e. `(accuracy score, recall score)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_two():\n",
    "    from sklearn.dummy import DummyClassifier\n",
    "    from sklearn.metrics import accuracy_score, recall_score\n",
    "     \n",
    "    dummy_majority = DummyClassifier(strategy = 'most_frequent').fit(X_train, y_train)\n",
    "    y_dummy_predictions = dummy_majority.predict(X_test)\n",
    "    \n",
    "    return (accuracy_score(y_test,y_dummy_predictions),recall_score(y_test,y_dummy_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "Using X_train, X_test, y_train, y_test (as defined above), train a SVC classifer using the default parameters. What is the accuracy, recall, and precision of this classifier?\n",
    "\n",
    "*This function should a return a tuple with three floats, i.e. `(accuracy score, recall score, precision score)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_three():\n",
    "    from sklearn.metrics import accuracy_score, recall_score, precision_score\n",
    "    from sklearn.svm import SVC\n",
    "\n",
    "    svm = SVC().fit(X_train, y_train)\n",
    "    y_svm = svm.predict(X_test)\n",
    "                       \n",
    "    return (accuracy_score(y_test,y_svm),recall_score(y_test,y_svm),precision_score(y_test,y_svm))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "Using the SVC classifier with parameters `{'C': 1e9, 'gamma': 1e-07}`, what is the confusion matrix when using a threshold of -220 on the decision function. Use X_test and y_test.\n",
    "\n",
    "*This function should return a confusion matrix, a 2x2 numpy array with 4 integers.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def answer_four():\n",
    "    from sklearn.metrics import confusion_matrix\n",
    "    from sklearn.svm import SVC\n",
    "\n",
    "    svm = SVC(C=1e9, gamma=1e-07).fit(X_train, y_train)\n",
    "     \n",
    "    prediction = svm.decision_function(X_test) > -220\n",
    "    \n",
    "    confusion = confusion_matrix(y_test, prediction)\n",
    "    \n",
    "    return confusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Train a logisitic regression classifier with default parameters using X_train and y_train.\n",
    "\n",
    "For the logisitic regression classifier, create a precision recall curve and a roc curve using y_test and the probability estimates for X_test (probability it is fraud).\n",
    "\n",
    "Looking at the precision recall curve, what is the recall when the precision is `0.75`?\n",
    "\n",
    "Looking at the roc curve, what is the true positive rate when the false positive rate is `0.16`?\n",
    "\n",
    "*This function should return a tuple with two floats, i.e. `(recall, true positive rate)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_five():\n",
    "        \n",
    "\n",
    "    return (0.85, 0.85) # Return your answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "Perform a grid search over the parameters listed below for a Logisitic Regression classifier, using recall for scoring and the default 3-fold cross validation.\n",
    "\n",
    "`'penalty': ['l1', 'l2']`\n",
    "\n",
    "`'C':[0.01, 0.1, 1, 10, 100]`\n",
    "\n",
    "From `.cv_results_`, create an array of the mean test scores of each parameter combination. i.e.\n",
    "\n",
    "|      \t| `l1` \t| `l2` \t|\n",
    "|:----:\t|----\t|----\t|\n",
    "| **`0.01`** \t|    ?\t|   ? \t|\n",
    "| **`0.1`**  \t|    ?\t|   ? \t|\n",
    "| **`1`**    \t|    ?\t|   ? \t|\n",
    "| **`10`**   \t|    ?\t|   ? \t|\n",
    "| **`100`**   \t|    ?\t|   ? \t|\n",
    "\n",
    "<br>\n",
    "\n",
    "*This function should return a 5 by 2 numpy array with 10 floats.* \n",
    "\n",
    "*Note: do not return a DataFrame, just the values denoted by '?' above in a numpy array. You might need to reshape your raw result to meet the format we are looking for.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_six():    \n",
    "    from sklearn.model_selection import GridSearchCV\n",
    "    from sklearn.linear_model import LogisticRegression\n",
    "    \n",
    "    grid_values = {'penalty': ['l1', 'l2'],'C':[0.01, 0.1, 1, 10, 100]}\n",
    "    \n",
    "    lr = LogisticRegression().fit(X_train, y_train)\n",
    "    \n",
    "    clf = GridSearchCV(lr, grid_values, scoring = 'recall')\n",
    "    clf.fit(X_train, y_train)\n",
    "    #lr_predicted = lr.predict(X_test)\n",
    "    \n",
    "    return clf.cv_results_['mean_test_score'].reshape(5,2) # Return your answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Use the following function to help visualize results from the grid search\n",
    "def GridSearch_Heatmap(scores):\n",
    "    %matplotlib notebook\n",
    "    import seaborn as sns\n",
    "    import matplotlib.pyplot as plt\n",
    "    plt.figure()\n",
    "    sns.heatmap(scores.reshape(5,2), xticklabels=['l1','l2'], yticklabels=[0.01, 0.1, 1, 10, 100])\n",
    "    plt.yticks(rotation=0);\n",
    "\n",
    "#GridSearch_Heatmap(answer_six())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-machine-learning",
   "graded_item_id": "5yX9Z",
   "launcher_item_id": "eqnV3",
   "part_id": "Msnj0"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}