ROC_curve_comparison_.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d89eb051-b775-94aa-7d61-baf5144aff04",
        "_uuid": "786568a75df6ac1ce698f0d6b0cd490684eed9b4"
      },
      "source": [
        "I am  going to apply 6 Supervised machine learning models on the given dataset.The strategy is to apply default model first with no tuning of the hyperparameter and then tuned them with different hyperparameter values and then I'll plot ROC curve to select the best machine learning model.The models used are as follows:\n",
        "**1) Principal Component Analysis\n",
        "**2)Logistic Regression**\n",
        " **3)Gaussian Naive Bayes**\n",
        "  **4)Support Vector Machine**\n",
        "  **5)Random Forest Classifier** \n",
        "  **6)Decision trees**\n",
        "   **7)Simple neural network**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e5bb6859-47c5-7612-d53b-1f6bd39cf320",
        "_uuid": "16365c74dd530e17edd2a0df7fcfa7472e377095"
      },
      "outputs": [],
      "source": [
        "# This Python 3 environment comes with many helpful analytics libraries installed\n",
        "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
        "# For example, here's several helpful packages to load in \n",
        "# Input data files are available in the \"../input/\" directory.\n",
        "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n",
        "\n",
        "from subprocess import check_output\n",
        "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n",
        "\n",
        "# Any results you write to the current directory are saved as output."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "dd3cebd3-cd87-dd42-fffa-c5f3aefde314",
        "_uuid": "f11ecccd5f04b6ec68765e8ffde8a56a5d87042e"
      },
      "source": [
        "###  Importing all the libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bb4b600a-c486-9f3c-3a1c-26567f217391",
        "_uuid": "78d06754047df004d5776da9d66efe49ce4a6518"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "import seaborn as sns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "080d387b-dd78-e7f0-36ac-dcb831910082",
        "_uuid": "56c011020f2eeffb91f3423642f16de37eb9666d"
      },
      "source": [
        "### Reading the file "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fe287208-001e-07e5-1a9b-dbf8d5a40c6a",
        "_uuid": "f4a36a4d4dd5a8a83e9332f77e7a78c3f36a6cac"
      },
      "outputs": [],
      "source": [
        "data = pd.read_csv(\"../input/mushrooms.csv\")\n",
        "data.head(6)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f9455820-2893-f798-b7ba-3f9090cfeac1",
        "_uuid": "72e8424a13408772cafa9792b9eb10df22fa6892"
      },
      "source": [
        "### Let us check if there is any null values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5499a675-fbba-d400-ecd3-57fc1a0f3915",
        "_uuid": "4d13cfe1b929b92fc17f3093a84f71c09a896474"
      },
      "outputs": [],
      "source": [
        "data.isnull().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "516a10c6-a887-10a1-af23-8846c6517cb2",
        "_uuid": "02811ac4e594a285c90deb527cad083088b205f1"
      },
      "outputs": [],
      "source": [
        "data['class'].unique()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2392cf2f-5d0b-0fbc-25d7-6ac8f6f9aa91",
        "_uuid": "d550638ac2c8411be39972a8097092514f4cd9fe"
      },
      "source": [
        "**Thus we have two claasification. Either the mushroom is poisonous or edible**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "abf7629e-1970-5b51-bd93-594b84eb30cf",
        "_uuid": "bc497236c5adf11c58d7bac0f050e97057627960"
      },
      "outputs": [],
      "source": [
        "data.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8198f725-e1d9-13d3-e29b-01aa7942f93a",
        "_uuid": "833b4d14df12086f7a3a2497b50cbe7066ca4a01"
      },
      "source": [
        "**Thus we have 22 features(1st one is label)  and 8124 instances.Now let us check which features constitutes maximum information.** "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b6e97737-4258-b9ae-62ab-dd6faebfd0a2",
        "_uuid": "d4ba71813bfd2c188673cbf8ba454f15c917f6a5"
      },
      "source": [
        "**We can see that the dataset has values in strings.We need to convert all the unique values to integers. Thus we perform label encoding on the data.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "af0ee2d3-add4-2b0c-140a-6815347c4d3c",
        "_uuid": "ab284b2d43e925e292d688a27a8635d1e2605665"
      },
      "outputs": [],
      "source": [
        "from sklearn.preprocessing import LabelEncoder\n",
        "labelencoder=LabelEncoder()\n",
        "for col in data.columns:\n",
        "    data[col] = labelencoder.fit_transform(data[col])\n",
        " \n",
        "data.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b49e7feb-8261-9056-d345-9215e0d384bd",
        "_uuid": "b4fe437adb0432a8a2e88f90b555103fc1ae6f46"
      },
      "source": [
        "### Checking the encoded values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "262c2fe1-8e68-f5fe-a879-3bd9400b469c",
        "_uuid": "65a4999e0a6d9a2da74e055dc90745f672c5571a"
      },
      "outputs": [],
      "source": [
        "data['stalk-color-above-ring'].unique()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9ced0f59-a411-46a8-92a4-f9ef9fe47a0d",
        "_uuid": "315f7e1554e43c1de380f24c83c22679c2f280bf"
      },
      "outputs": [],
      "source": [
        "print(data.groupby('class').size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8b40bb73-5d26-48f3-e97f-072b2467d0bc",
        "_uuid": "31d625b63e374172aafcae73e30d337193472db8"
      },
      "source": [
        "### Plotting boxplot to see the distribution of the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3c0414fe-dac5-12f7-80c5-517f13236eec",
        "_uuid": "f13fbc158b36ef385fdc609a4a00ff831281cfd5"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "# Create a figure instance\n",
        "fig, axes = plt.subplots(nrows=2 ,ncols=2 ,figsize=(9, 9))\n",
        "\n",
        "# Create an axes instance and the boxplot\n",
        "bp1 = axes[0,0].boxplot(data['stalk-color-above-ring'],patch_artist=True)\n",
        "\n",
        "bp2 = axes[0,1].boxplot(data['stalk-color-below-ring'],patch_artist=True)\n",
        "\n",
        "bp3 = axes[1,0].boxplot(data['stalk-surface-below-ring'],patch_artist=True)\n",
        "\n",
        "bp4 = axes[1,1].boxplot(data['stalk-surface-above-ring'],patch_artist=True)\n",
        "'''\n",
        "ax = sns.boxplot(x='class', y='stalk-color-above-ring', \n",
        "                data=data)\n",
        "ax = sns.stripplot(x=\"class\", y='stalk-color-above-ring',\n",
        "                   data=data, jitter=True,\n",
        "                   edgecolor=\"gray\")\n",
        "sns.plt.title(\"Class w.r.t stalkcolor above ring\",fontsize=12)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3594a475-f8a8-3599-f2b8-c0209e87aeec",
        "_uuid": "9018af8e638a03e93193ae9fb187f74a894717d1"
      },
      "source": [
        "**Separating features and label**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f15dc721-d721-9320-3964-66cc19b72a18",
        "_uuid": "cb784bf004a1a9fa96baa327c4fac13092747b1e"
      },
      "outputs": [],
      "source": [
        "X = data.iloc[:,1:23]  # all rows, all the features and no labels\n",
        "y = data.iloc[:, 0]  # all rows, label only\n",
        "X.head()\n",
        "y.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "19a00b00-b010-6918-11cb-1fa7b18aaa17",
        "_uuid": "dac81faac1dbb2bb8abf0058e73cd9186b214b44"
      },
      "outputs": [],
      "source": [
        "X.describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9f8ebf80-0488-93ea-9a10-d128c8519bf6",
        "_uuid": "daf21399b2443eb751ed0f5fe1d21e53b59b487d"
      },
      "outputs": [],
      "source": [
        "y.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "29c744c8-7da3-bcbd-d33d-e01d55833f07",
        "_uuid": "7b80dffdfaad294c40746ea97949d01fa4cea3f2"
      },
      "outputs": [],
      "source": [
        "data.corr()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5c52b254-e464-3262-05f8-523bec33fe44",
        "_uuid": "cf2cb9df27464a86437a02ba876d7f0862893647"
      },
      "source": [
        "# Standardising the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "54185e06-71a1-3738-e41b-cbc0ae4ca614",
        "_uuid": "ca49c4ef0a91b22f32d71145f03ad1dbb2d9ddb2"
      },
      "outputs": [],
      "source": [
        "# Scale the data to be between -1 and 1\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "scaler = StandardScaler()\n",
        "X=scaler.fit_transform(X)\n",
        "X"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "949d806b-30f9-2d26-0dde-65377690bef6",
        "_uuid": "252a822b8dcc636fb690697eb0fca5e491d717fd"
      },
      "source": [
        "**Note**: We can avoid PCA here since the dataset is very small."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "80eb90a3-f197-cf7e-13ad-126911b51064",
        "_uuid": "994f70bc40134a4cce266472a8ff2876bcdd3a00"
      },
      "source": [
        "# Principal Component Analysis"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0e3cc9bb-7d54-6133-8863-d10b6924ef5d",
        "_uuid": "286a68a1cf7a8711f48900837c474dce2549a16c"
      },
      "outputs": [],
      "source": [
        "from sklearn.decomposition import PCA\n",
        "pca = PCA()\n",
        "pca.fit_transform(X)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0904dbb8-7a4f-7cff-f978-37c84371b440",
        "_uuid": "45110aaafa26fe801c265c6a2eccc4d05fa57ed5"
      },
      "outputs": [],
      "source": [
        "covariance=pca.get_covariance()\n",
        "#covariance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a936fc92-793d-a887-bdb6-ca72905ef330",
        "_uuid": "0b8b7e48edc985f05f637bdd6fa1137ec08eedd1"
      },
      "outputs": [],
      "source": [
        "explained_variance=pca.explained_variance_\n",
        "explained_variance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "19dd9eca-ee82-bd65-002b-bdd11590956c",
        "_uuid": "42a057446a7afa8bc91c1ba5bc281c887e51b24b"
      },
      "outputs": [],
      "source": [
        "with plt.style.context('dark_background'):\n",
        "    plt.figure(figsize=(6, 4))\n",
        "    \n",
        "    plt.bar(range(22), explained_variance, alpha=0.5, align='center',\n",
        "            label='individual explained variance')\n",
        "    plt.ylabel('Explained variance ratio')\n",
        "    plt.xlabel('Principal components')\n",
        "    plt.legend(loc='best')\n",
        "    plt.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a2e6f6ff-bb71-a216-33ba-05a43a61cb83",
        "_uuid": "ec3e3ea21ff565682085f0c34df897b3247f503e"
      },
      "source": [
        "**We can see that the last 4 components has less amount of variance  of the data.The 1st 17 components retains more than 90% of the data.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cbcac3ce-62cf-5ac7-946e-a5de3571052a",
        "_uuid": "29e126263ce7126a6480cce546eba440be656885"
      },
      "source": [
        "### Let us take only first two principal components and visualise it using K-means clustering"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "12199b6a-ec63-32ca-4efb-2f930fcf004f",
        "_uuid": "a8b53b53a63697ad43b5dcb769cf573ed02d259f"
      },
      "outputs": [],
      "source": [
        "N=data.values\n",
        "pca = PCA(n_components=2)\n",
        "x = pca.fit_transform(N)\n",
        "plt.figure(figsize = (5,5))\n",
        "plt.scatter(x[:,0],x[:,1])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b504681c-c3b9-e808-fb8c-3b0ff036f8b7",
        "_uuid": "aae2d1c67f12ce9253ab106373ecdd158a2a6fb0"
      },
      "outputs": [],
      "source": [
        "from sklearn.cluster import KMeans\n",
        "kmeans = KMeans(n_clusters=2, random_state=5)\n",
        "X_clustered = kmeans.fit_predict(N)\n",
        "\n",
        "LABEL_COLOR_MAP = {0 : 'g',\n",
        "                   1 : 'y'\n",
        "                  }\n",
        "\n",
        "label_color = [LABEL_COLOR_MAP[l] for l in X_clustered]\n",
        "plt.figure(figsize = (5,5))\n",
        "plt.scatter(x[:,0],x[:,1], c= label_color)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0a9f3343-5f46-0572-27f0-8269ef162a56",
        "_uuid": "e21ed30d38472557d356edc6fa8556e263eb9e68"
      },
      "source": [
        "### Thus using K-means we are able segregate 2 classes well using the first two components with maximum variance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "90fa31fa-cac2-a8a5-d69a-9bd53fc52920",
        "_uuid": "aa6bcb8164fe4168a62c0604a612de1927c67252"
      },
      "source": [
        "# Performing PCA by taking 17 components with maximum Variance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d4f93ff7-0bf8-1ecf-e14f-ddb56a1b4bb4",
        "_uuid": "8cc9f256ee82aeaa287552b84c19adcc54e84961"
      },
      "outputs": [],
      "source": [
        "pca_modified=PCA(n_components=17)\n",
        "pca_modified.fit_transform(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a44859a3-c852-332c-3d29-ad48f8e090df",
        "_uuid": "d1587a4373a82994a3e38dfbc9223719e084d5d7"
      },
      "source": [
        "### Splitting the data into training and testing dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "200dc29e-9986-069a-7a59-2374dbf57b35",
        "_uuid": "75ec9e83cfd85cf463ecbe35d10879988051e9c5"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "801b54d9-d7d1-6ed2-722c-58fa17bb28f7",
        "_uuid": "5464234586b6bbc1c992a5bd69d24ee091417b03"
      },
      "source": [
        "# Default Logistic Regression"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "47b309b1-8b11-26d5-112c-3838427e7ae6",
        "_uuid": "cc2e8f2d5cd0101ae76b2a6b1919242e1c645167"
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn import metrics\n",
        "\n",
        "model_LR= LogisticRegression()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f1c6e97f-6207-1829-d4e2-4e5a5229c2a0",
        "_uuid": "1742429c5cd485ece083e6565abdc62a5b873ee8"
      },
      "outputs": [],
      "source": [
        "model_LR.fit(X_train,y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "24aa6045-a438-f648-5d8c-88e45e1102a0",
        "_uuid": "f434b6250096175cb22979c6b43da8d7369acbc1"
      },
      "outputs": [],
      "source": [
        "y_prob = model_LR.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "model_LR.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "31c2969e-7fd6-f6a9-04f5-16ffd8fd6396",
        "_uuid": "2dc131abd61792b16d6adf23780fe35dca442242"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "48f43519-85e2-fe26-9174-b8a8b1899812",
        "_uuid": "95b3fb6103fab9b16fb31cb0f9f8a08c66fa4eed"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "84dbc1dc-d945-2277-e1b7-e976769becd3",
        "_uuid": "131a8116d32238aa4b5addd2c1c6a319b6069749"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5ee904b5-7db8-e2ff-9953-02f34d868f25",
        "_uuid": "afb001763bd52c12baf41f75c8ba7aed70bf58da"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "10ae42f6-254e-6d33-a064-7751ba99e6a5",
        "_uuid": "30e0a3db82f30aec719add091c501baf30a69aee"
      },
      "source": [
        "# Logistic Regression(Tuned model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e0858772-da3a-e0df-ae26-5be570167287",
        "_uuid": "8fb58f073c199a36e9fdb0ac888a17c7a06f079a"
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn import metrics\n",
        "\n",
        "LR_model= LogisticRegression()\n",
        "\n",
        "tuned_parameters = {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000] ,\n",
        "              'penalty':['l1','l2']\n",
        "                   }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8140075f-2f17-91d3-a86f-c7d765cac500",
        "_uuid": "5ae0f30542def4c7db067dd5238528c3dae1dd70"
      },
      "source": [
        "**L1 and L2 are regularization parameters.They're used to avoid overfiting.Both L1 and L2 regularization prevents overfitting by shrinking (imposing a penalty) on the coefficients.**  \n",
        "    **L1 is the first moment norm |x1-x2| (|w| for regularization case) that is simply the absolute dıstance between two points where L2 is second moment norm corresponding to Eucledian Distance that is  |x1-x2|^2 (|w|^2 for regularization case).**     \n",
        "       **In simple words,L2 (Ridge) shrinks all the coefficient by the same proportions but eliminates none, while L1 (Lasso) can shrink some coefficients to zero, performing variable selection.\n",
        "If all the features are correlated with the label, ridge outperforms lasso, as the coefficients are never zero in ridge. If only a subset of features are correlated with the label, lasso outperforms ridge as in lasso model some coefficient can be shrunken to zero.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f802a8e6-9471-cd39-7fa5-8112e1bddf5c",
        "_uuid": "27a8b413d62cc10e48dc271656c35570ff54d6cd"
      },
      "source": [
        "### Taking a look at the correlation "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "be35cd37-d265-af2a-3c36-a699cca3ab12",
        "_uuid": "44be985ce957d155133f9b70417a2f837d1db3bc"
      },
      "outputs": [],
      "source": [
        "data.corr()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "30f38234-8a5b-4b16-b015-8f103d8eea97",
        "_uuid": "c0685973ec51c859681407c026e203521dea70ba"
      },
      "source": [
        "**The grid search provided by GridSearchCV exhaustively generates candidates from a grid of parameter values specified with the tuned_parameter.The GridSearchCV instance implements the usual estimator API: when “fitting” it on a dataset all the possible combinations of parameter values are evaluated and the best combination is retained.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a076d8d6-4c09-6768-12c9-fad9cc09497b",
        "_uuid": "58050d7b167bc53244c9177d34a87be5694fb130"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "LR= GridSearchCV(LR_model, tuned_parameters,cv=10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "15772de5-6f72-5a9c-3732-a5abea9db2d5",
        "_uuid": "63b3952d6831a8ad3275af2bfc10d0a28a2acba2"
      },
      "outputs": [],
      "source": [
        "LR.fit(X_train,y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7863a821-1ac5-5376-2d91-d4e1969af94f",
        "_uuid": "fb92efeaeeb32fc4eb3a0467ef28263700ee409f"
      },
      "outputs": [],
      "source": [
        "print(LR.best_params_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2242b89a-8495-1c07-db6c-c2f167a4be22",
        "_uuid": "1765707e72f09fe74c14a733cecaf5a1af48782c"
      },
      "outputs": [],
      "source": [
        "y_prob = LR.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "LR.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "60a50942-de06-9dc9-5118-d584bfa0b2eb",
        "_uuid": "2ea38ff65c47c6ceb8f81b57b8bd30746ab826c1"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5e486e4c-bde6-e849-bdfe-af9f9982185a",
        "_uuid": "a284298d724652f15cfb3c04ff12deff88afa766"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f69d035c-5c51-fded-43a6-ec9c322a835c",
        "_uuid": "6e3585746cb60ba999290ee25c70afce10a94194"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8a9ab5d3-e42f-973d-927f-b0fd68b0ab38",
        "_uuid": "222756c0a785cd47345af6093d79624a55f6f440"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "500e2710-55e4-548e-34e2-43cad4375cf2",
        "_uuid": "17ddd6650e4cebe4ba37e7e14d535d127bf82af2"
      },
      "outputs": [],
      "source": [
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3def21ce-7500-f16c-4283-607e07d851b2",
        "_uuid": "ab84b04b75cccb577a33226ebdb85d66ab74c434"
      },
      "outputs": [],
      "source": [
        "LR_ridge= LogisticRegression(penalty='l2')\n",
        "LR_ridge.fit(X_train,y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ac4b80d3-c4cb-1c6d-3755-dcc7af4f4172",
        "_uuid": "635c3c886ebd644d3464baeb169e2d3ed50d7bb8"
      },
      "outputs": [],
      "source": [
        "y_prob = LR_ridge.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "LR_ridge.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7bf05fdb-e10c-387f-edf8-99a87877e96a",
        "_uuid": "49a6834825de9fbb0ff622494dbeaf11bf0fc4cf"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fc4f82d2-4bf5-0709-3b9a-6beef4f731ad",
        "_uuid": "b4bb30639e15eea4ee10e76ca70f850b95d4779b"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f69d29c0-d88f-e5a7-c995-5004e033b31b",
        "_uuid": "de9f23b9b331c979bdd12806614a916378b954cc"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "549cea57-3c1a-5636-f8ee-dd161cd05764",
        "_uuid": "89350b1d366b4f697f17382bfa25cfc7d9bff6a0"
      },
      "outputs": [],
      "source": [
        "\n",
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ab06322c-ffe0-5fba-8498-cd60bde01be5",
        "_uuid": "5f8a35b343dea66ac1a20914f11ed539cb75458a"
      },
      "outputs": [],
      "source": [
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8159a67f-f410-9007-06cd-f59e00dd106a",
        "_uuid": "d00fcfaa0c8253f191eda817cf28f2f9b406e949"
      },
      "source": [
        "# Gaussian Naive Bayes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "59acd771-44d1-c91f-c0ec-5402446db5c1",
        "_uuid": "dd03ab44e86b12c0a6801cf64ae6224645d2f84a"
      },
      "outputs": [],
      "source": [
        "from sklearn.naive_bayes import GaussianNB\n",
        "model_naive = GaussianNB()\n",
        "model_naive.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "454e7a22-1323-fa6a-d451-20e2aca84f23",
        "_uuid": "ad4ab023fa4413b9228a54cc075ee071467cf247"
      },
      "outputs": [],
      "source": [
        "y_prob = model_naive.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "model_naive.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7c9facdd-4f7a-e378-d382-2bd64b4890ec",
        "_uuid": "993ea36ffb05d942668705101de4ef2d9b511d3f"
      },
      "outputs": [],
      "source": [
        "print(\"Number of mislabeled points from %d points : %d\"\n",
        "      % (X_test.shape[0],(y_test!= y_pred).sum()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "84675635-78a3-27e5-604e-f6da06ca06dc",
        "_uuid": "e622518aea2380e8205b526da906ea7c37721c55"
      },
      "outputs": [],
      "source": [
        "scores = cross_val_score(model_naive, X, y, cv=10, scoring='accuracy')\n",
        "print(scores)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7bbb2136-5c8d-ea94-30eb-8efe1c865c75",
        "_uuid": "0607bd34f114a0d019f6acf390aeb30c7ce27d7b"
      },
      "outputs": [],
      "source": [
        "scores.mean()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "624934f3-6f27-93f8-786c-6132112ddc20",
        "_uuid": "b3e7a091ec12ded2d5b5eadc49a22e66da0c6406"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4618ed82-cbf8-d1a6-d7a2-befe113077f0",
        "_uuid": "1709d1a9cf0ee3e2ab18863fdf713a4fdaa5597c"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "91bdc65e-a302-6ab3-ff47-37d0ee726a53",
        "_uuid": "604cfaa8076c60a4e675edaa0ca58e31c009c32d"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3c6b655c-826f-50eb-4d4f-79c1525708b9",
        "_uuid": "c6ea7ee6381d87ce613994d7e4670a21e27a2156"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cb33dfcd-950a-6372-3d96-2fc6fef3ec85",
        "_uuid": "16e6b8d90415d94f14e834c8e702e2535b4484d8"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "458e57b8-6935-176b-9211-83fce5d23b1a",
        "_uuid": "4f0b1f9ad9c027e191cf22bbff957923c6199b05"
      },
      "source": [
        "# Support Vector Machine"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7625c838-94db-c5c3-3fe2-6ac22512f8c5",
        "_uuid": "5a01be1060a9c92732777850c23c0382abbdf5a0"
      },
      "outputs": [],
      "source": [
        "from sklearn.svm import SVC\n",
        "svm_model= SVC()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1dff13e1-61f3-e9b5-a0d2-c275c208aa8e",
        "_uuid": "44ebb631538f51956d4213469d7775aba707cb79"
      },
      "source": [
        "The **gamma** parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’. The **gamma** parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.\n",
        "\n",
        "The **C** parameter trades off misclassification of training examples against simplicity of the decision surface. A low **C** makes the decision surface smooth, while a high **C** aims at classifying all training examples correctly by giving the model freedom to select more samples as support vectors."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f4335b19-f669-2529-e9c8-3cd9d8f2f8ef",
        "_uuid": "a9e8d5d99fd2e85c06b7684f1add399de1040324"
      },
      "source": [
        "# Support Vector Machine without polynomial kernel"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1acfd379-4698-8609-8d34-75568232c606",
        "_uuid": "9df67297271abdc255bfb91ebdb666f2be296a22"
      },
      "outputs": [],
      "source": [
        "tuned_parameters = {\n",
        " 'C': [1, 10, 100,500, 1000], 'kernel': ['linear','rbf'],\n",
        " 'C': [1, 10, 100,500, 1000], 'gamma': [1,0.1,0.01,0.001, 0.0001], 'kernel': ['rbf'],\n",
        " #'degree': [2,3,4,5,6] , 'C':[1,10,100,500,1000] , 'kernel':['poly']\n",
        "    }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d8a28afe-9c3a-9d18-cbe1-4610bc6cc00b",
        "_uuid": "306ed0241e2f157829b96a61ca70b0078018eed3"
      },
      "source": [
        "The grid search provided by GridSearchCV exhaustively generates candidates from a grid of parameter values specified with the tuned_parameter**.The GridSearchCV instance implements the usual estimator API: when “fitting” it on a dataset all the possible combinations of parameter values are evaluated and the best combination is retained.\n",
        "But it is proving computationally expensive here.So I am opting for RandomizedSearchCV.\n",
        "\n",
        "RandomizedSearchCV implements a randomized search over parameters, where each setting is sampled from a distribution over possible parameter values. This has two main benefits over an exhaustive search:\n",
        "1)A budget can be chosen independent of the number of parameters and possible values.\n",
        "2)Adding parameters that do not influence the performance does not decrease efficiency."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "41a288c2-3707-a9ab-6d35-5ba9878777bb",
        "_uuid": "ffeefa854f40f1746ed61cdc71cecc697065ee46"
      },
      "outputs": [],
      "source": [
        "from sklearn.grid_search import RandomizedSearchCV\n",
        "\n",
        "model_svm = RandomizedSearchCV(svm_model, tuned_parameters,cv=10,scoring='accuracy',n_iter=20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6cf1ce3a-6b6d-083e-9b48-c24557b2e5ff",
        "_uuid": "72195cd4b9d87a1035c7acb33434bc23a67d48b2"
      },
      "outputs": [],
      "source": [
        "model_svm.fit(X_train, y_train)\n",
        "print(model_svm.best_score_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fbab9fd3-65fe-3c00-27a5-0e9849b1a24d",
        "_uuid": "4c3b0eb06708c2869aa4494d81a07ff78812075b"
      },
      "outputs": [],
      "source": [
        "print(model_svm.grid_scores_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c28fdf03-1aaf-081c-1c19-d7daae4f2169",
        "_uuid": "cbf88dac5fb5d2d3c7e0d51a8a9f3e2785f1a99a"
      },
      "outputs": [],
      "source": [
        "print(model_svm.best_params_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e0ce400d-9a7c-9134-7bcc-30bfa23dbd72",
        "_uuid": "65c9c8e5c6fdeafcb7457ae43f39c317acde6817"
      },
      "outputs": [],
      "source": [
        "\n",
        "y_pred= model_svm.predict(X_test)\n",
        "print(metrics.accuracy_score(y_pred,y_test))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a42b9916-f91b-2f90-dfae-64e876aeb6c5",
        "_uuid": "7d0a0f60133afa53195b35529c8db150b945032c"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e5b968ae-af2a-d18c-8a0b-8aaa11bc3cd8",
        "_uuid": "29b67753a55864e73baa55b1bad76c3c39367e01"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "89963240-7f86-3a73-5f18-86bc6dcc813a",
        "_uuid": "243251bc506ec17358c2ae5ed4349572685165e1"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "39d255a4-4527-378d-6218-1cdd548d90e9",
        "_uuid": "a0d55a17d593742b13a43ce90a6c8dbac71a48aa"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_pred)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "05af8089-0adc-50fc-4140-6e842e6e8a66",
        "_uuid": "a23dae7bd8cc44a628421d1b71722ff5a9d99ce6"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c8453261-f76c-7134-64c8-e444aedda8f7",
        "_uuid": "c467e0bb1484ae9994484bb1800f345aaf4937d7"
      },
      "source": [
        "# Support Vector machine with polynomial Kernel"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d38b7986-8e96-d201-0c0d-c4ea48488e59",
        "_uuid": "4f7ab1a9e1d7e9f079cf1a39b62d48459e75456c"
      },
      "outputs": [],
      "source": [
        "tuned_parameters = {\n",
        " 'C': [1, 10, 100,500, 1000], 'kernel': ['linear','rbf'],\n",
        " 'C': [1, 10, 100,500, 1000], 'gamma': [1,0.1,0.01,0.001, 0.0001], 'kernel': ['rbf'],\n",
        " 'degree': [2,3,4,5,6] , 'C':[1,10,100,500,1000] , 'kernel':['poly']\n",
        "    }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5e58725a-ec33-9f5b-060c-93e5116233dc",
        "_uuid": "28ecbd118c80c5f1e3b6dc85e0e074615540f3e2"
      },
      "outputs": [],
      "source": [
        "from sklearn.grid_search import RandomizedSearchCV\n",
        "\n",
        "model_svm = RandomizedSearchCV(svm_model, tuned_parameters,cv=10,scoring='accuracy',n_iter=20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8d83fa49-8da2-f561-f1e8-197fbe6fffa3",
        "_uuid": "76ff20443055998b7435b4c7d0985b3b3b4a03b9"
      },
      "outputs": [],
      "source": [
        "model_svm.fit(X_train, y_train)\n",
        "print(model_svm.best_score_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "81ff3083-2ccb-cc2f-df29-25a196593f0b",
        "_uuid": "e0109029bcd549930e483fc1da5f93600cbfe217"
      },
      "outputs": [],
      "source": [
        "print(model_svm.grid_scores_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ac8aefd3-1164-f84f-94d2-f2e7ba8deb11",
        "_uuid": "ba7320f66da892adca87815fdb15a094e155bf45"
      },
      "outputs": [],
      "source": [
        "print(model_svm.best_params_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7e2b0071-81cd-73fd-28da-25cd70516611",
        "_uuid": "4f953dc87b6595b8730ffb4d00861da672a77fd6"
      },
      "outputs": [],
      "source": [
        "y_pred= model_svm.predict(X_test)\n",
        "print(metrics.accuracy_score(y_pred,y_test))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3ad7298e-af70-067e-7d5d-00a144d84a45",
        "_uuid": "15991f769ca68a6ca005f5ba491ca0562b4e16e2"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "90cb8fc6-59e8-caa1-daa3-0e125ec25248",
        "_uuid": "a96fa0e0b53e4436c8cc311bbfeb424e2e64e0ab"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "34497db1-b6ac-d7e5-1a04-1dc33e55cc47",
        "_uuid": "e1c5fa87833a4cf1a8a5b6ab85afa7dd16a5027e"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9b7af233-30d6-1bb0-ce31-dda8a44aeddf",
        "_uuid": "6099ba4128128c15ecaee1194a01dbe5215d9336"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_pred)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "25277b51-305a-4370-45b2-85ecf437e68b",
        "_uuid": "24b83cec037ccfeb99378fc544153765a1e9cce8"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "16016745-71f6-9ce1-0595-912a4bc784db",
        "_uuid": "48e4deeb6633eee4d5dce5d0bfd4f482babb419c"
      },
      "source": [
        "### Trying default model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0456a449-6b1b-1208-bc8b-d0873517e327",
        "_uuid": "1a4e730e75b0d397ba5ec17d2a2d8081c2f2679a"
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "model_RR=RandomForestClassifier()\n",
        "\n",
        "#tuned_parameters = {'min_samples_leaf': range(5,10,5), 'n_estimators' : range(50,200,50),\n",
        "                    #'max_depth': range(5,15,5), 'max_features':range(5,20,5)\n",
        "                    #}\n",
        "               "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d2e24231-f8a6-d34c-6edb-609c1ffaf4ec",
        "_uuid": "c841e5b9234e07bd2d26a7b1031bac0742b4a569"
      },
      "outputs": [],
      "source": [
        "model_RR.fit(X_train,y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3ef4d341-8f38-a8fa-d8a6-27bd44b6ec00",
        "_uuid": "0bfb00c62d64ccd891028be7c73d1e5c440ce228"
      },
      "outputs": [],
      "source": [
        "y_prob = model_RR.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "model_RR.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "125dc2ea-b909-fc61-114b-21e9f9f23625",
        "_uuid": "464f1579069470508797410a0c16e2205918be2a"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "14f1721a-17db-0aa2-53d0-abd93e142c31",
        "_uuid": "633c594998df978d4be1ca24eb575b7f84b2698e"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a8248452-b6e5-6479-e666-537927eb89dd",
        "_uuid": "75b9780eaa4b90a977520893384a85ae8111cadf"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c7a51e02-cf46-e6a9-80bd-96a4eaefaa42",
        "_uuid": "3d8cc940f92a5d68e99485285379d761e74b9805"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "76f4605b-4aa7-3689-3878-76f64d987e52",
        "_uuid": "fda943769c7aeceb4b09653f0c148ab9966d14cd"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ad487047-3130-5932-0ce5-2f7a47851a4c",
        "_uuid": "fafdbf6eac43627a640a5f84cdf059bdf7eef3a8"
      },
      "source": [
        "### Thus default Random forest model is giving us best accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ae95e479-db31-0877-2143-4a8467e78fba",
        "_uuid": "a3fa2c65f7cc0b7e1c9071b1def76c06290e212b"
      },
      "source": [
        "### Let us tuned the parameters of Random Forest just for the purpose of knowledge"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fe230638-8d8d-e978-a623-705568d8c0eb",
        "_uuid": "95a514e40a04566d2dfb1e197b2ca7993458565e"
      },
      "source": [
        "**There are 3 features which can be tuned to improve the performance of Random Forest**    \n",
        "\n",
        "**1) max_features 2) n_estimators  3) min_sample_leaf**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f4e6558a-16b5-b198-c83f-dd834b52c7e8",
        "_uuid": "7dbdbada2b5bf46d12a66fb38418c334b4d163ad"
      },
      "source": [
        " **A)max_features**: These are the maximum number of features Random Forest is allowed to try in individual tree.\n",
        "**1)Auto** : This will simply take all the features which make sense in every tree.Here we simply do not put any restrictions on the individual tree.\n",
        "**2)sqrt** : This option will take square root of the total number of features in individual run. For instance, if the total number of variables are 100, we can only take 10 of them in individual tree.\n",
        "**3)log2**:It  is another option which takes log to the base 2 of the features input.\n",
        "\n",
        "**Increasing max_features generally improves the performance of the model as at each node now we have a higher number of options to be considered.But, for sure, you decrease the speed of algorithm by increasing the max_features. Hence, you need to strike the right balance and choose the optimal max_features.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "14de4e9c-4a3e-1b1e-d036-f023e690d95c",
        "_uuid": "a3c8c75fd16c1c83bb2f069da829e11fd603da5c"
      },
      "source": [
        "**B) n_estimators** : This is the number of trees you want to build before taking the maximum voting or averages of predictions. Higher number of trees give you better performance but makes your code slower. You should choose as high value as your processor can handle because this makes your predictions stronger and more stable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "96f0f477-afe7-fc86-beab-75e85ecab3f3",
        "_uuid": "564cf0e86c26b6050b919d484967d3adcd88f0a5"
      },
      "source": [
        "**C)min_sample_leaf**:  Leaf is the end node of a decision tree. A smaller leaf makes the model more prone to capturing noise in train data. Hence it is important to try different values to get good estimate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "60ef6e9a-dcf4-e604-2d81-ef6a0c61492d",
        "_uuid": "468e0ce62a833fb2f016e9bf065196f1f0458fdd"
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "model_RR=RandomForestClassifier()\n",
        "\n",
        "tuned_parameters = {'min_samples_leaf': range(10,100,10), 'n_estimators' : range(10,100,10),\n",
        "                    'max_features':['auto','sqrt','log2']\n",
        "                    }\n",
        "    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "07e5baa5-9656-30b8-5b42-49894aa80442",
        "_uuid": "04d170809ba881c8d2ce0882144cbaed755e2b8b"
      },
      "source": [
        "### n_jobs\n",
        "**This parameter tells the engine how many processors is it allowed to use. A value of “-1” means there is no restriction whereas a value of “1” means it can only use one processor.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b333b5c3-785c-3fee-b089-b75ec777e5c2",
        "_uuid": "88cef3ae9e85794e6d81ab8c97c478b8f3c488b6"
      },
      "outputs": [],
      "source": [
        "from sklearn.grid_search import RandomizedSearchCV\n",
        "\n",
        "RR_model= RandomizedSearchCV(model_RR, tuned_parameters,cv=10,scoring='accuracy',n_iter=20,n_jobs= -1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f7221d0c-d309-cc8e-31e0-e872438733b1",
        "_uuid": "fb2c22904cfff4d4a100a5033647636e79a744e6"
      },
      "outputs": [],
      "source": [
        "RR_model.fit(X_train,y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5bed2aca-9482-1dc9-bd11-25ff1ce05fb1",
        "_uuid": "368e6a3dbcabc4fb27a692d06fe522953f8a1b7b"
      },
      "outputs": [],
      "source": [
        "print(RR_model.grid_scores_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a4553b00-8a25-09b1-fa6a-f8a6e8dbd177",
        "_uuid": "502ac030a68e29757464f9757342baafb041e0df"
      },
      "outputs": [],
      "source": [
        "print(RR_model.best_score_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "02255028-9468-efdb-1da8-3d9458e7b9a7",
        "_uuid": "79568934ac1eebe10074fccdcd22b7d319bb20e4"
      },
      "outputs": [],
      "source": [
        "print(RR_model.best_params_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5feb4edc-6c89-b982-ffce-92ee75577da4",
        "_uuid": "4bbb82cdfe737ed6c90c0d2851ea0435c59ecd94"
      },
      "outputs": [],
      "source": [
        "y_prob = RR_model.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "RR_model.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f4e72487-3075-97a9-3ec3-173d5fe2f887",
        "_uuid": "e9e8337b6b9295ca3d437cb3f4035eccad5518e3"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "582e0f14-8f68-c8ab-e2c9-f3fc6f51744b",
        "_uuid": "945275fba35d2cd528051c2550e8e119233a9021"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "14df7ec6-e138-d5b1-a9f5-0ec73eeb46f3",
        "_uuid": "7a9a8a023a5243e2231a3408308b2adb6ceae275"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5230b9e7-2f66-d169-5991-0823af6c3028",
        "_uuid": "7c5fbcf1fb7cc5251ed2a22606a0fb8a94f541d8"
      },
      "outputs": [],
      "source": [
        "\n",
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e16d56b7-0ae4-8991-cd58-5a3772d6d7f1",
        "_uuid": "c1faa2317af2af3a8a2c47b35cde6a9e8efe9bb2"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5973819d-4c75-576e-59a3-732fcd22157f",
        "_uuid": "28d122390fc2042015157cdc68b8648be3d25528"
      },
      "source": [
        "### Default Decision Tree model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2ebd85f1-6ad9-21be-ab6a-05905df317b0",
        "_uuid": "fe961b76b2aaf55e5e71d8507a9cd265aa6d84e5"
      },
      "outputs": [],
      "source": [
        "from sklearn.tree import DecisionTreeClassifier\n",
        "\n",
        "model_tree = DecisionTreeClassifier()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1446f642-8c09-76e2-18f9-871e88643083",
        "_uuid": "b4c51d484ef474a6d654663840dc25708033b6e3"
      },
      "outputs": [],
      "source": [
        "model_tree.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b1b91a2c-296f-f301-14b4-9d2648592a87",
        "_uuid": "51019150d4f4d1e492056b04bbc92cbfb497df56"
      },
      "outputs": [],
      "source": [
        "y_prob = model_tree.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "model_tree.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "aa6a00dc-7e3c-319a-5dcd-3ac083e79479",
        "_uuid": "f7a06378b5f114f1c300bbb9da207c3cbca6be3f"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ff010249-5dae-c9b4-197f-6c843ae97903",
        "_uuid": "46e7f42cf5b5635a5475ff75e7ec77eda53baf50"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "05adf0ac-898e-1aa3-c569-00b8dec3e263",
        "_uuid": "88998a15264664fefc9eeb2e013504ba88d9a5ca"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "db4b8464-8e5c-4eb3-06f8-f72382b6941f",
        "_uuid": "dc033e1e4eabf2ebd60174dffc42a38b6e43da11"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f6860830-c215-377f-c34c-5df3ad1142da",
        "_uuid": "ee3ca580b17682ca24d77f07cea1fd72764aa0bc"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ee50bd8b-735c-d65d-71c9-bac21bed725a",
        "_uuid": "340128e0928ea69089c7d3c74b7ee562435d5e33"
      },
      "source": [
        "### Thus default decision tree model is giving us best accuracy score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c2dff12c-8a12-33a5-16a3-b57490d0fbf4",
        "_uuid": "89a9443d54e4b9f4b3a13109a2b6639a6929003c"
      },
      "source": [
        "### Let us tune the hyperparameters of the Decision tree model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bb16436e-ad87-2a0f-3d68-fbed40f5ed82",
        "_uuid": "557a9535cc660cf2d1a44135057736842204faca"
      },
      "source": [
        "**1)Criterion:** Decision trees use multiple algorithms to decide to split a node in two or more sub-nodes.Decision tree splits the nodes on all available variables and then selects the split which results in most homogeneous sub-nodes. The details of Gini and entropy needs detail explanation.\n",
        "\n",
        "2)**max_depth(Maximum depth of tree (vertical depth)):**\n",
        "Used to control over-fitting as higher depth will allow model to learn relations very specific to a particular sample.\n",
        "\n",
        "**max_features** and **min_samples_leaf** is same as Random Forest classifier"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "471fc8d9-35a3-e59a-7e3a-791f56b15381",
        "_uuid": "b0fe4275c53542068e5b11efd4fc0f72d2e565dd"
      },
      "outputs": [],
      "source": [
        "from sklearn.tree import DecisionTreeClassifier\n",
        "\n",
        "model_DD = DecisionTreeClassifier()\n",
        "\n",
        "\n",
        "tuned_parameters= {'criterion': ['gini','entropy'], 'max_features': [\"auto\",\"sqrt\",\"log2\"],\n",
        "                   'min_samples_leaf': range(1,100,1) , 'max_depth': range(1,50,1)\n",
        "                  }\n",
        "           "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6e292088-c9c9-32b2-050b-d871ee7e15b2",
        "_uuid": "7366a2a44ba819ab45438d1a0cb7b41bc7206976"
      },
      "outputs": [],
      "source": [
        "from sklearn.grid_search import RandomizedSearchCV\n",
        "DD_model= RandomizedSearchCV(model_DD, tuned_parameters,cv=10,scoring='accuracy',n_iter=20,n_jobs= -1,random_state=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "60b59643-c853-6607-6c9e-c58b16fc4c76",
        "_uuid": "efa25fecff80965d008ae7c69251b4b4e4aea6de"
      },
      "outputs": [],
      "source": [
        "DD_model.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d348db12-1afd-9c30-83b2-b01f94469d9c",
        "_uuid": "a3cb5925f02d219e3e9e078c5f17e6f9ff640349"
      },
      "outputs": [],
      "source": [
        "print(DD_model.grid_scores_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "97affd45-ad62-915f-38e8-6b41149729dc",
        "_uuid": "f374f78dd8813abb236605848d0cdfe2bf55b066"
      },
      "outputs": [],
      "source": [
        "print(DD_model.best_score_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "896cb38d-4b2f-bad4-631e-e399be807817",
        "_uuid": "7c02757987a25750cc41a6a91c6c8b94f180705a"
      },
      "outputs": [],
      "source": [
        "print(DD_model.best_params_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "47b9dfa0-e529-abbd-95fb-80cebc41b81b",
        "_uuid": "0d52e5be564ed3b638e29e4487160980e3efd00b"
      },
      "outputs": [],
      "source": [
        "y_prob = DD_model.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "DD_model.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "515e2157-6408-6e85-399a-e6b56fa82157",
        "_uuid": "0052924c4ef71dde902ce6815f6c80e182e70f5d"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "07d47b4c-a626-0fb7-b96f-07f8ddb8cd95",
        "_uuid": "db421b92da8a36fae039ebe48cd2e05e04f74f24"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5fb54b29-2502-3e1a-f8a8-78a3ba5bad06",
        "_uuid": "5f78ba5fbda830aaa39f7452d2676a91a1871bb3"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "950dc38c-cfcb-b0a0-3b6c-e46cac880b7f",
        "_uuid": "8891efe2cc1f786f90e153ae8145e360c3119df0"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f895a8c3-1c2e-0805-8764-3e57a260651c",
        "_uuid": "f613e8d18b5daef554c615e03beee52a18f207f1"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "410f17b3-d60a-10e8-df53-19cc869894e4",
        "_uuid": "cc3261a2eeb01c0fe53814338eee7d9e4f2ee7f0"
      },
      "source": [
        "## Neural Network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e4e497d7-78e2-1f59-4236-d1d543ddb632",
        "_uuid": "2d155cb1762988d9849054d4615e87ee32113336"
      },
      "source": [
        "### Applying default Neural Network model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b50b16ac-705b-523e-0125-1f8280087879",
        "_uuid": "e0dd38ce8b8f7631192b3b8b2f715e9c3977b8a9"
      },
      "outputs": [],
      "source": [
        "from sklearn.neural_network import MLPClassifier"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "488e2203-595a-35f1-342f-7ff9a32a864e",
        "_uuid": "301f9cb41f2c140e1485d53bf0c8279550db5afa"
      },
      "outputs": [],
      "source": [
        "mlp = MLPClassifier()\n",
        "mlp.fit(X_train,y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3eba1830-b130-2afc-45d6-7f9e182d3bf5",
        "_uuid": "b14ff9d1d56a63095ab452e0fae45dbb8bd3a081"
      },
      "outputs": [],
      "source": [
        "y_prob = mlp.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "mlp.score(X_test, y_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "df1fdb7e-4d0f-e42d-4046-878d8420699d",
        "_uuid": "9d1d9b2ea0b11104f9dd71f867f576508f1b07b1"
      },
      "outputs": [],
      "source": [
        "confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b4f20678-d020-9c78-a794-b1b924c882bb",
        "_uuid": "0f0859e946de3caed1173e28b152527b304080f2"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6c5392b2-8de2-8686-809c-d545d9c318a4",
        "_uuid": "9b125c1bab05d6bb25fcdaeaa8e5871c36df2515"
      },
      "outputs": [],
      "source": [
        "auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ec742bf2-a75f-6482-f9a0-e2156c67a04a",
        "_uuid": "3c1981637a6e4604bbfc763dc2ab410c66a3f30b"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "roc_auc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a20af212-174c-bd5a-8465-6a980ca1dd32",
        "_uuid": "a250099cc6ea45e6976f09e955b0698f9ea31cc1"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "71b42e1b-5e31-e576-1459-fcad4ac140d7",
        "_uuid": "b6984b27c54df935ff0ca2154e77455c8fe4b508"
      },
      "source": [
        "### Tuning the hyperparameters of the neural network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2447b4bd-fd97-ef55-1e68-454e686a4889",
        "_uuid": "a2e3389e57e4d78c075211dfb3a11e78ef50816e"
      },
      "source": [
        "It is turning out to be computationally expensive for me with tuned model. Hence I am not running this. Also any suggestion to improvise it is welcome. :)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bd096438-f5d3-1f62-41a7-03b4f42be394",
        "_uuid": "b8e8107b5b7f27a18297a48ec08151a7ba756c7a"
      },
      "source": [
        "**1) hidden_layer_sizes**  : Number of hidden layers in the network.(default is 100).Large number may overfit the data.\n",
        "\n",
        "**2)activation**: Activation function for the hidden layer.\n",
        "A)‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)).\n",
        "B)‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x).\n",
        "C)‘relu’, the rectified linear unit function, returns f(x) = max(0, x)\n",
        "\n",
        "**3)alpha:** L2 penalty (regularization term) parameter.(default 0.0001)\n",
        "\n",
        "**4)max_iter:** Maximum number of iterations. The solver iterates until convergence (determined by ‘tol’) or this number of iterations.(default 200)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c134b79f-031c-9604-7003-d3bd25a5251c",
        "_uuid": "d6ee5d0d1cb7254446252839351303c3a39b7308"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "\n",
        "mlp = MLPClassifier()\n",
        "\n",
        "tuned_parameters={'hidden_layer_sizes': range(1,200,10) , 'activation': ['tanh','logistic','relu'],\n",
        "                  'alpha':[0.0001,0.001,0.01,0.1,1,10], 'max_iter': range(50,200,50)\n",
        "    \n",
        "}\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1897166d-0941-2348-5e96-3b1cc2c2d4f2",
        "_uuid": "57c9c3b090be1cf6589d92da2df5c818c86a4626"
      },
      "outputs": [],
      "source": [
        "#from sklearn.grid_search import RandomizedSearchCV\n",
        "#model_mlp= RandomizedSearchCV(mlp_model, tuned_parameters,cv=10,scoring='accuracy',n_iter=5,n_jobs= -1,random_state=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7272dbfc-550a-5a44-bbf5-98e041b815ee",
        "_uuid": "1a76d9d4fdc0e6cae53e7ca3afc2f966731c701b"
      },
      "outputs": [],
      "source": [
        "#model_mlp.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "83de0f83-1245-04a2-a11a-01dc73b6d763",
        "_uuid": "7a40de29ebb4214389f9aff7a6b1dbcb805669f6"
      },
      "outputs": [],
      "source": [
        "#print(model_mlp.grid_scores_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ed41d08a-ccb7-41d3-31b3-f4a3d262a81e",
        "_uuid": "bdebe2650463c7e26a81f750ed7d5c53bb04fe70"
      },
      "outputs": [],
      "source": [
        "#print(model_mlp.best_score_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "80c49de3-4ab4-ce7f-ad03-62fbca353380",
        "_uuid": "5f20fb36d30d2bc194b28cddf2f412a57214808b"
      },
      "outputs": [],
      "source": [
        "#print(model_mlp.best_params_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d5679052-0545-d3ce-e3ae-0f2246c9c833",
        "_uuid": "967cc1a0327c3a29376163ba3452694d9f3df171"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "y_prob = model_LR.predict_proba(X_test)[:,1] # This will give you positive class prediction probabilities  \n",
        "y_pred = np.where(y_prob > 0.5, 1, 0) # This will threshold the probabilities to give class predictions.\n",
        "model_LR.score(X_test, y_pred)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "65d57d14-8915-2d98-7c2b-81f64ab33e3c",
        "_uuid": "9493183d2bcedc2661cbf0a375478c366bca10a5"
      },
      "outputs": [],
      "source": [
        "#confusion_matrix=metrics.confusion_matrix(y_test,y_pred)\n",
        "#confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e2c734a0-e57f-3db2-4db7-b8b350518f98",
        "_uuid": "582311345e15e261c7fcd203ebef2cb009031941"
      },
      "outputs": [],
      "source": [
        "#auc_roc=metrics.classification_report(y_test,y_pred)\n",
        "#auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f5a50044-422f-6440-47aa-8c7a15a3b0fe",
        "_uuid": "edcf42de7f06c2bc68a8fc507cd3093d7ff427e2"
      },
      "outputs": [],
      "source": [
        "#auc_roc=metrics.roc_auc_score(y_test,y_pred)\n",
        "#auc_roc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b858023f-fa46-9912-f647-888741dbdbd2",
        "_uuid": "a22c3f8ec94103e2ca63bcb86382e900a1aa9d62"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "from sklearn.metrics import roc_curve, auc\n",
        "false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_prob)\n",
        "roc_auc = auc(false_positive_rate, true_positive_rate)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1c96335c-abf0-9dfc-b081-3f8b7525c525",
        "_uuid": "7981abee7ef22575c23651e067d169e2d1524e42"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(false_positive_rate,true_positive_rate, color='red',label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],linestyle='--')\n",
        "plt.axis('tight')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')\n",
        "'''"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}