reproduce_results.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reproduce results\n",
    "### Adrián Fernández Cid\n",
    "\n",
    "This notebook loads the input matrices and trained models to reproduce the validation and test results shown in train_models.ipynb. In addition, I show and comment on a sample of the test mistakes produced by the two best models: the neural network with a tf-idf vectorisation and the cosine similarity, and the logistic regression with the same preprocessing. The whole notebook executes in <1min."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# custom stuff\n",
    "from utils import *\n",
    "# basic stuff\n",
    "import pandas as pd\n",
    "import scipy as sp\n",
    "import numpy as np\n",
    "import os\n",
    "import pickle  #for saving matrices, models, etc.\n",
    "#sklearn stuff\n",
    "import sklearn\n",
    "from sklearn import *\n",
    "from xgboost.sklearn import XGBClassifier\n",
    "from sklearn.metrics import roc_auc_score\n",
    "#tensoflow stuff\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from keras.models import model_from_json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>qid1</th>\n",
       "      <th>qid2</th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>is_duplicate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>What is the step by step guide to invest in sh...</td>\n",
       "      <td>What is the step by step guide to invest in sh...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>What is the story of Kohinoor (Koh-i-Noor) Dia...</td>\n",
       "      <td>What would happen if the Indian government sto...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>How can I increase the speed of my internet co...</td>\n",
       "      <td>How can Internet speed be increased by hacking...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>Why am I mentally very lonely? How can I solve...</td>\n",
       "      <td>Find the remainder when [math]23^{24}[/math] i...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>Which one dissolve in water quikly sugar, salt...</td>\n",
       "      <td>Which fish would survive in salt water?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  qid1  qid2                                          question1  \\\n",
       "0   0     1     2  What is the step by step guide to invest in sh...   \n",
       "1   1     3     4  What is the story of Kohinoor (Koh-i-Noor) Dia...   \n",
       "2   2     5     6  How can I increase the speed of my internet co...   \n",
       "3   3     7     8  Why am I mentally very lonely? How can I solve...   \n",
       "4   4     9    10  Which one dissolve in water quikly sugar, salt...   \n",
       "\n",
       "                                           question2  is_duplicate  \n",
       "0  What is the step by step guide to invest in sh...             0  \n",
       "1  What would happen if the Indian government sto...             0  \n",
       "2  How can Internet speed be increased by hacking...             0  \n",
       "3  Find the remainder when [math]23^{24}[/math] i...             0  \n",
       "4            Which fish would survive in salt water?             0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_data =  os.path.expanduser('~') \n",
    "\n",
    "train_df = pd.read_csv(os.path.join(path_data,\n",
    "                                    os.path.join(\"Datasets\", \"kaggle_datasets\", \"quora\", \"quora_train_data.csv\")))\n",
    "\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tr_df.shape= (364871, 6)\n",
      "va_df.shape= (19204, 6)\n",
      "te_df.shape= (20215, 6)\n"
     ]
    }
   ],
   "source": [
    "A_df, te_df = sklearn.model_selection.train_test_split(train_df, test_size=0.05,random_state=123)\n",
    "\n",
    "tr_df, va_df = sklearn.model_selection.train_test_split(A_df, test_size=0.05,random_state=123)\n",
    "print('tr_df.shape=',tr_df.shape)\n",
    "print('va_df.shape=',va_df.shape)\n",
    "print('te_df.shape=',te_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_tr = tr_df[\"is_duplicate\"].values\n",
    "y_va = va_df[\"is_duplicate\"].values\n",
    "y_te = te_df[\"is_duplicate\"].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load input matrices and trained models, and print the validation results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "LOGISTIC REGRESSION\n",
      "==================================================\n",
      "--------------------------------------\n",
      "Count vectoriser\n",
      "--------------------------------------\n",
      "****With no new feature****\n",
      "-Validation\n",
      "AUC: 0.8058412271384804\n",
      "log loss: 0.5150855223485794\n",
      "****With Euclidean distance new feature****\n",
      "-Validation\n",
      "AUC: 0.8058355318707418\n",
      "log loss: 0.5150976791190962\n",
      "****With cosine similarity new feature****\n",
      "-Validation\n",
      "AUC: 0.8567268858027568\n",
      "log loss: 0.4532025503252208\n",
      "****With edit distance new feature****\n",
      "-Validation\n",
      "ERROR. Could not get metrics: there's an issue with input matrix.\n",
      "--------------------------------------\n",
      "Sklearn's tf-idf vectoriser\n",
      "--------------------------------------\n",
      "****With no new feature****\n",
      "-Validation\n",
      "AUC: 0.8033913596741575\n",
      "log loss: 0.5075859218079861\n",
      "****With Euclidean distance new feature****\n",
      "-Validation\n",
      "AUC: 0.8033859222168925\n",
      "log loss: 0.5076338630398345\n",
      "****With cosine similarity new feature****\n",
      "-Validation\n",
      "AUC: 0.8700305156195411\n",
      "log loss: 0.4330545686924139\n",
      "****With edit distance new feature****\n",
      "-Validation\n",
      "ERROR. Could not get metrics: there's an issue with input matrix.\n",
      "==================================================\n",
      "XGBOOST\n",
      "==================================================\n",
      "--------------------------------------\n",
      "Count vectoriser\n",
      "--------------------------------------\n",
      "****With no new feature****\n",
      "-Validation\n",
      "AUC: 0.8196534408487384\n",
      "log loss: 0.5044448192961917\n",
      "****With Euclidean distance new feature****\n",
      "-Validation\n",
      "AUC: 0.8196534408487384\n",
      "log loss: 0.5044448192961917\n",
      "****With cosine similarity new feature****\n",
      "-Validation\n",
      "AUC: 0.8552098352258263\n",
      "log loss: 0.4491335918600538\n",
      "****With edit distance new feature****\n",
      "-Validation\n",
      "AUC: 0.8200587775067956\n",
      "log loss: 0.49800502105421174\n",
      "--------------------------------------\n",
      "Sklearn's tf-idf vectoriser\n",
      "--------------------------------------\n",
      "****With no new feature****\n",
      "-Validation\n",
      "AUC: 0.8200587775067956\n",
      "log loss: 0.49800502105421174\n",
      "****With Euclidean distance new feature****\n",
      "-Validation\n",
      "AUC: 0.8200587775067956\n",
      "log loss: 0.49800502105421174\n",
      "****With cosine similarity new feature****\n",
      "-Validation\n",
      "AUC: 0.8639790827112718\n",
      "log loss: 0.43414601872006653\n",
      "****With edit distance new feature****\n",
      "-Validation\n",
      "AUC: 0.8200587775067956\n",
      "log loss: 0.49800502105421174\n",
      "CPU times: user 6.55 s, sys: 241 ms, total: 6.79 s\n",
      "Wall time: 5.07 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "mod_fnames = [\"logistic\", \"xgb\"]\n",
    "mod_print = [\"LOGISTIC REGRESSION\", \"XGBOOST\"]\n",
    "vec_fnames = [\"count\", \"skl_tfidf\"]\n",
    "vec_print = [\"Count\", \"Sklearn's tf-idf\"]\n",
    "feat_fnames = [\"raw\", \"euclid\", \"cos\", \"edit\"]\n",
    "feat_print = [\"no\", \"Euclidean distance\", \"cosine similarity\", \"edit distance\"]\n",
    "for i, mod in enumerate(mod_fnames):\n",
    "    print(\"==================================================\")\n",
    "    print(f\"{mod_print[i]}\")\n",
    "    print(\"==================================================\")\n",
    "    for j, vec in enumerate(vec_fnames):\n",
    "        print(\"--------------------------------------\")\n",
    "        print(f\"{vec_print[j]} vectoriser\")\n",
    "        print(\"--------------------------------------\")\n",
    "        for k, feat in enumerate(feat_fnames):\n",
    "            print(f\"****With {feat_print[k]} new feature****\")\n",
    "            print(f\"-Validation\")\n",
    "            # load input matrix\n",
    "            Xva_fname = \"results/Xva_\"+vec+\"_\"+feat+\".npz\"\n",
    "            Xva = sp.sparse.load_npz(Xva_fname)\n",
    "            # load model\n",
    "            filename = \"models/\"+mod+\"_\"+vec+\"_\"+feat+\".sav\"\n",
    "            clf = pickle.load(open(filename, 'rb'))\n",
    "            try:\n",
    "                print_auc_logloss(clf, Xva, y_va)\n",
    "            except:\n",
    "                print(\"ERROR. Could not get metrics: there's an issue with input matrix.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown in train_models.ipynb, the winner is the logistic regression with a tf-idf vectorisation and the cosine similarity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the test results for the previous winner. As we've established in train_models.ipynb, my tf-idf's and sklearn's results are the same up to 4 decimal digits, so I use my own vectoriser: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-Validation\n",
      "AUC: 0.8700340429355689\n",
      "log loss: 0.43305295164377394\n",
      "-Test\n",
      "AUC: 0.8757583837689608\n",
      "log loss: 0.42572321614875147\n"
     ]
    }
   ],
   "source": [
    "# load feature matrices\n",
    "Xval_fname = \"results/Xva_mytfidf_cos.npz\"\n",
    "Xval = sp.sparse.load_npz(Xval_fname)\n",
    "Xte_fname = \"results/Xte_mytfidf_cos.npz\"\n",
    "Xte = sp.sparse.load_npz(Xte_fname)\n",
    "# load model\n",
    "m_fname = \"models/logistic_mytfidf_cos.sav\"\n",
    "clf = pickle.load(open(m_fname, 'rb'))\n",
    "# get performance metrics\n",
    "print(\"-Validation\")\n",
    "print_auc_logloss(clf, Xval, y_va) \n",
    "print(\"-Test\")\n",
    "print_auc_logloss(clf, Xte, y_te)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the absolute best was the neural network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = \"mytfidf\"\n",
    "feat = \"cos\"\n",
    "# validation\n",
    "Xval_fname = \"results/Xva_\"+vec+\"_\"+feat+\".npz\"\n",
    "Xval = sp.sparse.load_npz(Xval_fname)\n",
    "Xval = convert_sparse_matrix_to_ordered_sparse_tensor(Xval)\n",
    "# test\n",
    "Xte_fname = \"results/Xte_\"+vec+\"_\"+feat+\".npz\"\n",
    "Xte = sp.sparse.load_npz(Xte_fname)\n",
    "Xte= convert_sparse_matrix_to_ordered_sparse_tensor(Xte)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded model neural_mytfidf_cos\n",
      "Validation AUC: 0.8765279958034079\n",
      "Test AUC: 0.8805521028374024\n"
     ]
    }
   ],
   "source": [
    "name = \"neural_mytfidf_cos\"\n",
    "# load json and create model\n",
    "json_file = open(\"models/\"+name+\".json\", \"r\")\n",
    "loaded_model_json = json_file.read()\n",
    "json_file.close()\n",
    "new_model = model_from_json(loaded_model_json)\n",
    "\n",
    "# load weights into new model\n",
    "new_model.load_weights(\"models/\"+name+\".h5\")\n",
    "print(f\"Loaded model {name}\")\n",
    "\n",
    "# check its auc\n",
    "in_list=[Xval, Xte] \n",
    "y_list = [y_va, y_te]\n",
    "steps = [\"Validation\", \"Test\"]\n",
    "for i, x in enumerate(in_list):\n",
    "    probs = new_model.predict(x)[:,1]\n",
    "    auc = roc_auc_score(y_list[i], probs)\n",
    "    print(f\"{steps[i]} AUC: {auc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Erroneus predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "error rate: 0.205392035617116 \n",
      " total mistakes: 4152\n",
      "==================================================================\n",
      "1\n",
      "How do I prepare for Gate Exam by myself?\n",
      "What are the best ways to prepare gate exam?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "2\n",
      "Does the black hole hold a gateway to another universe?\n",
      "Would a black hole be the exit of this universe?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "3\n",
      "How do I get rid of acne on my face? I workout daily and wash my face twice a day.\n",
      "What products should I use to get rid of acne quickly?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "4\n",
      "Why do we do rainwater harvesting?\n",
      "Why is rainwater harvesting illegal?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "5\n",
      "How can I get the girl I like?\n",
      "How do I get over a girl I like?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "6\n",
      "How can I stop my depression?\n",
      "What can I do to stop being depressed?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "7\n",
      "Which is the best site to book hotel online?\n",
      "What is the best hotel booking service?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "8\n",
      "Can H4 visa holders invest in stock markets?\n",
      "Can H4 visa holders invest in US stockmarkets?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "9\n",
      "What is the largest cell?\n",
      "Which is the biggest living cell?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "10\n",
      "What are Trump's most controversial cabinet picks?\n",
      "Are Trump's cabinet picks expected or are they all surprises?\n",
      "true class: 0\n",
      "prediction: 1\n"
     ]
    }
   ],
   "source": [
    "# test\n",
    "Xte_fname = \"results/Xte_\"+vec+\"_\"+feat+\".npz\"\n",
    "Xte = sp.sparse.load_npz(Xte_fname)\n",
    "\n",
    "mistake_indices, predictions = get_mistakes(clf,Xte, y_te)\n",
    "print(\"==================================================================\")\n",
    "for i in range(10):\n",
    "    print(i+1)\n",
    "    print_mistake_k(te_df, 100+i, mistake_indices, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some samples seem trickier (like the 5th), but others look quite straightforward (like the 7th). On the other hand, one could argue that predictions 1 and 8 are actually right, or at least reasonable. \n",
    "\n",
    "One thing that we verify is that sometimes the only thing telling two questions apart is a single word (e.g. sample 5), which highlights the limitations of a static vocabulary. One could therefore try using a dynamic or more complete vocabulary for vectorisation.\n",
    "\n",
    "It also stands out that the addition of information unrelated to the character of the question seems to offset the classifier in the 3rd case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "error rate: 0.2017313875834776 \n",
      " total mistakes: 4078\n",
      "==================================================================\n",
      "1\n",
      "Why was Tim Kaine chosen as the running mate for Hillary Clinton?\n",
      "How did Hillary Clinton decide to choose Tim Kaine as her running mate?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "2\n",
      "Why am I not able to post answers or comments on answers on Quora?\n",
      "Why am I not able to edit my answer on Quora?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "3\n",
      "What is the name of this T.V series ?\n",
      "What's the name of this TV series?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "4\n",
      "Why does my urine smell like garlic?\n",
      "Why does my urine smell like onions?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "5\n",
      "Which is the best laptop for mechanical?\n",
      "Which is the best laptop for engineers?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "6\n",
      "If you had the chance to meet with Sunil Gavaskar, what would you tell him?\n",
      "Which was Sunil Gavaskar's greatest innnings?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "7\n",
      "How are salt bridges used in galvanic cells?\n",
      "Why salt bridge are used in galvanic cell?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "8\n",
      "How does Zuckerberg earn the money, when facebook is free to the users?\n",
      "Besides advertisement, how does Facebook earn money?\n",
      "true class: 1\n",
      "prediction: 0\n",
      "9\n",
      "What steps are involved in fermentation?\n",
      "What are the steps involved in the solving of the tertiary butane formula?\n",
      "true class: 0\n",
      "prediction: 1\n",
      "10\n",
      "What is an SSL Certificate?\n",
      "What is SSL certificate?\n",
      "true class: 0\n",
      "prediction: 1\n"
     ]
    }
   ],
   "source": [
    "Xte_tf= convert_sparse_matrix_to_ordered_sparse_tensor(Xte)\n",
    "\n",
    "mistake_indices, predictions = get_mistakes(new_model,Xte_tf, y_te, neural_net=True)\n",
    "print(\"==================================================================\")\n",
    "for i in range(10):\n",
    "    print(i+1)\n",
    "    print_mistake_k(te_df, 100+i, mistake_indices, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some errors look reasonable (like the 4th, the 7th or the 8th) and others understandable (like the 5th or the 2nd). Aside from the already mentioned fact that single words sometimes make the difference, like in sample 5 (which suggests using a dynamic or more complete vocabulary), there does not seem to be a clear error pattern to be addressed at preprocessing.\n",
    "\n",
    "Nevertheless, the ground truth for samples 3 and 10 seems utterly wrong: either labels are poorly given or there is a problem with the reordering performed in the feature matrix for it to be compatible with tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}