3_textClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Standard Tools Pandas, Scikit, Numpy, Scipy, Matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#NOTES\n",
    "#Data Analysis PANDAS\n",
    "#Machine Learning SCIKIT LEARN\n",
    "#VIZUALIZATION Matplotlib, Seaborn, Pandas\n",
    "#Math, Linear Algebra, Matrix - Numpy and Scipy\n",
    "\n",
    "\n",
    "import pandas as pd\n",
    "import sklearn as sk\n",
    "import numpy as np\n",
    "import scipy as sc\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "np.random.seed(sum(map(ord, \"aesthetics\")))\n",
    "\n",
    "# Set some pandas options for controlling output\n",
    "pd.set_option('display.notebook_repr_html', False)\n",
    "pd.set_option('display.max_columns', 20)\n",
    "pd.set_option('display.max_rows', 20)\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "#Python Natural Language Processing ToolKit\n",
    "nltk.data.path.append(\"/usr/share/nltk_data-3\")\n",
    "\n",
    "\n",
    "#Other NLP Libraries OPENNLP from Stanford Univ, Apache Tika "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Extraction from Text: Bag of Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A collection of text documents is called a corpus. \n",
    "#A text document can be a short string, tweet or a large document. \n",
    "\n",
    "\n",
    "#List of 2 String Documents \n",
    "#The task is to respresent the documents as a Vector (List or Array) of Numericals\n",
    "corpus = [\n",
    "'UNC played Duke in basketball',\n",
    "'Duke lost the basketball game'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 1 0 1 0 1 0 1]\n",
      " [1 1 1 0 1 0 1 0]]\n",
      "{'basketball': 0, 'played': 5, 'lost': 4, 'in': 3, 'the': 6, 'unc': 7, 'duke': 1, 'game': 2}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\\n1. Given a corpus, generate the vocabulary of words that are in the corpus\\n2. Given a document, create a vector of 0 and 1 depending on whether a vocabulary\\nword is present or in the document\\n3. Thus, the feature vector size is the same as the size of the Vocabulary\\n\\nIssues:\\n4. Vocabulary size can be large (100s of thousands or more)\\n5. Some ways to reduce the vocabulary sizes are :\\n\\n  a. Some Prefiltering (throw some words) Stopwords, Stemming, Lemmazition\\n  b. Choosing only top-k important words (features) Feature Selection\\n  c. Dimensionality Reduction using Matrix Factorization \\n     (SVD is very common in Text Mining)\\n\\n'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#CountVectorizer is a class for feature extraction from text data\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    "\n",
    "#todense converts it into a Numpy dense matrix \n",
    "print (vectorizer.fit_transform(corpus).todense())\n",
    "print (vectorizer.vocabulary_)\n",
    "\n",
    "'''\n",
    "1. Given a corpus, generate the vocabulary of words that are in the corpus\n",
    "2. Given a document, create a vector of 0 and 1 depending on whether a vocabulary\n",
    "word is present or in the document\n",
    "3. Thus, the feature vector size is the same as the size of the Vocabulary\n",
    "\n",
    "Issues:\n",
    "4. Vocabulary size can be large (100s of thousands or more)\n",
    "5. Some ways to reduce the vocabulary sizes are :\n",
    "\n",
    "  a. Some Prefiltering (throw some words) Stopwords, Stemming, Lemmazition\n",
    "  b. Choosing only top-k important words (features) Feature Selection\n",
    "  c. Dimensionality Reduction using Matrix Factorization \n",
    "     (SVD is very common in Text Mining)\n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 1 0 1 0 1 0 0 1]\n",
      " [0 1 1 1 0 1 0 0 1 0]\n",
      " [1 0 0 0 0 0 0 1 0 0]]\n",
      "{'basketball': 1, 'played': 6, 'lost': 5, 'in': 4, 'the': 8, 'sandwich': 7, 'unc': 9, 'duke': 2, 'ate': 0, 'game': 3}\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "#Another example by adding a third statement different from the first two in the\n",
    "#words that it contains \n",
    "\n",
    "corpus = [\n",
    "'UNC played Duke in basketball',\n",
    "'Duke lost the basketball game',\n",
    "'I ate a sandwich'\n",
    "]\n",
    "\n",
    "print (vectorizer.fit_transform(corpus).todense())\n",
    "print (vectorizer.vocabulary_)\n",
    "print(len(vectorizer.vocabulary_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distance between Text Documents, Measurement of Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 1 0 1 0 1 0 0 1]\n",
      " [0 1 1 1 0 1 0 0 1 0]\n",
      " [1 0 0 0 0 0 0 1 0 0]]\n",
      "Distance between 1st and 2nd documents: [[ 2.44948974]]\n",
      "Distance between 1st and 3rd documents: [[ 2.64575131]]\n",
      "Distance between 2nd and 3rd documents: [[ 2.64575131]]\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Once u have a feature vector, there are many ways in which u can\n",
    "compute how similar/dissimilar they are. Another way is to view\n",
    "is to compute distance between feature vectors. Larger the distance,\n",
    "the less similar two vectors are. \n",
    "\n",
    "Example of such measures are:\n",
    "Euclidean Distance\n",
    "Manhattan Distance\n",
    "Cosine Similarity for Feature Vectors \n",
    "Jaccard Similarity for Sets of Data\n",
    "\n",
    "Similarity between two document feature vectors can tell you\n",
    "how similar they are . This can be used for document clustering\n",
    "or classification.\n",
    "\n",
    "'''\n",
    "\n",
    "\n",
    "from sklearn.metrics.pairwise import euclidean_distances\n",
    "\n",
    "array_ = vectorizer.fit_transform(corpus).todense()\n",
    "print(array_)\n",
    "\n",
    "print ('Distance between 1st and 2nd documents:'\n",
    "       , euclidean_distances(array_[0], array_[1]))\n",
    "\n",
    "print ('Distance between 1st and 3rd documents:'\n",
    "       , euclidean_distances(array_[0], array_[2]))\n",
    "\n",
    "print ('Distance between 2nd and 3rd documents:'\n",
    "       , euclidean_distances(array_[1], array_[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CosSim between 1st and 2nd documents: [[ 0.4]]\n",
      "CosSim between 2nd and 3rd documents: [[ 0.]]\n",
      "CosSim between 1st and 3rd documents: [[ 0.]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "print ('CosSim between 1st and 2nd documents:'\n",
    "       , cosine_similarity(array_[0], array_[1]))\n",
    "\n",
    "print ('CosSim between 2nd and 3rd documents:'\n",
    "       , cosine_similarity(array_[1], array_[2]))\n",
    "\n",
    "print ('CosSim between 1st and 3rd documents:'\n",
    "       , cosine_similarity(array_[0], array_[2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 1 0 0 1 0 1]\n",
      " [0 1 1 1 1 0 0 0]\n",
      " [1 0 0 0 0 0 1 0]]\n",
      "{'basketball': 1, 'played': 5, 'lost': 4, 'sandwich': 6, 'unc': 7, 'duke': 2, 'ate': 0, 'game': 3}\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "#STOPWORD FILTERING\n",
    "\n",
    "'''\n",
    "Stopwords are words that are common and do not add any value\n",
    "to document semantics. We can filter these before doing the\n",
    "feature generation.\n",
    "\n",
    "Example of such words are: a, at, an, the, in , or, etc.\n",
    "Many libraries have stopwords in various languages .\n",
    "\n",
    "As a developer, you can always add somewords to the stopword list\n",
    "if you think you wanna filter them our. So, stopwords lists can \n",
    "be extended. \n",
    "\n",
    "'''\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "corpus = [\n",
    "'UNC played Duke in basketball',\n",
    "'Duke lost the basketball game',\n",
    "'I ate a sandwich'\n",
    "]\n",
    "vectorizer = CountVectorizer(stop_words='english')\n",
    "print (vectorizer.fit_transform(corpus).todense())\n",
    "print (vectorizer.vocabulary_)\n",
    "print(len(vectorizer.vocabulary_))\n",
    "\n",
    "#the and in are filtered  after Stopword filtering stage\n",
    "#countevectorizer by default throws words like \"I\" , \"a\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 0 0 1]\n",
      " [0 1 1 0]]\n",
      "{'sandwich': 2, 'sandwiches': 3, 'eaten': 1, 'ate': 0}\n"
     ]
    }
   ],
   "source": [
    "#Stemming and lemmatization\n",
    "'''\n",
    "After stopword filtering, we can do some more steps. Most common\n",
    "are Stemming and maybe, Lemmatization. \n",
    "\n",
    "Stemming : Input is a word and output is the basic form \n",
    "e.g. sandwiches and sandwich both output sandwich. we dont want\n",
    "many variants of same word in vocabulary. [run, Run, ran, Ran, running] should\n",
    "all boild down to run.\n",
    "\n",
    "Example of stemmers :  Porter stemmer, Lancaster Stemmer, Snowball Stemmer \n",
    "\n",
    "Knowledge of Regular Expressions and Text Processing Libraries can be Useful.\n",
    "NLTK is one in Python. OPENNLP, STANFORDNLP, TIKA, etc. There are many text \n",
    "processing libraries. Quite a few in Java are available.\n",
    "\n",
    "Lemmatization usually refers to doing \n",
    "things properly with the use of a\n",
    "vocabulary and morphological analysis \n",
    "of words, normally aiming to remove inflectional\n",
    "endings only and to return the base or \n",
    "dictionary form of a word, which is known as the lemma \n",
    "\n",
    "Lemmatisation (or lemmatization) in linguistics, \n",
    "is the process of grouping together the different \n",
    "inflected forms of a word so they can be analysed as a single item.\n",
    "\n",
    "http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization\n",
    "'''\n",
    "\n",
    "#without stemming and lemmatization\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "corpus = [\n",
    "'He ate the sandwiches',\n",
    "'Every sandwich was eaten by him'\n",
    "]\n",
    "vectorizer = CountVectorizer(binary=True, stop_words='english')\n",
    "print (vectorizer.fit_transform(corpus).todense())\n",
    "print ( vectorizer.vocabulary_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sandwich\n",
      "sandwich\n",
      "presum\n",
      "owe\n",
      "abacus\n"
     ]
    }
   ],
   "source": [
    "# Example of stemming \n",
    "from nltk.stem.porter import PorterStemmer\n",
    "porter_stemmer = PorterStemmer()\n",
    "\n",
    "\n",
    "print(porter_stemmer.stem('sandwiches'))\n",
    "print(porter_stemmer.stem('sandwich'))\n",
    "print(porter_stemmer.stem('presumably'))\n",
    "print(porter_stemmer.stem('owed'))\n",
    "print(porter_stemmer.stem('abacuses'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sandwich\n",
      "dog\n",
      "abacus\n",
      "is\n",
      "be\n",
      "be\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\\nThe NLTK Lemmatization method is based on WordNet’s built-in morphy function.\\nHere is the introduction from WordNet official website:\\n\\n\\nWordNet® is a large lexical database of English. Nouns, verbs, \\nadjectives and adverbs are grouped into sets of cognitive synonyms\\n(synsets), each expressing a distinct concept. Synsets are interlinked \\nby means of conceptual-semantic and lexical relations. The \\nresulting network of meaningfully related words and concepts \\ncan be navigated with the browser. WordNet is also freely and \\npublicly available for download. WordNet’s structure makes it a \\nuseful tool for computational linguistics and natural language processing.\\n\\nWordNet superficially resembles a thesaurus, in that it \\ngroups words together based on their meanings. However, \\nthere are some important distinctions. First, WordNet interlinks \\nnot just word forms—strings of letters—but specific senses of\\nwords. As a result, words that are found in close proximity \\nto one another in the network are semantically disambiguated.\\nSecond, WordNet labels the semantic relations among words, \\nwhereas the groupings of words in a thesaurus does not follow\\nany explicit pattern other than meaning similarity\\n\\n\\nThe goal of both stemming and lemmatization is to reduce inflectional \\nforms and sometimes derivationally related forms of a word to a common \\nbase form.\\n\\nHowever, the two words differ in their flavor. Stemming usually \\nrefers to a crude heuristic process that chops off the ends of \\nwords in the hope of achieving this goal correctly most of the \\ntime, and often includes the removal of derivational affixes. \\nLemmatization usually refers to doing things properly with the \\nuse of a vocabulary and morphological analysis of words, \\nnormally aiming to remove inflectional endings only and to\\nreturn the base or dictionary form of a word, which is known as the lemma .\\n\\n\\n'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "\n",
    "wordnet_lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "print(wordnet_lemmatizer.lemmatize('sandwiches'))\n",
    "print(wordnet_lemmatizer.lemmatize('dogs'))\n",
    "print(wordnet_lemmatizer.lemmatize('abaci'))\n",
    "\n",
    "print(wordnet_lemmatizer.lemmatize('is'))\n",
    "print(wordnet_lemmatizer.lemmatize('is', pos='v'))\n",
    "\n",
    "print(wordnet_lemmatizer.lemmatize('are', pos='v'))\n",
    "\n",
    "\n",
    "'''\n",
    "The NLTK Lemmatization method is based on WordNet’s built-in morphy function.\n",
    "Here is the introduction from WordNet official website:\n",
    "\n",
    "\n",
    "WordNet® is a large lexical database of English. Nouns, verbs, \n",
    "adjectives and adverbs are grouped into sets of cognitive synonyms\n",
    "(synsets), each expressing a distinct concept. Synsets are interlinked \n",
    "by means of conceptual-semantic and lexical relations. The \n",
    "resulting network of meaningfully related words and concepts \n",
    "can be navigated with the browser. WordNet is also freely and \n",
    "publicly available for download. WordNet’s structure makes it a \n",
    "useful tool for computational linguistics and natural language processing.\n",
    "\n",
    "WordNet superficially resembles a thesaurus, in that it \n",
    "groups words together based on their meanings. However, \n",
    "there are some important distinctions. First, WordNet interlinks \n",
    "not just word forms—strings of letters—but specific senses of\n",
    "words. As a result, words that are found in close proximity \n",
    "to one another in the network are semantically disambiguated.\n",
    "Second, WordNet labels the semantic relations among words, \n",
    "whereas the groupings of words in a thesaurus does not follow\n",
    "any explicit pattern other than meaning similarity\n",
    "\n",
    "\n",
    "The goal of both stemming and lemmatization is to reduce inflectional \n",
    "forms and sometimes derivationally related forms of a word to a common \n",
    "base form.\n",
    "\n",
    "However, the two words differ in their flavor. Stemming usually \n",
    "refers to a crude heuristic process that chops off the ends of \n",
    "words in the hope of achieving this goal correctly most of the \n",
    "time, and often includes the removal of derivational affixes. \n",
    "Lemmatization usually refers to doing things properly with the \n",
    "use of a vocabulary and morphological analysis of words, \n",
    "normally aiming to remove inflectional endings only and to\n",
    "return the base or dictionary form of a word, which is known as the lemma .\n",
    "\n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gather\n",
      "POS INFO\n",
      "[[('He', 'PRP'), ('ate', 'VBP'), ('the', 'DT'), ('sandwiches', 'NNS')], [('Every', 'DT'), ('sandwich', 'NN'), ('was', 'VBD'), ('eaten', 'VBN'), ('by', 'IN'), ('him', 'PRP')]]\n",
      "After Stemming\n",
      "Stemmed: [['He', 'ate', 'the', 'sandwich'], ['Everi', 'sandwich', 'wa', 'eaten', 'by', 'him']]\n",
      "After Lemmatization\n",
      "Lemmatized: [['He', 'eat', 'the', 'sandwich'], ['Every', 'sandwich', 'be', 'eat', 'by', 'him']]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "import string,re\n",
    "\n",
    "#NLTK IMPORTS \n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk import pos_tag\n",
    "\n",
    "\n",
    "\n",
    "def lemmatize(token, tag):\n",
    "    if tag[0].lower() in ['n', 'v']:\n",
    "        return lemmatizer.lemmatize(token, tag[0].lower())\n",
    "    return token\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "stemmer = PorterStemmer()\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "\n",
    "print (stemmer.stem('gathering'))\n",
    "\n",
    "corpus = [\n",
    "'He ate the sandwiches',\n",
    "'Every sandwich was eaten by him'\n",
    "]\n",
    "\n",
    "\n",
    "\n",
    "#Python List Comprehension\n",
    "\n",
    "tagged_corpus = [pos_tag(word_tokenize(document)) for document in corpus]\n",
    "print(\"POS INFO\")\n",
    "print(tagged_corpus)\n",
    "\n",
    "print(\"After Stemming\")\n",
    "print ('Stemmed:', [[stemmer.stem(token) for token in \n",
    "                    word_tokenize(document)] for document in corpus])\n",
    "\n",
    "print(\"After Lemmatization\")\n",
    "print ('Lemmatized:', [[lemmatize(token, tag) for token, tag in\n",
    "document] for document in tagged_corpus])\n",
    "\n",
    "#\n",
    "#Through stemming and lemmatization, we can reduce the dimensionality of our\n",
    "#feature space.\n",
    "\n",
    "#Stemming was -> wa   , every -> every (stemming may producsing useless output)\n",
    "#Lemmatization was -> be , every -> every (lemmatization will procude meaningful words)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2 1 2 0 0]\n",
      " [0 0 1 1 1]]\n"
     ]
    }
   ],
   "source": [
    "#Motivate TF IDF Term Freq , Inverse Doc Frequency\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "corpus = ['The dog ate a sandwich and I ate a sandwich', 'the wizard transfigured a sandwich ']\n",
    "vectorizer = CountVectorizer(stop_words='english')\n",
    "print (vectorizer.fit_transform(corpus).todense())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CountVectorizer Feat Vectors\n",
      "[[2 1 2 0 0]\n",
      " [0 0 1 1 1]]\n",
      "{'sandwich': 2, 'transfigured': 3, 'ate': 0, 'dog': 1, 'wizard': 4}\n",
      "TFIDF Feat Vectors\n",
      "[[ 0.75458397  0.37729199  0.53689271  0.          0.        ]\n",
      " [ 0.          0.          0.44943642  0.6316672   0.6316672 ]]\n",
      "{'sandwich': 2, 'transfigured': 3, 'ate': 0, 'dog': 1, 'wizard': 4}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "corpus = ['The dog ate a sandwich and I ate a sandwich',\n",
    "          'The wizard transfigured a sandwich']\n",
    "\n",
    "print(\"CountVectorizer Feat Vectors\")\n",
    "vectorizer = CountVectorizer(stop_words='english')\n",
    "print (vectorizer.fit_transform(corpus).todense())\n",
    "print(vectorizer.vocabulary_)\n",
    "\n",
    "\n",
    "print(\"TFIDF Feat Vectors\")\n",
    "vectorizer = TfidfVectorizer(stop_words='english')\n",
    "print (vectorizer.fit_transform(corpus).todense())\n",
    "print(vectorizer.vocabulary_)\n",
    "\n",
    "#\n",
    "#By comparing the TF-IDF weights to the raw term frequencies, we can see that words\n",
    "#that are common to many of the documents in the corpus, such as sandwich, have\n",
    "#been penalized.\n",
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TFIDF Feat Vectors\n",
      "[[ 0.55416811  0.55416811  0.27708406  0.27708406  0.39429518  0.27708406\n",
      "   0.          0.          0.          0.        ]\n",
      " [ 0.          0.          0.          0.          0.33517574  0.\n",
      "   0.47107781  0.47107781  0.47107781  0.47107781]]\n",
      "{'transfigured sandwich': 7, 'wizard transfigured': 9, 'sandwich ate': 5, 'ate sandwich': 1, 'sandwich': 4, 'transfigured': 6, 'ate': 0, 'dog ate': 3, 'dog': 2, 'wizard': 8}\n"
     ]
    }
   ],
   "source": [
    "#N-grams\n",
    "\n",
    "print(\"TFIDF Feat Vectors\")\n",
    "\n",
    "\n",
    "vectorizer = TfidfVectorizer(stop_words='english',ngram_range=(1,2))\n",
    "print (vectorizer.fit_transform(corpus).todense())\n",
    "print(vectorizer.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SPAM CLASSIFICATION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5574\n"
     ]
    }
   ],
   "source": [
    "#read the spam/non-span data into a list\n",
    "messages = [line.rstrip() for line in open('./SMSSpamCollection')]\n",
    "print (len(messages))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\n",
      "1 ham\tOk lar... Joking wif u oni...\n",
      "2 spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\n",
      "3 ham\tU dun say so early hor... U c already then say...\n",
      "4 ham\tNah I don't think he goes to usf, he lives around here though\n",
      "5 spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\n",
      "6 ham\tEven my brother is not like to speak with me. They treat me like aids patent.\n",
      "7 ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\n",
      "8 spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\n",
      "9 spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\n"
     ]
    }
   ],
   "source": [
    "#Explore some lines  , Python ENUMERATE \n",
    "for message_no, message in enumerate(messages[:10]):\n",
    "    print (message_no, message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: './presentations/ml.png'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-65-0401c5af185f>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mIPython\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdisplay\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mImage\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mImage\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m'./presentations/ml.png'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/IPython/core/display.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, data, url, filename, format, embed, width, height, retina, unconfined, metadata)\u001b[0m\n\u001b[0;32m    737\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0munconfined\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0munconfined\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    738\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmetadata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmetadata\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 739\u001b[1;33m         \u001b[0msuper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mImage\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfilename\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mfilename\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    740\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    741\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mretina\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/IPython/core/display.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, data, url, filename)\u001b[0m\n\u001b[0;32m    384\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfilename\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mNone\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mfilename\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m \u001b[1;32melse\u001b[0m \u001b[0municode_type\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    385\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 386\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreload\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    387\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_check_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    388\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/IPython/core/display.py\u001b[0m in \u001b[0;36mreload\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    759\u001b[0m         \u001b[1;34m\"\"\"Reload the raw data from file or URL.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    760\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0membed\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 761\u001b[1;33m             \u001b[0msuper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mImage\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreload\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    762\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mretina\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    763\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_retina_shape\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/IPython/core/display.py\u001b[0m in \u001b[0;36mreload\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    402\u001b[0m         \u001b[1;34m\"\"\"Reload the raw data from file or URL.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    403\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfilename\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 404\u001b[1;33m             \u001b[1;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfilename\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_read_flags\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    405\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    406\u001b[0m         \u001b[1;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0murl\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: './presentations/ml.png'"
     ]
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='./presentations/ml.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Load data, look around EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     label                                            message\n",
      "0      ham  Go until jurong point, crazy.. Available only ...\n",
      "1      ham                      Ok lar... Joking wif u oni...\n",
      "2     spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
      "3      ham  U dun say so early hor... U c already then say...\n",
      "4      ham  Nah I don't think he goes to usf, he lives aro...\n",
      "5     spam  FreeMsg Hey there darling it's been 3 week's n...\n",
      "6      ham  Even my brother is not like to speak with me. ...\n",
      "7      ham  As per your request 'Melle Melle (Oru Minnamin...\n",
      "8     spam  WINNER!! As a valued network customer you have...\n",
      "9     spam  Had your mobile 11 months or more? U R entitle...\n",
      "...    ...                                                ...\n",
      "5564   ham  Ok lor... Sony ericsson salesman... I ask shuh...\n",
      "5565   ham                                Ard 6 like dat lor.\n",
      "5566   ham  Why don't you wait 'til at least wednesday to ...\n",
      "5567   ham                                       Huh y lei...\n",
      "5568  spam  REMINDER FROM O2: To get 2.50 pounds free call...\n",
      "5569  spam  This is the 2nd time we have tried 2 contact u...\n",
      "5570   ham               Will ü b going to esplanade fr home?\n",
      "5571   ham  Pity, * was in mood for that. So...any other s...\n",
      "5572   ham  The guy did some bitching but I acted like i'd...\n",
      "5573   ham                         Rofl. Its true to its name\n",
      "\n",
      "[5574 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "import csv\n",
    "messages = pd.read_csv('./SMSSpamCollection', sep='\\t', quoting=csv.QUOTE_NONE,names=[\"label\", \"message\"])\n",
    "print (messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "                                                        message\n",
       "label                                                          \n",
       "ham   count                                                4827\n",
       "      unique                                               4518\n",
       "      top                                Sorry, I'll call later\n",
       "      freq                                                   30\n",
       "spam  count                                                 747\n",
       "      unique                                                653\n",
       "      top     Please call our customer service representativ...\n",
       "      freq                                                    4"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Print spam ham stats\n",
    "messages.groupby('label').describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  label                                            message  length\n",
      "0   ham  Go until jurong point, crazy.. Available only ...     111\n",
      "1   ham                      Ok lar... Joking wif u oni...      29\n",
      "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...     155\n",
      "3   ham  U dun say so early hor... U c already then say...      49\n",
      "4   ham  Nah I don't think he goes to usf, he lives aro...      61\n"
     ]
    }
   ],
   "source": [
    "#add a new column to the dataframe which stroes the lenght of the messages (number of characters)\n",
    "#can length be a useful feature ???\n",
    "# u can create new features and add them as new columns to the dataframe \n",
    "\n",
    "messages['length'] = messages['message'].map(lambda text: len(text))\n",
    "print (messages.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7efe0e972048>"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEECAYAAAD+qJluAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFmBJREFUeJzt3X+QXeV93/H36sdaYK5JGYxAIANNxQRREjAJInUa77jU\no9oOkE71BbtgarCTRi4ojKctoh2jdDrEpLUDjAuuB2MEHSt8a9eq3AKG2N0xM7WRcc3giXAdeVCD\nFv3ANrJEAlg/tn+cs75Xa0l798dz79W979fMjs59znnOee53tPvZc86z5w6Nj48jSVJJ87o9AElS\n/zNsJEnFGTaSpOIMG0lScYaNJKk4w0aSVNyCThwkIpYCDwGnAePAZzPznohYB3wYeLne9LbMfKzu\nsxa4ATgI3JyZT9TtlwAPAouARzNzTSfegyRp5jp1ZrMfuCUzLwAuAz4aEedTBc+nMvPi+msiaJYD\nVwPLgZXAvRExVO/rPuDGzFwGLIuIlVMdPCJG5vwdHaesRZO1aLIWTdaiaS5r0ZGwycydmflsvfwq\n8DxwZr166AhdrgQ2ZOb+zNwGbAVWRMQZQCMzN9fbPQRc1cYQRmYx/H4z0u0B9JCRbg+gh4x0ewA9\nZKTbA+ghI3O1o45cRmsVEecAFwPfAt4B3BQRHwSeAT6WmXuAJfX6Cdupwml/vTxhjGZoSZJ6VEcn\nCETEScAXgTX1Gc59wLnARcAO4JOdHI8kqTM6dmYTEQuBLwH/JTM3AmTm7pb19wNfqV+OAUtbup9F\ndUYzVi+3to8d4VgjtJz+ZebtwO1z8DaOe5kJ1gKwFq2sRZO1aMpMIqK1aTQzR2eyr6FOPIizvrm/\nHvhxZt7S0n5GZu6ol28BfiMzP1BPEPgCcCnVZbI/B/5OZo5HxNPAzcBm4H8C92Tm41MMYfyll16a\n8/d1PGo0Guzbt6/bw+gJ1qLJWjRZi6YlS5bAke+rT1unzmzeAVwLPBcR363bbgPeHxEXUc1KewH4\nfYDM3BIRCWwBDgCrM3MiFVdTTX0+gWrq81RBI0nqso6c2fQAz2xq/tbWZC2arEWTtWiayzMbnyAg\nSSrOsJEkFWfYSJKKM2wkScUZNpKk4gwbSVJxho0kqTjDRpJUnGEjSSrOsJEkFWfYSJKKM2wkScUZ\nNpKk4gwbSVJxho0kqTjDRpJUnGEjSSrOsJEkFWfYSJKKM2wkScUZNpKk4gwbSVJxho0kqbgF3R5A\npwzNou/4nI1CkgbTwIQN9/9HDu15Zdrd5i1ewrxrPsKhhcMFBiVJg2Fgwubg1u8z/uPd0+/4N68y\nf+6HI0kDxXs2kqTiDBtJUnGGjSSpOMNGklScYSNJKs6wkSQVZ9hIkoozbCRJxRk2kqTiDBtJUnEd\neVxNRCwFHgJOo3qu5Wcz856IOAV4BDgb2AZEZu6p+6wFbgAOAjdn5hN1+yXAg8Ai4NHMXNOJ9yBJ\nmrlOndnsB27JzAuAy4CPRsT5wK3Ak5l5HvC1+jURsRy4GlgOrATujYiJBzffB9yYmcuAZRGxskPv\nQZI0Qx0Jm8zcmZnP1suvAs8DZwJXAOvrzdYDV9XLVwIbMnN/Zm4DtgIrIuIMoJGZm+vtHmrpI0nq\nUR2/ZxMR5wAXA08DizNzV71qF7C4Xl4CbG/ptp0qnCa3j9XtkqQe1tGPGIiIk4AvAWsyc19E/Hxd\nZo5HxJx8TllEjAAjLfue1f4WLVrE/BPfPLtB9Yjh4WEajUa3h9ETrEWTtWiyFoeLiHUtL0czc3Qm\n++lY2ETEQqqgeTgzN9bNuyLi9MzcWV8im/jAmTFgaUv3s6jOaMbq5db2scnHqosx2tJ0+2zG/vrr\nr3Po4KHZ7KJnNBoN9u3b1+1h9ARr0WQtmqxFU6PRIDPXzcW+OnIZrb65/zlgS2be1bJqE3B9vXw9\nsLGl/ZqIGI6Ic4FlwObM3AnsjYgV9T6va+kjSepRnTqzeQdwLfBcRHy3blsLfALIiLiReuozQGZu\niYgEtgAHgNWZOXGJbTXV1OcTqKY+P96h9yBJmqGh8fE5uU3S68a3f/A9M/pY6HlLz2X+2v/AoYXD\nBYbVeV4iaLIWTdaiyVo0LVmyBGBoqu3a4RMEJEnFGTaSpOIMG0lScYaNJKk4w0aSVJxhI0kqzrCR\nJBVn2EiSijNsJEnFGTaSpOIMG0lScYaNJKk4w0aSVJxhI0kqzrCRJBVn2EiSijNsJEnFGTaSpOIM\nG0lScYaNJKk4w0aSVJxhI0kqzrCRJBVn2EiSijNsJEnFGTaSpOIMG0lScYaNJKk4w0aSVJxhI0kq\nzrCRJBVn2EiSijNsJEnFGTaSpOIMG0lScQs6daCIeAB4L7A7My+s29YBHwZerje7LTMfq9etBW4A\nDgI3Z+YTdfslwIPAIuDRzFzTqfcgSZqZTp7ZfB5YOaltHPhUZl5cf00EzXLgamB53efeiBiq+9wH\n3JiZy4BlETF5n5KkHtOxsMnMp4BXjrBq6AhtVwIbMnN/Zm4DtgIrIuIMoJGZm+vtHgKuKjFeSdLc\n6dhltGO4KSI+CDwDfCwz9wBLgG+1bLMdOBPYXy9PGKvbJUk9rNsTBO4DzgUuAnYAn+zucCRJJXT1\nzCYzd08sR8T9wFfql2PA0pZNz6I6oxmrl1vbxybvNyJGgJGW48xqnAtfe5XxsR/PqO+8t57OwtOX\nzOr4c2l4eJhGo9HtYfQEa9FkLZqsxeHqiVwTRjNzdCb76WrYRMQZmbmjfvm7wPfq5U3AFyLiU1SX\nyZYBmzNzPCL2RsQKYDNwHXDP5P3WxRhtabp9NuM8uHsnb9x564z6Dt96J6+/uXf+4zYaDfbt29ft\nYfQEa9FkLZqsRVOj0SAz183Fvjo59XkD8E7g1Ih4kSoARiLiIqpZaS8Avw+QmVsiIoEtwAFgdWaO\n17taTTX1+QSqqc+Pd+o9SJJmZlphExHzgMUtZyNty8z3H6H5gWNsfwdwxxHavwNcON3jS5K6p62w\niYi/Bfwn4J9QnWmcGBFXAJdm5r8tOD5JUh9odzbaZ4C9wNnAG3XbN4FrSgxKktRf2g2bfwDc1Hr5\nLDNfBk4rMipJUl9pN2z2AG9tbYiItwEvzfmIJEl9p92wuR/4YkS8C5gXEb8JrAf+c7GRSZL6Rruz\n0e4EXqOaJLCQ6qGanwHuLjQuSVIfaSts6r9xuRvDRZI0A23/nU1EvJtq9tlpmfm+iPh14C2Z+fVi\no5Mk9YW27tlExE1UD838S+C36+bXgX9faFySpD7S7gSBW4DLM/OPqT45E+B54FeKjEqS1FfaDZuT\ngBcntQ3T/ANPSZKOqt2weQqY/Njjm4D/NbfDkST1o3YnCNwEfCUiPgKcFBE/APYB7ys2MklS35gy\nbOonPf8K8FvAr1I9H+2vqD5f5lDZ4UmS+sGUYZOZhyJiU2aeBDxdf0mS1LZ279l8o35EjSRJ09bu\nPZv/BzwWERuB7VSfrAkwnpkfLzIySVLfaDdsTgA21stn1v8O0QwdSZKOqt1no/2zwuOQJPWxdj8W\n+m8fZdUbwA5npUmSjqXdy2hbj7HuUERsAv4gM3fNwZgkSX2m3dlovwd8AVhGdf/mPOBhYDVwIVVo\n3VtigJKk41+7ZzbrgGWZ+Vr9emtE/AHwg8z8TERcz7HPfiRJA6zdM5t5wDmT2t4GzK+X/6ZlWZKk\nw7R7ZnMX8PWIeIDq6c9LgQ/R/OTO9wDfnPvhSZL6QbtTn/8kIp4DAng7sAO4ITMfr9d/GfhysVFK\nko5rbX8sdB0sjxcciySpT7X7dzaLgI8D1wCnZuZbIuLdwHmZ+emSA5QkHf/anSDwp8DfBf4pMPEH\nnH9BNfVZkqRjajdsfhf4QGZ+k/p5aJk5RvM5aZIkHVW7YfMGky65RcRbgR/N+YgkSX2n3bD5r8CD\nE89Ii4gzgE8Df1ZqYJKk/tFu2Pwb4AXgOeBkqqcF7AD+XaFxSZL6SLtTn38Z+L/AHXWfjZn5XLFR\nSZL6yjHDJiKGgM8B11N9QudLVJMCbo+Ih4EPZaYfoCZJOqapzmx+DxgBLsvMb080RsRvABuAfw7c\nV2x0kqS+MFXYfBBY0xo0AJn57Yj4Q2AtbYZN/Vy19wK7M/PCuu0U4BHgbGAbEJm5p163FrgBOAjc\nnJlP1O2XAA8Ci4BHM3NNO8eXJHXPVBMElgOjR1n3DeCCaRzr88DKSW23Ak9m5nnA1+rXRMRy4Or6\n+CuBe+tLelCF242ZuQxYFhGT9ylJ6jFThc38zNx3pBWZubeN/q3bPwW8Mqn5CmB9vbweuKpevhLY\nkJn7M3Mb1ey3FfWU60Zmbq63e6iljySpR011GW1BRLzrKOuG2ug/lcUtHyW9C1hcLy8BvtWy3Xaq\niQn76+UJPsVAko4DU4XFbqrZaEez6xjrpiUzxyPCmW2S1IeOGTaZeU7h4++KiNMzc2d9iWx33T5G\n9QFtE86iOqMZq5db28cm7zQiRqhm0QGQmbMa5Lx5bV8t/AXz5y/gxEZjVsefS8PDwzR6aDzdZC2a\nrEWTtThcRKxreTmamaMz2c9sL4PN1iaqv+G5s/53Y0v7FyLiU1SXyZYBm+uzn70RsQLYDFwH3DN5\np3UxRluabp/NIA8dOjT1Rkdx8OAB9u074m2vrmg0Gj01nm6yFk3WoslaNDUaDTJz3Vzsq2NhExEb\ngHcCp0bEi1Sfj/MJICPiRuqpzwCZuSUiEtgCHABWt/zx6Gqqqc8nUE199gPdJKnHdSxsMvP9R1l1\n+VG2v4Pq8TiT278DXDiHQ5MkFdbty2g6hgU//QnjP5r5HIyhUxdz4ORT5nBEkjQzhk0PG//RLn72\niX894/7Dt94Jho2kHjDzaVaSJLXJsJEkFWfYSJKKM2wkScUZNpKk4gwbSVJxho0kqTjDRpJUnGEj\nSSrOsJEkFWfYSJKK89lohc0bfhP88PkZ9R06cGCORyNJ3WHYFDb+01f42d1/NKO+b1ozq898k6Se\n4WU0SVJxho0kqTjDRpJUnGEjSSrOsJEkFWfYSJKKM2wkScUZNpKk4gwbSVJxho0kqTjDRpJUnGEj\nSSrOsJEkFWfYSJKKM2wkScUZNpKk4gwbSVJxho0kqTjDRpJUnGEjSSrOsJEkFbeg2wMAiIhtwF7g\nILA/My+NiFOAR4CzgW1AZOaeevu1wA319jdn5hPdGLckqT29cmYzDoxk5sWZeWnddivwZGaeB3yt\nfk1ELAeuBpYDK4F7I6JX3ock6Qh66Yf00KTXVwDr6+X1wFX18pXAhszcn5nbgK3ApUiSelavhM04\n8OcR8UxEfKRuW5yZu+rlXcDienkJsL2l73bgzM4MU5I0Ez1xzwZ4R2buiIi3Ak9GxPdbV2bmeESM\nH6P/YesiYgQYaek/q8HNmzfzTB4amnzC1pm+APPnL+DERuOwtuHhYRqT2gaVtWiyFk3W4nARsa7l\n5Whmjs5kPz0RNpm5o/735Yj4MtVlsV0RcXpm7oyIM4Dd9eZjwNKW7mfVba37GwVGW5pun834Dh06\nNOO+4+PHyshyfQEOHjzAvn37DmtrNBq/0DaorEWTtWiyFk2NRoPMXDcX++r6ZbSIODEiGvXym4F3\nA98DNgHX15tdD2yslzcB10TEcEScCywDNnd21JKk6eh62FDdi3kqIp4Fngb+Rz2V+RPAP4yIHwDv\nql+TmVuABLYAjwGrM3N2pwCSpKK6fhktM18ALjpC+0+Ay4/S5w7gjsJDkyTNkV44s5Ek9TnDRpJU\nnGEjSSrOsJEkFWfYSJKKM2wkScUZNpKk4gwbSVJxho0kqTjDRpJUnGEjSSrOsJEkFWfYSJKKM2wk\nScUZNpKk4gwbSVJxho0kqTjDRpJUnGEjSSrOsJEkFWfYSJKKM2wkScUZNpKk4gwbSVJxho0kqTjD\nRpJUnGEjSSrOsJEkFWfYSJKKM2wkScUZNpKk4gwbSVJxho0kqTjDRpJUnGEjSSpuQbcHMBMRsRK4\nC5gP3J+Zd3Z5SD1p3vCb4IfPH9b2xvwFzD94YMq+Q6cu5sDJp5QamqQBc9yFTUTMBz4NXA6MAd+O\niE2Z+fyxew6e8Z++ws/u/qMZ9R2+9U4wbCTNkeMubIBLga2ZuQ0gIv4MuBIwbObQkc6K2uVZkaTJ\njsewORN4seX1dmBFl8bSt2ZzVrTo43cx/0e7ZtTXoJL60/EYNuMz6bTwqmsZf+2vp91vqHHyTA43\n0LoVVPPe8ksc2run7e0n37+abv9WhqR0bMdj2IwBS1teL6U6u/m5iBgBRiZeZyaL//EHZnfU335m\n5n3f/Tvd6dvNY8923Oq4RqPR7SH0DGvRFBHrWl6OZubojHY0Pj5+XH2tWrVqwapVq364atWqc1at\nWjW8atWqZ1etWnX+FH3WdXvcvfJlLayFtbAW3ajFcfd3Npl5APgXwFeBLcAjzkSTpN52PF5GIzMf\nAx7r9jgkSe057s5sZmi02wPoIaPdHkAPGe32AHrIaLcH0ENGuz2AHjI6VzsaGh+f0eQuSZLaNihn\nNpKkLjJsJEnFHZcTBKZjkB7aGRFLgYeA06j++PWzmXlPRJwCPAKcDWwDIjP31H3WAjcAB4GbM/OJ\nboy9lPpZes8A2zPzdwa1FhHxS8D9wAVU/zc+BPwlg1mLtcC1wCHge1S1eDMDUIuIeAB4L7A7My+s\n26b9PRERlwAPAouARzNzzVTH7uszm5aHdq4ElgPvj4jzuzuqovYDt2TmBcBlwEfr93sr8GRmngd8\nrX5NRCwHrqaqzUrg3ojot/8Ta6imyE/cnBzUWtxN9UPhfOBXge8zgLWIiHOAjwBvr3/YzgeuYXBq\n8Xmq99FqOu99qO5zH3BjZi4DltW/1B/T8Vy0dvz8oZ2ZuR+YeGhnX8rMnZn5bL38KtXDSc8ErgDW\n15utB66ql68ENmTm/vrBplupatYXIuIs4D1Uv9FPfJMMXC0i4mTg72fmA1D9rVpm/pQBrAWwl+qX\nshMjYgFwIvASA1KLzHwKeGVS83Te+4qIOANoZObmeruHWvocVb9fRhvYh3bWv8FdDDwNLM7MiQeO\n7QIW18tLgG+1dNtOVbN+8afAvwTe0tI2iLU4F3g5Ij4P/BrwHeAPGcBaZOZPIuKTwF8BrwFfzcwn\nI2LgatFiuu99P4c/ImyMNmrS72c2AzmvOyJOAr4ErMnMfa3rMnOcY9elL2oWEe+jui79XZpnNYcZ\nlFpQ/VL5duDezHw78NfUl0omDEotIuKXqYL2HKofpidFxLWt2wxKLY6kjfc+Y/0eNlM+tLPfRMRC\nqqB5ODM31s27IuL0ev0ZwO66fXJ9zqrb+sHfA66IiBeADcC7IuJhBrMW26kmSHy7fv1FqvDZOYC1\n+HXgf2fmj+tHX/034DcZzFpMmM73xPa6/axJ7VPWpN/D5hmqm1fnRMQw1c2uTV0eUzH1zbvPAVsy\n866WVZuA6+vl64GNLe3XRMRwRJwLLAM20wcy87bMXJqZ51LdAP56Zl7HYNZiJ/BiRJxXN10O/AXw\nFQasFlQTIy6LiBPq75fLqSaQDGItJkzre6L+/7Q3IlbUNbyupc9R9f0TBCLiH9Gc+vy5zPzjLg+p\nmIj4LeAbwHM0T4XXUn1zJPA2fnFq421UUxsPUF12+2qHh11cRLwT+FhmXlFP8xy4WkTEr1FNlBgG\nfkg13Xc+g1mLf0X1Q/UQ8H+ADwMNBqAWEbEBeCdwKtX9mY8D/51pvveWqc8nUM1yvHmqY/d92EiS\nuq/fL6NJknqAYSNJKs6wkSQVZ9hIkoozbCRJxRk2kqTiDBtJUnGGjSSpuP8Pj1DllFioDe8AAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7efe3c3906a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messages.length.plot(bins=20, kind='hist')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    5574.000000\n",
       "mean       80.478292\n",
       "std        59.848302\n",
       "min         2.000000\n",
       "25%        36.000000\n",
       "50%        62.000000\n",
       "75%       122.000000\n",
       "max       910.000000\n",
       "Name: length, dtype: float64"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#describe function prints a 5 number of summary\n",
    "messages.length.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"For me the love should start with attraction.i should feel that I need her every time around me.she should be the first thing which comes in my thoughts.I would start the day and end it with her.she should be there every time I dream.love will be then when my every breath has her name.my life should happen around her.my life will be named to her.I would cry for her.will give all my happiness and take all her sorrows.I will be ready to fight with anyone for her.I will be in love when I will be doing the craziest things for her.love will be when I don't have to proove anyone that my girl is the most beautiful lady on the whole planet.I will always be singing praises for her.love will be when I start up making chicken curry and end up makiing sambar.life will be the most beautiful then.will get every morning and thank god for the day because she is with me.I would like to say a lot..will tell later..\"]\n"
     ]
    }
   ],
   "source": [
    "#Pandas filtering \n",
    "\n",
    "print (list(messages.message[messages.length > 900]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x7efe23c1a358>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x7efe38fd6da0>], dtype=object)"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X20HHWd5/H3L325QIYeNICBS7LEg2GXuOgITsCn4eqi\nE9EBZs7eL+MsioTRXeMYnPGBRHdM8Bxj4igCOxvOuAgTdIh8d8Zl4wNIhj2XVVGjjDysCSOJRrk3\n5AYMCVcUkntT+0dV31Q696H6ubr78zqnz+36VXV/q5P61rd/Vb+uClEUISIiMpNZrV4BERFpDyoY\nIiKSiQqGiIhkooIhIiKZqGCIiEgmKhgiIpKJCkaOhBAGQwj/o9XrISIyGRWMfImSh4hI7qhgiIhI\nJioY+RNCCH8dQngyhPCrEMJtIYTZyYxzQwh3hxBGQgijIYQtIYQ/LHvxzhDCJ0MIN4cQ9oUQdocQ\n3hdCOC6E8N9DCHtDCEMhhPe35uOJTC2E8PoQwndDCM8mj4dCCG8JISwIIRwKIfynEMJ9IYTfhBB+\nFkL4s7LXfyqEsDWE8FwI4ZdJHvxuav67QwgHQwj9IYRHk/f5PyGEU0MIb0zi/TqEsDmE0Nf8f4F8\nU8HIlwD8R+BFwIXAnwGXAR9N5heBO5J5rwK+BWwKISwse58PAP8KnAv8N+BvgbuAx4FXJ9M3hRDO\nbuSHEalECKEH2AR8j3j7fhWwCvgNhw/Vfga4BXgl8A/Al0II56Xe5jngPcDZwLuBfuCmslCzgE8A\nS4HXAfOA/wmsBt4LvD5pu75+n65DRFGkR04ewCDw47K2m4EHpnnNQ8DHUtM7ga+mpgOwH/jfZW17\ngWWt/sx66FF6AC8GDgEXTjJvQTLvurL27wJfmuY9/xh4PjX97uR9XpFq+3DS9qpU2weBp1r9b5K3\nh3oY+RIBD5e17QLmAoQQTgkhrA8hbAshPBNCGAVeDvybqd4jirf+p4BHytr2AC9pyKcQqUIURc8Q\n9x6+FUL4Zgjh2hDCWWWLfa9s+rvAotJECOFPQgj/N4QwnOTHl4FjQginpkMBj6amR5K/j5S1nRRC\nCDV8pI6jgpE/B8qmI+IeAcDfE3ehP0Lcbf494h5Gb9lrDk7yHpO16f9fciWKovcC5wGbiQ+9/r8Q\nwnunecnEDj2EcD7gxD31y4gPaf2XZJl0jhxKvjRNhE1ij5e3pd9ftMNoN28A1kdR9PUoin4C7AbO\nbPE6idRVFEU/iaLo81EUXQx8kfi8QmkH/pqyxV8LbE2evx54OoqiT0RR9MMoirYD85uy0l2ip9Ur\nIEcITP6NptT2r8AVIYTvEv/ffZK46IdJlp3s9TO1ibRMCOFM4uKwCRgC+oA/AH6UWmxpCOEx4EHg\nCuACYHky7zHglBDCUuJexuuB9zVl5buECka+TPbDvXTbVcDfAVuIexefAY4ve81kP/zL2ibSSs8B\nLwO+ApwC/Ar4OvFJ6ZOSZVYQF5ULgCeBd0VR9CBAFEXfCCF8ClgDnEBcND5CPJoqTTlSpXDkobwj\nmdmtwNuAPe5+Tqr9A8AyYBz4hrtfm7SvJB6qNg4sd/d7k/bziI+/Hwd8092vacinEWmBeuWJTC2E\nsAD4GfD6KIoeaO3adK+ZzmHcBixJN5jZG4FLgFe4+78HPpu0LwIuJx6xsARYb2alwx43A1e7+0Jg\noZkd8Z5TMbP+jJ+jrhRXcStUa57MeC6xg/6tFLeN4067obr7t4FnyprfB3za3Q8myzyVtF8KbHT3\ng+6+E9gOnG9mpwFFd9+SLHc78QiGLPozLldviqu4mdUhTxZnCNNfn7WtWJ7iNuMQ0WRxm6Et4lZz\nDmMh8AdmtgZ4Hviwu/+I+ATV91PLDQGnEw/nHEq1DyftIp2s0jyRaURRtBMotHo9ul01w2p7gBe7\n+wXEJ5S8vqsk0hEqyROdXJW2UE0PYwj4KoC7/9DMDpnZycQ9h/SY53nJssPJ83T78GRvnBxP6y9N\nu/sq4mvJNJW7o7idHdfM0k2D7j5Y5zCV5MlR+aBcUNxmxa0kF6opGHcBbwLuN7OzgF53f9rMNgF3\nmNn1xF3shcAWd4/M7FkzO594OOg7OfpiYKWVHyQeCleyateuXVWsYm2KxSKjo6OK26Fx+/r6cPfV\nDQ5TUZ6Uv1i5oLjNUGkuTFswzGwj8c/zTzKzJ4iv8HgrcKuZPUp8GYt3Abj7VjNz4l9djgHL3L3U\n1V5GPKz2eOJhtfdU8qFE8qyOeSKSa9P+DiMHIn2rUtx66+vrg/b7pbtyQXHrrtJc0LWkREQkExUM\nERHJRAVDREQyUcEQEZFMVDBERCQTFQwREclEBUNERDJRwRARkUxUMEREJBMVDBERyUQFQ0REMlHB\nEBGRTFQwREQkExUMERHJRAVDREQyUcEQEZFMqrlFa0v07N9L9PQIAOHkuYydOKfFayQi0l3apmBE\nT49wYO21APSuWAcqGCIiTTXTPb1vBd4G7HH3c8rmfQj4G+Bkd9+btK0ElgLjwHJ3vzdpP4/4nt7H\nEd/T+5o6fw6RlqlXnkj+lI5s6KhGbKZzGLcBS8obzWw+8GbgF6m2RcDlwKLkNevNrHSv2JuBq919\nIbDQzI56T5E2Vmue6FxiTpWObJQOh3e7aTdUd/828Mwks64HPlrWdimw0d0PuvtOYDtwvpmdBhTd\nfUuy3O3AZTWttUiO1CFPFjd2DUXqo+JvNmZ2KTDk7o+UzeoDhlLTQ8Dpk7QPJ+0iHauKPBHJvYpO\nepvZbOBjxN3skjDF4iJdqYo8iRq7RiL1UekoqTOBBcDDZgYwD3jQzM4n7jnMTy07j/jb03DyPN0+\nPNmbm1k/0F+adneKxSIALxQOr2qh0MPspL0Rent7J+I2k+I2j5mtTk0OuvtgHd++0jw5Kh+my4Vm\n6rZtozxuab/TqfscqCwXQhRN/+XGzBYAXysf/ZHM+zlwnrvvTU7m3UF8PPZ04J+Bl7l7ZGY/AJYD\nW4BvADe5+z0ZPku0a9cuAAo7th0xrHb8zLMzvLw6xWKR0dHRhr2/4rY2bl9fH9S5Z1yPPJkhxEQu\nNFO3bRvlcUv7nU7d51SaC9OewzCzjcADwFlm9oSZXVW2yMRG7u5bAQe2AncDy1JJsAy4BXgc2J6x\nWIi0hTrmiUiuzdjDaDH1MBS37hrRw2gC9TBaEFc9jCNp/LeIiGSigiEiIpmoYIiISCYqGCIikokK\nhoiIZKKCISIimahgiIhIJioYIiKSiQqGiIhkooIhIiKZqGCIiEgmKhgiIpKJCoaIiGSigiEiIpmo\nYIiISCYqGCIikokKhoiIZKKCISIimfRMN9PMbgXeBuwp3dzezP4GeDtwANgBXOXu+5N5K4GlwDiw\n3N3vTdrPA/4eOA74prtf05BPI9IC9coTkbybqYdxG7CkrO1e4OXu/krgp8BKADNbBFwOLEpes97M\nSveKvRm42t0XAgvNrPw9RdpZrXminr60hWk3VHf/NvBMWdtmdz+UTP4AmJc8vxTY6O4H3X0nsB04\n38xOA4ruviVZ7nbgsjqtv0jL1SFPFjdrXUVqUes3m6XAN5PnfcBQat4QcPok7cNJu0i3yJInIrlX\ndcEws48DB9z9jjquj0hHyZgnUbPWR6QW0570noqZvRu4GPgPqeZhYH5qeh7xt6dhDnfHS+3DU7xv\nP9BfmnZ3isUiAC8UDq9qodDD7KS9EXp7eyfiNpPiNo+ZrU5NDrr7YANivJtseXJUPkyXC83UbdtG\nedzSfqdT9zlQWS5UXDCSE9YfAS509+dTszYBd5jZ9cRd7IXAFnePzOxZMzsf2AK8E7hpsvdOVjS9\nsqtGR0cBKIyPTTSOj49Ram+EYrHY0PdX3NbHdffVjYxRaZ6Uv366XGimbtw20nFL+51O3udUkgsz\nDavdCFwInGxmTwCriEd79AKbzQzge+6+zN23mpkDW4ExYJm7l7ray4iH1R5PPKz2noo+lUiO1TFP\nRHItRFGut9Vo165dABR2bOPA2msB6F2xjvEzz25Y0Lx8u1Hcxujr6wMIMy2XMxO50Ezdtm0c1cNI\n9judus+pNBc0/ltERDJRwRARkUxUMEREJBMVDBERyUQFQ0REMlHBEBGRTFQwREQkExUMERHJRAVD\nREQyUcEQEZFMVDBERCQTFQwREclEBUNERDJRwRARkUyquuOeiEin6tm/l+jpEQDC2NgMS3cXFQwR\nkZTo6ZGJe+8ce82qFq9NvuiQlIiIZKKCISIimcx0T+9bgbcBe9z9nKRtDnAncAawEzB335fMWwks\nBcaB5e5+b9J+HvE9vY8jvqf3NY34MCKtUK88Ecm7mXoYtwFLytpWAJvd/SzgvmQaM1sEXA4sSl6z\n3sxK94q9Gbja3RcCC82s/D1F2lmteaKevrSFaTdUd/828ExZ8yXAhuT5BuCy5PmlwEZ3P+juO4Ht\nwPlmdhpQdPctyXK3p14j0vbqkCeLm7GeIrWq5pvNXHcfSZ6PAHOT533AUGq5IeD0SdqHk3aRTlZp\nnojkXk1dYXePgKhO6yLSkTLkiXJI2kI1v8MYMbNT3X13crhpT9I+DMxPLTeP+NvTcPI83T482Rub\nWT/QX5p2d4rFIgAvFA6vaqHQw+ykvRF6e3sn4jaT4jaPma1OTQ66+2CdQ1SSJ0flw3S50Ezdtm30\n9vYesa8JIT4N26n7HKgsF6opGJuAK4F1yd+7Uu13mNn1xF3shcAWd4/M7FkzOx/YArwTuGmyN05W\nNL2yq0ZHRwEojB/+xeX4+Bil9kYoFosNfX/FbX1cd1/d4DAV5Un5i6fLhWbqxm1jPLWviaK489fJ\n+5xKcmGmYbUbgQuBk83sCeATwFrAzexqkuGCAO6+1cwc2AqMAcuSrjjAMuJhtccTD6u9p4LPdJRZ\nvcfCjm2Ek+cyduKcWt5KpGZ1zBORXJu2YLj7O6aYddEUy68B1kzS/iBwTsVrN4Vo/zMcuPE6eles\nAxUMabF65YlI3mn8t4iIZKKCISIimahgiIhIJioYIiKSiQqGiIhkooIhIiKZqGCIiEgmKhgiIpKJ\nCoaIiGSigiEiIpmoYIiISCYqGCIikokKhoiIZKKCISIimahgiIhIJioYIiKSiQqGiIhkooIhIiKZ\nTHuL1umY2UrgCuAQ8ChwFfA7wJ3AGST3MXb3fanllwLjwHJ3v7emNRdpA5XmiUieVdXDMLMFwHuA\nc939HKAA/CmwAtjs7mcB9yXTmNki4HJgEbAEWG9m6t1IR6s0T0Tyrtqd9rPAQWC2mfUAs4FdwCXA\nhmSZDcBlyfNLgY3uftDddwLbgcXVrrRIm6g0T0RyraqC4e57gc8BvyROgH3uvhmY6+4jyWIjwNzk\neR8wlHqLIeD0qtZYpE1UkSciuVbtIakzgQ8CC4iLwQlmdkV6GXePgGiat5lunkjbq1OeiORGtSe9\nXw084O6/AjCzrwKvAXab2anuvtvMTgP2JMsPA/NTr5+XtB3BzPqB/tK0u1MsFgF4oXB4VUMIABQK\nPcxO5tdTb2/vRNxmUtzmMbPVqclBdx9sQJhK8yS9fv1MkQvN1G3bRm9vb1P3Nem47ZAL1RaMx4C/\nNrPjgeeBi4AtwHPAlcC65O9dyfKbgDvM7HriQ1ELk+WPkKxoemVXjY6OAlAYH5tojKL4C9n4+Bil\n+fVULBYb8r6Km5+47r66CaEqzZMJ0+VCM3XjtjHexH1NOm475EK15zAeBm4HfgQ8kjR/AVgLvNnM\nfgq8KZnG3bcCDmwF7gaWJV1xkY5VaZ6I5F3Vv8Nw988Anylr3kv8LWqy5dcAa6qNJ9KOKs0TkTzT\nbyFERCQTFQwREclEBUNERDJRwRARkUxUMEREJBMVDBERyUQFQ0REMlHBEBGRTKr+4Z6ISLeY1Xss\n7NgGQDh5LmMnzmnxGrWGCoaIyAyi/c9w4MbrAOhdsQ66tGDokJSIiGSigiEiIpmoYIiISCYqGCIi\nkokKhoiIZKKCISIimahgiIhIJioYIiKSSdU/3DOzFwG3AC8HIuAq4HHgTuAMYCdg7r4vWX4lsBQY\nB5a7+701rblIG6g0T0TyrJYexo3AN939bOAVwGPACmCzu58F3JdMY2aLgMuBRcASYL2ZqXcj3SBz\nnojkXVU7bTM7EXiDu98K4O5j7r4fuATYkCy2AbgseX4psNHdD7r7TmA7sLiWFRfJuyryRCTXqj0k\n9VLgKTO7DXgl8CDwQWCuu48ky4wAc5PnfcD3U68fAk6vMrZIu6g0T0RyrdrDQj3AucB6dz8XeI6y\nbrW7R8THbKcy3TyRTlCPPBHJjWp7GEPAkLv/MJn+R2AlsNvMTnX33WZ2GrAnmT8MzE+9fl7SdgQz\n6wf6S9PuTrFYBOCFwuFVDSEAUCj0MDuZX0+9vb0TcZtJcZvHzFanJgfdfbABYSrNk/T69TNFLjRT\np28bB3fv4tBTuwGYdcqp9J500qT7mtJfaMx+p11yoaqCkWzoT5jZWe7+U+Ai4CfJ40pgXfL3ruQl\nm4A7zOx64kNRC4Etk7zvIJBe2VWjo6MAFMbHJhqjKP5CNj4+Rml+PRWLxYa8r+LmJ667r250nCry\nJP3aQabIhWbq9G2jsHuIA2uvBeLLlh84tY/xSfY1pb/QmP1Ou+RCLffD+ADwD2bWC+wgHi5YANzM\nriYZLgjg7lvNzIGtwBiwLOmK10Q3NZE2kDlPRPKu6oLh7g8Dvz/JrIumWH4NsKbaeJPRTU0k7yrN\nE5E8028hREQkExUMERHJRAVDREQyUcEQEZFMVDBERCQTFQwREclEBUNERDJRwRARkUxUMEREJBMV\nDBERyUQFQ0SE+Np0Lzz6L4SxsZkX7lIqGCIixNem++2nPgQHD7R6VXJLBUNERDJRwRARkUxquR+G\niEhb6tm/l+jpEZ2vqJB6GCLSdaKnR+I77el8RUXUwxCRrlDqVQDqWVRJBUNEusJErwI49ppVLV6b\n9lRTwTCzAvAjYMjd/8jM5gB3AmeQ3KvY3fcly64ElgLjwHJ3v7eW2CLtopI8EcmzWs9hXANsBaJk\negWw2d3PAu5LpjGzRcDlwCJgCbDezHT+RLpFpjwRybuqd9pmNg+4GLgFCEnzJcCG5PkG4LLk+aXA\nRnc/6O47ge3A4mpjT2ZW77EUdmyjsGMbPfv31vOtRapWYZ6I5Fot3/I/D3wEOJRqm+vuI8nzEWBu\n8rwPGEotNwScXkPso0T7n+HA2ms5sPbaiRNbIjlQSZ6I5FpVBcPM3g7scfcfc/hb0xHcPeJwF3wy\n080TaXt1yhOR3Kj2pPdrgUvM7GLgOOB3zexLwIiZneruu83sNGBPsvwwMD/1+nlJ2xHMrB/oL027\nO8ViEYAXCodXNYRwxN/y54VCD7OT11Wjt7d3Im4zKW7zmNnq1OSguw82IEyleZJev36myIVm6qRt\nI+s+ZKa2Wvcvk2mXXKiqYLj7x4CPJcEuBD7s7u80s88AVwLrkr93JS/ZBNxhZtcTH4paCGyZ5H0H\ngfTKrhodHQWgMH543HQURUf8LX8+Pj5G6XXVKBaLNb1ecfMf191XNzpOFXmSfu0gU+RCM3XStpF1\nHzJTW637l8m0Sy7Ua6RS6V9zLfBmM/sp8KZkGnffCjjxSJG7gWVJV1ykm0ybJyJ5V/MP99z9fuD+\n5Ple4KIpllsDrKk1nkg7yponInmm30KIiEgmKhgiIpKJCoaIiGSigiEiIpmoYIiISCYqGCIikokK\nhoiIZKKCISIimahgiIhIJioYIiKSiQqGiIhkooIhIiKZ1HzxwXbUs3/vxF35wslzGTtxTovXSEQk\n/7qyYERPj3Bg7bUA9K5YByoYIiIz6sqCISLdo3REIYyNzbywTKsjz2HM6j2Wwo5tHPPUkxR2bKOw\nYxs9+/e2erVEpAUmjigcPNDqVWl7HVkwov3PcGDttUQjuziw9tr4eXLOQkREqtORBUNEROqvqnMY\nZjYfuB14CfF9ir/g7jeZ2RzgTuAMYCdg7r4vec1KYCkwDix393trX32R/KomT0TyrNoexkHgL939\n5cAFwPvN7GxgBbDZ3c8C7kumMbNFwOXAImAJsN7M1LuRTldRnojkXVU7bXff7e4PJc9/DWwDTgcu\nATYki20ALkueXwpsdPeD7r4T2A4srmG9K1Y6EV7YsU2jJaQpqsgTkVyr+Vu+mS0AXgX8AJjr7qWz\nyyPA3OR5HzCUetkQceI0TelEuEZLSCtkzBORXKupYJjZCcA/Ade4+2h6nrtHxMdtpzLdPJGOUWOe\nyAx69u/V8PkmqfqHe2Z2DHESfMnd70qaR8zsVHffbWanAXuS9mFgfurl85K28vfsB/pL0+5OsVgE\n4IXC4VUNIRzxd6rnM80HKBR6mJ3EKOnt7Z2I20yK2zxmtjo1Oejugw2KU0mepF/XzxS50EztsG28\nsPNxfptcueH4j3+O4rwzjpyf7Duy7iNmaptsn1GrdsmFakdJBeCLwFZ3vyE1axNwJbAu+XtXqv0O\nM7ue+FDUQmBL+fsmK5pe2VWjo/EXssL44fMOURQd8Xeq5zPNBxgfH6MUo6RYLB7V1gyK27y47r66\n0XGqyJMJ0+VCM7XDtpHeN0yWz6X5WfcRM7VNFqNW7ZIL1fYwXgdcATxiZj9O2lYCawE3s6tJhgsC\nuPtWM3NgKzAGLEu64iKdrKI8Ecm7qgqGu3+Hqc9/XDTFa9YAa6qJJ9KOqskTkTzL/cUH08cORUSk\ndXL/47lo/RpmPfjdVq+GiEjXy33BGP+X73Fo+BetXg0Rka6X+4IhIpJV6YoO+j1GY6hgiEjulX6c\nN1MhmLi1gW5n0BAqGCKSe6WbIKkQtFbuR0mJSPco3U71hUIPPS8+ibET51T1PrN6j4Ud2wB0sdE6\nUsEQkdyYuJ0q0LtiHVRZMKL9z3DgxusAOPaaVXVbv26ngiEibanUG1EPonl0DkNE2tJEb0S3K2ga\n9TBEpG3o3ERrqWCISF2VDhUBhJPnVn3iejJ5ODdRKlr1/mztQAVDRI5SzWiliUIxNsaBz34cOHzi\nupFFpNlKRauWk/LtSucwROQopfMDv/3UhzL/9mG6cwqlefotRXtTwRARkUx0SEpEWm6yIbI6wZ0/\nXd/D0MXKRFqjdH2owo5tsHv4qMNZpetCaehsfnR9wdDFykRaI31eQwWhPTT1kJSZLQFuAArALe6+\nrpnxRfKiVblQ79FKR7xflx02Sh8ym/W7L+LQs/vafgTYTJrWwzCzAvC3wBJgEfAOMzu7WfFF8qKV\nuTDZaKX0oaFKD81W00soHQZu9wKTPmQWjew66khF1kuyt5NmHpJaDGx3953ufhD4CnBpE+NPq7QR\nF3Zs4+DuXa1eHelsucqF9E4/PLvvqJ15Lef5JisOpR1tpx+G6sRLsjezYJwOPJGaHkraciH9beHQ\nvl9NFI9jnnpy2m9ftXw7k65V91wojO6nMLqfWS88X9OKTbYzr+U8X7cUh27RzHMYUVWvOqFI6D22\nzqsyvWjfXg7csBqILz9QuhTBcZ+4gcLTI0ccp0xfjrk0H7rnmKZUpbpcmEKIDnHIv8j4zsc55uq/\noufFJxE9PTKxDcLk5ysmLnGR8dDQpMfs2/ywUr1VMhR4YijxyXOhWKwoTvq1zdy/hCiq67Y7JTO7\nAFjt7kuS6ZXAofTJPjPrB/pL0+6uC9lLQ5jZdanJQXcfbGJs5YLkRkW5EEVRUx4DAwM9AwMDOwYG\nBhYMDAz0DgwMPDQwMHD2DK9Z3az1U1zFbWJ85YLitmXcpp3DcPcx4C+AbwFbgTvdfVuz4ovkhXJB\n2lVTf4fh7ncDdzczpkgeKRekHeX9l96Diqu4HRS3FoOKq7itjtu0k94iItLe8t7DEBGRnFDBEBGR\nTHJzP4zkWjqXcvgXr0PAJo0ekW6jXJC8ysU5DDO7FngH8TV1hpLm+cDlxEMOP93A2LOIr+1zOvEv\ncIeBLe7e0H8YxW1O3HbTqlwwsxcBK4DLgLnE/0d7gLuAte6+rxFxk9hdtU22c9y8FIzHgUXJhdjS\n7b3AVnd/WYPivgVYD2zncHLOAxYCy9z9W4rbvnGT2EuId4Klb+vDwF3ufk+jYtaihblwL3AfsAEY\ncffIzE4DrgTe5O5vaVDcrtom2z1uXg5JjRMn9M6y9r5kXqPcBFzk7kfENbOXEo+R/3eK275xzexG\n4oS4nbhQQJwky83sYndf3oi4NWpVLiwovyeHuz8JrDWzpQ2M21XbZLvHzUvB+CDwz2a2ncNX8ZxP\nnOx/0cC4BQ7vSNKGaey/jeI2J+7F7r6wvNHMvgI8DuSxYLQqF35hZh8FNrj7CICZnUrcw/hlA+N2\n2zbZ1nFzUTDc/R4z+7ccfXztR8llFBrlVuCHZraRI48X/2kyT3HbO+7zZrbY3beUtS8GftvAuFVr\nYS5cTnwO434zm5u0jQCbAGtg3G7bJts6bi7OYbSSmS0iHpHSlzQNE49I2aq47R3XzM4DbgaKHHnc\n9lni47YPNip2uzOzNxAXrUfd/d4Gx+qabbLd43Z9wZDOl5y8nUgSd9/dyvXJIzPb4u6Lk+fvAd4P\n/C/gLcDXGzlSUdpHLg5JtUqrhhIqblOHbAbgDA6PkuoxsxEN5z3KMann/xl4s7s/ZWafBX4AdNRw\nXjN7a3IByNI6fI6kRwX8Zek8TgPitvXn7fZfejvwDPGNaua4+xzgjcC+ZJ7itnHcZCjh48Bq4K3J\n4zpgu5n9YaPitqmCmc0xs5OAgrs/BeDuzwGNPHfSqm1yTer554AngT8Cfgj8XQPjtvfnbcVNO/Ly\nGBgY+Gk18xS3beI+NjAwsGCS9pcODAw81qi47fgYGBjYOTAw8PPk8bOBgYHTkvbiwMDAQx24bfw4\n9fzhgYGBkJ7W55380dWHpGjdUELF7ewhm23H3RdMMWsc+OMGhm7VtnGKmf0VEIATy+aFBsZt68/b\n7UnTqqGEeYq7G/hah8Zt1RDGjuHuvwF+3sAQrcqFW4hHzwHcBpwC7EkGSDzUwLht/Xm7fpRUcqG3\n04EfuPtoqn1JMy8fYWZfcvd3NjjG+cBj7r7fzH6HeMM9F/gJ8Cl339+guMcS76R3uftmM7sCeA3x\n7Um/UH7Iu74nAAACmklEQVQZjDrHbskQRqmdmV3l7re1IO5Sd2/YF4pW7XNScb/v7r9OtU+cEJ9J\nV/cwzGw58fDBbcAXzewad78rmf1poCH/eWb2NeLREemu4JuS0QuRu1/SiLjE3yxekTy/AXiO+HNe\nlMz7kwbGLQCzzexdwAnAV5O4i4m74w2RFAYVh/b0SeJtp9muo0E90Bbuc6aLu4aMtwvu6oIBvBc4\nz91/bWYLgH8yswXufkOD484j3ondAhwiLhyvBj7b4Lgh9Wvh89z93OT5d8zs4QbGPcfdzzGzHmAX\n0OfuY2b2ZeCRRgVt5XBeycbMHp1m9txp5rVlXFq3zymP+4/VxO32YbWh1DVLLsp1IfBWM/s8jT3x\n9WrgQeDjwLPuPgg87+73u/v9DYz7k9SF5B42s98HMLOzgAMNjDsrOSxVBI7n8Em342jsNtiqIYyS\n3UuAdxEP8Sx/PN2BcVu1zymP219N3G7vYewxs99z94cAkur7duCLHD50U3fuPg5cb2YOfN7M9tCc\n/4s/B240s/8KPAU8YGZDxBe5+/MGxv0ycVf4IPAh4Ntm9gBwAfHltBulVVdgley+AZzg7j8un2Fm\njfzy1Kq4Ldnn1CtuV5/0NrP5wMHyS0Ukvw5+nbt/p0nr8Xbgte7+sSbFOxF4KXGRGmrGpTKSbvCz\n7r7XzM4k7mU95u4NOxRmZpuBzUw+hPHN7n5Ro2KLTKZV+5x6xe3qgiGdzczmEJ/DuITDx6VLQxjX\nuvveVq2bSDtSwZCu1KohmyLtrNtPekv3+mSrV0Ck3XT7SW/pYC0cOinSkVQwpJO9BFhCPLS23ANN\nXheRtqeCIZ2sVUMnRTqSTnqLiEgmOuktIiKZqGCIiEgmKhgiIpKJCoaIiGSigiEiIpn8f/xO0TL/\nlGoaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7efe38f84c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot data len distribution by label (ham or spam)\n",
    "messages.hist(column='length', by='label', bins=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Data preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#take a message, split into list of tokens/words based on whitespace\n",
    "def split_into_tokens(message):\n",
    "  \n",
    "    #message = unicode(message, 'utf8')  # convert bytes into proper unicode\n",
    "    return message.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Go until jurong point, crazy.. Available only ...\n",
       "1                        Ok lar... Joking wif u oni...\n",
       "2    Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3    U dun say so early hor... U c already then say...\n",
       "4    Nah I don't think he goes to usf, he lives aro...\n",
       "Name: message, dtype: object"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages.message.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [Go, until, jurong, point,, crazy.., Available...\n",
       "1                 [Ok, lar..., Joking, wif, u, oni...]\n",
       "2    [Free, entry, in, 2, a, wkly, comp, to, win, F...\n",
       "3    [U, dun, say, so, early, hor..., U, c, already...\n",
       "4    [Nah, I, don't, think, he, goes, to, usf,, he,...\n",
       "Name: message, dtype: object"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#split message into tokens\n",
    "messages.message.head().apply(split_into_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['label', 'message', 'length'], dtype='object')\n",
      "  label                                            message  length\n",
      "0   ham  Go until jurong point, crazy.. Available only ...     111\n",
      "1   ham                      Ok lar... Joking wif u oni...      29\n",
      "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...     155\n",
      "3   ham  U dun say so early hor... U c already then say...      49\n",
      "4   ham  Nah I don't think he goes to usf, he lives aro...      61\n"
     ]
    }
   ],
   "source": [
    "#columns in the dataset\n",
    "print(messages.columns)\n",
    "print(messages.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\n",
      "go until jurong point crazi avail onli in bugi n great world la e buffet cine there got amor wat\n"
     ]
    }
   ],
   "source": [
    "#Lets Try  Stemming \n",
    "\n",
    "#STEMMING , No Lemmatization\n",
    "\n",
    "import string\n",
    "\n",
    "def stem_tokens(tokens,stemmer):\n",
    "    tokenLower=[token.lower() for token in tokens]\n",
    "    return [stemmer.stem(token) for token in  tokenLower]\n",
    "\n",
    "def tokenize(text):\n",
    "    stemmer=PorterStemmer()\n",
    "    text = \"\".join([ch for ch in text if ch not in string.punctuation])\n",
    "    tokens = nltk.word_tokenize(text)\n",
    "    tokensNoStop=[]\n",
    "\n",
    "    stems = stem_tokens(tokens, stemmer)\n",
    "    return \" \".join(stems)\n",
    "\n",
    "\n",
    "messages['messageStemmed'] = messages['message'].apply(tokenize)\n",
    "\n",
    "\n",
    "#Data in Raw Form\n",
    "print(messages['message'].ix[0,:])\n",
    "\n",
    "\n",
    "#Stemming and Convert to Lower Case \n",
    "print(messages['messageStemmed'].ix[0,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FIRST MODEL NAIVE BAYES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'scipy.sparse.csr.csr_matrix'>\n",
      "<class 'numpy.matrixlib.defmatrix.matrix'>\n",
      "(5574, 7970)\n",
      "[1 1 0 ..., 1 1 1]\n",
      "Index(['label', 'message', 'length', 'messageStemmed', 'labelNew'], dtype='object')\n",
      "Shapes on Feat Mat and Target Matrix is (5574, 7970) (5574,) \n",
      "The first feature vector is :\n",
      "  (0, 4036)\t1\n",
      "  (0, 5500)\t1\n",
      "  (0, 2207)\t1\n",
      "  (0, 1323)\t1\n",
      "  (0, 5167)\t1\n",
      "  (0, 1722)\t1\n",
      "  (0, 3352)\t1\n",
      "  (0, 7730)\t1\n",
      "  (0, 4180)\t1\n",
      "  (0, 1720)\t1\n",
      "  (0, 1993)\t1\n",
      "  (0, 3315)\t1\n",
      "  (0, 1138)\t1\n",
      "  (0, 7536)\t1\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#Init the Feature Extracter\n",
    "count_vectorizer = CountVectorizer(stop_words='english')\n",
    "\n",
    "#Fit countvectorizer\n",
    "counts = count_vectorizer.fit_transform(messages['messageStemmed'].values)\n",
    "print(type(counts))\n",
    "#<class 'scipy.sparse.csr.csr_matrix'>\n",
    "\n",
    "\n",
    "#convert to dense matrix\n",
    "countsDense=counts.todense()\n",
    "print(type(countsDense))\n",
    "\n",
    "#convert to Pandas DataFrame\n",
    "X=pd.DataFrame(countsDense)\n",
    "print(X.shape)\n",
    "\n",
    "\n",
    "#get the targets to train/predict\n",
    "#lets represent ham by 1 and spam by 0 \n",
    "codingDict = {\"ham\":1,\"spam\":0}\n",
    "messages['labelNew']=messages['label'].map(codingDict)\n",
    "targets = messages['labelNew'].values\n",
    "print(targets)\n",
    "print(messages.columns) #create new label 0/1 from spam/ham 'label', 'message', 'length', 'labelNew'\n",
    "\n",
    "Y=targets\n",
    "\n",
    "\n",
    "print(\"Shapes on Feat Mat and Target Matrix is %s %s \" % (counts.shape,targets.shape))\n",
    "\n",
    "#print the first feature vector \n",
    "print(\"The first feature vector is :\")\n",
    "print(counts[0,:])\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction Socre is  0.993720846789\n"
     ]
    }
   ],
   "source": [
    "#Naive Bayes Classifier\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(X,Y)\n",
    "\n",
    "#Predict \n",
    "pred=classifier.predict(X)\n",
    "\n",
    "print(\"Prediction Socre is \" , classifier.score(X,Y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Set Shapes (4459, 7970) (4459,) (1115, 7970) (1115,)\n"
     ]
    }
   ],
   "source": [
    "# You cannot use the whole data for training and then test it on the model which used the test data also\n",
    "#so lets create Training and Test Data ???\n",
    "\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size=0.20, random_state=4)\n",
    "\n",
    "print(\"Data Set Shapes %s %s %s %s\" %(X_train.shape,Y_train.shape,X_test.shape,Y_test.shape))\n",
    "\n",
    "#DONT USE TEST DATA IN MODEL BUILDING PROCESS\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Accuracy: 0.99416909621\n",
      "Test Accuracy: 0.970403587444\n"
     ]
    }
   ],
   "source": [
    "#Build Model \n",
    "#fit the training data\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(X_train, Y_train) \n",
    "\n",
    "#predict on training data\n",
    "train_pred = classifier.predict(X_train)\n",
    "\n",
    "#predict on test data \n",
    "test_pred = classifier.predict(X_test)\n",
    "\n",
    "from sklearn import metrics\n",
    "print( \"Train Accuracy:\", metrics.accuracy_score(Y_train, train_pred))\n",
    "print( \"Test Accuracy:\", metrics.accuracy_score(Y_test, test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix for Training Data\n",
      "[[ 575   15]\n",
      " [  11 3858]]\n",
      "Confusion Matrix for Test Data\n",
      "[[141  16]\n",
      " [ 17 941]]\n"
     ]
    }
   ],
   "source": [
    "# Confusion Matrix is used to compute TruePos, FalsePos, TrueNeg, FalseNeg\n",
    "#This can be used to compute metrics like Precision, Recall, F1-Score which\n",
    "#are used to report classifier performance \n",
    "\n",
    "cmTest = metrics.confusion_matrix(Y_test, test_pred)\n",
    "cmTrain = metrics.confusion_matrix(Y_train, train_pred)\n",
    "\n",
    "print(\"Confusion Matrix for Training Data\")\n",
    "print(cmTrain)\n",
    "\n",
    "print(\"Confusion Matrix for Test Data\")\n",
    "print(cmTest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision: 0.983281086729\n",
      "Recall: 0.982254697286\n",
      "F1 Score: 0.982767624021\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "#of the items we predicted to belong to a class C, how many did we get right\n",
    "print (\"Precision:\", metrics.precision_score(Y_test, test_pred, average='binary'))\n",
    "\n",
    "#of the items which belong to class C, how many did we get right\n",
    "print (\"Recall:\", metrics.recall_score(Y_test, test_pred, average='binary'))\n",
    "\n",
    "# a single number to represent performance, F1_Score = (2PR/(P+R))\n",
    "print (\"F1 Score:\", metrics.f1_score(Y_test, test_pred, average='binary'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use TfIDFVectorizer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'scipy.sparse.csr.csr_matrix'>\n",
      "<class 'numpy.matrixlib.defmatrix.matrix'>\n",
      "(5574, 7970)\n",
      "[1 1 0 ..., 1 1 1]\n",
      "Index(['label', 'message', 'length', 'messageStemmed', 'labelNew'], dtype='object')\n",
      "Shapes on Feat Mat and Target Matrix is (5574, 7970) (5574,) \n",
      "The first feature vector is :\n",
      "  (0, 7536)\t0.193710510317\n",
      "  (0, 1138)\t0.348686924906\n",
      "  (0, 3315)\t0.161811923942\n",
      "  (0, 1993)\t0.294574296835\n",
      "  (0, 1720)\t0.332859995787\n",
      "  (0, 4180)\t0.294574296835\n",
      "  (0, 7730)\t0.236963746858\n",
      "  (0, 3352)\t0.194080503704\n",
      "  (0, 1722)\t0.294574296835\n",
      "  (0, 5167)\t0.168155689475\n",
      "  (0, 1323)\t0.2651515607\n",
      "  (0, 2207)\t0.270037180084\n",
      "  (0, 5500)\t0.238095246665\n",
      "  (0, 4036)\t0.348686924906\n"
     ]
    }
   ],
   "source": [
    "tf_vectorizer = TfidfVectorizer(stop_words='english')\n",
    "\n",
    "#Fit tfidfvectorizer\n",
    "counts = tf_vectorizer.fit_transform(messages['messageStemmed'].values)\n",
    "print(type(counts))\n",
    "#<class 'scipy.sparse.csr.csr_matrix'>\n",
    "\n",
    "\n",
    "#convert to dense matrix\n",
    "countsDense=counts.todense()\n",
    "print(type(countsDense))\n",
    "\n",
    "#convert to Pandas DataFrame\n",
    "X=pd.DataFrame(countsDense)\n",
    "print(X.shape)\n",
    "\n",
    "\n",
    "#get the targets to train/predict\n",
    "#lets represent ham by 1 and spam by 0 \n",
    "codingDict = {\"ham\":1,\"spam\":0}\n",
    "messages['labelNew']=messages['label'].map(codingDict)\n",
    "targets = messages['labelNew'].values\n",
    "print(targets)\n",
    "print(messages.columns) #create new label 0/1 from spam/ham 'label', 'message', 'length', 'labelNew'\n",
    "\n",
    "Y=targets\n",
    "\n",
    "\n",
    "print(\"Shapes on Feat Mat and Target Matrix is %s %s \" % (counts.shape,targets.shape))\n",
    "\n",
    "#print the first feature vector \n",
    "print(\"The first feature vector is :\")\n",
    "print(counts[0,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction Socre is  0.979906709724\n",
      "Data Set Shapes (4459, 7970) (4459,) (1115, 7970) (1115,)\n",
      "Train Accuracy: 0.976003588248\n",
      "Test Accuracy: 0.959641255605\n"
     ]
    }
   ],
   "source": [
    "#Naive Bayes Classifier\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(X,Y)\n",
    "\n",
    "#Predict \n",
    "pred=classifier.predict(X)\n",
    "\n",
    "print(\"Prediction Socre is \" , classifier.score(X,Y))\n",
    "\n",
    "# You cannot use the whole data for training and then test it on the model which used the test data also\n",
    "#so lets create Training and Test Data ???\n",
    "\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size=0.20, random_state=4)\n",
    "\n",
    "print(\"Data Set Shapes %s %s %s %s\" %(X_train.shape,Y_train.shape,X_test.shape,Y_test.shape))\n",
    "\n",
    "#DONT USE TEST DATA IN MODEL BUILDING PROCESS\n",
    "\n",
    "#Build Model \n",
    "#fit the training data\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(X_train, Y_train) \n",
    "\n",
    "#predict on training data\n",
    "train_pred = classifier.predict(X_train)\n",
    "\n",
    "#predict on test data \n",
    "test_pred = classifier.predict(X_test)\n",
    "\n",
    "from sklearn import metrics\n",
    "print( \"Train Accuracy:\", metrics.accuracy_score(Y_train, train_pred))\n",
    "print( \"Test Accuracy:\", metrics.accuracy_score(Y_test, test_pred))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Accuracy: 0.967032967033\n",
      "Test Accuracy: 0.943497757848\n"
     ]
    }
   ],
   "source": [
    "#Build Model \n",
    "#fit the training data\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "classifier =LogisticRegression()\n",
    "classifier.fit(X_train, Y_train) \n",
    "\n",
    "#predict on training data\n",
    "train_pred = classifier.predict(X_train)\n",
    "\n",
    "#predict on test data \n",
    "test_pred = classifier.predict(X_test)\n",
    "\n",
    "from sklearn import metrics\n",
    "print( \"Train Accuracy:\", metrics.accuracy_score(Y_train, train_pred))\n",
    "print( \"Test Accuracy:\", metrics.accuracy_score(Y_test, test_pred))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CountVectorizer worked better in our case for Spam Classification compared to TfIDF Vectorizer"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}