Full+Exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build our our own machine learning algorithm that uses the principles in the first Obama/Trump tweet example to learn what Trump tweets and Obama tweets and be able to predict whether an unknown tweet is Trump or Obama.\n",
    "\n",
    "We have 9 trump tweets and 10 Obama tweets. We will use the first 6 of each to teach our system what is an Obama Tweet and what is a Trump tweet. We will then give the system our 3 Trump tweets and our 4 Obama tweets and ask it to tell us who tweeted. \n",
    "\n",
    "## 1 Load the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Author</th>\n",
       "      <th>Tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Trump</td>\n",
       "      <td>The current tax code is a burden on American t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Trump</td>\n",
       "      <td>I am supportive of Lamar as a person &amp; also of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Trump</td>\n",
       "      <td>Democrat Congresswoman totally fabricated what...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Trump</td>\n",
       "      <td>The NFL has decided that it will not force pla...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Trump</td>\n",
       "      <td>As it has turned out, James Comey lied and lea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Trump</td>\n",
       "      <td>The Democrats will only vote for Tax Increases...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Trump</td>\n",
       "      <td>...people not interviewed, including Clinton h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Trump</td>\n",
       "      <td>Wow, FBI confirms report that James Comey draf...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Trump</td>\n",
       "      <td>The most important truth our FOUNDERS understo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Obama</td>\n",
       "      <td>I'm grateful to @SenJohnMcCain for his lifetim...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Obama</td>\n",
       "      <td>Michelle &amp; I are praying for the victims in La...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Obama</td>\n",
       "      <td>Proud to cheer on Team USA at the Invictus Gam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Obama</td>\n",
       "      <td>We're expanding our efforts to help Puerto Ric...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Obama</td>\n",
       "      <td>Prosecutor, soldier, family man, citizen. Beau...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Obama</td>\n",
       "      <td>Thinking about our neighbors in Mexico and all...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Obama</td>\n",
       "      <td>Coding is important – and fun. @CSforAll, than...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Obama</td>\n",
       "      <td>Michelle and I want the @ObamaFoundation to in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Obama</td>\n",
       "      <td>We remember everyone we lost on 9/11 and honor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Obama</td>\n",
       "      <td>Proud of these McKinley Tech students—inspirin...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Author                                              Tweet\n",
       "0   Trump  The current tax code is a burden on American t...\n",
       "1   Trump  I am supportive of Lamar as a person & also of...\n",
       "2   Trump  Democrat Congresswoman totally fabricated what...\n",
       "3   Trump  The NFL has decided that it will not force pla...\n",
       "4   Trump  As it has turned out, James Comey lied and lea...\n",
       "5   Trump  The Democrats will only vote for Tax Increases...\n",
       "6   Trump  ...people not interviewed, including Clinton h...\n",
       "7   Trump  Wow, FBI confirms report that James Comey draf...\n",
       "8   Trump  The most important truth our FOUNDERS understo...\n",
       "9   Obama  I'm grateful to @SenJohnMcCain for his lifetim...\n",
       "10  Obama  Michelle & I are praying for the victims in La...\n",
       "11  Obama  Proud to cheer on Team USA at the Invictus Gam...\n",
       "12  Obama  We're expanding our efforts to help Puerto Ric...\n",
       "13  Obama  Prosecutor, soldier, family man, citizen. Beau...\n",
       "14  Obama  Thinking about our neighbors in Mexico and all...\n",
       "15  Obama  Coding is important – and fun. @CSforAll, than...\n",
       "16  Obama  Michelle and I want the @ObamaFoundation to in...\n",
       "17  Obama  We remember everyone we lost on 9/11 and honor...\n",
       "18  Obama  Proud of these McKinley Tech students—inspirin..."
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "tweets = pd.read_csv(\"trump-obama-tweets - Sheet1.csv\")\n",
    "tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Create the training and test sets:\n",
    "\n",
    "Before we start coding our machine learning algorithm we need to prepare our data. Typically 80% of any machine learning or NLP activity is data preparation. Here we will take our 19 tweets and create a 'training set' of 12 tweets and a test set of 7 tweets - 4 Obama and 3 trump.\n",
    "\n",
    "We will train our algorithm with the 12 training tweets by giving feeding it each tweet and telling it the author so the algorithm can learn what an Obama tweet and a Trump tweet is like.\n",
    "\n",
    "Once the machine has been trained we will give it the remaining 7 tweets without telling it who is author and see how accurately it classifies the 7 unknown tweets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Obama Training:\n",
      "[\"i'm grateful @senjohnmccain lifetime service country. congratulations, john, receiving year's liberty medal.\", 'michelle & i praying victims las vegas. thoughts families & enduring senseless tragedy.', 'proud cheer team usa invictus games today friend joe. represent best country.', \"we're expanding efforts help puerto rico & usvi, fellow americans need right now. join http://oneamericaappeal.org \", 'prosecutor, soldier, family man, citizen. beau want better. legacy leave. testament @joebiden.', 'thinking neighbors mexico mexican-american friends tonight. cuidense mucho y fuerte abrazo para todos.']\n",
      "\n",
      "Obama Test:\n",
      "['coding important – fun. @csforall, thanks work make sure kid compete high-tech, global economy.', \"michelle i want @obamafoundation inspire empower people change world. here's we're getting started fall.\", 'we remember lost 9/11 honor defend country ideals. act terror change are.', 'proud mckinley tech students—inspiring young minds make hopeful future.']\n",
      "\n",
      "Trump Training:\n",
      "['the current tax code burden american taxpayers & harmful job-creators. americans need #taxreform! more: http://45.wh.gov/stqh9z ', \"i supportive lamar person & process, i support bailing ins co's fortune w/ o'care.\", 'democrat congresswoman totally fabricated i said wife soldier died action (and i proof). sad!', 'the nfl decided force players stand playing national anthem. total disrespect great country!', 'as turned out, james comey lied leaked totally protected hillary clinton. best thing happened her!', 'the democrats vote tax increases. hopefully, senate republicans vote largest tax cuts u.s. history.']\n",
      "\n",
      "Trump Test:\n",
      "[\"...people interviewed, including clinton herself. comey stated oath didn't this-obviously fix? justice dept?\", 'wow, fbi confirms report james comey drafted letter exonerating crooked hillary clinton long investigation complete. many..', 'the important truth founders understood was: freedom gift govt. freedom gift god.']\n"
     ]
    }
   ],
   "source": [
    "def clean(tweet):\n",
    "  stopwords = [\"a\", \"about\", \"above\", \"above\", \"across\", \"after\", \"afterwards\", \"again\", \"against\", \"all\", \"almost\", \"alone\", \"along\", \"already\", \"also\",\"although\",\"always\",\"am\",\"among\", \"amongst\", \"amoungst\", \"amount\",  \"an\", \"and\", \"another\", \"any\",\"anyhow\",\"anyone\",\"anything\",\"anyway\", \"anywhere\", \"are\", \"around\", \"as\",  \"at\", \"back\",\"be\",\"became\", \"because\",\"become\",\"becomes\", \"becoming\", \"been\", \"before\", \"beforehand\", \"behind\", \"being\", \"below\", \"beside\", \"besides\", \"between\", \"beyond\", \"bill\", \"both\", \"bottom\",\"but\", \"by\", \"call\", \"can\", \"cannot\", \"cant\", \"co\", \"con\", \"could\", \"couldnt\", \"cry\", \"de\", \"describe\", \"detail\", \"do\", \"done\", \"down\", \"due\", \"during\", \"each\", \"eg\", \"eight\", \"either\", \"eleven\",\"else\", \"elsewhere\", \"empty\", \"enough\", \"etc\", \"even\", \"ever\", \"every\", \"everyone\", \"everything\", \"everywhere\", \"except\", \"few\", \"fifteen\", \"fify\", \"fill\", \"find\", \"fire\", \"first\", \"five\", \"for\", \"former\", \"formerly\", \"forty\", \"found\", \"four\", \"from\", \"front\", \"full\", \"further\", \"get\", \"give\", \"go\", \"had\", \"has\", \"hasnt\", \"have\", \"he\", \"hence\", \"her\", \"here\", \"hereafter\", \"hereby\", \"herein\", \"hereupon\", \"hers\", \"herself\", \"him\", \"himself\", \"his\", \"how\", \"however\", \"hundred\", \"ie\", \"if\", \"in\", \"inc\", \"indeed\", \"interest\", \"into\", \"is\", \"it\", \"its\", \"itself\", \"keep\", \"last\", \"latter\", \"latterly\", \"least\", \"less\", \"ltd\", \"made\", \"many\", \"may\", \"me\", \"meanwhile\", \"might\", \"mill\", \"mine\", \"more\", \"moreover\", \"most\", \"mostly\", \"move\", \"much\", \"must\", \"my\", \"myself\", \"name\", \"namely\", \"neither\", \"never\", \"nevertheless\", \"next\", \"nine\", \"no\", \"nobody\", \"none\", \"noone\", \"nor\", \"not\", \"nothing\", \"now\", \"nowhere\", \"of\", \"off\", \"often\", \"on\", \"once\", \"one\", \"only\", \"onto\", \"or\", \"other\", \"others\", \"otherwise\", \"our\", \"ours\", \"ourselves\", \"out\", \"over\", \"own\",\"part\", \"per\", \"perhaps\", \"please\", \"put\", \"rather\", \"re\", \"same\", \"see\", \"seem\", \"seemed\", \"seeming\", \"seems\", \"serious\", \"several\", \"she\", \"should\", \"show\", \"side\", \"since\", \"sincere\", \"six\", \"sixty\", \"so\", \"some\", \"somehow\", \"someone\", \"something\", \"sometime\", \"sometimes\", \"somewhere\", \"still\", \"such\", \"system\", \"take\", \"ten\", \"than\", \"that\", \"the\", \"their\", \"them\", \"themselves\", \"then\", \"thence\", \"there\", \"thereafter\", \"thereby\", \"therefore\", \"therein\", \"thereupon\", \"these\", \"they\", \"thickv\", \"thin\", \"third\", \"this\", \"those\", \"though\", \"three\", \"through\", \"throughout\", \"thru\", \"thus\", \"to\", \"together\", \"too\", \"top\", \"toward\", \"towards\", \"twelve\", \"twenty\", \"two\", \"un\", \"under\", \"until\", \"up\", \"upon\", \"us\", \"very\", \"via\", \"was\", \"we\", \"well\", \"were\", \"what\", \"whatever\", \"when\", \"whence\", \"whenever\", \"where\", \"whereafter\", \"whereas\", \"whereby\", \"wherein\", \"whereupon\", \"wherever\", \"whether\", \"which\", \"while\", \"whither\", \"who\", \"whoever\", \"whole\", \"whom\", \"whose\", \"why\", \"will\", \"with\", \"within\", \"without\", \"would\", \"yet\", \"you\", \"your\", \"yours\", \"yourself\", \"yourselves\", \"the\"]\n",
    "  #stopwords = []\n",
    "  tweet = tweet.lower()\n",
    "  for stopword in stopwords:\n",
    "    tweet = tweet.replace(\" \" + stopword + \" \",\" \")\n",
    "  return tweet\n",
    "\n",
    "obama_training_tweets = []\n",
    "trump_training_tweets = []\n",
    "obama_test_tweets = []\n",
    "trump_test_tweets = []\n",
    "\n",
    "for row in tweets.itertuples():\n",
    "    if (row[0] < 15 and row[1] == 'Obama'):\n",
    "        obama_training_tweets.append(clean(row[2]))\n",
    "    elif (row[0] > 14 and row[1] == 'Obama'):\n",
    "        obama_test_tweets.append(clean(row[2]))\n",
    "    elif (row[0] < 6 and row[1] == 'Trump'):\n",
    "        trump_training_tweets.append(clean(row[2]))\n",
    "    elif (row[0] > 5 and row[1] == 'Trump'):\n",
    "        trump_test_tweets.append(clean(row[2]))\n",
    "        \n",
    "        \n",
    "\n",
    "print(\"Obama Training:\")\n",
    "print(obama_training_tweets)\n",
    "print(\"\")\n",
    "print(\"Obama Test:\")\n",
    "print(obama_test_tweets)\n",
    "print(\"\")\n",
    "print(\"Trump Training:\")\n",
    "print(trump_training_tweets)\n",
    "print(\"\")\n",
    "print(\"Trump Test:\")\n",
    "print(trump_test_tweets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Probability\n",
    "\n",
    "In this corpus there are 6 training tweets for Trump and 6 for Obama so by inspection we can deduce that:\n",
    "\n",
    "P(Obama) = 0.5\n",
    "\n",
    "P(Trump) = 0.5\n",
    "\n",
    "Thus without applying Bayes Theorem there is a 50/50 chance the training tweets are either Trump or Obama.\n",
    "\n",
    "## 4 Prior Probability\n",
    "\n",
    "### 4.1 Naive Bayes Learning\n",
    "\n",
    "Now let's calculate the probabilty of Obama and Trump using each word in our corpus. We will concatenate all of the tweets for an author (Obama then Trump) then create we will iterate through every word in every tweet and build a learnings dictionary that will be the probability of that word being used by its author.\n",
    "\n",
    "For example the word grateful may occur two times in a total of 130 words so our dictionary entry would look like:\n",
    "\n",
    "`grateful: 0.015` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Obama learning\n",
      "{\"i'm\": 0.012987012987012988, 'grateful': 0.012987012987012988, '@senjohnmccain': 0.012987012987012988, 'lifetime': 0.012987012987012988, 'service': 0.012987012987012988, 'country.': 0.025974025974025976, 'congratulations,': 0.012987012987012988, 'john,': 0.012987012987012988, 'receiving': 0.012987012987012988, \"year's\": 0.012987012987012988, 'liberty': 0.012987012987012988, 'medal.': 0.012987012987012988, 'michelle': 0.012987012987012988, '&': 0.03896103896103896, 'i': 0.012987012987012988, 'praying': 0.012987012987012988, 'victims': 0.012987012987012988, 'las': 0.012987012987012988, 'vegas.': 0.012987012987012988, 'thoughts': 0.012987012987012988, 'families': 0.012987012987012988, 'enduring': 0.012987012987012988, 'senseless': 0.012987012987012988, 'tragedy.': 0.012987012987012988, 'proud': 0.012987012987012988, 'cheer': 0.012987012987012988, 'team': 0.012987012987012988, 'usa': 0.012987012987012988, 'invictus': 0.012987012987012988, 'games': 0.012987012987012988, 'today': 0.012987012987012988, 'friend': 0.012987012987012988, 'joe.': 0.012987012987012988, 'represent': 0.012987012987012988, 'best': 0.012987012987012988, \"we're\": 0.012987012987012988, 'expanding': 0.012987012987012988, 'efforts': 0.012987012987012988, 'help': 0.012987012987012988, 'puerto': 0.012987012987012988, 'rico': 0.012987012987012988, 'usvi,': 0.012987012987012988, 'fellow': 0.012987012987012988, 'americans': 0.012987012987012988, 'need': 0.012987012987012988, 'right': 0.012987012987012988, 'now.': 0.012987012987012988, 'join': 0.012987012987012988, 'http://oneamericaappeal.org': 0.012987012987012988, 'prosecutor,': 0.012987012987012988, 'soldier,': 0.012987012987012988, 'family': 0.012987012987012988, 'man,': 0.012987012987012988, 'citizen.': 0.012987012987012988, 'beau': 0.012987012987012988, 'want': 0.012987012987012988, 'better.': 0.012987012987012988, 'legacy': 0.012987012987012988, 'leave.': 0.012987012987012988, 'testament': 0.012987012987012988, '@joebiden.': 0.012987012987012988, 'thinking': 0.012987012987012988, 'neighbors': 0.012987012987012988, 'mexico': 0.012987012987012988, 'mexican-american': 0.012987012987012988, 'friends': 0.012987012987012988, 'tonight.': 0.012987012987012988, 'cuidense': 0.012987012987012988, 'mucho': 0.012987012987012988, 'y': 0.012987012987012988, 'fuerte': 0.012987012987012988, 'abrazo': 0.012987012987012988, 'para': 0.012987012987012988, 'todos.': 0.012987012987012988}\n",
      "Trump learning\n",
      "{'the': 0.03529411764705882, 'current': 0.011764705882352941, 'tax': 0.03529411764705882, 'code': 0.011764705882352941, 'burden': 0.011764705882352941, 'american': 0.011764705882352941, 'taxpayers': 0.011764705882352941, '&': 0.023529411764705882, 'harmful': 0.011764705882352941, 'job-creators.': 0.011764705882352941, 'americans': 0.011764705882352941, 'need': 0.011764705882352941, '#taxreform!': 0.011764705882352941, 'more:': 0.011764705882352941, 'http://45.wh.gov/stqh9z': 0.011764705882352941, 'i': 0.047058823529411764, 'supportive': 0.011764705882352941, 'lamar': 0.011764705882352941, 'person': 0.011764705882352941, 'process,': 0.011764705882352941, 'support': 0.011764705882352941, 'bailing': 0.011764705882352941, 'ins': 0.011764705882352941, \"co's\": 0.011764705882352941, 'fortune': 0.011764705882352941, 'w/': 0.011764705882352941, \"o'care.\": 0.011764705882352941, 'democrat': 0.011764705882352941, 'congresswoman': 0.011764705882352941, 'totally': 0.023529411764705882, 'fabricated': 0.011764705882352941, 'said': 0.011764705882352941, 'wife': 0.011764705882352941, 'soldier': 0.011764705882352941, 'died': 0.011764705882352941, 'action': 0.011764705882352941, '(and': 0.011764705882352941, 'proof).': 0.011764705882352941, 'sad!': 0.011764705882352941, 'nfl': 0.011764705882352941, 'decided': 0.011764705882352941, 'force': 0.011764705882352941, 'players': 0.011764705882352941, 'stand': 0.011764705882352941, 'playing': 0.011764705882352941, 'national': 0.011764705882352941, 'anthem.': 0.011764705882352941, 'total': 0.011764705882352941, 'disrespect': 0.011764705882352941, 'great': 0.011764705882352941, 'country!': 0.011764705882352941, 'as': 0.011764705882352941, 'turned': 0.011764705882352941, 'out,': 0.011764705882352941, 'james': 0.011764705882352941, 'comey': 0.011764705882352941, 'lied': 0.011764705882352941, 'leaked': 0.011764705882352941, 'protected': 0.011764705882352941, 'hillary': 0.011764705882352941, 'clinton.': 0.011764705882352941, 'best': 0.011764705882352941, 'thing': 0.011764705882352941, 'happened': 0.011764705882352941, 'her!': 0.011764705882352941, 'democrats': 0.011764705882352941, 'vote': 0.023529411764705882, 'increases.': 0.011764705882352941, 'hopefully,': 0.011764705882352941, 'senate': 0.011764705882352941, 'republicans': 0.011764705882352941, 'largest': 0.011764705882352941, 'cuts': 0.011764705882352941, 'u.s.': 0.011764705882352941, 'history.': 0.011764705882352941}\n"
     ]
    }
   ],
   "source": [
    "def p_word(training_tweets, learnings):\n",
    "  all_tweets = \"\"\n",
    "  for tweet in training_tweets:\n",
    "    all_tweets = all_tweets + \" \" + tweet\n",
    "  \n",
    "  words = all_tweets.split()\n",
    "  for word in words:\n",
    "    learnings[word] = ( float(words.count(word))) / float(len(words))\n",
    "  return learnings;\n",
    "\n",
    "\n",
    "obama_learnings = {}\n",
    "trump_learnings = {}\n",
    "\n",
    "p_word(obama_training_tweets, obama_learnings)\n",
    "    \n",
    "p_word(trump_training_tweets, trump_learnings)\n",
    "   \n",
    "print(\"Obama learning\")    \n",
    "print(obama_learnings)\n",
    "print(\"Trump learning\")    \n",
    "print(trump_learnings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Naive Bayes Likelihood\n",
    "\n",
    "We'll create an algorithm that uses the data we have learned to calculate the likelihood of each of our TEST tweets being either Obama or Trump. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def p_likelihood(tweet, learnings):\n",
    "  likelihood = 1.0;\n",
    "  for word in tweet.split():\n",
    "    likelihood = likelihood * (learnings.get(word,0.0) + 1.0);\n",
    "  return likelihood;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 Naive Bayes Classification\n",
    "\n",
    "Let's use our probabilty and likelihood in Bayes Theorem to classify our test tweets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"coding important – fun. @csforall, thanks work make sure kid compete high-tech, global economy.\"\n",
      "actual author: Obama\n",
      "Trump or Obama - equal probability: 0.50 0.50\n",
      "\n",
      "\"michelle i want @obamafoundation inspire empower people change world. here's we're getting started fall.\"\n",
      "actual author: Obama\n",
      "predicted author: 50.14% Obama\n",
      "\n",
      "\"we remember lost 9/11 honor defend country ideals. act terror change are.\"\n",
      "actual author: Obama\n",
      "Trump or Obama - equal probability: 0.50 0.50\n",
      "\n",
      "\"proud mckinley tech students—inspiring young minds make hopeful future.\"\n",
      "actual author: Obama\n",
      "predicted author: 50.32% Obama\n",
      "\n",
      "\"...people interviewed, including clinton herself. comey stated oath didn't this-obviously fix? justice dept?\"\n",
      "actual author: Trump\n",
      "predicted author: 50.29% Trump\n",
      "\n",
      "\"wow, fbi confirms report james comey drafted letter exonerating crooked hillary clinton long investigation complete. many..\"\n",
      "actual author: Trump\n",
      "predicted author: 50.88% Trump\n",
      "\n",
      "\"the important truth founders understood was: freedom gift govt. freedom gift god.\"\n",
      "actual author: Trump\n",
      "predicted author: 50.87% Trump\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def print_result(predicted_author, predicted_author_bayes_probability, other_author_bayes_probability):\n",
    "  percent = predicted_author_bayes_probability / (other_author_bayes_probability + predicted_author_bayes_probability) * 100.0\n",
    "  print(\"predicted author: {:0.2f}\".format(percent) + \"% \" + predicted_author)\n",
    "\n",
    "def classify(author, tweet, obama_learnings, trump_learnings):\n",
    "  obama_probability = 0.5 #half of our 'known' tweets are obama.\n",
    "  obama_likelihood = p_likelihood(tweet, obama_learnings)\n",
    "  obama_bayes_probability = obama_probability * obama_likelihood\n",
    "\n",
    "  trump_probability = 0.5 #half of our 'known' tweets are obama.\n",
    "  trump_likelihood = p_likelihood(tweet, trump_learnings)\n",
    "  trump_bayes_probability = trump_probability * trump_likelihood\n",
    "    \n",
    "  \n",
    "  print('\"' + tweet + '\"')\n",
    "  print(\"actual author: \" + author)\n",
    "\n",
    "  if obama_bayes_probability > trump_bayes_probability:\n",
    "    print_result(\"Obama\", obama_bayes_probability, trump_bayes_probability)\n",
    "  elif obama_bayes_probability < trump_bayes_probability:\n",
    "    print_result(\"Trump\", trump_bayes_probability, obama_bayes_probability)\n",
    "  else:\n",
    "    print(\"Trump or Obama - equal probability: {:0.2f} {:0.2f}\".format(obama_bayes_probability, trump_bayes_probability))\n",
    "  print(\"\")\n",
    "\n",
    "#classify the Obama test tweets\n",
    "for tweet in obama_test_tweets:\n",
    "  classify(\"Obama\", tweet, obama_learnings, trump_learnings)\n",
    "\n",
    "#classify the Trump test tweets\n",
    "for tweet in trump_test_tweets:\n",
    "  classify(\"Trump\", tweet, obama_learnings, trump_learnings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "Exercise 1:\n",
    "\n",
    "Using our code.txt file copy the \n",
    "\n",
    "Exercise 2:\n",
    "\n",
    "If you've got it right then you should see every result as __Trump or Obama - equal probability: 0.00 0.00__ which isn't very helpful. This is because in every tweet there is at least one word that the predicted author did not use. For example, if the tweet contains \"@csforall\" and we are predicting the likelihood of Trump using that word our __p_likelihood__ algorithm will return 0 meaning the product of all the words in that likelihood will be 0: 0.0 * 0.1 * 0.2 will always be 0. To fix this we apply Laplace Smoothing:\n",
    "\n",
    "change this line in p_likelihood to apply Laplace Smoothing: \n",
    "\n",
    "`likelihood = likelihood * (learnings.get(word,0.0));` \n",
    "\n",
    "to \n",
    "\n",
    "`likelihood = likelihood * (learnings.get(word,0.0) + 1.0);`\n",
    "\n",
    "Now run the code in sections 4.2 and 5 again. You should see classifications now with the first two Obama tweets being classified incorrectly as Trump tweets. Note that generally the classifications are very similar and not far off 50%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise 3:\n",
    "\n",
    "Our data set is very small and in most cases all but the most common words (stop words) are present only once. We added the Laplace Smoothing as 1 too so in many cases a word we have applied smoothing to because it's not in our training set.\n",
    "\n",
    "Let's attempt to offset that bias by multiplying every real word occrence by 10 and reducing the effect of the Laplace Smoothing:\n",
    "\n",
    "change this line in p_words \n",
    "\n",
    "`learnings[word] = ( float(words.count(word))) / float(len(words))`\n",
    "\n",
    "to\n",
    "\n",
    "`learnings[word] = ( float(words.count(word) * 10.0)) / float(len(words))`\n",
    "\n",
    "Now run the code in sections 4.1, 4.2 and 5 again and we should now see more significant differences in our classification. The first tweet will still be incorrectly predicted.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise 4:\n",
    "\n",
    "Stopwords are the everyday words like 'a', 'the', 'and', 'but' etc.. if we remove these words then we are left with the more significant words which should increase the accuracy of our predictions:\n",
    "\n",
    "In `def clean` uncomment\n",
    "\n",
    "`stopwords = [\"a\", \"about\", \"above\", \"above\", \"across\"....]`\n",
    "\n",
    "and comment out\n",
    "\n",
    "`stopwords = []`\n",
    "\n",
    "This will remove any stop words from the tweets. Run sections 2,4.1,4.2 and 5 again. Now we have two tweets that are exact matches - 50/50. Why is that ?\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### All of this in scikit is quite short and coulde be even less code with a simpler data representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "import numpy as np\n",
    "\n",
    "count_vect = CountVectorizer()\n",
    "\n",
    "training_counts = count_vect.fit_transform(obama_training_tweets+trump_training_tweets)\n",
    "target_training = ([1]*len(obama_training_tweets) + [0]*len(trump_training_tweets))\n",
    "test_counts = count_vect.transform(obama_test_tweets+trump_test_tweets)\n",
    "target_test = ([1]*len(obama_test_tweets)+[0]*len(trump_test_tweets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first number is the probability a tweet is coming from Obama or Trump:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.5  0.5]] coding important – fun. @csforall, thanks work make sure kid compete high-tech, global economy.\n",
      "[[ 0.05220252  0.94779748]] michelle i want @obamafoundation inspire empower people change world. here's we're getting started fall.\n",
      "[[ 0.2383362  0.7616638]] we remember lost 9/11 honor defend country ideals. act terror change are.\n",
      "[[ 0.32634731  0.67365269]] proud mckinley tech students—inspiring young minds make hopeful future.\n",
      "[[ 0.78969429  0.21030571]] ...people interviewed, including clinton herself. comey stated oath didn't this-obviously fix? justice dept?\n",
      "[[ 0.93377437  0.06622563]] wow, fbi confirms report james comey drafted letter exonerating crooked hillary clinton long investigation complete. many..\n",
      "[[ 0.79489517  0.20510483]] the important truth founders understood was: freedom gift govt. freedom gift god.\n"
     ]
    }
   ],
   "source": [
    "clf = MultinomialNB().fit(training_counts, target_training)\n",
    "\n",
    "for t in (obama_test_tweets+trump_test_tweets):\n",
    "    count = count_vect.transform([t])\n",
    "    print(clf.predict_proba(count), t)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding how our model made it's decisions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.coef_.argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-4.2858827741209788"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.coef_[0,22]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'45': 0,\n",
       " 'abrazo': 1,\n",
       " 'action': 2,\n",
       " 'american': 3,\n",
       " 'americans': 4,\n",
       " 'and': 5,\n",
       " 'anthem': 6,\n",
       " 'as': 7,\n",
       " 'bailing': 8,\n",
       " 'beau': 9,\n",
       " 'best': 10,\n",
       " 'better': 11,\n",
       " 'burden': 12,\n",
       " 'care': 13,\n",
       " 'cheer': 14,\n",
       " 'citizen': 15,\n",
       " 'clinton': 16,\n",
       " 'co': 17,\n",
       " 'code': 18,\n",
       " 'comey': 19,\n",
       " 'congratulations': 20,\n",
       " 'congresswoman': 21,\n",
       " 'country': 22,\n",
       " 'creators': 23,\n",
       " 'cuidense': 24,\n",
       " 'current': 25,\n",
       " 'cuts': 26,\n",
       " 'decided': 27,\n",
       " 'democrat': 28,\n",
       " 'democrats': 29,\n",
       " 'died': 30,\n",
       " 'disrespect': 31,\n",
       " 'efforts': 32,\n",
       " 'enduring': 33,\n",
       " 'expanding': 34,\n",
       " 'fabricated': 35,\n",
       " 'families': 36,\n",
       " 'family': 37,\n",
       " 'fellow': 38,\n",
       " 'force': 39,\n",
       " 'fortune': 40,\n",
       " 'friend': 41,\n",
       " 'friends': 42,\n",
       " 'fuerte': 43,\n",
       " 'games': 44,\n",
       " 'gov': 45,\n",
       " 'grateful': 46,\n",
       " 'great': 47,\n",
       " 'happened': 48,\n",
       " 'harmful': 49,\n",
       " 'help': 50,\n",
       " 'her': 51,\n",
       " 'hillary': 52,\n",
       " 'history': 53,\n",
       " 'hopefully': 54,\n",
       " 'http': 55,\n",
       " 'increases': 56,\n",
       " 'ins': 57,\n",
       " 'invictus': 58,\n",
       " 'james': 59,\n",
       " 'job': 60,\n",
       " 'joe': 61,\n",
       " 'joebiden': 62,\n",
       " 'john': 63,\n",
       " 'join': 64,\n",
       " 'lamar': 65,\n",
       " 'largest': 66,\n",
       " 'las': 67,\n",
       " 'leaked': 68,\n",
       " 'leave': 69,\n",
       " 'legacy': 70,\n",
       " 'liberty': 71,\n",
       " 'lied': 72,\n",
       " 'lifetime': 73,\n",
       " 'man': 74,\n",
       " 'medal': 75,\n",
       " 'mexican': 76,\n",
       " 'mexico': 77,\n",
       " 'michelle': 78,\n",
       " 'more': 79,\n",
       " 'mucho': 80,\n",
       " 'national': 81,\n",
       " 'need': 82,\n",
       " 'neighbors': 83,\n",
       " 'nfl': 84,\n",
       " 'now': 85,\n",
       " 'oneamericaappeal': 86,\n",
       " 'org': 87,\n",
       " 'out': 88,\n",
       " 'para': 89,\n",
       " 'person': 90,\n",
       " 'players': 91,\n",
       " 'playing': 92,\n",
       " 'praying': 93,\n",
       " 'process': 94,\n",
       " 'proof': 95,\n",
       " 'prosecutor': 96,\n",
       " 'protected': 97,\n",
       " 'proud': 98,\n",
       " 'puerto': 99,\n",
       " 're': 100,\n",
       " 'receiving': 101,\n",
       " 'represent': 102,\n",
       " 'republicans': 103,\n",
       " 'rico': 104,\n",
       " 'right': 105,\n",
       " 'sad': 106,\n",
       " 'said': 107,\n",
       " 'senate': 108,\n",
       " 'senjohnmccain': 109,\n",
       " 'senseless': 110,\n",
       " 'service': 111,\n",
       " 'soldier': 112,\n",
       " 'stand': 113,\n",
       " 'stqh9z': 114,\n",
       " 'support': 115,\n",
       " 'supportive': 116,\n",
       " 'tax': 117,\n",
       " 'taxpayers': 118,\n",
       " 'taxreform': 119,\n",
       " 'team': 120,\n",
       " 'testament': 121,\n",
       " 'the': 122,\n",
       " 'thing': 123,\n",
       " 'thinking': 124,\n",
       " 'thoughts': 125,\n",
       " 'today': 126,\n",
       " 'todos': 127,\n",
       " 'tonight': 128,\n",
       " 'total': 129,\n",
       " 'totally': 130,\n",
       " 'tragedy': 131,\n",
       " 'turned': 132,\n",
       " 'usa': 133,\n",
       " 'usvi': 134,\n",
       " 'vegas': 135,\n",
       " 'victims': 136,\n",
       " 'vote': 137,\n",
       " 'want': 138,\n",
       " 'we': 139,\n",
       " 'wh': 140,\n",
       " 'wife': 141,\n",
       " 'year': 142}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-5.38449506, -4.69134788, -5.38449506, -4.69134788, -4.69134788,\n",
       "        -5.38449506, -5.38449506, -5.38449506, -5.38449506, -4.69134788,\n",
       "        -4.69134788, -4.69134788, -5.38449506, -5.38449506, -4.69134788,\n",
       "        -4.69134788, -5.38449506, -5.38449506, -5.38449506, -5.38449506,\n",
       "        -4.69134788, -5.38449506, -4.28588277, -5.38449506, -4.69134788,\n",
       "        -5.38449506, -5.38449506, -5.38449506, -5.38449506, -5.38449506,\n",
       "        -5.38449506, -5.38449506, -4.69134788, -4.69134788, -4.69134788,\n",
       "        -5.38449506, -4.69134788, -4.69134788, -4.69134788, -5.38449506,\n",
       "        -5.38449506, -4.69134788, -4.69134788, -4.69134788, -4.69134788,\n",
       "        -5.38449506, -4.69134788, -5.38449506, -5.38449506, -5.38449506,\n",
       "        -4.69134788, -5.38449506, -5.38449506, -5.38449506, -5.38449506,\n",
       "        -4.69134788, -5.38449506, -5.38449506, -4.69134788, -5.38449506,\n",
       "        -5.38449506, -4.69134788, -4.69134788, -4.69134788, -4.69134788,\n",
       "        -5.38449506, -5.38449506, -4.69134788, -5.38449506, -4.69134788,\n",
       "        -4.69134788, -4.69134788, -5.38449506, -4.69134788, -4.69134788,\n",
       "        -4.69134788, -4.69134788, -4.69134788, -4.69134788, -5.38449506,\n",
       "        -4.69134788, -5.38449506, -4.69134788, -4.69134788, -5.38449506,\n",
       "        -4.69134788, -4.69134788, -4.69134788, -5.38449506, -4.69134788,\n",
       "        -5.38449506, -5.38449506, -5.38449506, -4.69134788, -5.38449506,\n",
       "        -5.38449506, -4.69134788, -5.38449506, -4.69134788, -4.69134788,\n",
       "        -4.69134788, -4.69134788, -4.69134788, -5.38449506, -4.69134788,\n",
       "        -4.69134788, -5.38449506, -5.38449506, -5.38449506, -4.69134788,\n",
       "        -4.69134788, -4.69134788, -4.69134788, -5.38449506, -5.38449506,\n",
       "        -5.38449506, -5.38449506, -5.38449506, -5.38449506, -5.38449506,\n",
       "        -4.69134788, -4.69134788, -5.38449506, -5.38449506, -4.69134788,\n",
       "        -4.69134788, -4.69134788, -4.69134788, -4.69134788, -5.38449506,\n",
       "        -5.38449506, -4.69134788, -5.38449506, -4.69134788, -4.69134788,\n",
       "        -4.69134788, -4.69134788, -5.38449506, -4.69134788, -4.69134788,\n",
       "        -5.38449506, -5.38449506, -4.69134788]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.coef_"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}