07_nltk_movie_reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language processing with `nltk`\n",
    "\n",
    "`nltk` is the most popular Python package for Natural Language processing, it provides algorithms for importing, cleaning, pre-processing text data in human language and then apply computational linguistics algorithms like sentiment analysis, "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the movie reviews dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also includes many easy-to-use datasets in the `nltk.compus` package, we can download for example the `movie_reviews` package using the `nltk.download` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package movie_reviews to\n",
      "[nltk_data]     /home/zonca/nltk_data...\n",
      "[nltk_data]   Package movie_reviews is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download(\"movie_reviews\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also list and download other datasets interactively just typing:\n",
    "\n",
    "    nltk.download()\n",
    "    \n",
    "in the Jupyter Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the data have been downloaded, we can import them from `nltk.corpus`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import movie_reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fileids` method provided by all the datasets in `nltk.corpus` gives access to a list of all the files available.\n",
    "\n",
    "In particular in the movie_reviews dataset we have 2000 text files, each of them is a review of a movie, and they are already split in a `neg` folder for the negative reviews and a `pos` folder for the positive reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(movie_reviews.fileids())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['neg/cv000_29416.txt',\n",
       " 'neg/cv001_19502.txt',\n",
       " 'neg/cv002_17424.txt',\n",
       " 'neg/cv003_12683.txt',\n",
       " 'neg/cv004_12641.txt']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_reviews.fileids()[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pos/cv995_21821.txt',\n",
       " 'pos/cv996_11592.txt',\n",
       " 'pos/cv997_5046.txt',\n",
       " 'pos/cv998_14111.txt',\n",
       " 'pos/cv999_13106.txt']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_reviews.fileids()[-5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fileids` can also filter the available files based on their category, which is the name of the subfolders they are located in. Therefore we can have lists of positive and negative reviews separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "negative_fileids = movie_reviews.fileids('neg')\n",
    "positive_fileids = movie_reviews.fileids('pos')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 1000)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(negative_fileids), len(positive_fileids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can inspect one of the reviews using the `raw` method of `movie_reviews`, each file is split into sentences, the curators of this dataset also removed from each review from any direct mention of the rating of the movie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "films adapted from comic books have had plenty of success , whether they're about superheroes ( batman , superman , spawn ) , or geared toward kids ( casper ) or the arthouse crowd ( ghost world ) , but there's never really been a comic book like from hell before . \n",
      "for starters , it was created by alan moore ( and eddie campbell ) , who brought the medium to a whole new level in the mid '80s with a 12-part series called the watchmen . \n",
      "to say moore and campbell thoroughly researched the subject of jack the ripper would be like saying michael jackson is starting to look a little odd . \n",
      "the book ( or \" graphic novel , \" if you will ) is over 500 pages long and includes nearly 30 more that consist of nothing but footnotes . \n",
      "in other words , don't dismiss this film because of its source . \n",
      "if you can get past the whole comic book thing , you might find another stumbling block in from hell's directors , albert and allen hughes . \n",
      "getting the hughes brothers to direct this seems almost as ludicrous as casting carrot top in , well , anything , but riddle me this : who better to direct a film that's set in the ghetto and features really violent street crime than the mad geniuses behind menace ii society ? \n",
      "the ghetto in question is , of course , whitechapel in 1888 london's east end . \n",
      "it's a filthy , sooty place where the whores ( called \" unfortunates \" ) are starting to get a little nervous about this mysterious psychopath who has been carving through their profession with surgical precision . \n",
      "when the first stiff turns up , copper peter godley ( robbie coltrane , the world is not enough ) calls in inspector frederick abberline ( johnny depp , blow ) to crack the case . \n",
      "abberline , a widower , has prophetic dreams he unsuccessfully tries to quell with copious amounts of absinthe and opium . \n",
      "upon arriving in whitechapel , he befriends an unfortunate named mary kelly ( heather graham , say it isn't so ) and proceeds to investigate the horribly gruesome crimes that even the police surgeon can't stomach . \n",
      "i don't think anyone needs to be briefed on jack the ripper , so i won't go into the particulars here , other than to say moore and campbell have a unique and interesting theory about both the identity of the killer and the reasons he chooses to slay . \n",
      "in the comic , they don't bother cloaking the identity of the ripper , but screenwriters terry hayes ( vertical limit ) and rafael yglesias ( les mis ? rables ) do a good job of keeping him hidden from viewers until the very end . \n",
      "it's funny to watch the locals blindly point the finger of blame at jews and indians because , after all , an englishman could never be capable of committing such ghastly acts . \n",
      "and from hell's ending had me whistling the stonecutters song from the simpsons for days ( \" who holds back the electric car/who made steve guttenberg a star ? \" ) . \n",
      "don't worry - it'll all make sense when you see it . \n",
      "now onto from hell's appearance : it's certainly dark and bleak enough , and it's surprising to see how much more it looks like a tim burton film than planet of the apes did ( at times , it seems like sleepy hollow 2 ) . \n",
      "the print i saw wasn't completely finished ( both color and music had not been finalized , so no comments about marilyn manson ) , but cinematographer peter deming ( don't say a word ) ably captures the dreariness of victorian-era london and helped make the flashy killing scenes remind me of the crazy flashbacks in twin peaks , even though the violence in the film pales in comparison to that in the black-and-white comic . \n",
      "oscar winner martin childs' ( shakespeare in love ) production design turns the original prague surroundings into one creepy place . \n",
      "even the acting in from hell is solid , with the dreamy depp turning in a typically strong performance and deftly handling a british accent . \n",
      "ians holm ( joe gould's secret ) and richardson ( 102 dalmatians ) log in great supporting roles , but the big surprise here is graham . \n",
      "i cringed the first time she opened her mouth , imagining her attempt at an irish accent , but it actually wasn't half bad . \n",
      "the film , however , is all good . \n",
      "2 : 00 - r for strong violence/gore , sexuality , language and drug content \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(movie_reviews.raw(fileids=positive_fileids[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize text in words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "romeo_text = \"\"\"Why then, O brawling love! O loving hate!\n",
    "O any thing, of nothing first create!\n",
    "O heavy lightness, serious vanity,\n",
    "Misshapen chaos of well-seeming forms,\n",
    "Feather of lead, bright smoke, cold fire, sick health,\n",
    "Still-waking sleep, that is not what it is!\n",
    "This love feel I, that feel no love in this.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step in Natural Language processing is generally to split the text into words, this process might appear simple but it is very tedious to handle all corner cases, see for example all the issues with punctuation we have to solve if we just start with a split on whitespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Why',\n",
       " 'then,',\n",
       " 'O',\n",
       " 'brawling',\n",
       " 'love!',\n",
       " 'O',\n",
       " 'loving',\n",
       " 'hate!',\n",
       " 'O',\n",
       " 'any',\n",
       " 'thing,',\n",
       " 'of',\n",
       " 'nothing',\n",
       " 'first',\n",
       " 'create!',\n",
       " 'O',\n",
       " 'heavy',\n",
       " 'lightness,',\n",
       " 'serious',\n",
       " 'vanity,',\n",
       " 'Misshapen',\n",
       " 'chaos',\n",
       " 'of',\n",
       " 'well-seeming',\n",
       " 'forms,',\n",
       " 'Feather',\n",
       " 'of',\n",
       " 'lead,',\n",
       " 'bright',\n",
       " 'smoke,',\n",
       " 'cold',\n",
       " 'fire,',\n",
       " 'sick',\n",
       " 'health,',\n",
       " 'Still-waking',\n",
       " 'sleep,',\n",
       " 'that',\n",
       " 'is',\n",
       " 'not',\n",
       " 'what',\n",
       " 'it',\n",
       " 'is!',\n",
       " 'This',\n",
       " 'love',\n",
       " 'feel',\n",
       " 'I,',\n",
       " 'that',\n",
       " 'feel',\n",
       " 'no',\n",
       " 'love',\n",
       " 'in',\n",
       " 'this.']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "romeo_text.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`nltk` has a sophisticated word tokenizer trained on English named `punkt`, we first have to download its parameters: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/zonca/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download(\"punkt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can use the `word_tokenize` function to properly tokenize this text, compare to the whitespace splitting we used above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "romeo_words = nltk.word_tokenize(romeo_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Why',\n",
       " 'then',\n",
       " ',',\n",
       " 'O',\n",
       " 'brawling',\n",
       " 'love',\n",
       " '!',\n",
       " 'O',\n",
       " 'loving',\n",
       " 'hate',\n",
       " '!',\n",
       " 'O',\n",
       " 'any',\n",
       " 'thing',\n",
       " ',',\n",
       " 'of',\n",
       " 'nothing',\n",
       " 'first',\n",
       " 'create',\n",
       " '!',\n",
       " 'O',\n",
       " 'heavy',\n",
       " 'lightness',\n",
       " ',',\n",
       " 'serious',\n",
       " 'vanity',\n",
       " ',',\n",
       " 'Misshapen',\n",
       " 'chaos',\n",
       " 'of',\n",
       " 'well-seeming',\n",
       " 'forms',\n",
       " ',',\n",
       " 'Feather',\n",
       " 'of',\n",
       " 'lead',\n",
       " ',',\n",
       " 'bright',\n",
       " 'smoke',\n",
       " ',',\n",
       " 'cold',\n",
       " 'fire',\n",
       " ',',\n",
       " 'sick',\n",
       " 'health',\n",
       " ',',\n",
       " 'Still-waking',\n",
       " 'sleep',\n",
       " ',',\n",
       " 'that',\n",
       " 'is',\n",
       " 'not',\n",
       " 'what',\n",
       " 'it',\n",
       " 'is',\n",
       " '!',\n",
       " 'This',\n",
       " 'love',\n",
       " 'feel',\n",
       " 'I',\n",
       " ',',\n",
       " 'that',\n",
       " 'feel',\n",
       " 'no',\n",
       " 'love',\n",
       " 'in',\n",
       " 'this',\n",
       " '.']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "romeo_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good news is that the `movie_reviews` corpus already has direct access to tokenized text with the `words` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['films', 'adapted', 'from', 'comic', 'books', 'have', ...]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_reviews.words(fileids=positive_fileids[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a bag-of-words model\n",
    "\n",
    "The simplest model for analyzing text is just to think about text as an unordered collection of words (bag-of-words). This can generally allow to infer from the text the category, the topic or the sentiment.\n",
    "\n",
    "From the bag-of-words model we can build features to be used by a classifier, here we assume that each word is a feature that can either be `True` or `False`.\n",
    "We implement this in Python as a dictionary where for each word in a sentence we associate `True`, if a word is missing, that would be the same as assigning `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_bag_of_words_features(words):\n",
    "    return {word:True for word in words}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'!': True,\n",
       " ',': True,\n",
       " '.': True,\n",
       " 'Feather': True,\n",
       " 'I': True,\n",
       " 'Misshapen': True,\n",
       " 'O': True,\n",
       " 'Still-waking': True,\n",
       " 'This': True,\n",
       " 'Why': True,\n",
       " 'any': True,\n",
       " 'brawling': True,\n",
       " 'bright': True,\n",
       " 'chaos': True,\n",
       " 'cold': True,\n",
       " 'create': True,\n",
       " 'feel': True,\n",
       " 'fire': True,\n",
       " 'first': True,\n",
       " 'forms': True,\n",
       " 'hate': True,\n",
       " 'health': True,\n",
       " 'heavy': True,\n",
       " 'in': True,\n",
       " 'is': True,\n",
       " 'it': True,\n",
       " 'lead': True,\n",
       " 'lightness': True,\n",
       " 'love': True,\n",
       " 'loving': True,\n",
       " 'no': True,\n",
       " 'not': True,\n",
       " 'nothing': True,\n",
       " 'of': True,\n",
       " 'serious': True,\n",
       " 'sick': True,\n",
       " 'sleep': True,\n",
       " 'smoke': True,\n",
       " 'that': True,\n",
       " 'then': True,\n",
       " 'thing': True,\n",
       " 'this': True,\n",
       " 'vanity': True,\n",
       " 'well-seeming': True,\n",
       " 'what': True}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build_bag_of_words_features(romeo_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is what we wanted, but we notice that also punctuation like \"!\" and words useless for calssification purposes like \"of\" or \"that\" are also included.\n",
    "Those words are named \"stopwords\" and `nltk` has a convenient corpus we can download:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /home/zonca/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download(\"stopwords\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the Python `string.punctuation` list and the English stopwords we can build better features by filtering out those words that would not help in the classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "useless_words = nltk.corpus.stopwords.words(\"english\") + list(string.punctuation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_bag_of_words_features_filtered(words):\n",
    "    return {\n",
    "        word:1 for word in words \\\n",
    "        if not word in useless_words}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Feather': 1,\n",
       " 'I': 1,\n",
       " 'Misshapen': 1,\n",
       " 'O': 1,\n",
       " 'Still-waking': 1,\n",
       " 'This': 1,\n",
       " 'Why': 1,\n",
       " 'brawling': 1,\n",
       " 'bright': 1,\n",
       " 'chaos': 1,\n",
       " 'cold': 1,\n",
       " 'create': 1,\n",
       " 'feel': 1,\n",
       " 'fire': 1,\n",
       " 'first': 1,\n",
       " 'forms': 1,\n",
       " 'hate': 1,\n",
       " 'health': 1,\n",
       " 'heavy': 1,\n",
       " 'lead': 1,\n",
       " 'lightness': 1,\n",
       " 'love': 1,\n",
       " 'loving': 1,\n",
       " 'nothing': 1,\n",
       " 'serious': 1,\n",
       " 'sick': 1,\n",
       " 'sleep': 1,\n",
       " 'smoke': 1,\n",
       " 'thing': 1,\n",
       " 'vanity': 1,\n",
       " 'well-seeming': 1}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build_bag_of_words_features_filtered(romeo_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting frequencies of words\n",
    "\n",
    "It is common to explore a dataset before starting the analysis, in this section we will find the most common words and plot their frequency.\n",
    "\n",
    "Using the `.words()` function with no argument we can extract the words from the entire dataset and check that it is about 1.6 millions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_words = movie_reviews.words()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.58382"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(all_words)/1e6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we want to filter out `useless_words` as defined in the previous section, this will reduce the lenght of the dataset by more than a factor of 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "filtered_words = [word for word in movie_reviews.words() if not word in useless_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.710579"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(filtered_words)/1e6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `collection` package of the standard library contains a `Counter` class that is handy for counting frequencies of words in our list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "counter = Counter(filtered_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also has a `most_common()` method to access the words with the higher count:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "most_common_words = counter.most_common()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('film', 9517),\n",
       " ('one', 5852),\n",
       " ('movie', 5771),\n",
       " ('like', 3690),\n",
       " ('even', 2565),\n",
       " ('good', 2411),\n",
       " ('time', 2411),\n",
       " ('story', 2169),\n",
       " ('would', 2109),\n",
       " ('much', 2049)]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_common_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we would like to have a visualization of this using `matplotlib`.\n",
    "\n",
    "First we want to use the Jupyter magic function\n",
    "\n",
    "    %matplotlib inline\n",
    "    \n",
    "to setup the Notebook to show the plot embedded into the Jupyter Notebook page, you can also test:\n",
    "\n",
    "    %matplotlib notebook\n",
    "    \n",
    "for a more interactive plotting interface which however is not as well supported on all platforms and browsers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can sort the word counts and plot their values on Logarithmic axes to check the shape of the distribution. This visualization is particularly useful if comparing 2 or more datasets, a flatter distribution indicates a large vocabulary while a peaked distribution a restricted vocabulary often due to a focused topic or specialized language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEOCAYAAACTqoDjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VdW9xvHv75wQAiSEeUpAwAAaGSWKAlUQVCggVr1V\nbG21Fq5VHG+tdtAOt4PD1arVllKrdLAqxSrghFIFFEEJyoxAQIEwJUwBAiQkWfePc9SYnsAJyck+\n++T9PA8P2fvsvfNjibxZe629lznnEBERqSrgdQEiIhKfFBAiIhKRAkJERCJSQIiISEQKCBERiUgB\nISIiESkgREQkIgWEiIhEpIAQEZGIFBAiIhJRktcF1EabNm1c165dvS5DRMRXli5duts51/ZEx/k6\nILp27Upubq7XZYiI+IqZbY7mON1iEhGRiBQQIiISUVwFhJk1M7OlZjbW61pERBq6mAaEmT1lZgVm\ntqrK/lFmts7M8szs7kof3QVMj2VNIiISnVj3IKYBoyrvMLMg8AQwGsgGJphZtpmNBNYAu2Jck4iI\nRCGms5iccwvMrGuV3WcDec65TQBm9hwwHkgFmhEKjSNm9qpzriKW9YmISPW8mOaaAWyttJ0PDHLO\nTQYws2uB3dWFg5lNAiYBdOnSJbaViog0YF4MUluEfZ8vjO2cm+ace7m6k51zU51zOc65nLZtT/ic\nh4iInCQvAiIf6FxpOxPY7kEdIiJyHF4ExBKgh5l1M7Nk4CpgVk0uYGbjzGxqUVFRTAoUEZHYT3N9\nFlgE9DKzfDO73jlXBkwG5gBrgenOudU1ua5zbrZzblJ6enrdFy0iIkDsZzFNqGb/q8CrJ3tdMxsH\njMvKyjrZS4iIyAnE1ZPU0VIPQkQk9nwZECIiEnu+DAgNUouIxJ4vA0K3mEREYs+XASEiIrHnyxXl\najuL6R/vb+FvizeTFDCCAfvi96ARsM+2A6Hfg/Yfx4W+DnxpX7c2zbhiYCZmkR4UFxHxH18GhHNu\nNjA7Jydn4smc37xJEpktm1Be4SircJRXVFBW7ig5VhHe/uJXWUVFpeNCv1d8aTt0blmFY8GG3Tx4\nRV9SGgXr+E8sIlL/fBkQtTW2byfG9u1UZ9dzzvGH+Rt5cM46Nu8pZuo1OXRIT6mz64uIeEFjEHXA\nzLhxWBZTr8lhY8EhLnn8XZZv3e91WSIiteLLgIjXaa4XZrfnhRsHk5wU4Ot/XMTMZdu8LklE5KT5\nMiDieZrraR2aM/OmIfTLbMGtzy3joTfWUVHhTnyiiEic8WVAxLvWqY35+3cHcdVZnfndW3l875ml\nFJeUeV2WiEiNKCBiJDkpwG8u68O9Y7N5c80uLv/De+TvO+x1WSIiUVNAxJCZ8Z2h3Xj6urPZtv8I\n4x9fSO6ne70uS0QkKr4MiHgdpK7O+T3b8uKNQ0hLSWLCnxbzz9ytJz5JRMRjvgyIeB6krk5Wu1Re\numkIg7q15s4ZK3jg9Y9xToPXIhK/fBkQftWiaTLTrjuLCWd34ffzNvLTWas1w0lE4laDfJLaS0nB\nAL/+Wm/SUpKYumATxSXl3H95H5KCymoRiS8KCA+YGT8cfRqpjZN4+M31HC4t49GrBpCcpJAQkfih\nf5E8YmbcMqIH94zN5rVVO5n0t1yOHiv3uiwRkc/5MiD8NovpeK4f2o37LuvD/PWFfPupDzikB+pE\nJE74MiD8OIvpeK46uwuPXjWApZv38Y0n32f/4VKvSxIR8WdAJKJL+nViyjcHsnbHAa6aupjCgyVe\nlyQiDZz5eS5+Tk6Oy83N9bqMOrUwbzff/UsuZtA3M51+mS3om9mCfp3TyWjRRCvWiUitmdlS51zO\niY7TLKY4MySrDf+84Vym525l+db9PL3wU0rLKwBo3SyZ7E7NaZr85RXrDOO8nm25YmCmZkKJSJ1R\nDyLOlZSVs27nQZZv3c/y/CI+3nmAsvIv/zc7XFrOlr2H6dyqCbeO6Mml/TvpuQoRqVa0PQgFRAJw\nzjFvfSEPvbGOVdsO0L1NM777le70yUjn1HbNaJqsjqKIfEG3mBoQM2N4r3YM69mWN9bs4uE31vOj\nF1d+/nlGiyb0aJ9KVtvU0O/tUslqm0Z600YeVi0i8U4BkUDMjIvP6MCFp7dn0+5D5BUcYsOuQ+QV\nhn5ftHEPJWUVnx/fLq0xWe1S6dEuFBqntG5G69RkWjVLpn1aCoGABsRFGjJf3mIys3HAuKysrIkb\nNmzwuhzfKK9wbNt3hA0FB0PhURAKkbyCQ//xgF7T5CCndUgjo2VTkoMB2qY1ZkhWa3JOaUWTKoPk\nIuIvGoOQqDnn2HWghC17D7O3uJTdh0rIKzjEmh0H2H2whJKyCgoOHuVYucMMOjZPoWubZnRr04yv\n9unI4FNba/qtiI9oDEKiZmZ0SE+hQ3pKtccUl5Tx/id7WJl/gE/3FPPJ7mJmLdvOM+9voX/nFkwe\nnsWI09spKEQSiHoQctKOHitnxtJ8pszfSP6+I5zWIY0bh2cxpk9Hghq/EIlbusUk9aasvIJZy7fz\n+3kbySs4RNfWTbliYCZDstrQJyNdz2SIxBkFhNS7igrHG2t2MnXBJj7csh+AtMZJDOremuyOaXy1\nb0dO69Dc4ypFRAEhntpbXMp7G3ezMG83izftZfOeYgAuOzOTb517Cn0zW3hcoUjDpYCQuLL/cCmP\nzN3A80u2cuRYOW1Sk0kOBshs2ZRrh3Tlq306el2iSIOhgJC4dODoMV76aBtrth+grMLx0ZZ9bCws\n5r8GZnLvuGzSUvR0t0isaZqrxKXmKY341rldP98uK6/g0X9v4PG385i5fDu92qdxy4gejNSUWRHP\nqQchcWFlfhGzlm/j7XWFn8+EmnTeqVw9qIvXpYkkHN/dYjKz04FbgTbAv51zfzjROQqIxFNaVsFL\nH23j+dytLN28jzF9OnLXqNPIaNlEz1aI1JG4CAgzewoYCxQ453pX2j8KeBQIAk865+6r9FkA+JNz\n7voTXV8BkbjKKxxT5m/k0bkbKC2vIC0liXvGZDO6TweNU4jUUrwExHnAIeCvnwWEmQWB9cCFQD6w\nBJjgnFtjZpcAdwOPO+f+caLrKyAS36bCQyzatIfpS7ayPL8IgKSA0aVVU8b3z2B0nw70bJ/mcZUi\n/hIXAREupCvwcqWAOBf4mXPu4vD2DwGcc7+pdM4rzrkxJ7q2AqLhKK9w5H66l9zN+zhUUsbSzfv4\n4JO9BAPGbSN68L1hp+qJbZEoxfMspgxga6XtfGCQmQ0DLgMaA69Wd7KZTQImAXTpogHMhiIYMAZ1\nb82g7q0/31dw8Cg3PfMhD725noKDJdw7LptGCgmROuNFQEQaaXTOuXnAvBOd7JybCkyFUA+iTisT\nX2mXlsI/bxjMjc8s5W+LN/Nu3m5ObduMzq2aMrp3R/p1TqdxktauEDlZXgREPtC50nYmsL0mF6i0\nYFBd1iU+9cTVZzJ7xQ5eWJpP/r4jLNiwm6cXfkqb1GS+NiCDUb070qtDGqmN9diPSE14MQaRRGiQ\negSwjdAg9dXOudU1vbbGICSSfcWlvLZqJ7OXb2fpln2UhpdZbd+8Mf07t+CXl/ahbVpjj6sU8U5c\nDFKb2bPAMELPNuwCfuqc+7OZfRV4hNA016ecc786mesrIORE9h8uZfGmPWwsLGb9roO8unIHZRWO\nnu3SGNS9FeP7d2JA55Zaf1salLgIiFjRmtRyslZtK+KNNbtYvnU/izbtobSsgowWTbj/8r4M7dHG\n6/JE6kVCB8Rn1IOQ2ig4cJT56wv5w/yNbCos5s6Le3HD+afqiW1JeAoIkSgdPVbON558n6Wb99Gj\nXSrfOvcU/iunMymNNANKElO0AeHLSeNmNs7MphYVFXldiiSAlEZBnp14Dr/+Wh+SggHumbma0Y++\nw4yl+RwqKfO6PBHPqAchUklFhWPm8m387q08NhUW07JpI+4Zm834/hm69SQJQ7eYRGrBOcfiTXu5\n/fll7DxwlFNaN+XBK/pxdrdWXpcmUmsJHRCaxST15Vh5BXNW7+TuF1ZyqKSMb57ThUv7Z9A7I11j\nFOJbCR0Qn1EPQurL5j3F3PXCChZv2gtAauMkbhqexcSvdNNLAsV3FBAiMbD7UAmLN+3hXx9u462P\nC+jQPIVvD+7KDed31xKp4hsJPYtJxCttUhsztm8n/vztHKZeM5DOrZpw/+sfc8f05ZSUlXtdnkid\n8mVAaJqreM3MuOiMDkz/73O55YIsXvxoG1f8YREfbtnndWkidcaXAeGcm+2cm5Senu51KdLAmRl3\nXNSLBy7vS/6+w3x9yiKeeDuPo8fUmxD/0xiESB0pOnKMu2as4PXVO+mUnsLXz+rM2L6dyGqX6nVp\nIl+iQWoRj7yXt5vfvPYxK7eFboGO6duRywZkMLxXO701VuKCAkLEY4UHS3h64Sc8+c4nlJZXkNUu\nlStzOnPpgAytRyGeSuiA0INy4ieHS8t4fdVO/rZ4Mx9t2U9yMMAPRvXiO0O6qUchnkjogPiMehDi\nN+t3HeSB19cxd+0u+nVuwffO787FZ3TQMxRSr/QchEgc6tk+jT99ayD3jM1mb3EJN/z9Q66btoS9\nxaVelybyHxQQIvXMzLh+aDfe/p9h3Ds2m0Ub93DxIwuYuWwb5RX+7dFL4lFAiHgkKRjgO0O78cL3\nBtOheQq3PreMr/1+Iet2HvS6NBFAASHiud4Z6bx00xAevKIv+fuOMOaxd5gyfyNHSvWwnXjLl4PU\nmsUkiarwYAk//NdK5q7dRatmyXzr3FP47/NOpUmyXi0udUezmER8yjnHgg27efiNdSzPL6J3RnPu\nHXsGZ3VtqdlOUic0i0nEp8yM83u2ZebkofxuwgC27TvC1/+4iJuf/YjCgyVelycNiAJCJI6N69eJ\nd+66gDsu7MkrK3cw7MG3mblsG6VlFV6XJg2AAkIkzqU2TuKWET145eav0D4822nwfW/xyNz1GsiW\nmFJAiPhEdqfmvH7beUy9ZiC9M5rzyNwNXPv0B2zde9jr0iRBaZBaxKem527lZ7NWU1buGN2nAxdl\nd2B07w56v5OcULSD1En1UYyI1L2v53Tm3O6tefjN9byxeiczl20nu2Nzfnf1AE5tqzUopPbUgxBJ\nAGXlFTyfu5X/fXkNATOuOecUbhvZU89PSEQJPc1Va1KLfFlSMMA3Bp3CzJuG8pUebfjjgk1c/MgC\nXl+1k7JyzXiSk6MehEgCemXFDu6csZzDpeW0SW3M7Rf24MqcziQFffkzodSxhO5BiMjxjenbkcU/\nGsFPx2UD8OMXV3H1k+9TobfFSg0oIEQSVPOURlw3pBsf/GgE3zynCx98spfLp7xHXsEhr0sTn1BA\niCS4QMD4xSW9+eY5Xfhoy35GPjyf38/Lw8+3l6V+KCBEGoBAwPjlpX2Ye8d5dG/bjAdeX8fEvy6l\npExPYkv1FBAiDUhWuzTm3n4+E87uzNy1uxj24Dzey9vtdVkSp6J6UM7M7jje5865h+umHBGJtUDA\n+M1lfcnu2Jx7Zq7m6iff59rBXfnBqF40Tdazs/KFaHsQOcD3gIzwrxuAbCAt/EtEfOaac7vy9veH\n0TujOdPe+5ScX85lRf5+r8uSOBLVcxBm9gZwuXPuYHg7Dfinc25UjOs7Lj0HIVI3Zi7bxq3PLQPg\nhvNP5QcX99I7nRJYXT8H0QUorbRdCnQ9ibpEJA6N75/BrMlD6Nk+lSnzN3Lp7xeybf8Rr8sSj0Ub\nEH8DPjCzn5nZT4H3gb/WdTFmdqmZ/cnMZprZRXV9fRGpXt/MFsy57TxuHdGDFflFDHvwbeau2eV1\nWeKhqALCOfcr4DpgH7AfuM459+tozjWzp8yswMxWVdk/yszWmVmemd0d/j4vOecmAtcCV9bgzyEi\ndcDMuP3Cnrx442AaJwX57l9zuXfmKj2B3UDVZJprU+CAc+5RIN/MukV53jTgS2MVZhYEngBGExrs\nnmBm2ZUO+Un4cxHxwIAuLVnwg+H0yUjnr4s2c85v/q3psA1QVAERvq10F/DD8K5GwN+jOdc5twDY\nW2X32UCec26Tc64UeA4YbyH3A6855z6M5voiEhutmiUz++ahfP+inhQcLOHqJ99n3O/epejwMa9L\nk3oSbQ/ia8AlQDGAc247tZvemgFsrbSdH953MzASuMLMboh0oplNMrNcM8stLCysRQkiEo3JF/Qg\n9ycj6ZeZzsptRQx/aB7vbND/ew1BtAFR6kLzYR2AmTWr5feNNH/OOecec84NdM7d4JybEulE59xU\n51yOcy6nbdu2tSxDRKLRJrUxMycP5aH/6sfe4lKu+fMHfPcvSyg6ot5EIos2IKab2R+BFmY2EZgL\n/KkW3zcf6FxpOxPYHu3JWjBIxBuXD8zk3buGEwwYc9cW0O/nb2hsIoFFvWCQmV0IXETop/85zrk3\no/4mZl2Bl51zvcPbScB6YASwDVgCXO2cW12T4vWgnIg3Kiocv3h5DdPe+xSA0b078PjVZxLUw3W+\nUGcPyplZ0MzmOufedM7d6Zz7fg3D4VlgEdDLzPLN7HrnXBkwGZgDrAWm1zQcRMQ7gYDxs0vO4NmJ\n5xAMGK+t2kmvn7zG+l0HvS5N6lC0r9qYBVzjnIuLezpmNg4Yl5WVNXHDhg1elyPSoJVXOL791Ae8\nG77VdN9lfbjq7C4eVyXHE20PItqAmA6cA7xJeCYTgHPultoUWVu6xSQSP6bnbuUHM1YAcF7Ptky9\nZiApjYIeVyWR1HVAfDvSfufcX06itlpTD0IkPq3beZCLH1nw+fa87w+ja5vaTnqUulYnAWFmXZxz\nW+q0sjqkHoRI/Ckrr+DW55fxyoodAMy57Tx6ddCqAPGkrgapX6p0wRdqXZWIJLykYIAnrj6T20b2\nAODiRxawbKvWmfCjEwVE5Tlr3WNZiIgklttG9uSOC3sCcOkTC3ni7TyPK5KaOlFAuGq+9pQelBPx\nh1tG9ODRq/oD8OCcdVz02/mUlVd4XJVE60RjEOWEZi0Z0AQ4/NlHhF6N0TzmFR6HxiBE/GFH0RGG\n3v825eHXhi//6UWkN2nkcVUNV52MQTjngs655s65NOdcUvjrz7Y9DQcR8Y+O6U1Y/8vR9MlIB6Df\nz9/QuIQP1GQ9CBGRkxYMGDNvGsI3BoUeorv0iYX8bNZqon3dj9Q/XwaExiBE/CkQMH71tT48fd1Z\nAEx771OG/988DpWUeVyZROLLgHDOzXbOTUpPT/e6FBE5CcN7tWPJj0fSKGh8uucwvX86h9xPq64r\nJl7zZUCIiP+1TWvM2l+M4tL+nQC4YsoiXl4R9Vv/pR4oIETEM0nBAI9cNYD7L+8DwOR/fMST72zy\nuCr5jAJCRDx35VldmPLNgQD88pW13PLsR1RUaPDaa74MCA1SiySeUb078MotQwGYtXw7ox99RyHh\nMV8GhAapRRLTGZ3SWXbvhZjBul0HGfC/b3LgqNa99oovA0JEEleLpsms/vnFZLZsQtGRY+T871wO\nl2oarBcUECISd5omJzH/zuGc1iGN0vIKsu+dw9a9h098otQpBYSIxKVgwJh981AGntISgK888Lae\nlahnCggRiVuNggFm3HAut4wIrS1xxZRFzFy2zeOqGg5fBoRmMYk0HGbGHRf25BfjzwDg1ueWcd9r\nH3tcVcPgy4DQLCaRhudb53bl6WtD73CaMn8jtz+/zOOKEp8vA0JEGqbhp7VjwZ3DAXjxo2189y9L\ntABRDCkgRMRXurRuyqIfXkDAYO7aAi56ZAFFh/WsRCwoIETEdzqmN+G9u0fQsmkjNhUWM+T+tyg8\nWOJ1WQlHASEivtQhPYV53x/O6R2bc6ikjBEPzWP7/iNel5VQFBAi4lvpTRvx4o2D6ZORzoGjZYx8\neD6f7i72uqyEoYAQEV9LaRRkxvfOJeeUlhwuLWfkw/PZskdPXdcFXwaEnoMQkcoaJwV5ZuIgzu/Z\nlrIKx9VPLmZn0VGvy/I9XwaEnoMQkaoaJwX54zUDueC0duTvO8Lg+/6tkKglXwaEiEgkKY2CPHpV\nf/7nwp5UODhP72+qFQWEiCSUtJRG/Pf5p/L9i3pSWl7B//xzOS8szfe6LF9SQIhIwklOCnDT8Cwm\nnN2FfcWlPDEvjzmrd3pdlu8oIEQkIZkZv7msD+P7Z7CpsJgbn/mQg1qdrkYUECKS0H4x/gx+Ni6b\n8gpH/1+8yertmv0YLQWEiCQ0M+OygZncPrIn5RWOB+es48Mt+7wuyxcUECKS8JqnNGLSed3p2T6V\nBesL+fO7n3hdki8oIESkQWiSHOSN28/nzC4teWXFDs761VyKjmhM4ngUECLSoNx5cS/G9OlI4cES\nXlu5g4IDepiuOnETEGbW3cz+bGYzvK5FRBLXoO6tmXRedwDu/tdKJv1tqccVxa+YBoSZPWVmBWa2\nqsr+UWa2zszyzOxuAOfcJufc9bGsR0QEoG9mOi/fPJRhvdqyeU8x/8zdyo4ivSq8qlj3IKYBoyrv\nMLMg8AQwGsgGJphZdozrEBH5nJnROyOdM7u0ZN/hY9w5YwX/N2e912XFnZgGhHNuAVD1RShnA3nh\nHkMp8BwwPpZ1iIhEcvMFWbx713B6tk8lr+Agr6/awZ5DWpnuM16MQWQAWytt5wMZZtbazKYAA8zs\nh9WdbGaTzCzXzHILCwtjXauIJDAzI7NlU7q0asby/CJu+PuHPPD6Oq/LihtJHnxPi7DPOef2ADec\n6GTn3FRgKkBOTo6r49pEpAF6bEJ/Nu85zE3PfEjhoRKKS8pomhzELNI/Vw2HFz2IfKBzpe1MYLsH\ndYiIANA0OYnTOzanTVpj3vq4gDN+Oodfv7rW67I850VALAF6mFk3M0sGrgJm1eQCWlFORGLhnjHZ\n/Oirp9EpPYUNBYe8LsdzsZ7m+iywCOhlZvlmdr1zrgyYDMwB1gLTnXOra3JdrSgnIrHQJzOdSeed\nSudWTdmw6xAPv7memcu2eV2WZ2I6BuGcm1DN/leBV0/2umY2DhiXlZV1spcQEalWn4x03v9kL4/9\newPBgHFJv04Ncjwibp6krgn1IEQkln4yNptP7xvDD0b1orzCUVJW4XVJnvBiFpOIiC80aRQE4K4X\nVtAoGODKszpzVtdWHldVf3wZELrFJCL1oV/nFnRt3ZTcT/ex88BRyitcgwoI3WISEanGmV1aMu/O\n4Sy8+wJObduMo8fKvS6pXvmyByEiUt8aJwXZtv8I/167i2DAOKd7a1LCt6ASlS97EHoOQkTqW5vU\nZFbkF3H9X3K59uklTM/deuKTfM6XAaFbTCJS3x6bMIBZk4fw4o2DASg6nPir0ekWk4hIFNJSGtE3\nswXOOcygtDzxp74qIEREasDMaJwUYNW2Iv71YT4AA09pySmtm3lcWd3zZUBomquIeKlD8xTeXlfI\n2+tCSw6c37Mtf/nO2R5XVfd8GRDOudnA7JycnIle1yIiDc/Lt3zl84WFbn9+GYdLyzyuKDZ8GRAi\nIl5KbZxEauPQP5+pKY0oOpKYA9a+nMUkIhIvkoPGsQR9V5N6ECIitdAoGGBH0RF+9cqaz/cN79WO\nwVltPKyqbvgyIDRILSLxol/nFsxfX8gz728B4OixclbkFyVEQJhz/l3WOScnx+Xm5npdhojI5675\n8/sUl5TxrxuHeF1KtcxsqXMu50THaQxCRKQOBQNGWYV/f/CuTAEhIlKHkgIBjpUrIEREpIqkgFFe\nkRizmnw5SC0iEq+Sgsb+w8eYvXz7l/b36pBGz/ZpHlV1cnwZEJrFJCLxqk1qYwoOlnDzsx99aX9W\nu1Tm3nG+R1WdHM1iEhGpQ8fKK9i8p/hL+x54fR2rtx9g4d0XeFTVl0U7i8mXPQgRkXjVKBggq92X\nbyW1aNqIMh+OS2iQWkQkxoKBAH5cPkIBISISY8EAVPjwdr4CQkQkxoJmlPvw4TkFhIhIjAUCCggR\nEYkgyacBoVlMIiIxFggYJWXl/PjFlRE/H5LVhq/26VjPVZ2YLwNCD8qJiJ/0z2xB69TGzFm98z8+\nO3CkjNxP98VlQOhBORERD930zIes33WQN+vxKWu97ltExA8sfqfAKiBERDwUMCNO80EBISLipYB6\nECIiEknAjHidAauAEBHxkKkHISIikWgMQkREItIYhIiIRBQag1BAiIhIFRbHg9Rx86oNM2sG/B4o\nBeY5557xuCQRkZgLGMTrGy1i2oMws6fMrMDMVlXZP8rM1plZnpndHd59GTDDOTcRuCSWdYmIxIvQ\nLCavq4gs1reYpgGjKu8wsyDwBDAayAYmmFk2kAlsDR9WHuO6RETiQjyPQcT0FpNzboGZda2y+2wg\nzzm3CcDMngPGA/mEQmIZGhsRkQYiYEZxSRl3PL+sRuddeVZnBnVvHaOqQrwYg8jgi54ChIJhEPAY\n8LiZjQFmV3eymU0CJgF06dIlhmWKiMTeWV1b8dbHBSzZvLdG5404vX2MKvqCFwFhEfY551wxcN2J\nTnbOTQWmQuh133Vcm4hIvRrTtyNj+sbfWhDgza2cfKBzpe1MYHtNLmBm48xsalFRUZ0WJiIiX/Ai\nIJYAPcysm5klA1cBs2pyAefcbOfcpPT09JgUKCIisZ/m+iywCOhlZvlmdr1zrgyYDMwB1gLTnXOr\nY1mHiIjUXKxnMU2oZv+rwKsne12tSS0iEnu+nE6qW0wiIrHny4AQEZHY82VAaBaTiEjs+TIgdItJ\nRCT2LF7fIhgNMysE9gOVuxLpx9mu/HUbYHcdllP1+9b2+ON9Humz4/25q26rHSJve9kW0Rxb3THR\ntEPVfWqHyNvx0g7RHF+Tdqi6/xTnXNsTVuCc8/UvYGq021W+zo1lHbU9/nifR/pM7VDzdointojm\n2OqOiaYdjvd3oCG3w/Haxa//b9R0//F++fIWUxVV39t0vO1q3/EUgzpqe/zxPo/0mdoh8r6abtel\nmlw7mmOrOyaadqi6T+0QeTte2iGa42vSDifz/f19i6k2zCzXOZfjdR1eUzt8QW0RonYIUTv4dJC6\njkz1uoA4oXb4gtoiRO0Q0uDbocH2IERE5Pgacg9CRESOQwEhIiIRKSBERCQiBUSYmTUzs7+Y2Z/M\n7Bte1+OqSNEuAAAFGElEQVQVM+tuZn82sxle1+IlM7s0/Hdhppld5HU9XjGz081sipnNMLPveV2P\n18L/Tiw1s7Fe11IfEjogzOwpMysws1VV9o8ys3Vmlmdmd4d3XwbMcM5NBC6p92JjqCbt4Jzb5Jy7\n3ptKY6uG7fBS+O/CtcCVHpQbMzVsh7XOuRuArwMJN+Wzhv9GANwFTK/fKr2T0AEBTANGVd5hZkHg\nCWA0kA1MMLNsQkufbg0fVl6PNdaHaUTfDolsGjVvh5+EP08k06hBO5jZJcC7wL/rt8x6MY0o28LM\nRgJrgF31XaRXEjognHMLgL1Vdp8N5IV/Ui4FngPGE1orOzN8TEK1Sw3bIWHVpB0s5H7gNefch/Vd\nayzV9O+Dc26Wc24wkHC3XmvYFsOBc4CrgYlmllD/TkQS0xXl4lQGX/QUIBQMg4DHgMfNbAyxfdw+\nXkRsBzNrDfwKGGBmP3TO/caT6upPdX8fbgZGAulmluWcm+JFcfWour8Pwwjdfm1MLVaB9JmIbeGc\nmwxgZtcCu51zFR7UVq8aYkBYhH3OOVcMXFffxXiounbYA9xQ38V4qLp2eIzQDw0NRXXtMA+YV7+l\neC5iW3z+hXPT6q8UbyV8FymCfKBzpe1MYLtHtXhJ7RCidghRO3xBbRHWEANiCdDDzLqZWTJwFTDL\n45q8oHYIUTuEqB2+oLYIS+iAMLNngUVALzPLN7PrnXNlwGRgDrAWmO6cW+1lnbGmdghRO4SoHb6g\ntjg+vaxPREQiSugehIiInDwFhIiIRKSAEBGRiBQQIiISkQJCREQiUkCIiEhECghpMMzst2Z2W6Xt\nOWb2ZKXth8zsjlpc/2dm9v1q9m8zs2VmtsbMJtTiewwzs5dP9nyRmlBASEPyHjAYIPwmzjbAGZU+\nHwwsjOZC4VdC18RvnXP9Cb0V9I9m1qiG54vUOwWENCQLCQcEoWBYBRw0s5Zm1hg4Hfgo/KrvB81s\nlZmtNLMr4fOf3t82s38AK8P7fhxeWGYu0OtEBTjnNgCHgZbh8yea2RIzW25mL5hZ0/D+aWb2mJm9\nZ2abzOyKqtcys7PM7CMz617bhhGJpCG+zVUaKOfcdjMrM7MuhIJiEaFXO58LFAErnHOlZnY50B/o\nR6iXscTMFoQvczbQ2zn3iZkNJPSengGE/l/6EFh6vBrM7Exgg3OuILzrX865P4U/+yVwPfC78Gcd\ngaHAaYTeBTSj0nUGh48b75zbcrJtInI8CghpaD7rRQwGHiYUEIMJBcR74WOGAs8658qBXWY2HzgL\nOAB84Jz7JHzcV4AXnXOHAczseC90u93MJgLd+fIKZr3DwdACSCX0/p/PvBRec2CNmbWvtP90YCpw\nkXOuQb5lVOqHbjFJQ/PZOEQfQreYFhPqQVQef4i0HsBniqtsR/sys98653oRWt/6r2aWEt4/DZjs\nnOsD/BxIqXROSaWvK9e0AzhKqOciEjMKCGloFgJjgb3OuXLn3F5CP72fS+iWE8AC4EozC5pZW+A8\n4IMI11oAfM3MmphZGjDuRN/cOfcvIBf4dnhXGrAjPGgd7ZKe+4ExwK/DK76JxIQCQhqalYTGFRZX\n2VfknNsd3n4RWAEsB94CfuCc21n1QuG1qp8HlgEvAO9EWcMvgDvCM6nuAd4H3gQ+jvYP4ZzbRSiQ\nnjCzQdGeJ1ITet23iIhEpB6EiIhEpIAQEZGIFBAiIhKRAkJERCJSQIiISEQKCBERiUgBISIiESkg\nREQkov8HMZNhYhn2dqkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa8b3a7a518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "word_counts = sorted(list(counter.values()), reverse=True)\n",
    "\n",
    "plt.loglog(word_counts)\n",
    "plt.ylabel(\"Freq\")\n",
    "plt.xlabel(\"Word Rank\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another related plot is the histogram of `word_counts`, which displays how many words have a count in a specific range.\n",
    "\n",
    "Of course the distribution is highly peaked at low counts, i.e. most of the words appear which a low count, so we better display it on semilogarithmic axes to inspect the tail of the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAD8CAYAAACyyUlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF3VJREFUeJzt3X+sX/V93/HnqzYQljSxCTeRZ1uy01pbnEg15A6cZZoy\nyIwhVU0lIhlVxU2Z3GUgJVu1xrR/0PxACtMaIrSE1i1uTJTGMJIOi5p5FqGqIiWGSyGAIZ5vgIUb\ne/hmBkIWjdTkvT++HyffmK99r++5P3zx8yEdfc95n8859/P5notfnB/f+01VIUlSF7801x2QJM1/\nhokkqTPDRJLUmWEiSerMMJEkdWaYSJI6M0wkSZ0ZJpKkzgwTSVJnC+e6A1N1/vnn14oVK+a6G5I0\nrzz88MM/qKqh6d7vvA2TFStWMDIyMtfdkKR5Jcn/mon9TvoyV5IFSR5Jcm9bXplkb5IDSe5Mcnar\nn9OWR9v6FX37uKHV9ye5rK++vtVGk2yZvuFJkmbDqdwz+SjwVN/yzcAtVbUKeAG4ttWvBV6oql8F\nbmntSLIa2Ai8C1gPfKEF1ALg88DlwGrg6tZWkjRPTCpMkiwDPgj8RVsOcAlwd2uyHbiyzW9oy7T1\nl7b2G4AdVfVKVT0DjAIXtWm0qp6uqp8AO1pbSdI8Mdkzk88BfwD8tC2/FXixqo625TFgaZtfCjwH\n0Na/1Nr/rH7cNieqv0aSzUlGkoyMj49PsuuSpJk2YZgk+XXgcFU93F8e0LQmWHeq9dcWq7ZW1XBV\nDQ8NTfvDCJKkKZrM01zvA34jyRXAG4A30ztTWZRkYTv7WAYcbO3HgOXAWJKFwFuAI331Y/q3OVFd\nkjQPTHhmUlU3VNWyqlpB7wb616vqt4AHgKtas03APW1+Z1umrf969b7OcSewsT3ttRJYBTwIPASs\nak+Hnd1+xs5pGZ0kaVZ0+ZzJx4EdST4NPALc3uq3A19KMkrvjGQjQFXtS3IX8CRwFLiuql4FSHI9\nsBtYAGyrqn0d+iVJmmWZr98BPzw8XH5oUZJOTZKHq2p4uvc7bz8B38WKLX8zsP7sZz44yz2RpNcH\n/9CjJKkzw0SS1JlhIknqzDCRJHVmmEiSOjNMJEmdGSaSpM4ME0lSZ4aJJKkzw0SS1JlhIknqzDCR\nJHVmmEiSOjNMJEmdGSaSpM4ME0lSZxOGSZI3JHkwybeT7EvyiVb/YpJnkjzapjWtniS3JhlN8liS\nC/v2tSnJgTZt6qu/J8njbZtbk2QmBitJmhmT+abFV4BLqupHSc4CvpHkvrbuP1bV3ce1vxxY1aaL\ngduAi5OcB9wIDAMFPJxkZ1W90NpsBr4F7ALWA/chSZoXJjwzqZ4ftcWz2nSyL47fANzRtvsWsCjJ\nEuAyYE9VHWkBsgdY39a9uaq+Wb0vpL8DuLLDmCRJs2xS90ySLEjyKHCYXiDsbatuapeybklyTqst\nBZ7r23ys1U5WHxtQlyTNE5MKk6p6tarWAMuAi5K8G7gB+KfAPwPOAz7emg+631FTqL9Gks1JRpKM\njI+PT6brkqRZcEpPc1XVi8DfAuur6lC7lPUK8JfARa3ZGLC8b7NlwMEJ6ssG1Af9/K1VNVxVw0ND\nQ6fSdUnSDJrM01xDSRa1+XOBDwDfafc6aE9eXQk80TbZCVzTnupaC7xUVYeA3cC6JIuTLAbWAbvb\nupeTrG37uga4Z3qHKUmaSZN5mmsJsD3JAnrhc1dV3Zvk60mG6F2mehT4t639LuAKYBT4MfBhgKo6\nkuRTwEOt3Ser6kib/wjwReBcek9x+SSXJM0jE4ZJVT0GXDCgfskJ2hdw3QnWbQO2DaiPAO+eqC+S\npNOTn4CXJHVmmEiSOjNMJEmdGSaSpM4ME0lSZ4aJJKkzw0SS1JlhIknqzDCRJHVmmEiSOjNMJEmd\nGSaSpM4ME0lSZ4aJJKkzw0SS1JlhIknqzDCRJHVmmEiSOpswTJK8IcmDSb6dZF+ST7T6yiR7kxxI\ncmeSs1v9nLY82tav6NvXDa2+P8llffX1rTaaZMv0D1OSNJMmc2byCnBJVf0asAZYn2QtcDNwS1Wt\nAl4Arm3trwVeqKpfBW5p7UiyGtgIvAtYD3whyYIkC4DPA5cDq4GrW1tJ0jwxYZhUz4/a4lltKuAS\n4O5W3w5c2eY3tGXa+kuTpNV3VNUrVfUMMApc1KbRqnq6qn4C7GhtJUnzxKTumbQziEeBw8Ae4LvA\ni1V1tDUZA5a2+aXAcwBt/UvAW/vrx21zovqgfmxOMpJkZHx8fDJdlyTNgkmFSVW9WlVrgGX0ziTe\nOahZe80J1p1qfVA/tlbVcFUNDw0NTdxxSdKsOKWnuarqReBvgbXAoiQL26plwME2PwYsB2jr3wIc\n6a8ft82J6pKkeWIyT3MNJVnU5s8FPgA8BTwAXNWabQLuafM72zJt/derqlp9Y3vaayWwCngQeAhY\n1Z4OO5veTfqd0zE4SdLsWDhxE5YA29tTV78E3FVV9yZ5EtiR5NPAI8Dtrf3twJeSjNI7I9kIUFX7\nktwFPAkcBa6rqlcBklwP7AYWANuqat+0jVCSNOMmDJOqegy4YED9aXr3T46v/z/gQyfY103ATQPq\nu4Bdk+ivJOk05CfgJUmdGSaSpM4ME0lSZ4aJJKkzw0SS1JlhIknqzDCRJHVmmEiSOjNMJEmdGSaS\npM4ME0lSZ4aJJKkzw0SS1JlhIknqzDCRJHVmmEiSOjNMJEmdTeY74JcneSDJU0n2Jfloq/9xku8n\nebRNV/Rtc0OS0ST7k1zWV1/faqNJtvTVVybZm+RAkjvbd8FLkuaJyZyZHAV+v6reCawFrkuyuq27\nparWtGkXQFu3EXgXsB74QpIF7TvkPw9cDqwGru7bz81tX6uAF4Brp2l8kqRZMGGYVNWhqvr7Nv8y\n8BSw9CSbbAB2VNUrVfUMMErvu+IvAkar6umq+gmwA9iQJMAlwN1t++3AlVMdkCRp9p3SPZMkK4AL\ngL2tdH2Sx5JsS7K41ZYCz/VtNtZqJ6q/FXixqo4eV5ckzROTDpMkbwK+Cnysqn4I3Ab8CrAGOAT8\nybGmAzavKdQH9WFzkpEkI+Pj45PtuiRphk0qTJKcRS9IvlxVXwOoquer6tWq+inw5/QuY0HvzGJ5\n3+bLgIMnqf8AWJRk4XH116iqrVU1XFXDQ0NDk+m6JGkWTOZprgC3A09V1Wf76kv6mv0m8ESb3wls\nTHJOkpXAKuBB4CFgVXty62x6N+l3VlUBDwBXte03Afd0G5YkaTYtnLgJ7wN+G3g8yaOt9of0nsZa\nQ++S1LPA7wFU1b4kdwFP0nsS7LqqehUgyfXAbmABsK2q9rX9fRzYkeTTwCP0wkuSNE9MGCZV9Q0G\n39fYdZJtbgJuGlDfNWi7qnqan18mkyTNM34CXpLUmWEiSerMMJEkdWaYSJI6M0wkSZ0ZJpKkzgwT\nSVJnhokkqTPDRJLUmWEiSerMMJEkdWaYSJI6M0wkSZ0ZJpKkzgwTSVJnhokkqTPDRJLUmWEiSeps\nwjBJsjzJA0meSrIvyUdb/bwke5IcaK+LWz1Jbk0ymuSxJBf27WtTa38gyaa++nuSPN62uTXJoK8J\nliSdpiZzZnIU+P2qeiewFrguyWpgC3B/Va0C7m/LAJcDq9q0GbgNeuED3AhcTO/73m88FkCtzea+\n7dZ3H5okabZMGCZVdaiq/r7Nvww8BSwFNgDbW7PtwJVtfgNwR/V8C1iUZAlwGbCnqo5U1QvAHmB9\nW/fmqvpmVRVwR9++JEnzwCndM0myArgA2Au8vaoOQS9wgLe1ZkuB5/o2G2u1k9XHBtQlSfPEpMMk\nyZuArwIfq6ofnqzpgFpNoT6oD5uTjCQZGR8fn6jLkqRZMqkwSXIWvSD5clV9rZWfb5eoaK+HW30M\nWN63+TLg4AT1ZQPqr1FVW6tquKqGh4aGJtN1SdIsmMzTXAFuB56qqs/2rdoJHHsiaxNwT1/9mvZU\n11rgpXYZbDewLsniduN9HbC7rXs5ydr2s67p25ckaR5YOIk27wN+G3g8yaOt9ofAZ4C7klwLfA/4\nUFu3C7gCGAV+DHwYoKqOJPkU8FBr98mqOtLmPwJ8ETgXuK9NkqR5YsIwqapvMPi+BsClA9oXcN0J\n9rUN2DagPgK8e6K+SJJOT34CXpLUmWEiSerMMJEkdWaYSJI6M0wkSZ0ZJpKkzgwTSVJnhokkqTPD\nRJLUmWEiSerMMJEkdWaYSJI6M0wkSZ0ZJpKkzgwTSVJnhokkqTPDRJLU2WS+A35bksNJnuir/XGS\n7yd5tE1X9K27Iclokv1JLuurr2+10SRb+uork+xNciDJnUnOns4BSpJm3mTOTL4IrB9Qv6Wq1rRp\nF0CS1cBG4F1tmy8kWZBkAfB54HJgNXB1awtwc9vXKuAF4NouA5Ikzb4Jw6Sq/g44Msn9bQB2VNUr\nVfUMMApc1KbRqnq6qn4C7AA2JAlwCXB32347cOUpjkGSNMe63DO5Pslj7TLY4lZbCjzX12as1U5U\nfyvwYlUdPa4uSZpHphomtwG/AqwBDgF/0uoZ0LamUB8oyeYkI0lGxsfHT63HkqQZM6Uwqarnq+rV\nqvop8Of0LmNB78xieV/TZcDBk9R/ACxKsvC4+ol+7taqGq6q4aGhoal0XZI0A6YUJkmW9C3+JnDs\nSa+dwMYk5yRZCawCHgQeAla1J7fOpneTfmdVFfAAcFXbfhNwz1T6JEmaOwsnapDkK8D7gfOTjAE3\nAu9PsobeJalngd8DqKp9Se4CngSOAtdV1attP9cDu4EFwLaq2td+xMeBHUk+DTwC3D5to5MkzYoJ\nw6Sqrh5QPuE/+FV1E3DTgPouYNeA+tP8/DKZJGke8hPwkqTODBNJUmeGiSSpM8NEktSZYSJJ6sww\nkSR1ZphIkjozTCRJnRkmkqTODBNJUmeGiSSpM8NEktSZYSJJ6swwkSR1ZphIkjozTCRJnRkmkqTO\nDBNJUmcThkmSbUkOJ3mir3Zekj1JDrTXxa2eJLcmGU3yWJIL+7bZ1NofSLKpr/6eJI+3bW5Nkuke\npCRpZk3mzOSLwPrjaluA+6tqFXB/Wwa4HFjVps3AbdALH+BG4GJ63/d+47EAam029213/M+SJJ3m\nJgyTqvo74Mhx5Q3A9ja/Hbiyr35H9XwLWJRkCXAZsKeqjlTVC8AeYH1b9+aq+mZVFXBH374kSfPE\nVO+ZvL2qDgG017e1+lLgub52Y612svrYgLokaR6Z7hvwg+531BTqg3eebE4ykmRkfHx8il2UJE23\nqYbJ8+0SFe31cKuPAcv72i0DDk5QXzagPlBVba2q4aoaHhoammLXJUnTbaphshM49kTWJuCevvo1\n7amutcBL7TLYbmBdksXtxvs6YHdb93KSte0prmv69iVJmicWTtQgyVeA9wPnJxmj91TWZ4C7klwL\nfA/4UGu+C7gCGAV+DHwYoKqOJPkU8FBr98mqOnZT/yP0nhg7F7ivTZKkeWTCMKmqq0+w6tIBbQu4\n7gT72QZsG1AfAd49UT8kSacvPwEvSerMMJEkdWaYSJI6M0wkSZ0ZJpKkzgwTSVJnhokkqTPDRJLU\nmWEiSerMMJEkdWaYSJI6M0wkSZ0ZJpKkzgwTSVJnhokkqTPDRJLUmWEiSerMMJEkddYpTJI8m+Tx\nJI8mGWm185LsSXKgvS5u9SS5NclokseSXNi3n02t/YEkm7oNSZI026bjzORfVdWaqhpuy1uA+6tq\nFXB/Wwa4HFjVps3AbdALH+BG4GLgIuDGYwEkSZofZuIy1wZge5vfDlzZV7+jer4FLEqyBLgM2FNV\nR6rqBWAPsH4G+iVJmiFdw6SA/5Hk4SSbW+3tVXUIoL2+rdWXAs/1bTvWaieqv0aSzUlGkoyMj493\n7Lokabos7Lj9+6rqYJK3AXuSfOckbTOgViepv7ZYtRXYCjA8PDywjSRp9nU6M6mqg+31MPDX9O55\nPN8uX9FeD7fmY8Dyvs2XAQdPUpckzRNTDpMkb0zyy8fmgXXAE8BO4NgTWZuAe9r8TuCa9lTXWuCl\ndhlsN7AuyeJ2431dq0mS5okul7neDvx1kmP7+auq+u9JHgLuSnIt8D3gQ639LuAKYBT4MfBhgKo6\nkuRTwEOt3Ser6kiHfkmSZtmUw6SqngZ+bUD9/wCXDqgXcN0J9rUN2DbVvkiS5pafgJckdWaYSJI6\nM0wkSZ0ZJpKkzgwTSVJnhokkqTPDRJLUmWEiSerMMJEkdWaYSJI6M0wkSZ0ZJpKkzgwTSVJnhokk\nqTPDRJLUmWEiSerMMJEkdXbahEmS9Un2JxlNsmWu+yNJmrzTIkySLAA+D1wOrAauTrJ6bnslSZqs\nKX8H/DS7CBht3ytPkh3ABuDJ2ezEii1/M7D+7Gc+OJvdkKR553QJk6XAc33LY8DFc9SX1zBkJOnk\nTpcwyYBavaZRshnY3BZ/lGT/FH/e+cAPprjtz/tzc9c9zJlpGf885vgd/5k8/n8yEzs9XcJkDFje\nt7wMOHh8o6raCmzt+sOSjFTVcNf9zFeO3/E7/jN7/DOx39PiBjzwELAqycokZwMbgZ1z3CdJ0iSd\nFmcmVXU0yfXAbmABsK2q9s1xtyRJk3RahAlAVe0Cds3Sj+t8qWyec/xnNsd/ZpuR8afqNfe5JUk6\nJafLPRNJ0jx2RoXJ6/VPtiRZnuSBJE8l2Zfko61+XpI9SQ6018WtniS3tvfhsSQX9u1rU2t/IMmm\nuRrTVCRZkOSRJPe25ZVJ9rax3Nke7iDJOW15tK1f0bePG1p9f5LL5mYkpy7JoiR3J/lO+z1475l0\n/JP8+/a7/0SSryR5w+v5+CfZluRwkif6atN2vJO8J8njbZtbkwz6+MYvqqozYqJ3Y/+7wDuAs4Fv\nA6vnul/TNLYlwIVt/peB/0nvz9L8J2BLq28Bbm7zVwD30ft8z1pgb6ufBzzdXhe3+cVzPb5TeB/+\nA/BXwL1t+S5gY5v/U+Ajbf7fAX/a5jcCd7b51e334hxgZft9WTDX45rk2LcD/6bNnw0sOlOOP70P\nPT8DnNt33H/n9Xz8gX8JXAg80VebtuMNPAi8t21zH3D5hH2a6zdlFt/89wK7+5ZvAG6Y637N0Fjv\nAf41sB9Y0mpLgP1t/s+Aq/va72/rrwb+rK/+C+1O54neZ5PuBy4B7m3/EfwAWHj88af31OB72/zC\n1i7H/070tzudJ+DN7R/THFc/I44/P/8LGue143kvcNnr/fgDK44Lk2k53m3dd/rqv9DuRNOZdJlr\n0J9sWTpHfZkx7ZT9AmAv8PaqOgTQXt/Wmp3ovZjP79HngD8AftqW3wq8WFVH23L/WH42zrb+pdZ+\nvo7/HcA48JftMt9fJHkjZ8jxr6rvA/8Z+B5wiN7xfJgz5/gfM13He2mbP75+UmdSmEzqT7bMZ0ne\nBHwV+FhV/fBkTQfU6iT101qSXwcOV9XD/eUBTWuCdfNy/PT+7/pC4LaqugD4v/Quc5zI62r87d7A\nBnqXpv4x8EZ6f4H8eK/X4z+RUx3vlN6HMylMJvUnW+arJGfRC5IvV9XXWvn5JEva+iXA4VY/0Xsx\nX9+j9wG/keRZYAe9S12fAxYlOfZZqv6x/Gycbf1bgCPM3/GPAWNVtbct300vXM6U4/8B4JmqGq+q\nfwC+Bvxzzpzjf8x0He+xNn98/aTOpDB53f7Jlvakxe3AU1X12b5VO4FjT2hsoncv5Vj9mvaUx1rg\npXZavBtYl2Rx+7+9da12WquqG6pqWVWtoHdcv15VvwU8AFzVmh0//mPvy1WtfbX6xva0z0pgFb0b\nkae1qvrfwHNJjv0Bv0vpfX3DGXH86V3eWpvkH7X/Fo6N/4w4/n2m5Xi3dS8nWdvez2v69nVic30T\naZZvWF1B70mn7wJ/NNf9mcZx/Qt6p6GPAY+26Qp614HvBw601/Na+9D7MrLvAo8Dw337+l1gtE0f\nnuuxTeG9eD8/f5rrHfT+MRgF/itwTqu/oS2PtvXv6Nv+j9r7sp9JPMFyukzAGmCk/Q78N3pP55wx\nxx/4BPAd4AngS/SeyHrdHn/gK/TuD/0DvTOJa6fzeAPD7b38LvBfOO7hjkGTn4CXJHV2Jl3mkiTN\nEMNEktSZYSJJ6swwkSR1ZphIkjozTCRJnRkmkqTODBNJUmf/H6G4PJULeh8GAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa8b39fda20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(word_counts, bins=50);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADz5JREFUeJzt3W2MXNddx/HvDwenkJY0aQIKdqx15CjC4gUNozQpCFVQ\nWrupG1RVyFalPhBitSiIhxfgqEiIdymqUIkaSE1rghA4DaFq7dSVQaVVhBSFODy0Tl3TbZqSJaV2\niDCoQkpD/7yYm2bOajeZ3bnrmfV+P9Jo55659845907y8z3nPqSqkCTpBd837QpIkmaLwSBJahgM\nkqSGwSBJahgMkqSGwSBJahgMkqSGwSBJahgMkqTGRdOuAMAVV1xRc3Nz066GJK0rjz322DNVdWXf\n652JYJibm+PEiRPTroYkrStJvrEW651qV1KSPUkOnjt3bprVkCSNmGowVNXRqtp/6aWXTrMakqQR\nDj5LkhoGgySp4RiDJKnhGIMkqWFXkiSpYTBIkhpTvcAtyR5gz44dO1a9jrkDn1my/Mk7b171OiVp\nI3OMQZLUsCtJktQwGCRJDYNBktTwAjdJUsPBZ0lSw64kSVLDYJAkNQwGSVLDYJAkNQwGSVLDYJAk\nNbyOQZLU8DoGSVLDriRJUsNgkCQ1DAZJUsNgkCQ1DAZJUsNgkCQ1DAZJUmNNgiHJJUkeS/LWtVi/\nJGntjBUMSQ4lOZPk5KLyXUlOJ5lPcmDko98G7u+zopKk82PcI4Z7gV2jBUk2AXcDu4GdwL4kO5O8\nEfgy8K0e6ylJOk8uGmemqnooydyi4huA+ap6AiDJfcAtwCuBSxiGxf8mOVZV3+2txpKkNTVWMCxj\nC/DUyPQC8Lqquh0gyXuAZ5YLhST7gf0A27Ztm6AakqQ+TTL4nCXK6ntvqu6tqgeXW7iqDlbVoKoG\nV1555QTVkCT1aZJgWACuHpneCjy9khV4221Jmj2TBMOjwLVJtifZDOwFjqxkBd52W5Jmz7inqx4G\nHgauS7KQ5Naqeh64HTgOnALur6rHV/LlHjFI0uwZ96ykfcuUHwOOrfbLq+oocHQwGNy22nVIkvrl\noz0lSQ0f7SlJangTPUlSw64kSVLDriRJUsOuJElSw2CQJDUcY5AkNRxjkCQ17EqSJDUMBklSwzEG\nSVLDMQZJUsOuJElSw2CQJDUMBklSw8FnSVLDwWdJUsOuJElSw2CQJDUMBklSw2CQJDUMBklSw2CQ\nJDW8jkGS1PA6BklSw64kSVLDYJAkNQwGSVLDYJAkNQwGSVLDYJAkNQwGSVKj92BI8mNJ7knyQJL3\n971+SdLaGisYkhxKcibJyUXlu5KcTjKf5ABAVZ2qqvcBvwgM+q+yJGktjXvEcC+wa7QgySbgbmA3\nsBPYl2Rn99nbgL8HPtdbTSVJ58VYwVBVDwHPLiq+AZivqieq6jngPuCWbv4jVfV64J19VlaStPYu\nmmDZLcBTI9MLwOuSvAF4O3AxcGy5hZPsB/YDbNu2bYJqSJL6NEkwZImyqqovAF94uYWr6iBwEGAw\nGNQE9ZAk9WiSs5IWgKtHprcCT69kBd52W5JmzyTB8ChwbZLtSTYDe4EjK1mBt92WpNkz7umqh4GH\ngeuSLCS5taqeB24HjgOngPur6vGVfLlHDJI0e8YaY6iqfcuUH+MlBpjHWO9R4OhgMLhtteuQJPXL\nR3tKkho+2lOS1PAmepKkhl1JkqSGXUmSpIZdSZKkhsEgSWo4xiBJajjGIElq2JUkSWoYDJKkhmMM\nkqSGYwySpMYkT3CbaXMHPrNk+ZN33nyeayJJ64tjDJKkhsEgSWo4+CxJajj4LElq2JUkSWoYDJKk\nhsEgSWoYDJKkhsEgSWoYDJKkhtcxSJIaXscgSWrYlSRJahgMkqSGwSBJahgMkqSGwSBJahgMkqSG\nwSBJaqxJMCT5hSR/kuTTSd60Ft8hSVobF407Y5JDwFuBM1X14yPlu4A/BDYBH6uqO6vqU8CnklwG\nfAj4m36rvXpzBz6z7GdP3nnzeayJJM2mlRwx3AvsGi1Isgm4G9gN7AT2Jdk5MsvvdJ9LktaJsYOh\nqh4Cnl1UfAMwX1VPVNVzwH3ALRn6IPDZqvrH/qorSVprk44xbAGeGple6Mp+FXgj8I4k71tqwST7\nk5xIcuLs2bMTVkOS1JexxxiWkSXKqqruAu56qQWr6iBwEGAwGNSE9ZAk9WTSI4YF4OqR6a3A0+Mu\n7G23JWn2TBoMjwLXJtmeZDOwFzgy7sLedluSZs/YwZDkMPAwcF2ShSS3VtXzwO3AceAUcH9VPb6C\ndXrEIEkzZuwxhqrat0z5MeDYar68qo4CRweDwW2rWV6S1D8f7SlJavhoT0lSw5voSZIadiVJkhp2\nJUmSGnYlSZIaBoMkqeEYgySp4RiDJKlhV5IkqTHpbbcvKMs99tNHfkraSKYaDEn2AHt27NgxzWq8\nLAND0kbiGIMkqeEYgySpYTBIkhoGgySp4QVukqSGg8+SpIZdSZKkhsEgSWoYDJKkhsEgSWoYDJKk\nhsEgSWp4HYMkqeF1DJKkhl1JkqSGwSBJahgMkqSGj/acgE92k3Qh8ohBktQwGCRJDYNBktToPRiS\nXJPk40ke6HvdkqS1N1YwJDmU5EySk4vKdyU5nWQ+yQGAqnqiqm5di8pKktbeuEcM9wK7RguSbALu\nBnYDO4F9SXb2WjtJ0nk3VjBU1UPAs4uKbwDmuyOE54D7gFt6rp8k6TybZIxhC/DUyPQCsCXJa5Lc\nA7w2yR3LLZxkf5ITSU6cPXt2gmpIkvo0yQVuWaKsquo/gfe93MJVdRA4CDAYDGqCekiSejRJMCwA\nV49MbwWeXskKkuwB9uzYsWOCaswer4iWtJ5N0pX0KHBtku1JNgN7gSMrWYG33Zak2TPWEUOSw8Ab\ngCuSLAC/W1UfT3I7cBzYBByqqsdX8uUX6hHDcjySkLQejBUMVbVvmfJjwLHVfnlVHQWODgaD21a7\nDklSv7wlhiSp4TOfJUkNn/ksSWpM9UE9G23wuS8OYktaSx4xSJIaDj5LkhoGgySp4RjDDHDMQNIs\ncYxBktSwK0mS1DAYJEkNxxhm2HJjD5K0lhxjkCQ17EqSJDUMBklSw2CQJDW87bYkqeHgsySpYVeS\nJKlhMEiSGgaDJKlhMEiSGgaDJKlhMEiSGt5E7wKy0pvu+SCg2eYDnDQtXscgSWrYlSRJahgMkqSG\nwSBJahgMkqSGwSBJahgMkqSGwSBJavR+gVuSS4A/Ap4DvlBVf9H3d0iS1s5YRwxJDiU5k+TkovJd\nSU4nmU9yoCt+O/BAVd0GvK3n+kqS1ti4XUn3ArtGC5JsAu4GdgM7gX1JdgJbgae62f6vn2pKks6X\nsYKhqh4Cnl1UfAMwX1VPVNVzwH3ALcACw3AYe/2SpNkxyRjDFl48MoBhILwOuAv4SJKbgaPLLZxk\nP7AfYNu2bRNUQ6s1izdpm8U6SX1YT7/tSYIhS5RVVX0beO/LLVxVB4GDAIPBoCaohySpR5N09SwA\nV49MbwWeXskKkuxJcvDcuXMTVEOS1KdJguFR4Nok25NsBvYCR1ayAm+7LUmzZ9zTVQ8DDwPXJVlI\ncmtVPQ/cDhwHTgH3V9XjK/lyjxgkafaMNcZQVfuWKT8GHFvtl1fVUeDoYDC4bbXrkCT1y9NJJUmN\nqQaDXUmSNHt85rMkqeERgySpkarpX1uW5CzwjVUufgXwTI/VWW9sv+23/RvXdVX1qr5X2vttt1ej\nqq5c7bJJTlTVoM/6rCe23/bb/o3d/rVYr2clSZIaBoMkqXEhBMPBaVdgymz/xmb7N7Y1af9MDD5L\nkmbHhXDEIEnq0boOhmWeOb2uJbk6yeeTnEryeJJf68ovT/K3Sb7a/b2sK0+Su7pt8MUk14+s693d\n/F9N8u5ptWk1kmxK8k9JHuymtyd5pGvLJ7o7+pLk4m56vvt8bmQdd3Tlp5O8eTotWbkkr07yQJKv\ndL+DmzbS/k/yG91v/2SSw0lecaHv/ySHkpxJcnKkrLd9nuQnk3ypW+auJEs9T+dFVbUuX8Am4GvA\nNcBm4F+AndOuVw/tugq4vnv/KuBfGT5T+/eBA135AeCD3fu3AJ9l+OCkG4FHuvLLgSe6v5d17y+b\ndvtWsB1+E/hL4MFu+n5gb/f+HuD93ftfAe7p3u8FPtG939n9Ji4Gtne/lU3TbteYbf8z4Je795uB\nV2+U/c/wyZBfB35gZL+/50Lf/8DPANcDJ0fKetvnwD8AN3XLfBbY/ZL1mfYGmWBD3gQcH5m+A7hj\n2vVag3Z+Gvh54DRwVVd2FXC6e/9RYN/I/Ke7z/cBHx0pb+ab5RfDhz59DvhZ4MHux/wMcNHifc/w\ntu83de8v6ubL4t/D6Hyz/AJ+qPsfYxaVb4j9z4uPDL68258PAm/eCPsfmFsUDL3s8+6zr4yUN/Mt\n9VrPXUlLPXN6y5Tqsia6w+LXAo8AP1JV3wTo/v5wN9ty22E9b58PA78FfLebfg3wXzV8Bgi0bfle\nO7vPz3Xzr9f2XwOcBf6060r7WJJL2CD7v6r+HfgQ8G/ANxnuz8fYOPt/VF/7fEv3fnH5stZzMCz5\nzOnzXos1kuSVwF8Dv15V//1Ssy5RVi9RPtOSvBU4U1WPjRYvMWu9zGfrsv0M/9V7PfDHVfVa4NsM\nuxGWc0G1v+tHv4Vh98+PApcAu5eY9ULd/+NYaZtXvC3WczBM/MzpWZXk+xmGwl9U1Se74m8luar7\n/CrgTFe+3HZYr9vnp4C3JXkSuI9hd9KHgVcneeEWLqNt+V47u88vBZ5l/bZ/AVioqke66QcYBsVG\n2f9vBL5eVWer6jvAJ4HXs3H2/6i+9vlC935x+bLWczBM/MzpWdSdLfBx4FRV/cHIR0eAF84yeDfD\nsYcXyt/VnalwI3CuO+w8DrwpyWXdv8Le1JXNtKq6o6q2VtUcw336d1X1TuDzwDu62Ra3/4Xt8o5u\n/urK93ZnrWwHrmU4ADfTquo/gKeSXNcV/RzwZTbI/mfYhXRjkh/s/lt4of0bYv8v0ss+7z77nyQ3\ndtv0XSPrWtq0B1wmHKx5C8Ozdr4GfGDa9empTT/N8DDvi8A/d6+3MOw3/Rzw1e7v5d38Ae7utsGX\ngMHIun4JmO9e751221axLd7Ai2clXcPwP+x54K+Ai7vyV3TT893n14ws/4Fuu5zmZc7CmKUX8BPA\nie438CmGZ5hsmP0P/B7wFeAk8OcMzyy6oPc/cJjhmMp3GP4L/9Y+9zkw6Lbn14CPsOjkhsUvr3yW\nJDXWc1eSJGkNGAySpIbBIElqGAySpIbBIElqGAySpIbBIElqGAySpMb/A9Eqlo/ulYqaAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa8b37d0e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(word_counts, bins=50, log=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a classifier for Sentiment Analysis\n",
    "\n",
    "Using our `build_bag_of_words_features` function we can build separately the negative and positive features.\n",
    "Basically for each of the 1000 negative and for the 1000 positive review, we create one dictionary of the words and we associate the label \"neg\" and \"pos\" to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "negative_features = [\n",
    "    (build_bag_of_words_features_filtered(movie_reviews.words(fileids=[f])), 'neg') \\\n",
    "    for f in negative_fileids\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "positive_features = [\n",
    "    (build_bag_of_words_features_filtered(movie_reviews.words(fileids=[f])), 'pos') \\\n",
    "    for f in positive_fileids\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.classify import NaiveBayesClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the simplest supervised machine learning classifiers is the Naive Bayes Classifier, it can be trained on 80% of the data to learn what words are generally associated with positive or with negative reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "split = 800"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "classifier = NaiveBayesClassifier.train(positive_features[:split]+negative_features[:split])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check after training what is the accuracy on the training set, i.e. the same data used for training, we expect this to be a very high number because the algorithm already \"saw\" those data. Accuracy is the fraction of the data that is classified correctly, we can turn it into percent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "98.0625"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.classify.util.accuracy(classifier, positive_features[:split]+negative_features[:split])*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy above is mostly a check that nothing went very wrong in the training, the real measure of accuracy is on the remaining 20% of the data that wasn't used in training, the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "71.75"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.classify.util.accuracy(classifier, positive_features[split:]+negative_features[split:])*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy here is around 70% which is pretty good for such a simple model if we consider that the estimated accuracy for a person is about 80%.\n",
    "We can finally print the most informative features, i.e. the words that mostly identify a positive or a negative review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "             outstanding = 1                 pos : neg    =     13.9 : 1.0\n",
      "               insulting = 1                 neg : pos    =     13.7 : 1.0\n",
      "              vulnerable = 1                 pos : neg    =     13.0 : 1.0\n",
      "               ludicrous = 1                 neg : pos    =     12.6 : 1.0\n",
      "             uninvolving = 1                 neg : pos    =     12.3 : 1.0\n",
      "                  avoids = 1                 pos : neg    =     11.7 : 1.0\n",
      "              astounding = 1                 pos : neg    =     11.7 : 1.0\n",
      "             fascination = 1                 pos : neg    =     11.0 : 1.0\n",
      "                  symbol = 1                 pos : neg    =     10.3 : 1.0\n",
      "               animators = 1                 pos : neg    =     10.3 : 1.0\n"
     ]
    }
   ],
   "source": [
    "classifier.show_most_informative_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}