Programming for Humanities Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Humanities data: what is it really?\n",
    "\n",
    "Humanities work can take many many forms, but usually involves text.  This text may be:\n",
    "\n",
    "* derivatives from text, e.g. bag of words\n",
    "* transcriptions of intererviews\n",
    "* unstructured free text\n",
    "* semi-strucutred data\n",
    "* qualitative coded values\n",
    "* lists of resources\n",
    "* and many more\n",
    "\n",
    "The data area where Python really shines is the manipulation of text structures.\n",
    "\n",
    "## But what about R?\n",
    "\n",
    "R is another language, focused on statistics.  It has great support for modeling, viz, etc.  Many text modeling/analysis tools are written in R.  So you may need to focus your efforts in learning R.  That said, the basic concepts that we're covered this morning and will look at this afternoon will carry over to working in R.  The structures in R are a bit different, so it isn't 1:1, but knowing the concept of a variable, the concept of a string, the concept of numerical types, etc. will serve you very well in the transition.\n",
    "\n",
    "## You don't have to choose just one platform\n",
    "\n",
    "Choosing the right tool for the right job is a key concept for programming.  Sometimes an entirely different language can save your a lot of trouble.  Also, the digital humanities crowd is pretty split between R and Python, so being functional in both will serve you well.\n",
    "\n",
    "Python handles un-/semi-structured data very well, so you might find it easier to wrangle all your raw data in Python, then produce your analysis data file for processing in another program (e.g. produce coordinates for ArcGIS, network data for Gephi, categorical codes for R, etc.)\n",
    "\n",
    "You don't need to pick and work with only one program.  You'll want to use many tools.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Breaking down a problem\n",
    "\n",
    "I want to focus more on design and give you some of the essential patterns to experiment with.  This is the best way to learn.  From a pratitioner standpoint, starting with the core patterns and tinkering with them allows you to get things done first and then you'll fill in the theory along the way.\n",
    "\n",
    "### A basic problem:  Count the words in the Raven\n",
    "\n",
    "In this problem we'll explore reading in text, manipulating it to get our data out, and then writing out data for ourselves.  Writing programs is a very iterative and non-linear process.  Much like starting off with an outline of a paper, you'll have clearer ideas on how to handle certain tasks that others.\n",
    "\n",
    "Even though you've never seen this code, we can start somewhere!  We know all these things need to happen, even if we aren't sure how yet.\n",
    "\n",
    "1. Read in the text\n",
    "2. Count the words. I might need to toss out some stop words, but I don't know what those are yet.\n",
    "3. Assemble some data\n",
    "4. Make a data file where I have:  word, count\n",
    "\n",
    "This is a great start, and pinpoints many of the problems that we need to solve.  I usually like to start filling in the easy chunks first, so I can focus on adapting the middle chunk.  Usually the juicy peices, e.g. \"counting the words\" appear as just a few lines in middle of a whole load of prep."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading in text\n",
    "\n",
    "There are multiple methods of reading in text, but I'm going to show you one of the classic methods.  We're very used to programs handling file opening, writing, editing, etc. for us, but when it comes to programming languages, we have to do that all ourselves.  We're used to opening and closing files within a GUI, but we'll be doing that more explicitly inside of our code.\n",
    "\n",
    "`variable = open(filename, mode)` is our basic pattern for file handling.  The filename is just a string of the file path, and the mode is a string (https://docs.python.org/3.6/library/functions.html#open) to declare which mode you'd like to oepn the file as.  There are many, but you're going to usually use:\n",
    "\n",
    "* 'r' for reading in a file\n",
    "* 'w' for writing to a file\n",
    "* 'a' for append (not the most common)\n",
    "\n",
    "Once you're done doing what you need to a file, you'll want to run:  `variable.close()`.  Once you close a file you will no longer have read or write access to that IO object.\n",
    "\n",
    "The more specific pattern for handling files is:\n",
    "\n",
    "1. Create a file IO object in the desired mode\n",
    "    * `file_in = open('raven.txt', 'r')`\n",
    "    * `file_out = open('raven_data.txt', 'w')`\n",
    "    * When writing out a file, if that file name does not exist it will create it.  If it does exist, in `'w'` mode it'll overwrite it, and `'a'` mode will add stuff to the end of it. \n",
    "2. Do your stuff to the file\n",
    "    * We haven't learned these yet!\n",
    "3. Close the file\n",
    "    * `file_in.close()`\n",
    "    * `file_out.close()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## .read() the oldie but goodie\n",
    "\n",
    "There are many methods of reading in a file, but .read() is sort of the \"when unsure...\" option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "file_in = open('raven.txt', 'r') # create IO object in read mode\n",
    "text = file_in.read() # apply .read() to the IO object and store the results as variable text\n",
    "file_in.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Once upon a midnight dreary, while I pondered, weak and weary, \n",
      "Over many a quaint and curious volume of forgotten lore, \n",
      "While I nodded, nearly napping, suddenly there came a tapping, \n",
      "As of some one gently rapping, rapping at my chamber door. \n",
      "\"'Tis some visitor,\" I muttered, \"tapping at my chamber door- \n",
      "                Only this, and nothing more.\" \n",
      "\n",
      "Ah, distinctly I remember it was in the bleak December, \n",
      "And each separate dying ember wrought its ghost upon the floor. \n",
      "Eagerly I wished the morrow;- vainly I had sought to borrow \n",
      "From my books surcease of sorrow- sorrow for the lost Lenore- \n",
      "For the rare and radiant maiden whom the angels name Lenore- \n",
      "                Nameless here for evermore. \n",
      "\n",
      "And the silken, sad, uncertain rustling of each purple curtain \n",
      "Thrilled me- filled me with fantastic terrors never felt before; \n",
      "So that now, to still the beating of my heart, I stood repeating, \n",
      "\"'Tis some visitor entreating entrance at my chamber door- \n",
      "Some late visitor entreating entrance at my chamber door;- \n",
      "                This it is, and nothing more.\" \n",
      "\n",
      "Presently my soul grew stronger; hesitating then no longer, \n",
      "\"Sir,\" said I, \"or Madam, truly your forgiveness I implore; \n",
      "But the fact is I was napping, and so gently you came rapping, \n",
      "And so faintly you came tapping, tapping at my chamber door, \n",
      "That I scarce was sure I heard you\"- here I opened wide the door;- \n",
      "                Darkness there, and nothing more. \n",
      "\n",
      "Deep into that darkness peering, long I stood there wondering, fearing, \n",
      "Doubting, dreaming dreams no mortal ever dared to dream before; \n",
      "But the silence was unbroken, and the stillness gave no token, \n",
      "And the only word there spoken was the whispered word, \"Lenore?\" \n",
      "This I whispered, and an echo murmured back the word, \"Lenore!\"- \n",
      "                Merely this, and nothing more. \n",
      "\n",
      "Back into the chamber turning, all my soul within me burning, \n",
      "Soon again I heard a tapping somewhat louder than before. \n",
      "\"Surely,\" said I, \"surely that is something at my window lattice: \n",
      "Let me see, then, what thereat is, and this mystery explore- \n",
      "Let my heart be still a moment and this mystery explore;- \n",
      "                'Tis the wind and nothing more!\" \n",
      "\n",
      "Open here I flung the shutter, when, with many a flirt and flutter, \n",
      "In there stepped a stately Raven of the saintly days of yore; \n",
      "Not the least obeisance made he; not a minute stopped or stayed he; \n",
      "But, with mien of lord or lady, perched above my chamber door- \n",
      "Perched upon a bust of Pallas just above my chamber door- \n",
      "                Perched, and sat, and nothing more. \n",
      "\n",
      "Then this ebony bird beguiling my sad fancy into smiling, \n",
      "By the grave and stern decorum of the countenance it wore. \n",
      "\"Though thy crest be shorn and shaven, thou,\" I said, \"art sure no craven, \n",
      "Ghastly grim and ancient Raven wandering from the Nightly shore- \n",
      "Tell me what thy lordly name is on the Night's Plutonian shore!\" \n",
      "                Quoth the Raven, \"Nevermore.\" \n",
      "\n",
      "Much I marvelled this ungainly fowl to hear discourse so plainly, \n",
      "Though its answer little meaning- little relevancy bore; \n",
      "For we cannot help agreeing that no living human being \n",
      "Ever yet was blessed with seeing bird above his chamber door- \n",
      "Bird or beast upon the sculptured bust above his chamber door, \n",
      "                With such name as \"Nevermore.\" \n",
      "\n",
      "But the Raven, sitting lonely on the placid bust, spoke only \n",
      "That one word, as if his soul in that one word he did outpour. \n",
      "Nothing further then he uttered- not a feather then he fluttered- \n",
      "Till I scarcely more than muttered, \"Other friends have flown before- \n",
      "On the morrow he will leave me, as my hopes have flown before.\" \n",
      "                Then the bird said, \"Nevermore.\" \n",
      "\n",
      "Startled at the stillness broken by reply so aptly spoken, \n",
      "\"Doubtless,\" said I, \"what it utters is its only stock and store, \n",
      "Caught from some unhappy master whom unmerciful Disaster \n",
      "Followed fast and followed faster till his songs one burden bore- \n",
      "Till the dirges of his Hope that melancholy burden bore \n",
      "                Of 'Never- nevermore'.\" \n",
      "\n",
      "But the Raven still beguiling all my fancy into smiling, \n",
      "Straight I wheeled a cushioned seat in front of bird, and bust and door; \n",
      "Then upon the velvet sinking, I betook myself to linking \n",
      "Fancy unto fancy, thinking what this ominous bird of yore- \n",
      "What this grim, ungainly, ghastly, gaunt and ominous bird of yore \n",
      "                Meant in croaking \"Nevermore.\" \n",
      "\n",
      "This I sat engaged in guessing, but no syllable expressing \n",
      "To the fowl whose fiery eyes now burned into my bosom's core; \n",
      "This and more I sat divining, with my head at ease reclining \n",
      "On the cushion's velvet lining that the lamp-light gloated o'er, \n",
      "But whose velvet violet lining with the lamp-light gloating o'er, \n",
      "                She shall press, ah, nevermore! \n",
      "\n",
      "Then methought the air grew denser, perfumed from an unseen censer \n",
      "Swung by Seraphim whose footfalls tinkled on the tufted floor. \n",
      "\"Wretch,\" I cried, \"thy God hath lent thee- by these angels he hath sent thee \n",
      "Respite- respite and nepenthe, from thy memories of Lenore! \n",
      "Quaff, oh quaff this kind nepenthe and forget this lost Lenore!\" \n",
      "                Quoth the Raven, \"Nevermore.\" \n",
      "\n",
      "\"Prophet!\" said I, \"thing of evil! - prophet still, if bird or devil! - \n",
      "Whether Tempter sent, or whether tempest tossed thee here ashore, \n",
      "Desolate yet all undaunted, on this desert land enchanted- \n",
      "On this home by Horror haunted- tell me truly, I implore- \n",
      "Is there- is there balm in Gilead?- tell me- tell me, I implore!\" \n",
      "                Quoth the Raven, \"Nevermore.\" \n",
      "\n",
      "\"Prophet!\" said I, \"thing of evil! - prophet still, if bird or devil! \n",
      "By that Heaven that bends above us- by that God we both adore- \n",
      "Tell this soul with sorrow laden if, within the distant Aidenn, \n",
      "It shall clasp a sainted maiden whom the angels name Lenore- \n",
      "Clasp a rare and radiant maiden whom the angels name Lenore.\" \n",
      "                Quoth the Raven, \"Nevermore.\" \n",
      "\n",
      "\"Be that word our sign in parting, bird or fiend,\" I shrieked, upstarting- \n",
      "\"Get thee back into the tempest and the Night's Plutonian shore! \n",
      "Leave no black plume as a token of that lie thy soul hath spoken! \n",
      "Leave my loneliness unbroken!- quit the bust above my door! \n",
      "Take thy beak from out my heart, and take thy form from off my door!\" \n",
      "                Quoth the Raven, \"Nevermore.\" \n",
      "\n",
      "And the Raven, never flitting, still is sitting, still is sitting \n",
      "On the pallid bust of Pallas just above my chamber door; \n",
      "And his eyes have all the seeming of a demon's that is dreaming, \n",
      "And the lamp-light o'er him streaming throws his shadow on the floor; \n",
      "And my soul from out that shadow that lies floating on the floor \n",
      "                Shall be lifted- nevermore! \n"
     ]
    }
   ],
   "source": [
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Once upon a midnight dreary, while I pondered, weak and weary, \\nOver many a quaint and curious volume of forgotten lore, \\nWhile I nodded, nearly napping, suddenly there came a tapping, \\nAs of some one gently rapping, rapping at my chamber door. \\n\"\\'Tis some visitor,\" I muttered, \"tapping at my chamber door- \\n                Only this, and nothing more.\" \\n\\nAh, distinctly I remember it was in the bleak December, \\nAnd each separate dying ember wrought its ghost upon the floor. \\nEagerly I wished the morrow;- vainly I had sought to borrow \\nFrom my books surcease of sorrow- sorrow for the lost Lenore- \\nFor the rare and radiant maiden whom the angels name Lenore- \\n                Nameless here for evermore. \\n\\nAnd the silken, sad, uncertain rustling of each purple curtain \\nThrilled me- filled me with fantastic terrors never felt before; \\nSo that now, to still the beating of my heart, I stood repeating, \\n\"\\'Tis some visitor entreating entrance at my chamber door- \\nSome late visitor entreating entrance at my chamber door;- \\n                This it is, and nothing more.\" \\n\\nPresently my soul grew stronger; hesitating then no longer, \\n\"Sir,\" said I, \"or Madam, truly your forgiveness I implore; \\nBut the fact is I was napping, and so gently you came rapping, \\nAnd so faintly you came tapping, tapping at my chamber door, \\nThat I scarce was sure I heard you\"- here I opened wide the door;- \\n                Darkness there, and nothing more. \\n\\nDeep into that darkness peering, long I stood there wondering, fearing, \\nDoubting, dreaming dreams no mortal ever dared to dream before; \\nBut the silence was unbroken, and the stillness gave no token, \\nAnd the only word there spoken was the whispered word, \"Lenore?\" \\nThis I whispered, and an echo murmured back the word, \"Lenore!\"- \\n                Merely this, and nothing more. \\n\\nBack into the chamber turning, all my soul within me burning, \\nSoon again I heard a tapping somewhat louder than before. \\n\"Surely,\" said I, \"surely that is something at my window lattice: \\nLet me see, then, what thereat is, and this mystery explore- \\nLet my heart be still a moment and this mystery explore;- \\n                \\'Tis the wind and nothing more!\" \\n\\nOpen here I flung the shutter, when, with many a flirt and flutter, \\nIn there stepped a stately Raven of the saintly days of yore; \\nNot the least obeisance made he; not a minute stopped or stayed he; \\nBut, with mien of lord or lady, perched above my chamber door- \\nPerched upon a bust of Pallas just above my chamber door- \\n                Perched, and sat, and nothing more. \\n\\nThen this ebony bird beguiling my sad fancy into smiling, \\nBy the grave and stern decorum of the countenance it wore. \\n\"Though thy crest be shorn and shaven, thou,\" I said, \"art sure no craven, \\nGhastly grim and ancient Raven wandering from the Nightly shore- \\nTell me what thy lordly name is on the Night\\'s Plutonian shore!\" \\n                Quoth the Raven, \"Nevermore.\" \\n\\nMuch I marvelled this ungainly fowl to hear discourse so plainly, \\nThough its answer little meaning- little relevancy bore; \\nFor we cannot help agreeing that no living human being \\nEver yet was blessed with seeing bird above his chamber door- \\nBird or beast upon the sculptured bust above his chamber door, \\n                With such name as \"Nevermore.\" \\n\\nBut the Raven, sitting lonely on the placid bust, spoke only \\nThat one word, as if his soul in that one word he did outpour. \\nNothing further then he uttered- not a feather then he fluttered- \\nTill I scarcely more than muttered, \"Other friends have flown before- \\nOn the morrow he will leave me, as my hopes have flown before.\" \\n                Then the bird said, \"Nevermore.\" \\n\\nStartled at the stillness broken by reply so aptly spoken, \\n\"Doubtless,\" said I, \"what it utters is its only stock and store, \\nCaught from some unhappy master whom unmerciful Disaster \\nFollowed fast and followed faster till his songs one burden bore- \\nTill the dirges of his Hope that melancholy burden bore \\n                Of \\'Never- nevermore\\'.\" \\n\\nBut the Raven still beguiling all my fancy into smiling, \\nStraight I wheeled a cushioned seat in front of bird, and bust and door; \\nThen upon the velvet sinking, I betook myself to linking \\nFancy unto fancy, thinking what this ominous bird of yore- \\nWhat this grim, ungainly, ghastly, gaunt and ominous bird of yore \\n                Meant in croaking \"Nevermore.\" \\n\\nThis I sat engaged in guessing, but no syllable expressing \\nTo the fowl whose fiery eyes now burned into my bosom\\'s core; \\nThis and more I sat divining, with my head at ease reclining \\nOn the cushion\\'s velvet lining that the lamp-light gloated o\\'er, \\nBut whose velvet violet lining with the lamp-light gloating o\\'er, \\n                She shall press, ah, nevermore! \\n\\nThen methought the air grew denser, perfumed from an unseen censer \\nSwung by Seraphim whose footfalls tinkled on the tufted floor. \\n\"Wretch,\" I cried, \"thy God hath lent thee- by these angels he hath sent thee \\nRespite- respite and nepenthe, from thy memories of Lenore! \\nQuaff, oh quaff this kind nepenthe and forget this lost Lenore!\" \\n                Quoth the Raven, \"Nevermore.\" \\n\\n\"Prophet!\" said I, \"thing of evil! - prophet still, if bird or devil! - \\nWhether Tempter sent, or whether tempest tossed thee here ashore, \\nDesolate yet all undaunted, on this desert land enchanted- \\nOn this home by Horror haunted- tell me truly, I implore- \\nIs there- is there balm in Gilead?- tell me- tell me, I implore!\" \\n                Quoth the Raven, \"Nevermore.\" \\n\\n\"Prophet!\" said I, \"thing of evil! - prophet still, if bird or devil! \\nBy that Heaven that bends above us- by that God we both adore- \\nTell this soul with sorrow laden if, within the distant Aidenn, \\nIt shall clasp a sainted maiden whom the angels name Lenore- \\nClasp a rare and radiant maiden whom the angels name Lenore.\" \\n                Quoth the Raven, \"Nevermore.\" \\n\\n\"Be that word our sign in parting, bird or fiend,\" I shrieked, upstarting- \\n\"Get thee back into the tempest and the Night\\'s Plutonian shore! \\nLeave no black plume as a token of that lie thy soul hath spoken! \\nLeave my loneliness unbroken!- quit the bust above my door! \\nTake thy beak from out my heart, and take thy form from off my door!\" \\n                Quoth the Raven, \"Nevermore.\" \\n\\nAnd the Raven, never flitting, still is sitting, still is sitting \\nOn the pallid bust of Pallas just above my chamber door; \\nAnd his eyes have all the seeming of a demon\\'s that is dreaming, \\nAnd the lamp-light o\\'er him streaming throws his shadow on the floor; \\nAnd my soul from out that shadow that lies floating on the floor \\n                Shall be lifted- nevermore! '"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text # running it like this better shows the raw content of the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## .readlines() for more of a delicate touch\n",
    "\n",
    "The other main method of reading in text is .readlines(), which does a lot of stuff for you.  But you do need to use it more purposefully.  We know that the text is organized by lines, and there are newlines aplenty.  .readlines() will create a list where each line of text is a separate string within that list.\n",
    "\n",
    "So a text that looks like:\n",
    "\n",
    "```\n",
    "I am a meat popsicle.\n",
    "I live to be eaten.\n",
    "My soul is frozen.\n",
    "There is a stick.```\n",
    "\n",
    "Will be turned into:\n",
    "\n",
    "```\n",
    "[ ['I am a meat popsicle.\\n'], \n",
    "  ['I live to be eaten.\\n'],\n",
    "  ['My soul is frozen.\\n'],\n",
    "  ['There is a stick.\\n'] ]\n",
    "```\n",
    "\n",
    "Sometimes you want this structure because your plan is to do stuff line by line.  But somethings you just want access to the entire document in a single flat structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## There is no one perfect way to read in a file\n",
    "\n",
    "There is no one perfect method of reading in a file, exept for the method that is right for your project.  You'll also develop your own personal style over time.  Which is fine!  Program development is an iterative process, so you may start down one path and then have to backtrack and pivot to another method.  Eventually you'll get a feel for which method is better for certain processes.\n",
    "\n",
    "The goal is to have the content in the structure that you want.  How you accomplish that isn't as important as the accuracy of the end product.  Focus on that rather than number of lines of code, etc.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## But which one do we want in this case?\n",
    "\n",
    "\n",
    "Let's go back to our original goal: counting the words.  Our first question is, what is the unit being measured?  In this case, we want the counts for the whole document.  Every word treated equally.  We aren't splitting it apart, and we don't care about lines.  We just want a bag of words for the entire thing.\n",
    "\n",
    "That's a pretty easy case for `.read()` since the newline breaks are a little meaningless.  We've already got that code above, so we'll copy it down here again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "file_in = open('raven.txt', 'r') # create IO object in read mode\n",
    "text = file_in.read() # apply .read() to the IO object and store the results as variable text\n",
    "file_in.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Once upon a midnight dreary, while I pondered, weak and weary, \n",
      "Over many a quaint and curious volume of forgotten lore, \n",
      "While I nodded, nearly napping, suddenly there came a tapping, \n",
      "As of some one\n"
     ]
    }
   ],
   "source": [
    "# for the sake of brevity, we'll be printing out the first 300 characters of the file\n",
    "# rather than the entire thing\n",
    "print(text[:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Essential string processing\n",
    "\n",
    "When it come to counting things in text, we need to remember that Python doesn't entirely know what a word is.  Nor does it care.  This `text` object that we have stored in Python is a series of characters.  This means we'll need to describe what a word is on a character level.  We won't dive super deep into this question because that isn't the point of this lesson, but there are a few things we can do to explore the application of some of our theoretical structures.\n",
    "\n",
    "As a first pass, let's say that \"words are things separated by white space\".  Which we know isn't the complete story, but it is sufficiently true that it is a good place to start.\n",
    "\n",
    "## `.split()`\n",
    "\n",
    "This function is a real work horse for text processing.  Say you have a long string with regular delimiters and you want to break it apart into those components.  `.split()` will do just that.  \n",
    "\n",
    "* It operates on:  a string you want to be broken up\n",
    "* You give it:  a string that is the delimiter\n",
    "* It gives you:  a list, with each element being the chunk of characters between each of those delimiters.  The delimiter you pass it will be 'consumed', meaning that it will be erased from the content returned back to you.\n",
    "* Pattern:  `new_text = original_text.split(my_delimiter)`  This method will not change the original string and returns a new list to you.  Meaning that you'll have to save that list value in a variable in order to keep it around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1378\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "maybe_words = text.split(\" \") # see that single space in the quotes?\n",
    "\n",
    "# let's investigate this object before we look at it\n",
    "\n",
    "print(len(maybe_words)) # so there are a bunch, this seems in the right area\n",
    "print(type(maybe_words)) # this is a list, so I need to manipulate it like one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activity!  \n",
    "\n",
    "Pair up and discuss what you're seeing inside of `maybe_words[:100]`.\n",
    "\n",
    "1. What content do you see in the strings?\n",
    "2. What words?\n",
    "3. Punctuation?\n",
    "4. Other observations?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Once', 'upon', 'a', 'midnight', 'dreary,', 'while', 'I', 'pondered,', 'weak', 'and', 'weary,', '\\nOver', 'many', 'a', 'quaint', 'and', 'curious', 'volume', 'of', 'forgotten', 'lore,', '\\nWhile', 'I', 'nodded,', 'nearly', 'napping,', 'suddenly', 'there', 'came', 'a', 'tapping,', '\\nAs', 'of', 'some', 'one', 'gently', 'rapping,', 'rapping', 'at', 'my', 'chamber', 'door.', '\\n\"\\'Tis', 'some', 'visitor,\"', 'I', 'muttered,', '\"tapping', 'at', 'my', 'chamber', 'door-', '\\n', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'Only', 'this,', 'and', 'nothing', 'more.\"', '\\n\\nAh,', 'distinctly', 'I', 'remember', 'it', 'was', 'in', 'the', 'bleak', 'December,', '\\nAnd', 'each', 'separate', 'dying', 'ember', 'wrought', 'its', 'ghost', 'upon', 'the', 'floor.', '\\nEagerly', 'I', 'wished', 'the', 'morrow;-', 'vainly']\n"
     ]
    }
   ],
   "source": [
    "# given that there are 1000+ words, let's just look at a slice\n",
    "\n",
    "print(maybe_words[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we take a closer look at what this has done, it's chosen to separate out long stretches of white space into a group of empty strings.  It wants to find the content betwee delimiters, but when there's no content?  This actully beneficial in other contexts, if super annoying in this one.\n",
    "\n",
    "So you can see that it turns a list of 4 spaces into a list of list of 5 empty strings.  You'll also see that newline characters were retained and added to the subsequent words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['', '', '', '', '']\n"
     ]
    }
   ],
   "source": [
    "print(\"    \".split(\" \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other way to use `.split()` is by giving it no arguments.  By default, it'll split on any amount of white space.  It'll also treat repeating instances of white space characters as a single delimiter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(\"    \".split()) # so now it's empty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1089\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "maybe_words = text.split() # not passing it anything!\n",
    "\n",
    "# let's investigate this object before we look at it\n",
    "\n",
    "print(len(maybe_words)) # fewer words this time!\n",
    "print(type(maybe_words)) # still a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Once', 'upon', 'a', 'midnight', 'dreary,', 'while', 'I', 'pondered,', 'weak', 'and', 'weary,', 'Over', 'many', 'a', 'quaint', 'and', 'curious', 'volume', 'of', 'forgotten', 'lore,', 'While', 'I', 'nodded,', 'nearly', 'napping,', 'suddenly', 'there', 'came', 'a', 'tapping,', 'As', 'of', 'some', 'one', 'gently', 'rapping,', 'rapping', 'at', 'my', 'chamber', 'door.', '\"\\'Tis', 'some', 'visitor,\"', 'I', 'muttered,', '\"tapping', 'at', 'my', 'chamber', 'door-', 'Only', 'this,', 'and', 'nothing', 'more.\"', 'Ah,', 'distinctly', 'I', 'remember', 'it', 'was', 'in', 'the', 'bleak', 'December,', 'And', 'each', 'separate', 'dying', 'ember', 'wrought', 'its', 'ghost', 'upon', 'the', 'floor.', 'Eagerly', 'I', 'wished', 'the', 'morrow;-', 'vainly', 'I', 'had', 'sought', 'to', 'borrow', 'From', 'my', 'books', 'surcease', 'of', 'sorrow-', 'sorrow', 'for', 'the', 'lost', 'Lenore-']\n"
     ]
    }
   ],
   "source": [
    "print(maybe_words[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now this is looking a lot more reasonable!  So let's play with a neat tool, the `Counter()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "560\n",
      "<class 'collections.Counter'>\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "counted_words = Counter(maybe_words)\n",
    "\n",
    "#hmmm, what is this\n",
    "print(len(counted_words)) # so there are fewer items in here\n",
    "print(type(counted_words)) # not a list?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unhashable type: 'slice'",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-6ed48c46a595>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcounted_words\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: unhashable type: 'slice'"
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "print(counted_words[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notation doesn't work because we're not dealing with a list anymore!  We're dealing with a new object type, a Counter object.  Under the hood, it look sa lot like a dictionary.  Anyhow, this object is unordered, so we can't go by index position.  When we print the entire thing out:\n",
    "\n",
    "`print(counted_words)` \n",
    "\n",
    "We can see that it starts with `Counter({'the': 56, 'and': 30, 'I': 27,`...  These are key:value pairs.  So `'the'` is a key, and that `56` is the value.  In this case, all the unique words in the list appear as the keys, and the number of times they appear are the values.  \n",
    "\n",
    "Don't believe me? We can check this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1089\n",
      "1089\n"
     ]
    }
   ],
   "source": [
    "print(len(maybe_words))\n",
    "print(sum(counted_words.values()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Counter` objects have all kinds of fun data exploration methods.  Most importantly, we can look at the most common."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activity!\n",
    "\n",
    "Work with your partner to discuss the results of `print(counted_words.most_common(20))`\n",
    "\n",
    "1.  What are you observations of the counts?\n",
    "2.  What's happening with the words that you see?\n",
    "3.  Is this analysis done?  Can we do more to refine our text for better results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 56), ('and', 30), ('I', 27), ('my', 24), ('of', 20), ('that', 15), ('a', 15), ('chamber', 11), ('this', 11), ('\"Nevermore.\"', 8), ('And', 8), ('at', 8), ('bird', 8), ('is', 8), ('from', 7), ('with', 7), ('in', 7), ('no', 7), ('above', 7), ('or', 7)]\n"
     ]
    }
   ],
   "source": [
    "print(counted_words.most_common(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the 12 most common words in the document, and all that counting has been done for us.  While cool, we can already see some limitations to our methods.\n",
    "\n",
    "* `'and'` and `'And'` have been counted as separate words\n",
    "* `'Nevermore.`' has a period on the end\n",
    "* very few of these words are pretty interesting\n",
    "\n",
    "We can tackle all three of these issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lowercase things for better counting\n",
    "\n",
    "Recall our discussion of that ASCII table and that `A` and `a` are totally separate letters in the eyes of Python.  Sometimes we care about this, such as an instance of someone named Rose versus someone receiving a rose.  But more often than not, we need to lowercase everything to get better counts.  You could uppercase everything to standardize stuff for counting...but pain.  \n",
    "\n",
    "## `.lower()`\n",
    "\n",
    "We can use the `.lower()` method to lowercase every character.\n",
    "\n",
    "* It operates on:  a string you want to lowercase\n",
    "* You give it:  usually nothing, so leave the `()` empty\n",
    "* It gives you:  A new string, with all characters lowercased (that can be). This method will not change the original string and returns a new list to you.  Meaning that you'll have to save that list value in a variable in order to keep it around.\n",
    "* Basic pattern:  `lower_string = original_string.lower()`\n",
    "\n",
    "Let's start from the top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 56), ('and', 38), ('i', 27), ('my', 24), ('of', 21), ('that', 17), ('this', 15), ('a', 15), ('chamber', 11), ('on', 10), ('bird', 9), ('is', 9), ('from', 8), ('with', 8), ('at', 8), ('in', 8), ('\"nevermore.\"', 8), ('by', 7), ('above', 7), ('then', 7)]\n"
     ]
    }
   ],
   "source": [
    "file_in = open('raven.txt', 'r') # create IO object in read mode\n",
    "text = file_in.read() # apply .read() to the IO object and store the results as variable text\n",
    "file_in.close()\n",
    "\n",
    "lower_text = text.lower() # our new addition!\n",
    "\n",
    "maybe_words = lower_text.split()\n",
    "counted_words = Counter(maybe_words)\n",
    "\n",
    "print(counted_words.most_common(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the `('and', 30)` and `('And', 8)` have been combined into `('and', 38)`.  \n",
    "\n",
    "But we've still got that punctuation thing going.  We can see this is a problem with `('\"nevermore.\"', 8)`. I can show you a bit more advanced work to highlight that we are actually losing data with this punctuation thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"nevermore.\"\n",
      "\"nevermore.\"\n",
      "\"nevermore.\"\n",
      "nevermore'.\"\n",
      "\"nevermore.\"\n",
      "nevermore!\n",
      "\"nevermore.\"\n",
      "\"nevermore.\"\n",
      "\"nevermore.\"\n",
      "\"nevermore.\"\n",
      "nevermore!\n"
     ]
    }
   ],
   "source": [
    "for word in maybe_words:\n",
    "    if 'nevermore' in word:\n",
    "        print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This checks for any unbroken instance of 'nevermore' inside of a string and prints it out if it is found.  We can see that there are several versions of this.  There are 11 results here with three different kinds of punctuation issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting rid of punctuation\n",
    "\n",
    "This is something a little more involved, so hang on with me.\n",
    "\n",
    "It would be great to be able to say \"replace all these things with nothing\", but when you have multiple things to replace with one thing...things get weird.  There are many ways to solve this problem, this is just one.\n",
    "\n",
    "What we're going to do is take advantage of the regular expression module.  That is a whole other conversation, but regualr expressions are a way to describe text patterns.  We create a character class that has all the desired punctuation characters in it, and tell it to replace any match with an empty string (effectively, nothing).\n",
    "\n",
    "Character classes in regular expressions are denoted by square brackets.  For example `[abc]` would match `a`, or `b`, or `c`.  We'll need to first import the re module, which contains all the regular expression tools for Python, and then use the `re.sub()` function.\n",
    "\n",
    "`re.sub()` works a little differently from the other methods that we've explored.  Instead of it being called on the string you want to change, you call the function and pass it three arguments:  the pattern for replacement (in our case the character class), the replacement value (an empty string for this), and the string to be changed.\n",
    "\n",
    "This is also one of our first instances of calling a function from another module.  We sort of did this with Counter, but when we imported it directly from the module we don't need to use the module name.  The more generic `import module_name` will require that we put `module_name` before the function name.\n",
    "\n",
    "We've got a variable with a string of the letters to erase, and need to transform that into a character class.  Remember that character classes are a series of characters inside square brackets.  So `'[' + string + ']'` should do the trick."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hll, hmn mt crtr\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "original = \"Hello, human meat creature\"\n",
    "\n",
    "letters_to_erase= 'aeiou'\n",
    "\n",
    "print(re.sub(\"[\" + letters_to_erase + \"]\", '', original))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now capable of taking a string, making a character class for items to remove, and then running that through `re.sub()` to create a new string without those characters.\n",
    "\n",
    "The only thing missing is how to get all of the punctuation characters.  The handy `string` module is our friend in this case.  It has a series of commands that produce a variety of latin-1 based letter sequences.  This may not save characters in our code, but makes our intent clearer by having dedicated names.\n",
    "\n",
    "We'll need to import string like we did with re, meaning that we need to add `string.` before all our calls.  You can read all about the other items in the string module here: https://docs.python.org/3.6/library/string.html\n",
    "\n",
    "Special note: while we've gotten used to having `()` on the end of our function calls, this is a little different.  Internally, these aren't actually functions, they're variables inside the module that we're getting values for.  Anyhow, you don't care why, just remember that you omit the `()` when trying to access the string values from the string module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
     ]
    }
   ],
   "source": [
    "import string\n",
    "\n",
    "print(string.punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a lot easier than trying to type that a whole bunch of times.  \n",
    "\n",
    "Let's put all this together now.  Recall that our `lower_text` variable has our lowecased text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "once upon a midnight dreary while i pondered weak and weary \n",
      "over many a quaint and curious volume of forgotten lore \n",
      "while i nodded nearly napping suddenly there came a tapping \n",
      "as of some one gently rapping rapping at my chamber door \n",
      "tis some visitor i muttered tapping at my chamber door \n",
      "                only this and nothing more \n",
      "\n",
      "ah distinctly i remember it was in the bleak december \n",
      "and each separate dying ember wrought its ghost upon the floor \n",
      "eagerly i wished the morrow vainly i had sought to borrow \n",
      "from my books surcease of sorrow sorrow for the lost lenore \n",
      "for the rare and radiant maiden whom the angels name lenore \n",
      "                nameless here for evermore \n",
      "\n",
      "and the silken sad uncertain rustling of each purple curtain \n",
      "thrilled me filled me with fantastic terrors never felt before \n",
      "so that now to still the beating of my heart i stood repeating \n",
      "tis some visitor entreating entrance at my chamber door \n",
      "some late visitor entreating entrance at my chamber door \n",
      "                this it is and nothing more \n",
      "\n",
      "presently my soul grew stronger hesitating then no longer \n",
      "sir said i or madam truly your forgiveness i implore \n",
      "but the fact is i was napping and so gently you came rapping \n",
      "and so faintly you came tapping tapping at my chamber door \n",
      "that i scarce was sure i heard you here i opened wide the door \n",
      "                darkness there and nothing more \n",
      "\n",
      "deep into that darkness peering long i stood there wondering fearing \n",
      "doubting dreaming dreams no mortal ever dared to dream before \n",
      "but the silence was unbroken and the stillness gave no token \n",
      "and the only word there spoken was the whispered word lenore \n",
      "this i whispered and an echo murmured back the word lenore \n",
      "                merely this and nothing more \n",
      "\n",
      "back into the chamber turning all my soul within me burning \n",
      "soon again i heard a tapping somewhat louder than before \n",
      "surely said i surely that is something at my window lattice \n",
      "let me see then what thereat is and this mystery explore \n",
      "let my heart be still a moment and this mystery explore \n",
      "                tis the wind and nothing more \n",
      "\n",
      "open here i flung the shutter when with many a flirt and flutter \n",
      "in there stepped a stately raven of the saintly days of yore \n",
      "not the least obeisance made he not a minute stopped or stayed he \n",
      "but with mien of lord or lady perched above my chamber door \n",
      "perched upon a bust of pallas just above my chamber door \n",
      "                perched and sat and nothing more \n",
      "\n",
      "then this ebony bird beguiling my sad fancy into smiling \n",
      "by the grave and stern decorum of the countenance it wore \n",
      "though thy crest be shorn and shaven thou i said art sure no craven \n",
      "ghastly grim and ancient raven wandering from the nightly shore \n",
      "tell me what thy lordly name is on the nights plutonian shore \n",
      "                quoth the raven nevermore \n",
      "\n",
      "much i marvelled this ungainly fowl to hear discourse so plainly \n",
      "though its answer little meaning little relevancy bore \n",
      "for we cannot help agreeing that no living human being \n",
      "ever yet was blessed with seeing bird above his chamber door \n",
      "bird or beast upon the sculptured bust above his chamber door \n",
      "                with such name as nevermore \n",
      "\n",
      "but the raven sitting lonely on the placid bust spoke only \n",
      "that one word as if his soul in that one word he did outpour \n",
      "nothing further then he uttered not a feather then he fluttered \n",
      "till i scarcely more than muttered other friends have flown before \n",
      "on the morrow he will leave me as my hopes have flown before \n",
      "                then the bird said nevermore \n",
      "\n",
      "startled at the stillness broken by reply so aptly spoken \n",
      "doubtless said i what it utters is its only stock and store \n",
      "caught from some unhappy master whom unmerciful disaster \n",
      "followed fast and followed faster till his songs one burden bore \n",
      "till the dirges of his hope that melancholy burden bore \n",
      "                of never nevermore \n",
      "\n",
      "but the raven still beguiling all my fancy into smiling \n",
      "straight i wheeled a cushioned seat in front of bird and bust and door \n",
      "then upon the velvet sinking i betook myself to linking \n",
      "fancy unto fancy thinking what this ominous bird of yore \n",
      "what this grim ungainly ghastly gaunt and ominous bird of yore \n",
      "                meant in croaking nevermore \n",
      "\n",
      "this i sat engaged in guessing but no syllable expressing \n",
      "to the fowl whose fiery eyes now burned into my bosoms core \n",
      "this and more i sat divining with my head at ease reclining \n",
      "on the cushions velvet lining that the lamplight gloated oer \n",
      "but whose velvet violet lining with the lamplight gloating oer \n",
      "                she shall press ah nevermore \n",
      "\n",
      "then methought the air grew denser perfumed from an unseen censer \n",
      "swung by seraphim whose footfalls tinkled on the tufted floor \n",
      "wretch i cried thy god hath lent thee by these angels he hath sent thee \n",
      "respite respite and nepenthe from thy memories of lenore \n",
      "quaff oh quaff this kind nepenthe and forget this lost lenore \n",
      "                quoth the raven nevermore \n",
      "\n",
      "prophet said i thing of evil  prophet still if bird or devil  \n",
      "whether tempter sent or whether tempest tossed thee here ashore \n",
      "desolate yet all undaunted on this desert land enchanted \n",
      "on this home by horror haunted tell me truly i implore \n",
      "is there is there balm in gilead tell me tell me i implore \n",
      "                quoth the raven nevermore \n",
      "\n",
      "prophet said i thing of evil  prophet still if bird or devil \n",
      "by that heaven that bends above us by that god we both adore \n",
      "tell this soul with sorrow laden if within the distant aidenn \n",
      "it shall clasp a sainted maiden whom the angels name lenore \n",
      "clasp a rare and radiant maiden whom the angels name lenore \n",
      "                quoth the raven nevermore \n",
      "\n",
      "be that word our sign in parting bird or fiend i shrieked upstarting \n",
      "get thee back into the tempest and the nights plutonian shore \n",
      "leave no black plume as a token of that lie thy soul hath spoken \n",
      "leave my loneliness unbroken quit the bust above my door \n",
      "take thy beak from out my heart and take thy form from off my door \n",
      "                quoth the raven nevermore \n",
      "\n",
      "and the raven never flitting still is sitting still is sitting \n",
      "on the pallid bust of pallas just above my chamber door \n",
      "and his eyes have all the seeming of a demons that is dreaming \n",
      "and the lamplight oer him streaming throws his shadow on the floor \n",
      "and my soul from out that shadow that lies floating on the floor \n",
      "                shall be lifted nevermore \n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import string\n",
    "\n",
    "file_in = open('raven.txt', 'r') # create IO object in read mode\n",
    "text = file_in.read() # apply .read() to the IO object and store the results as variable text\n",
    "file_in.close()\n",
    "\n",
    "lower_text = text.lower() # our new addition!\n",
    "\n",
    "print(re.sub('[' + string.punctuation + ']', '', lower_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 56), ('and', 38), ('i', 32), ('my', 24), ('of', 21), ('this', 17), ('that', 17), ('a', 15), ('door', 14), ('chamber', 11), ('nevermore', 11), ('is', 11), ('bird', 10), ('on', 10), ('raven', 10)]\n"
     ]
    }
   ],
   "source": [
    "# and now we recount things using this cleaned text\n",
    "\n",
    "import re\n",
    "import string\n",
    "from collections import Counter\n",
    "\n",
    "file_in = open('raven.txt', 'r') # create IO object in read mode\n",
    "text = file_in.read() # apply .read() to the IO object and store the results as variable text\n",
    "file_in.close()\n",
    "\n",
    "lower_text = text.lower() # our new addition!\n",
    "\n",
    "no_punc_lower_text = re.sub('[' + string.punctuation + ']', '', lower_text)\n",
    "\n",
    "maybe_words = no_punc_lower_text.split()\n",
    "\n",
    "counted_words = Counter(maybe_words)\n",
    "\n",
    "print(counted_words.most_common(15))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This isn't too terribly different from before, but we can see that ` ('\"nevermore.\"', 8)` has actually changed to `('nevermore', 11)`.  So there must have been other variations of the word floating around our original list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excluding certain words\n",
    "\n",
    "This will not be a complete discussion of stop words, why you want to exclude them, and what's happening here.  We can see that many fo our most common words are common words required to construct syntatically correct English, rather than words that hold interesting meaning within the poem.  We'd like to remove them before counting.\n",
    "\n",
    "This will have us explore a basic accumulator pattern.  Meaning that we will traverse over our list of words, check if they do or don't belong, and then gather up all the words that we want.  Accumulator patterns are classic and prevalent in nearly every coding pattern when dealing with data.\n",
    "\n",
    "Basic accumulator pattern:\n",
    "\n",
    "```python\n",
    "empty_bucket = [] # an empty list to gather our words\n",
    "\n",
    "for word in list_of_words: # loop over your list of words\n",
    "    # do something to either change the word or test something about it\n",
    "    empty_bucket.append(word) # add the word to our accumulator variable\n",
    "```\n",
    "\n",
    "Our first step will be to make the list of words we don't want, which we can do so by looking at the top list of counted words and picking some out that don't interest us.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words_to_omit = ['the', 'and', 'i', 'my', 'of', 'this', 'that', 'a', 'is', 'on']\n",
    "\n",
    "# we can leave them lower case because we're only going to be\n",
    "# traversing over our list of lowercase words\n",
    "# these are from my top 15 words, so let's see what happens\n",
    "\n",
    "words_i_want = []\n",
    "\n",
    "for word in maybe_words:\n",
    "    if word in words_to_omit:\n",
    "        continue\n",
    "    else:\n",
    "        words_i_want.append(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're using this `in` keyword again.  We previously used it to check if there's an unbroken instance of 'nevermore' in a larger string.  Now we're using it to check membership of an item in a list.\n",
    "\n",
    "We've also added in an `else:` statement, which pairs up with that `if`.  It will execute in conditions where the if statement is false.\n",
    "\n",
    "But let's view our newly accumulated list of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['once', 'upon', 'midnight', 'dreary', 'while', 'pondered', 'weak', 'weary', 'over', 'many', 'quaint', 'curious', 'volume', 'forgotten', 'lore', 'while', 'nodded', 'nearly', 'napping', 'suddenly', 'there', 'came', 'tapping', 'as', 'some', 'one', 'gently', 'rapping', 'rapping', 'at', 'chamber', 'door', 'tis', 'some', 'visitor', 'muttered', 'tapping', 'at', 'chamber', 'door', 'only', 'nothing', 'more', 'ah', 'distinctly', 'remember', 'it', 'was', 'in', 'bleak', 'december', 'each', 'separate', 'dying', 'ember', 'wrought', 'its', 'ghost', 'upon', 'floor', 'eagerly', 'wished', 'morrow', 'vainly', 'had', 'sought', 'to', 'borrow', 'from', 'books', 'surcease', 'sorrow', 'sorrow', 'for', 'lost', 'lenore', 'for', 'rare', 'radiant', 'maiden', 'whom', 'angels', 'name', 'lenore', 'nameless', 'here', 'for', 'evermore', 'silken', 'sad', 'uncertain', 'rustling', 'each', 'purple', 'curtain', 'thrilled', 'me', 'filled', 'me', 'with', 'fantastic', 'terrors', 'never', 'felt', 'before', 'so', 'now', 'to', 'still', 'beating', 'heart', 'stood', 'repeating', 'tis', 'some', 'visitor', 'entreating', 'entrance', 'at', 'chamber', 'door', 'some', 'late', 'visitor', 'entreating', 'entrance', 'at', 'chamber', 'door', 'it', 'nothing', 'more', 'presently', 'soul', 'grew', 'stronger', 'hesitating', 'then', 'no', 'longer', 'sir', 'said', 'or', 'madam', 'truly', 'your', 'forgiveness', 'implore', 'but', 'fact', 'was', 'napping', 'so', 'gently', 'you', 'came', 'rapping', 'so', 'faintly', 'you', 'came', 'tapping', 'tapping', 'at', 'chamber', 'door', 'scarce', 'was', 'sure', 'heard', 'you', 'here', 'opened', 'wide', 'door', 'darkness', 'there', 'nothing', 'more', 'deep', 'into', 'darkness', 'peering', 'long', 'stood', 'there', 'wondering', 'fearing', 'doubting', 'dreaming', 'dreams', 'no', 'mortal', 'ever', 'dared', 'to', 'dream', 'before', 'but', 'silence', 'was', 'unbroken', 'stillness', 'gave', 'no', 'token', 'only', 'word', 'there', 'spoken', 'was', 'whispered', 'word', 'lenore', 'whispered', 'an', 'echo', 'murmured', 'back', 'word', 'lenore', 'merely', 'nothing', 'more', 'back', 'into', 'chamber', 'turning', 'all', 'soul', 'within', 'me', 'burning', 'soon', 'again', 'heard', 'tapping', 'somewhat', 'louder', 'than', 'before', 'surely', 'said', 'surely', 'something', 'at', 'window', 'lattice', 'let', 'me', 'see', 'then', 'what', 'thereat', 'mystery', 'explore', 'let', 'heart', 'be', 'still', 'moment', 'mystery', 'explore', 'tis', 'wind', 'nothing', 'more', 'open', 'here', 'flung', 'shutter', 'when', 'with', 'many', 'flirt', 'flutter', 'in', 'there', 'stepped', 'stately', 'raven', 'saintly', 'days', 'yore', 'not', 'least', 'obeisance', 'made', 'he', 'not', 'minute', 'stopped', 'or', 'stayed', 'he', 'but', 'with', 'mien', 'lord', 'or', 'lady', 'perched', 'above', 'chamber', 'door', 'perched', 'upon', 'bust', 'pallas', 'just', 'above', 'chamber', 'door', 'perched', 'sat', 'nothing', 'more', 'then', 'ebony', 'bird', 'beguiling', 'sad', 'fancy', 'into', 'smiling', 'by', 'grave', 'stern', 'decorum', 'countenance', 'it', 'wore', 'though', 'thy', 'crest', 'be', 'shorn', 'shaven', 'thou', 'said', 'art', 'sure', 'no', 'craven', 'ghastly', 'grim', 'ancient', 'raven', 'wandering', 'from', 'nightly', 'shore', 'tell', 'me', 'what', 'thy', 'lordly', 'name', 'nights', 'plutonian', 'shore', 'quoth', 'raven', 'nevermore', 'much', 'marvelled', 'ungainly', 'fowl', 'to', 'hear', 'discourse', 'so', 'plainly', 'though', 'its', 'answer', 'little', 'meaning', 'little', 'relevancy', 'bore', 'for', 'we', 'cannot', 'help', 'agreeing', 'no', 'living', 'human', 'being', 'ever', 'yet', 'was', 'blessed', 'with', 'seeing', 'bird', 'above', 'his', 'chamber', 'door', 'bird', 'or', 'beast', 'upon', 'sculptured', 'bust', 'above', 'his', 'chamber', 'door', 'with', 'such', 'name', 'as', 'nevermore', 'but', 'raven', 'sitting', 'lonely', 'placid', 'bust', 'spoke', 'only', 'one', 'word', 'as', 'if', 'his', 'soul', 'in', 'one', 'word', 'he', 'did', 'outpour', 'nothing', 'further', 'then', 'he', 'uttered', 'not', 'feather', 'then', 'he', 'fluttered', 'till', 'scarcely', 'more', 'than', 'muttered', 'other', 'friends', 'have', 'flown', 'before', 'morrow', 'he', 'will', 'leave', 'me', 'as', 'hopes', 'have', 'flown', 'before', 'then', 'bird', 'said', 'nevermore', 'startled', 'at', 'stillness', 'broken', 'by', 'reply', 'so', 'aptly', 'spoken', 'doubtless', 'said', 'what', 'it', 'utters', 'its', 'only', 'stock', 'store', 'caught', 'from', 'some', 'unhappy', 'master', 'whom', 'unmerciful', 'disaster', 'followed', 'fast', 'followed', 'faster', 'till', 'his', 'songs', 'one', 'burden', 'bore', 'till', 'dirges', 'his', 'hope', 'melancholy', 'burden', 'bore', 'never', 'nevermore', 'but', 'raven', 'still', 'beguiling', 'all', 'fancy', 'into', 'smiling', 'straight', 'wheeled', 'cushioned', 'seat', 'in', 'front', 'bird', 'bust', 'door', 'then', 'upon', 'velvet', 'sinking', 'betook', 'myself', 'to', 'linking', 'fancy', 'unto', 'fancy', 'thinking', 'what', 'ominous', 'bird', 'yore', 'what', 'grim', 'ungainly', 'ghastly', 'gaunt', 'ominous', 'bird', 'yore', 'meant', 'in', 'croaking', 'nevermore', 'sat', 'engaged', 'in', 'guessing', 'but', 'no', 'syllable', 'expressing', 'to', 'fowl', 'whose', 'fiery', 'eyes', 'now', 'burned', 'into', 'bosoms', 'core', 'more', 'sat', 'divining', 'with', 'head', 'at', 'ease', 'reclining', 'cushions', 'velvet', 'lining', 'lamplight', 'gloated', 'oer', 'but', 'whose', 'velvet', 'violet', 'lining', 'with', 'lamplight', 'gloating', 'oer', 'she', 'shall', 'press', 'ah', 'nevermore', 'then', 'methought', 'air', 'grew', 'denser', 'perfumed', 'from', 'an', 'unseen', 'censer', 'swung', 'by', 'seraphim', 'whose', 'footfalls', 'tinkled', 'tufted', 'floor', 'wretch', 'cried', 'thy', 'god', 'hath', 'lent', 'thee', 'by', 'these', 'angels', 'he', 'hath', 'sent', 'thee', 'respite', 'respite', 'nepenthe', 'from', 'thy', 'memories', 'lenore', 'quaff', 'oh', 'quaff', 'kind', 'nepenthe', 'forget', 'lost', 'lenore', 'quoth', 'raven', 'nevermore', 'prophet', 'said', 'thing', 'evil', 'prophet', 'still', 'if', 'bird', 'or', 'devil', 'whether', 'tempter', 'sent', 'or', 'whether', 'tempest', 'tossed', 'thee', 'here', 'ashore', 'desolate', 'yet', 'all', 'undaunted', 'desert', 'land', 'enchanted', 'home', 'by', 'horror', 'haunted', 'tell', 'me', 'truly', 'implore', 'there', 'there', 'balm', 'in', 'gilead', 'tell', 'me', 'tell', 'me', 'implore', 'quoth', 'raven', 'nevermore', 'prophet', 'said', 'thing', 'evil', 'prophet', 'still', 'if', 'bird', 'or', 'devil', 'by', 'heaven', 'bends', 'above', 'us', 'by', 'god', 'we', 'both', 'adore', 'tell', 'soul', 'with', 'sorrow', 'laden', 'if', 'within', 'distant', 'aidenn', 'it', 'shall', 'clasp', 'sainted', 'maiden', 'whom', 'angels', 'name', 'lenore', 'clasp', 'rare', 'radiant', 'maiden', 'whom', 'angels', 'name', 'lenore', 'quoth', 'raven', 'nevermore', 'be', 'word', 'our', 'sign', 'in', 'parting', 'bird', 'or', 'fiend', 'shrieked', 'upstarting', 'get', 'thee', 'back', 'into', 'tempest', 'nights', 'plutonian', 'shore', 'leave', 'no', 'black', 'plume', 'as', 'token', 'lie', 'thy', 'soul', 'hath', 'spoken', 'leave', 'loneliness', 'unbroken', 'quit', 'bust', 'above', 'door', 'take', 'thy', 'beak', 'from', 'out', 'heart', 'take', 'thy', 'form', 'from', 'off', 'door', 'quoth', 'raven', 'nevermore', 'raven', 'never', 'flitting', 'still', 'sitting', 'still', 'sitting', 'pallid', 'bust', 'pallas', 'just', 'above', 'chamber', 'door', 'his', 'eyes', 'have', 'all', 'seeming', 'demons', 'dreaming', 'lamplight', 'oer', 'him', 'streaming', 'throws', 'his', 'shadow', 'floor', 'soul', 'from', 'out', 'shadow', 'lies', 'floating', 'floor', 'shall', 'be', 'lifted', 'nevermore']\n"
     ]
    }
   ],
   "source": [
    "print(words_i_want)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('door', 14), ('chamber', 11), ('nevermore', 11), ('bird', 10), ('raven', 10), ('me', 9), ('from', 8), ('more', 8), ('then', 8), ('with', 8), ('at', 8), ('in', 8), ('or', 8), ('lenore', 8), ('thy', 7)]\n"
     ]
    }
   ],
   "source": [
    "# and then add this to our larger script\n",
    "\n",
    "import re\n",
    "import string\n",
    "from collections import Counter\n",
    "\n",
    "file_in = open('raven.txt', 'r') # create IO object in read mode\n",
    "text = file_in.read() # apply .read() to the IO object and store the results as variable text\n",
    "file_in.close()\n",
    "\n",
    "lower_text = text.lower() # our new addition!\n",
    "\n",
    "no_punc_lower_text = re.sub('[' + string.punctuation + ']', '', lower_text)\n",
    "\n",
    "maybe_words = no_punc_lower_text.split()\n",
    "\n",
    "words_to_omit = ['the', 'and', 'i', 'my', 'of', 'this', 'that', 'a', 'is', 'on']\n",
    "\n",
    "words_i_want = []\n",
    "\n",
    "for word in maybe_words:\n",
    "    if word in words_to_omit:\n",
    "        continue\n",
    "    else:\n",
    "        words_i_want.append(word)\n",
    "\n",
    "counted_words = Counter(words_i_want) # I want to count this new list instead\n",
    "\n",
    "print(counted_words.most_common(15))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activity!\n",
    "\n",
    "Work with your partner to expand the list of words to omit and explore the results on the data being generated.  Take about 10 minutes to explore this and we'll discuss which words you chose and why."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('door', 14), ('nevermore', 11), ('chamber', 11), ('raven', 10), ('bird', 10), ('lenore', 8), ('said', 7), ('still', 7), ('nothing', 7), ('no', 7), ('above', 7), ('bust', 6), ('was', 6), ('to', 6), ('word', 6)]\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import string\n",
    "from collections import Counter\n",
    "\n",
    "file_in = open('raven.txt', 'r') # create IO object in read mode\n",
    "text = file_in.read() # apply .read() to the IO object and store the results as variable text\n",
    "file_in.close()\n",
    "\n",
    "lower_text = text.lower() # our new addition!\n",
    "\n",
    "no_punc_lower_text = re.sub('[' + string.punctuation + ']', '', lower_text)\n",
    "\n",
    "maybe_words = no_punc_lower_text.split()\n",
    "\n",
    "words_to_omit = ['the', 'and', 'i', 'my', 'of', 'this', 'that', 'a', 'is', \\\n",
    "                 'on', 'me', 'from', 'then', 'with', 'at', 'in', 'or', 'by', 'he', 'but',\\\n",
    "                 'there', 'his', 'more', 'thy', 'but']\n",
    "\n",
    "words_i_want = []\n",
    "\n",
    "for word in maybe_words:\n",
    "    if word in words_to_omit:\n",
    "        continue\n",
    "    else:\n",
    "        words_i_want.append(word)\n",
    "\n",
    "counted_words = Counter(words_i_want) # I want to count this new list instead\n",
    "\n",
    "print(counted_words.most_common(15))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# And we're done...\n",
    "\n",
    "Hopefully now you're more comfortable with working inside of a scripting system, perhaps even working inside of Jupyter Notebooks.  If you were working in the notebooks, you won't need to have repeated so much, but if you're working inside of a script you'll be editing the same document.  \n",
    "\n",
    "The process of developing a script will happen in the process of you develping your analyses.  Doing this inside of the jpyter notebook, as we've done, will help you narrate the choices and presumptions you've made.\n",
    "\n",
    "What this hasn't covered is how to write out these results, but that's another conversation.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus template: writing out the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "print_me = []\n",
    "\n",
    "for word, count in counted_words.items():\n",
    "#     print(word, count)\n",
    "    print_me.append([word, count])\n",
    "    \n",
    "with open('word_counts.csv', 'w') as file_out:\n",
    "    writeme = csv.writer(file_out)\n",
    "    writeme.writerow(['word', 'count'])\n",
    "    writeme.writerows(print_me)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    ""
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3.0
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}