02_UNIX_reading.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UNIX Commands for Data Scientists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this reading we will go through the UNIX commands introduced in this week's video again so you can familiarize more with their syntax.\n",
    "\n",
    "At any point feel free to modify the code and explore yourself the functionality of the UNIX shell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to execute the commands\n",
    "\n",
    "On **Windows** you need to open Git Bash and paste the command into the terminal, either using the mouse right click or right clicking on the top window border and select edit -> paste.\n",
    "\n",
    "On **Mac OS** or **Linux** you can choose to either execute commands through this Jupyter Notebook or copy paste them into a terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Declare Filename\n",
    "\n",
    "First we want to create a variable to hold the filename of the text file we want to analyze, so that if we want to change it later, we can change it only in this line.\n",
    "\n",
    "This is the only case where the syntax is different in the Jupyter Notebook and running directly in the shell.\n",
    "\n",
    "In the Notebook, each command is run on a separate shell process therefore we need to store `filename` in an enviromental variable, which is a way to set a persistent variable. This is performed using the `%env` IPython Magic function, execute `%env?` to learn more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shakespeare.txt\r\n"
     ]
    }
   ],
   "source": [
    "!ls ./unix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: filename=./unix/shakespeare.txt\n"
     ]
    }
   ],
   "source": [
    "%env filename=./unix/shakespeare.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are instead running in a shell, you can just define a shell variable named filename with this syntax:\n",
    "\n",
    "    filename=./unix/shakespeare.txt\n",
    "    \n",
    "Make sure that there are **no spaces** around the equal sign."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can verify that the variable is now defined by printing it out with `echo`. For the rest of this reading we will use this variable to point to the filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./unix/shakespeare.txt\r\n"
     ]
    }
   ],
   "source": [
    "!echo $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`head` prints some lines from the top of the file, you can specify how many with `-n`, what happens if you don't specify a number of lines?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the 100th Etext file presented by Project Gutenberg, and\r",
      "\r\n",
      "is presented in cooperation with World Library, Inc., from their\r",
      "\r\n",
      "Library of the Future and Shakespeare CDROMS.  Project Gutenberg\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 3 $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PERSONAL USE ONLY, AND (2) ARE NOT DISTRIBUTED OR USED\r",
      "\r\n",
      "COMMERCIALLY.  PROHIBITED COMMERCIAL DISTRIBUTION INCLUDES BY ANY\r",
      "\r\n",
      "SERVICE THAT CHARGES FOR DOWNLOAD TIME OR FOR MEMBERSHIP.>>\r",
      "\r\n",
      "\r",
      "\r\n",
      "\r",
      "\r\n",
      "\r",
      "\r\n",
      "End of this Etext of The Complete Works of William Shakespeare\r",
      "\r\n",
      "\r",
      "\r\n",
      "\r",
      "\r\n",
      "\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!tail -n 10 $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## wc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`wc`, which stands for wordcount, prints the number of lines, words and characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 124505  901447 5583442 ./unix/shakespeare.txt\r\n"
     ]
    }
   ],
   "source": [
    "!wc $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you can specify `-l` to only print the number of lines. Execute (in Git Bash on Windows or on Linux):\n",
    "\n",
    "    wc --help\n",
    "    \n",
    "or (on Mac or on Linux):\n",
    "\n",
    "    man wc\n",
    "        \n",
    "to find out how to print only words instead. Or guess!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "124505 ./unix/shakespeare.txt\r\n"
     ]
    }
   ],
   "source": [
    "!wc -l $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use pipes with `|` to stream the output of a command to the input of another, this is useful to compone many tools together to achieve a more complicated output.\n",
    "\n",
    "For example `cat` dumps the content of a file, then we can pipe it to `wc`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "124505\r\n"
     ]
    }
   ],
   "source": [
    "!cat $filename | wc -l "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## grep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`grep` is an extremely powerful tool to look for text in one or more files. For example in the next command we are looking for all the lines that contain a word, we also specify with `-i` that we are interested in case insensitive matching, i.e. don't care about case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  If the skin were parchment, and the blows you gave were ink,\r",
      "\r\n",
      "  Ham. Is not parchment made of sheepskins?\r",
      "\r\n",
      "    of the skin of an innocent lamb should be made parchment? That\r",
      "\r\n",
      "    parchment, being scribbl'd o'er, should undo a man? Some say the\r",
      "\r\n",
      "    Upon a parchment, and against this fire\r",
      "\r\n",
      "    But here's a parchment with the seal of Caesar;  \r",
      "\r\n",
      "    With inky blots and rotten parchment bonds;\r",
      "\r\n",
      "    Nor brass, nor stone, nor parchment, bears not one,\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!grep -i 'parchment' $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine `grep` and `wc` to count the number of lines in a file that contain a specific word: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "72\r\n"
     ]
    }
   ],
   "source": [
    "!grep -i 'liberty' $filename | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sed` is a powerful stream editor, it works similarly to `grep`, but it also modifies the output text, it uses regular expressions, which are a language to define pattern matching and replacement.\n",
    "\n",
    "For example:\n",
    "\n",
    "    s/from/to/g\n",
    "    \n",
    "means:\n",
    "\n",
    "* `s` for substitution\n",
    "* `from` is the word to match\n",
    "* `to` is the replacement string\n",
    "* `g` specifies to apply this to all occurrences on a line, not just the first\n",
    "\n",
    "In the following we are replacing all instances of 'parchment' to 'manuscript'\n",
    "\n",
    "Also we are redirecting the output to a file with `>`. Therefore the output instead of being printed to screen is saved in the text file `temp.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#replace all instances of 'parchment' to 'manuscript'\n",
    "\n",
    "!sed -e 's/parchment/manuscript/g' $filename > temp.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we are checking with `grep` that `temp.txt` contains the word \"manuscript\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  If the skin were manuscript, and the blows you gave were ink,\r",
      "\r\n",
      "  Ham. Is not manuscript made of sheepskins?\r",
      "\r\n",
      "    of the skin of an innocent lamb should be made manuscript? That\r",
      "\r\n",
      "    manuscript, being scribbl'd o'er, should undo a man? Some say the\r",
      "\r\n",
      "    Upon a manuscript, and against this fire\r",
      "\r\n",
      "    But here's a manuscript with the seal of Caesar;  \r",
      "\r\n",
      "    With inky blots and rotten manuscript bonds;\r",
      "\r\n",
      "    Nor brass, nor stone, nor manuscript, bears not one,\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!grep -i 'manuscript' temp.txt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the 100th Etext file presented by Project Gutenberg, and\r",
      "\r\n",
      "is presented in cooperation with World Library, Inc., from their\r",
      "\r\n",
      "Library of the Future and Shakespeare CDROMS.  Project Gutenberg\r",
      "\r\n",
      "often releases Etexts that are NOT placed in the Public Domain!!\r",
      "\r\n",
      "\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 5 $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can sort in alphabetical order the first 5 lines in the file, see that we are just ordering by the first letter in each line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\r\n",
      "is presented in cooperation with World Library, Inc., from their\r",
      "\r\n",
      "Library of the Future and Shakespeare CDROMS.  Project Gutenberg\r",
      "\r\n",
      "often releases Etexts that are NOT placed in the Public Domain!!\r",
      "\r\n",
      "This is the 100th Etext file presented by Project Gutenberg, and\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 5 $filename | sort"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can specify that we would like to sort on the second word of each line, we specify that the delimiter is space with `-t' '` and then specify we want to sort on column 2 `-k2`.\n",
    "\n",
    "Therefore we are sorting on \"is, of, presented, releases\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\r\n",
      "This is the 100th Etext file presented by Project Gutenberg, and\r",
      "\r\n",
      "Library of the Future and Shakespeare CDROMS.  Project Gutenberg\r",
      "\r\n",
      "is presented in cooperation with World Library, Inc., from their\r",
      "\r\n",
      "often releases Etexts that are NOT placed in the Public Domain!!\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 5 $filename | sort -t' ' -k2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sort` is often used in combination with `uniq` to remove duplicated lines.\n",
    "\n",
    "`uniq -u` eliminates duplicated lines, but they need to be consecutive, therefore we first use `sort` to have equal lines consecutive and then we can filter them out easily with `uniq`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "124505\r\n"
     ]
    }
   ],
   "source": [
    "!sort $filename | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "110834\r\n"
     ]
    }
   ],
   "source": [
    "!sort $filename | uniq -u | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lets bring it all together\n",
    "\n",
    "The \"UNIX philosophy\" is \"Do one thing, do it well\" (https://en.wikipedia.org/wiki/Unix_philosophy). The point is to have specialized tools with just 1 well defined function and then compose them together with pipes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Count the most frequent words\n",
    "\n",
    "For example we want to find the 15 most frequent words with their count. We can achieve this combining the tools we learned in this reading.\n",
    "\n",
    "First try it yourself, copy/paste this line many times run it piece by piece and try to understand what each step is doing, read documentation with `--help` or `man`, then will go through it together:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning for MAC OS**: Mac OS has a different version of `sed` that has a special treatment of line feed `\\n` and carriage return `\\n`. Therefore on Mac we need to replace each occurrence of:\n",
    "\n",
    "    sed -e 's/ /\\n/g' -e 's/\\r//g'\n",
    "\n",
    "with:\n",
    "\n",
    "    sed -e 's/ /\\'$'\\n/g' -e $'s/\\r//g'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  23244 the\r\n",
      "  19542 I\r\n",
      "  18302 and\r\n",
      "  15623 to\r\n",
      "  15551 of\r\n",
      "  12532 a\r\n",
      "  10824 my\r\n",
      "   9576 in\r\n",
      "   9081 you\r\n",
      "   7851 is\r\n",
      "   7531 that\r\n",
      "   7068 And\r\n",
      "   6948 not\r\n",
      "   6722 with\r\n",
      "   6218 his\r\n",
      "sort: write failed: 'standard output': Broken pipe\r\n",
      "sort: write error\r\n"
     ]
    }
   ],
   "source": [
    "!sed -e 's/ /\\n/g' -e 's/\\r//g' $filename  | sed '/^$/d'| sort | uniq -c | sort -nr | head -15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**do not worry** about the Broken Pipe error, it is due to the fact that `head` is closing the pipe after the first 15 lines, and `sort` is complaining that it would have more text to write "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    !sed -e 's/ /\\n/g' -e 's/\\r//g' $filename\n",
    "    \n",
    "`sed` is making 2 replacements. The first replaces each space with `\\n`, which is the symbol for a newline character, basically this is splitting all of the words in a text on separate lines. See yourself below!\n",
    "\n",
    "The second replacement is more complicated, `shakespeare.txt` is using the Windows convention of using `\\r\\n` to indicate a new line. `\\r` is carriage return, we want to get rid of it, so we are replacing it with nothing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This\r\n",
      "is\r\n",
      "the\r\n",
      "100th\r\n",
      "Etext\r\n",
      "file\r\n",
      "presented\r\n",
      "by\r\n",
      "Project\r\n",
      "Gutenberg,\r\n",
      "sed: couldn't write 48 items to stdout: Broken pipe\r\n"
     ]
    }
   ],
   "source": [
    "!sed -e 's/ /\\n/g' -e 's/\\r//g' < $filename | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we are not interested in counting empty lines, so we can remove them with:\n",
    "\n",
    "     sed '/^$/d'\n",
    "     \n",
    "* `^` indicates the beginning of a line\n",
    "* `$` indicates the end of a line\n",
    "\n",
    "Therefore `/^$/` matches empty lines. `/d` instructs `sed` to delete them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'd like to count the occurrence of each word, here we can use `uniq` with the `-c` option, but as with the `-u` option, it needs equal lines to be consecutive, so we do a sort first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      1 __\n",
      "      9 -\n",
      "      2 ?\n",
      "      1 /\n",
      "     51 .\n",
      "    241 \"\n",
      "      1 (~),\n",
      "      1 (_)\n",
      "      1 (*)\n",
      "     14 [\n",
      "uniq: write error: Broken pipe\n"
     ]
    }
   ],
   "source": [
    "!sed -e 's/ /\\n/g' -e 's/\\r//g' $filename  | sed '/^$/d' | sort | uniq -c | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good so we have counted the words, so we need to sort but we need to sort in numeric ordering instead of alphabetical so we specify `-n`, also we need reverse order `-r`, bigger first!\n",
    "\n",
    "And finally we take the first 15 lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  23244 the\r\n",
      "  19542 I\r\n",
      "  18302 and\r\n",
      "  15623 to\r\n",
      "  15551 of\r\n",
      "  12532 a\r\n",
      "  10824 my\r\n",
      "   9576 in\r\n",
      "   9081 you\r\n",
      "   7851 is\r\n",
      "   7531 that\r\n",
      "   7068 And\r\n",
      "   6948 not\r\n",
      "   6722 with\r\n",
      "   6218 his\r\n",
      "sort: write failed: 'standard output': Broken pipe\r\n",
      "sort: write error\r\n"
     ]
    }
   ],
   "source": [
    "!sed -e 's/ /\\n/g' -e 's/\\r//g' $filename | sed '/^$/d' | sort | uniq -c | sort -nr | head -15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Write the output to a file\n",
    "\n",
    "We can also do the same and save the output to a file for later usage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sort: write failed: 'standard output': Broken pipe\r\n",
      "sort: write error\r\n"
     ]
    }
   ],
   "source": [
    "!sed -e 's/ /\\n/g' -e 's/\\r//g' < $filename | sed '/^$/d' | sort | sed '/^$/d' | uniq -c | sort -nr | head -15 > count_vs_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  23244 the\r\n",
      "  19542 I\r\n",
      "  18302 and\r\n",
      "  15623 to\r\n",
      "  15551 of\r\n",
      "  12532 a\r\n",
      "  10824 my\r\n",
      "   9576 in\r\n",
      "   9081 you\r\n",
      "   7851 is\r\n",
      "   7531 that\r\n",
      "   7068 And\r\n",
      "   6948 not\r\n",
      "   6722 with\r\n",
      "   6218 his\r\n"
     ]
    }
   ],
   "source": [
    "!cat count_vs_words"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}