Lab_ex_2_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ex 2.1 Hadoop MapReduce with Python\n",
    "There are two prominent *Python* APIs for interfacing *Hadoop MapReduce* clusters:\n",
    "\n",
    "## *Snakebite* for *HDFS* access\n",
    "The [Snakebite Lib](https://github.com/spotify/snakebite) allows easy access to *HDFS* file systems:  \n",
    "```\n",
    ">>> from snakebite.client import Client\n",
    ">>> client = Client(\"localhost\", 8020, use_trash=False)\n",
    ">>> for x in client.ls(['/']):\n",
    "...     print x\n",
    "```\n",
    "\n",
    "See [documentation](https://snakebite.readthedocs.io/en/latest/) for details.\n",
    "\n",
    "\n",
    "## *MRJOB* for *MapReduce* job execution\n",
    "The ``mrjob`` lib -> [see docu](https://mrjob.readthedocs.io/en/latest/index.html) is a power full *MapReduce* client for *Python*. Some of the key features are:\n",
    "\n",
    "* local emulation (single and multi-core) a *Hadoop* cluster for development and debugging\n",
    "* simple access, authentication and file transfer to *Hadoop* clusters\n",
    "* powerful API for common cloud services, such as AWS or Azure   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing our environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: mrjob in /home/keuper/anaconda3/lib/python3.7/site-packages (0.7.3)\n",
      "Collecting boto3\n",
      "  Downloading boto3-1.14.1-py2.py3-none-any.whl (128 kB)\n",
      "\u001b[K     |████████████████████████████████| 128 kB 2.2 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: PyYAML>=3.10 in /home/keuper/anaconda3/lib/python3.7/site-packages (from mrjob) (5.3)\n",
      "Collecting botocore<1.18.0,>=1.17.1\n",
      "  Downloading botocore-1.17.1.tar.gz (6.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 6.5 MB 10.6 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n",
      "  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)\n",
      "Collecting s3transfer<0.4.0,>=0.3.0\n",
      "  Downloading s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)\n",
      "\u001b[K     |████████████████████████████████| 69 kB 9.7 MB/s  eta 0:00:01\n",
      "\u001b[?25hCollecting docutils<0.16,>=0.10\n",
      "  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)\n",
      "\u001b[K     |████████████████████████████████| 547 kB 17.8 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/keuper/anaconda3/lib/python3.7/site-packages (from botocore<1.18.0,>=1.17.1->boto3) (2.8.1)\n",
      "Requirement already satisfied: urllib3<1.26,>=1.20 in /home/keuper/anaconda3/lib/python3.7/site-packages (from botocore<1.18.0,>=1.17.1->boto3) (1.25.8)\n",
      "Requirement already satisfied: six>=1.5 in /home/keuper/anaconda3/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.18.0,>=1.17.1->boto3) (1.14.0)\n",
      "Building wheels for collected packages: botocore\n",
      "  Building wheel for botocore (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for botocore: filename=botocore-1.17.1-py2.py3-none-any.whl size=6262073 sha256=68c321947081e5a1d0d1f60cba1db3166cd7213ddc72c3a7f559de11bf3d0b0e\n",
      "  Stored in directory: /home/keuper/.cache/pip/wheels/26/b9/b7/b0a578bf00238864f15970f0cf362709c62a56d5469d74f2e1\n",
      "Successfully built botocore\n",
      "Installing collected packages: jmespath, docutils, botocore, s3transfer, boto3\n",
      "  Attempting uninstall: docutils\n",
      "    Found existing installation: docutils 0.16\n",
      "    Uninstalling docutils-0.16:\n",
      "      Successfully uninstalled docutils-0.16\n",
      "Successfully installed boto3-1.14.1 botocore-1.17.1 docutils-0.15.2 jmespath-0.10.0 s3transfer-0.3.3\n"
     ]
    }
   ],
   "source": [
    "#install mrjob lib and boto3 for AWS S3 access\n",
    "!conda install -c conda-forge -y mrjob boto3\n",
    "\n",
    "#or !pip install mrjob boto3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A *MRJOB* Example: WordCount (again)\n",
    "Since *Hadoop* works only on file in- and outputs, we do not have usual function based API. We need to pass our code (implementation of *Map* and *Reduce*) as executable *Python* scripts:\n",
    "\n",
    "* use *Jupyter's* ``%%file`` magic command to write the cell to file\n",
    "* create a executable script with ``__main__`` method\n",
    "* inherit from the ``MRJob`` class\n",
    "* implement ``mapper()`` and ``reducer()`` methods\n",
    "* call ``run()`` at start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing wordcount.py\n"
     ]
    }
   ],
   "source": [
    "%%file wordcount.py \n",
    "#this will save this cell as file\n",
    "\n",
    "from mrjob.job import MRJob\n",
    "\n",
    "class MRWordCount(MRJob):\n",
    "    def mapper(self, _, line):\n",
    "        for word in line.split():\n",
    "            yield(word, 1)\n",
    " \n",
    "    def reducer(self, word, counts):\n",
    "        yield(word, sum(counts))\n",
    "        \n",
    "if __name__ == '__main__':\n",
    "    MRWordCount.run()\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### execute script from cmd\n",
    "* ``-r local`` causes local multi-core emulation a *Hadoop* cluster.\n",
    "* Input files are cmd arguments\n",
    "* define ouput-file (see docs) or use streams: `` > out.txt``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No configs found; falling back on auto-configuration\n",
      "No configs specified for local runner\n",
      "Creating temp directory /tmp/wordcount.keuper.20200612.040613.470702\n",
      "Running step 1 of 1...\n",
      "job output is in /tmp/wordcount.keuper.20200612.040613.470702/output\n",
      "Streaming final output from /tmp/wordcount.keuper.20200612.040613.470702/output...\n",
      "\"-\"\t3\n",
      "\"--\"\t6\n",
      "\"A\"\t16\n",
      "\"Adjusting\"\t3\n",
      "\"Alex\"\t3\n",
      "\"All\"\t3\n",
      "\"Almighty,\"\t3\n",
      "\"Alphabet\"\t3\n",
      "\"Amazingly\"\t3\n",
      "\"And\"\t3\n",
      "\"Bawds\"\t3\n",
      "\"Baz,\"\t3\n",
      "\"Big\"\t6\n",
      "\"Blind\"\t9\n",
      "\"Blowzy\"\t3\n",
      "\"Bookmarksgrove\"\t4\n",
      "\"Bookmarksgrove,\"\t3\n",
      "\"Brawny\"\t3\n",
      "\"Brick\"\t3\n",
      "\"Bright\"\t3\n",
      "\"But\"\t3\n",
      "\"Commas,\"\t3\n",
      "\"Consonantia,\"\t4\n",
      "\"Copy\"\t3\n",
      "\"Cozy\"\t3\n",
      "\"Crazy\"\t3\n",
      "\"DJs\"\t4\n",
      "\"Duden\"\t4\n",
      "\"Even\"\t3\n",
      "\"Far\"\t2\n",
      "\"Few\"\t3\n",
      "\"Five\"\t3\n",
      "\"Flummoxed\"\t3\n",
      "\"Fox\"\t3\n",
      "\"Foxy\"\t3\n",
      "\"Fredericka\"\t3\n",
      "\"God!\"\t3\n",
      "\"Grammar.\"\t3\n",
      "\"Have\"\t3\n",
      "\"How\"\t3\n",
      "\"I\"\t44\n",
      "\"Ipsum\"\t3\n",
      "\"Iraq.\"\t3\n",
      "\"It\"\t4\n",
      "\"Italic\"\t3\n",
      "\"Jack!\\\"\"\t3\n",
      "\"Jack\"\t3\n",
      "\"Japan\"\t3\n",
      "\"Jim.\"\t6\n",
      "\"Joaquin\"\t3\n",
      "\"July\"\t3\n",
      "\"Junk\"\t3\n",
      "\"Lane.\"\t3\n",
      "\"Line\"\t3\n",
      "\"Little\"\t9\n",
      "\"Longe\"\t3\n",
      "\"Lorem\"\t3\n",
      "\"MTV\"\t9\n",
      "\"Marks\"\t3\n",
      "\"Mountains,\"\t3\n",
      "\"My\"\t6\n",
      "\"O\"\t3\n",
      "\"Oh,\"\t3\n",
      "\"On\"\t3\n",
      "\"One\"\t3\n",
      "\"Oxmox\"\t3\n",
      "\"Parole\"\t3\n",
      "\"Phoenix\"\t3\n",
      "\"Pityful\"\t3\n",
      "\"Pointing\"\t3\n",
      "\"Question\"\t3\n",
      "\"Quick\"\t9\n",
      "\"Quick,\"\t3\n",
      "\"Semantics,\"\t4\n",
      "\"Semikoli,\"\t3\n",
      "\"Separated\"\t4\n",
      "\"Sex-charged\"\t3\n",
      "\"She\"\t3\n",
      "\"Six\"\t3\n",
      "\"Sixty\"\t3\n",
      "\"TV\"\t9\n",
      "\"Text\"\t6\n",
      "\"Text,\"\t3\n",
      "\"The\"\t12\n",
      "\"Trebek's\"\t3\n",
      "\"Two\"\t3\n",
      "\"Village\"\t3\n",
      "\"Vokalia\"\t4\n",
      "\"W.\"\t3\n",
      "\"Waltz,\"\t3\n",
      "\"Watch\"\t3\n",
      "\"When\"\t3\n",
      "\"When,\"\t3\n",
      "\"World\"\t3\n",
      "\"Woven\"\t3\n",
      "\"Writers\"\t3\n",
      "\"Zompyc[1]\"\t3\n",
      "\"\\\"Jeopardy!\\\",\"\t3\n",
      "\"\\\"Now\"\t3\n",
      "\"\\\"and\\\"\"\t3\n",
      "\"\\u001b[200~Far\"\t1\n",
      "\"a\"\t56\n",
      "\"about\"\t3\n",
      "\"absorb\"\t3\n",
      "\"absorbed\"\t3\n",
      "\"abused\"\t3\n",
      "\"advised\"\t3\n",
      "\"again\"\t3\n",
      "\"again.\"\t3\n",
      "\"agency,\"\t3\n",
      "\"all\"\t3\n",
      "\"all-powerful\"\t3\n",
      "\"almost\"\t3\n",
      "\"alone,\"\t4\n",
      "\"am\"\t7\n",
      "\"amazed\"\t3\n",
      "\"ambushed\"\t3\n",
      "\"among\"\t6\n",
      "\"an\"\t6\n",
      "\"and\"\t93\n",
      "\"and,\"\t3\n",
      "\"are\"\t6\n",
      "\"around\"\t9\n",
      "\"artist\"\t3\n",
      "\"as\"\t9\n",
      "\"asked\"\t3\n",
      "\"at\"\t7\n",
      "\"away,\"\t4\n",
      "\"ax\"\t3\n",
      "\"back\"\t3\n",
      "\"bad\"\t12\n",
      "\"bag.\"\t3\n",
      "\"ball\"\t3\n",
      "\"be\"\t9\n",
      "\"bears\"\t3\n",
      "\"because\"\t3\n",
      "\"bed.\"\t3\n",
      "\"been\"\t6\n",
      "\"before\"\t3\n",
      "\"behind\"\t4\n",
      "\"beloved\"\t3\n",
      "\"belt\"\t3\n",
      "\"big\"\t9\n",
      "\"blew\"\t3\n",
      "\"blind\"\t10\n",
      "\"bliss\"\t4\n",
      "\"bliss;\"\t3\n",
      "\"blow,\"\t3\n",
      "\"blue\"\t3\n",
      "\"bold\"\t3\n",
      "\"bought\"\t3\n",
      "\"bow,\"\t3\n",
      "\"box.\"\t3\n",
      "\"brave\"\t3\n",
      "\"breath\"\t3\n",
      "\"too\"\t3\n",
      "\"tranquil\"\t3\n",
      "\"trees,\"\t3\n",
      "\"trickling\"\t3\n",
      "\"turn\"\t3\n",
      "\"twenty\"\t3\n",
      "\"under\"\t3\n",
      "\"universal\"\t3\n",
      "\"unknown\"\t3\n",
      "\"unorthographic\"\t3\n",
      "\"until\"\t3\n",
      "\"up\"\t6\n",
      "\"upon\"\t3\n",
      "\"upper\"\t3\n",
      "\"us\"\t6\n",
      "\"us,\"\t3\n",
      "\"using\"\t3\n",
      "\"valley\"\t3\n",
      "\"vapour\"\t3\n",
      "\"veldt\"\t3\n",
      "\"versalia,\"\t3\n",
      "\"very\"\t6\n",
      "\"vex!\"\t3\n",
      "\"vex\"\t12\n",
      "\"vex.\"\t3\n",
      "\"vexing\"\t3\n",
      "\"view\"\t3\n",
      "\"visions!\"\t2\n",
      "\"visions!A\"\t1\n",
      "\"vixens\"\t6\n",
      "\"vow.\"\t3\n",
      "\"wafting\"\t3\n",
      "\"waltz.\"\t6\n",
      "\"warm\"\t3\n",
      "\"warned\"\t3\n",
      "\"was\"\t13\n",
      "\"watch\"\t3\n",
      "\"waves\"\t3\n",
      "\"wax\"\t3\n",
      "\"wax.\"\t2\n",
      "\"wax.The\"\t1\n",
      "\"way\"\t3\n",
      "\"way.\"\t6\n",
      "\"weight\"\t3\n",
      "\"were\"\t6\n",
      "\"whangs\"\t3\n",
      "\"whelps.\"\t3\n",
      "\"when\"\t9\n",
      "\"where\"\t6\n",
      "\"which\"\t14\n",
      "\"while\"\t3\n",
      "\"who\"\t3\n",
      "\"whole\"\t4\n",
      "\"wiki-girl.\"\t3\n",
      "\"wild\"\t3\n",
      "\"will\"\t3\n",
      "\"with\"\t20\n",
      "\"within\"\t3\n",
      "\"wizard\\u2019s\"\t3\n",
      "\"wolves\"\t3\n",
      "\"won\"\t3\n",
      "\"wonderful\"\t4\n",
      "\"word\"\t7\n",
      "\"world\"\t3\n",
      "\"would\"\t9\n",
      "\"wove\"\t3\n",
      "\"woven\"\t6\n",
      "\"yet\"\t3\n",
      "\"your\"\t3\n",
      "\"zany\"\t3\n",
      "\"zaps\"\t3\n",
      "\"zebra,\"\t3\n",
      "\"zebras\"\t3\n",
      "\"zephyrs\"\t9\n",
      "\"zippers\"\t3\n",
      "\"zippy\"\t3\n",
      "\"often\"\t3\n",
      "\"on\"\t6\n",
      "\"opal\"\t3\n",
      "\"origin\"\t3\n",
      "\"over\"\t10\n",
      "\"overspreads\"\t3\n",
      "\"own\"\t6\n",
      "\"own,\"\t3\n",
      "\"packed\"\t3\n",
      "\"pager\"\t3\n",
      "\"paper\"\t3\n",
      "\"paradisematic\"\t3\n",
      "\"parsons\"\t3\n",
      "\"parts\"\t3\n",
      "\"pick:\"\t3\n",
      "\"picked\"\t3\n",
      "\"pig,\"\t3\n",
      "\"place\"\t4\n",
      "\"plaid\"\t3\n",
      "\"plants\"\t3\n",
      "\"pled.\"\t3\n",
      "\"possession\"\t4\n",
      "\"power,\"\t3\n",
      "\"presence\"\t3\n",
      "\"present\"\t3\n",
      "\"prog.\"\t3\n",
      "\"projects\"\t3\n",
      "\"provide\"\t3\n",
      "\"put\"\t3\n",
      "\"pyjamas\"\t3\n",
      "\"quack!\"\t3\n",
      "\"quack\"\t3\n",
      "\"quack.\"\t3\n",
      "\"quacking\"\t3\n",
      "\"quart\"\t3\n",
      "\"quartz,\"\t3\n",
      "\"quartz.\"\t3\n",
      "\"question\"\t3\n",
      "\"questions\"\t3\n",
      "\"quick\"\t9\n",
      "\"quick,\"\t4\n",
      "\"quick-jived\"\t3\n",
      "\"quickly\"\t12\n",
      "\"quietly\"\t3\n",
      "\"quips\"\t3\n",
      "\"quit.\"\t3\n",
      "\"quiver\"\t3\n",
      "\"quiz!\"\t3\n",
      "\"quiz\"\t24\n",
      "\"quiz.\"\t6\n",
      "\"ran\"\t3\n",
      "\"reached\"\t3\n",
      "\"red\"\t3\n",
      "\"regelialia.\"\t4\n",
      "\"rethoric\"\t3\n",
      "\"return\"\t3\n",
      "\"rewritten\"\t3\n",
      "\"rewritten,\"\t3\n",
      "\"right\"\t4\n",
      "\"river\"\t4\n",
      "\"road,\"\t3\n",
      "\"roasted\"\t3\n",
      "\"safe\"\t3\n",
      "\"said\"\t3\n",
      "\"sanctuary,\"\t3\n",
      "\"seem\"\t3\n",
      "\"seized\"\t3\n",
      "\"sense\"\t3\n",
      "\"sentences\"\t3\n",
      "\"serenity\"\t4\n",
      "\"seven\"\t3\n",
      "\"she\"\t18\n",
      "\"should\"\t6\n",
      "\"show.\"\t3\n",
      "\"silk\"\t3\n",
      "\"single\"\t3\n",
      "\"sink\"\t3\n",
      "\"six\"\t9\n",
      "\"skyline\"\t3\n",
      "\"small\"\t7\n",
      "\"so\"\t12\n",
      "\"so,\"\t3\n",
      "\"soul\"\t6\n",
      "\"soul,\"\t7\n",
      "\"souls\"\t4\n",
      "\"sphinx\"\t3\n",
      "\"splendour\"\t3\n",
      "\"spot,\"\t4\n",
      "\"spring\"\t4\n",
      "\"stalks,\"\t3\n",
      "\"steal\"\t3\n",
      "\"still\"\t3\n",
      "\"stray\"\t3\n",
      "\"stream;\"\t3\n",
      "\"strength\"\t3\n",
      "\"strikes\"\t3\n",
      "\"stroke\"\t3\n",
      "\"subline\"\t3\n",
      "\"sun\"\t3\n",
      "\"supplies\"\t4\n",
      "\"surface\"\t3\n",
      "\"sustains\"\t3\n",
      "\"sweet\"\t4\n",
      "\"take\"\t3\n",
      "\"taken\"\t4\n",
      "\"talents.\"\t3\n",
      "\"tall\"\t3\n",
      "\"teems\"\t3\n",
      "\"text\"\t3\n",
      "\"texts\"\t3\n",
      "\"texts.\"\t4\n",
      "\"than\"\t3\n",
      "\"that\"\t21\n",
      "\"the\"\t182\n",
      "\"their\"\t10\n",
      "\"then\"\t12\n",
      "\"then,\"\t3\n",
      "\"there\"\t7\n",
      "\"these\"\t10\n",
      "\"they\"\t10\n",
      "\"think\"\t3\n",
      "\"this\"\t4\n",
      "\"thousand\"\t6\n",
      "\"thousands\"\t3\n",
      "\"throw\"\t3\n",
      "\"times\"\t3\n",
      "\"to\"\t24\n",
      "\"judge.\"\t3\n",
      "\"jug\"\t3\n",
      "\"jukeboxes.\"\t3\n",
      "\"jumbled\"\t3\n",
      "\"jump\"\t3\n",
      "\"jump.\"\t3\n",
      "\"jump;\"\t3\n",
      "\"jumping\"\t3\n",
      "\"jumps\"\t4\n",
      "\"jumpy\"\t3\n",
      "\"junk\"\t3\n",
      "\"jury\"\t3\n",
      "\"just\"\t3\n",
      "\"jute\"\t3\n",
      "\"killed\"\t3\n",
      "\"kvetching\"\t3\n",
      "\"language\"\t4\n",
      "\"large\"\t4\n",
      "\"last\"\t3\n",
      "\"lazy\"\t7\n",
      "\"leave\"\t3\n",
      "\"left\"\t3\n",
      "\"letters\"\t3\n",
      "\"lie\"\t3\n",
      "\"life\"\t3\n",
      "\"like\"\t10\n",
      "\"line\"\t3\n",
      "\"listen.\"\t3\n",
      "\"little\"\t3\n",
      "\"live\"\t8\n",
      "\"living\"\t3\n",
      "\"long\"\t3\n",
      "\"longing,\"\t3\n",
      "\"lovably\"\t3\n",
      "\"love\"\t3\n",
      "\"lovely\"\t3\n",
      "\"luck.\"\t3\n",
      "\"made\"\t6\n",
      "\"many\"\t3\n",
      "\"me,\"\t6\n",
      "\"me:\"\t3\n",
      "\"mere\"\t3\n",
      "\"meridian\"\t3\n",
      "\"met\"\t3\n",
      "\"might\"\t6\n",
      "\"milk.\"\t3\n",
      "\"mine.\"\t3\n",
      "\"mirror\"\t6\n",
      "\"mistress,\"\t3\n",
      "\"mock\"\t3\n",
      "\"moment;\"\t3\n",
      "\"mornings\"\t4\n",
      "\"mountains,\"\t4\n",
      "\"mouth.\"\t3\n",
      "\"moved\"\t3\n",
      "\"movement\"\t3\n",
      "\"much\"\t3\n",
      "\"my\"\t56\n",
      "\"myself\"\t3\n",
      "\"name\"\t3\n",
      "\"named\"\t4\n",
      "\"necessary\"\t4\n",
      "\"neglect\"\t3\n",
      "\"never\"\t3\n",
      "\"no\"\t6\n",
      "\"not\"\t3\n",
      "\"nothing\"\t3\n",
      "\"noticed\"\t3\n",
      "\"now.\"\t3\n",
      "\"nymph,\"\t3\n",
      "\"nymphs\"\t3\n",
      "\"nymphs.\"\t3\n",
      "\"ocean.\"\t4\n",
      "\"of\"\t98\n",
      "\"forcing\"\t3\n",
      "\"forgot\"\t3\n",
      "\"form\"\t3\n",
      "\"formed\"\t3\n",
      "\"forms\"\t3\n",
      "\"fowl\"\t3\n",
      "\"fowls.\"\t3\n",
      "\"fox\"\t7\n",
      "\"fox,\"\t3\n",
      "\"fox.\"\t9\n",
      "\"friend\"\t3\n",
      "\"friend,\"\t6\n",
      "\"from\"\t16\n",
      "\"front\"\t3\n",
      "\"full\"\t3\n",
      "\"fun\"\t3\n",
      "\"galvanized\"\t3\n",
      "\"game.\"\t3\n",
      "\"gazed\"\t3\n",
      "\"get\"\t3\n",
      "\"ghost\"\t3\n",
      "\"girl\"\t3\n",
      "\"gleams\"\t3\n",
      "\"gods\"\t3\n",
      "\"grab\"\t3\n",
      "\"graced\"\t3\n",
      "\"grass\"\t3\n",
      "\"greater\"\t3\n",
      "\"grow\"\t3\n",
      "\"gunboats.\"\t3\n",
      "\"had\"\t3\n",
      "\"happy,\"\t3\n",
      "\"has\"\t7\n",
      "\"hasn\\u2019t\"\t3\n",
      "\"have\"\t3\n",
      "\"headline\"\t3\n",
      "\"hear\"\t3\n",
      "\"heart.\"\t4\n",
      "\"heaven\"\t3\n",
      "\"help\"\t3\n",
      "\"her\"\t36\n",
      "\"her,\"\t3\n",
      "\"her.\"\t2\n",
      "\"her.Far\"\t1\n",
      "\"herself\"\t3\n",
      "\"hills\"\t3\n",
      "\"him.\"\t3\n",
      "\"his\"\t3\n",
      "\"hometown\"\t3\n",
      "\"how\"\t3\n",
      "\"however\"\t3\n",
      "\"if\"\t3\n",
      "\"image,\"\t3\n",
      "\"impenetrable\"\t3\n",
      "\"impress\"\t3\n",
      "\"in\"\t29\n",
      "\"incapable\"\t3\n",
      "\"indescribable\"\t3\n",
      "\"infinite\"\t3\n",
      "\"initial\"\t3\n",
      "\"inner\"\t3\n",
      "\"insects\"\t3\n",
      "\"insidious\"\t3\n",
      "\"into\"\t12\n",
      "\"is\"\t18\n",
      "\"it\"\t25\n",
      "\"its\"\t9\n",
      "\"jackets\"\t3\n",
      "\"jay,\"\t3\n",
      "\"jeopardize\"\t3\n",
      "\"jewels.\"\t3\n",
      "\"jigs\"\t3\n",
      "\"jinx\"\t3\n",
      "\"job\"\t3\n",
      "\"job,\"\t3\n",
      "\"jocks\"\t3\n",
      "\"jodhpurs!\"\t3\n",
      "\"jog,\"\t3\n",
      "\"joke\"\t3\n",
      "\"jolt\"\t3\n",
      "\"brown\"\t7\n",
      "\"but\"\t9\n",
      "\"buzz\"\t3\n",
      "\"by\"\t28\n",
      "\"cable\"\t3\n",
      "\"cajole\"\t3\n",
      "\"came\"\t3\n",
      "\"charm\"\t4\n",
      "\"cheek,\"\t3\n",
      "\"chumps\"\t3\n",
      "\"close\"\t3\n",
      "\"coast\"\t4\n",
      "\"conceptions,\"\t3\n",
      "\"confound\"\t3\n",
      "\"continued\"\t3\n",
      "\"control\"\t3\n",
      "\"convince\"\t3\n",
      "\"copy\"\t6\n",
      "\"copy.\"\t3\n",
      "\"could\"\t9\n",
      "\"countless\"\t3\n",
      "\"countries\"\t4\n",
      "\"country,\"\t3\n",
      "\"country.\"\t3\n",
      "\"created\"\t4\n",
      "\"daft\"\t6\n",
      "\"darkness\"\t3\n",
      "\"day\"\t3\n",
      "\"dear\"\t3\n",
      "\"decided\"\t3\n",
      "\"describe\"\t3\n",
      "\"devils\"\t3\n",
      "\"devious\"\t3\n",
      "\"didn\\u2019t\"\t6\n",
      "\"dim\"\t3\n",
      "\"discotheques\"\t3\n",
      "\"do\"\t3\n",
      "\"dog.\"\t4\n",
      "\"dogs\"\t3\n",
      "\"down\"\t3\n",
      "\"dozen\"\t3\n",
      "\"dozy\"\t3\n",
      "\"dragged\"\t3\n",
      "\"drawing\"\t3\n",
      "\"driven\"\t3\n",
      "\"drunk\"\t3\n",
      "\"dwell\"\t3\n",
      "\"earth\"\t3\n",
      "\"earth,\"\t3\n",
      "\"earthquakes\"\t3\n",
      "\"enemy\"\t3\n",
      "\"enjoy\"\t4\n",
      "\"entire\"\t4\n",
      "\"eternity\"\t3\n",
      "\"everything\"\t3\n",
      "\"exchanged\"\t3\n",
      "\"existence\"\t4\n",
      "\"existence,\"\t3\n",
      "\"experimental\"\t3\n",
      "\"experts\"\t3\n",
      "\"exquisite\"\t6\n",
      "\"eyes,\"\t3\n",
      "\"familiar\"\t3\n",
      "\"far\"\t11\n",
      "\"fax\"\t6\n",
      "\"faxed\"\t3\n",
      "\"feel\"\t10\n",
      "\"few\"\t9\n",
      "\"fight\"\t3\n",
      "\"first\"\t3\n",
      "\"five\"\t3\n",
      "\"flax\"\t3\n",
      "\"flick\"\t3\n",
      "\"flies,\"\t3\n",
      "\"floats\"\t3\n",
      "\"flock\"\t3\n",
      "\"flocked\"\t3\n",
      "\"flows\"\t4\n",
      "\"fly\"\t3\n",
      "\"fog.\"\t3\n",
      "\"foliage\"\t3\n",
      "\"fop\"\t3\n",
      "\"for\"\t25\n",
      "Removing temp directory /tmp/wordcount.keuper.20200612.040613.470702...\n"
     ]
    }
   ],
   "source": [
    "! python wordcount.py -r local text1.rst text2.rst text3.rst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execution on AWS EMR\n",
    "AWS EMR is a clound formation service which allows you to create *Hadoop*, *Spark* and other data analytics clusters with a few clicks.\n",
    "\n",
    "**NOTE**: we are not endorsing AWS specifically, other cloud service providers have similar offers\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Case 1: create cluster on the fly \n",
    "We create a cluster just for a single job:\n",
    "* simple solution for large jobs that run only once (or only at sparse points in time)\n",
    "* this approach cause a lot of over head: not suitable for small and frequent jobs  \n",
    "\n",
    "First, we need a config file for the connection to EMR:\n",
    "**fill in YOUR AWS credentials**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mrjob.conf\n"
     ]
    }
   ],
   "source": [
    "%%file mrjob.conf\n",
    "runners:\n",
    "  emr:\n",
    "    aws_access_key_id: YOUR_ID\n",
    "    aws_secret_access_key: YOUR_KEY\n",
    "    instance_type: m5.xlarge\n",
    "    num_core_instances: 2\n",
    "    region: eu-west-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using s3://mrjob-42e7145df80ebe94/tmp/ as our temp dir on S3\n",
      "Creating temp directory /tmp/wordcount.keuper.20200611.214401.921119\n",
      "writing master bootstrap script to /tmp/wordcount.keuper.20200611.214401.921119/b.sh\n",
      "uploading working dir files to s3://mrjob-42e7145df80ebe94/tmp/wordcount.keuper.20200611.214401.921119/files/wd...\n",
      "Copying other local files to s3://mrjob-42e7145df80ebe94/tmp/wordcount.keuper.20200611.214401.921119/files/\n",
      "Can't access IAM API, trying default instance profile: EMR_EC2_DefaultRole\n",
      "Can't access IAM API, trying default service role: EMR_DefaultRole\n",
      "Created new cluster j-APC543JKKD6H\n",
      "Added EMR tags to cluster j-APC543JKKD6H: __mrjob_label=wordcount, __mrjob_owner=keuper, __mrjob_version=0.7.3\n",
      "Waiting for Step 1 of 1 (s-358KPGWRE3U5X) to complete...\n",
      "  PENDING (cluster is STARTING)\n",
      "  PENDING (cluster is STARTING)\n",
      "  PENDING (cluster is STARTING)\n",
      "  PENDING (cluster is STARTING: Configuring cluster software)\n",
      "  PENDING (cluster is STARTING: Configuring cluster software)\n",
      "  PENDING (cluster is BOOTSTRAPPING: Running bootstrap actions)\n",
      "  master node is ec2-54-229-242-142.eu-west-1.compute.amazonaws.com\n",
      "  RUNNING for 0:00:05\n",
      "  RUNNING for 0:00:36\n",
      "  COMPLETED\n",
      "Attempting to fetch counters from logs...\n",
      "Waiting for cluster (j-APC543JKKD6H) to terminate...\n",
      "  TERMINATING\n",
      "  TERMINATING\n",
      "  TERMINATED\n",
      "Looking for step log in s3://mrjob-42e7145df80ebe94/tmp/logs/j-APC543JKKD6H/steps/s-358KPGWRE3U5X...\n",
      "  Parsing step log: s3://mrjob-42e7145df80ebe94/tmp/logs/j-APC543JKKD6H/steps/s-358KPGWRE3U5X/syslog.gz\n",
      "Counters: 55\n",
      "\tFile Input Format Counters \n",
      "\t\tBytes Read=27487\n",
      "\tFile Output Format Counters \n",
      "\t\tBytes Written=3958\n",
      "\tFile System Counters\n",
      "\t\tFILE: Number of bytes read=4887\n",
      "\t\tFILE: Number of bytes written=2093945\n",
      "\t\tFILE: Number of large read operations=0\n",
      "\t\tFILE: Number of read operations=0\n",
      "\t\tFILE: Number of write operations=0\n",
      "\t\tHDFS: Number of bytes read=1251\n",
      "\t\tHDFS: Number of bytes written=0\n",
      "\t\tHDFS: Number of large read operations=0\n",
      "\t\tHDFS: Number of read operations=9\n",
      "\t\tHDFS: Number of write operations=0\n",
      "\t\tS3: Number of bytes read=27487\n",
      "\t\tS3: Number of bytes written=3958\n",
      "\t\tS3: Number of large read operations=0\n",
      "\t\tS3: Number of read operations=0\n",
      "\t\tS3: Number of write operations=0\n",
      "\tJob Counters \n",
      "\t\tData-local map tasks=9\n",
      "\t\tKilled map tasks=1\n",
      "\t\tLaunched map tasks=9\n",
      "\t\tLaunched reduce tasks=3\n",
      "\t\tTotal megabyte-milliseconds taken by all map tasks=216388608\n",
      "\t\tTotal megabyte-milliseconds taken by all reduce tasks=87257088\n",
      "\t\tTotal time spent by all map tasks (ms)=70439\n",
      "\t\tTotal time spent by all maps in occupied slots (ms)=6762144\n",
      "\t\tTotal time spent by all reduce tasks (ms)=14202\n",
      "\t\tTotal time spent by all reduces in occupied slots (ms)=2726784\n",
      "\t\tTotal vcore-milliseconds taken by all map tasks=70439\n",
      "\t\tTotal vcore-milliseconds taken by all reduce tasks=14202\n",
      "\tMap-Reduce Framework\n",
      "\t\tCPU time spent (ms)=21770\n",
      "\t\tCombine input records=0\n",
      "\t\tCombine output records=0\n",
      "\t\tFailed Shuffles=0\n",
      "\t\tGC time elapsed (ms)=2749\n",
      "\t\tInput split bytes=1251\n",
      "\t\tMap input records=2\n",
      "\t\tMap output bytes=18952\n",
      "\t\tMap output materialized bytes=5681\n",
      "\t\tMap output records=1998\n",
      "\t\tMerged Map outputs=27\n",
      "\t\tPhysical memory (bytes) snapshot=6741250048\n",
      "\t\tReduce input groups=370\n",
      "\t\tReduce input records=1998\n",
      "\t\tReduce output records=370\n",
      "\t\tReduce shuffle bytes=5681\n",
      "\t\tShuffled Maps =27\n",
      "\t\tSpilled Records=3996\n",
      "\t\tTotal committed heap usage (bytes)=6729236480\n",
      "\t\tVirtual memory (bytes) snapshot=63693524992\n",
      "\tShuffle Errors\n",
      "\t\tBAD_ID=0\n",
      "\t\tCONNECTION=0\n",
      "\t\tIO_ERROR=0\n",
      "\t\tWRONG_LENGTH=0\n",
      "\t\tWRONG_MAP=0\n",
      "\t\tWRONG_REDUCE=0\n",
      "job output is in s3://mrjob-42e7145df80ebe94/tmp/wordcount.keuper.20200611.214401.921119/output/\n",
      "Streaming final output from s3://mrjob-42e7145df80ebe94/tmp/wordcount.keuper.20200611.214401.921119/output/...\n",
      "\"--\"\t6\n",
      "\"Alphabet\"\t3\n",
      "\"Blind\"\t9\n",
      "\"Bookmarksgrove\"\t4\n",
      "\"Consonantia,\"\t4\n",
      "\"Copy\"\t3\n",
      "\"God!\"\t3\n",
      "\"It\"\t4\n",
      "\"Longe\"\t3\n",
      "\"Marks\"\t3\n",
      "\"On\"\t3\n",
      "\"Separated\"\t4\n",
      "\"She\"\t3\n",
      "\"Village\"\t3\n",
      "\"Vokalia\"\t4\n",
      "\"When\"\t3\n",
      "\"When,\"\t3\n",
      "\"about\"\t3\n",
      "\"again\"\t3\n",
      "\"almost\"\t3\n",
      "\"ambushed\"\t3\n",
      "\"among\"\t6\n",
      "\"an\"\t6\n",
      "\"and,\"\t3\n",
      "\"are\"\t6\n",
      "\"at\"\t7\n",
      "\"belt\"\t3\n",
      "\"blind\"\t10\n",
      "\"buzz\"\t3\n",
      "\"by\"\t13\n",
      "\"cheek,\"\t3\n",
      "\"control\"\t3\n",
      "\"convince\"\t3\n",
      "\"countries\"\t4\n",
      "\"country,\"\t3\n",
      "\"day\"\t3\n",
      "\"devious\"\t3\n",
      "\"drunk\"\t3\n",
      "\"dwell\"\t3\n",
      "\"existence\"\t4\n",
      "\"eyes,\"\t3\n",
      "\"familiar\"\t3\n",
      "\"for\"\t13\n",
      "\"forms\"\t3\n",
      "\"friend,\"\t6\n",
      "\"full\"\t3\n",
      "\"gleams\"\t3\n",
      "\"grow\"\t3\n",
      "\"happy,\"\t3\n",
      "\"have\"\t3\n",
      "\"heart.\"\t4\n",
      "\"her,\"\t3\n",
      "\"herself\"\t3\n",
      "\"his\"\t3\n",
      "\"however\"\t3\n",
      "\"if\"\t3\n",
      "\"impenetrable\"\t3\n",
      "\"impress\"\t3\n",
      "\"indescribable\"\t3\n",
      "\"its\"\t9\n",
      "\"language\"\t4\n",
      "\"live\"\t8\n",
      "\"living\"\t3\n",
      "\"long\"\t3\n",
      "\"longing,\"\t3\n",
      "\"love\"\t3\n",
      "\"lovely\"\t3\n",
      "\"mountains,\"\t4\n",
      "\"mouth.\"\t3\n",
      "\"much\"\t3\n",
      "\"name\"\t3\n",
      "\"neglect\"\t3\n",
      "\"noticed\"\t3\n",
      "\"ocean.\"\t4\n",
      "\"of\"\t89\n",
      "\"over\"\t3\n",
      "\"own,\"\t3\n",
      "\"paper\"\t3\n",
      "\"paradisematic\"\t3\n",
      "\"parts\"\t3\n",
      "\"possession\"\t4\n",
      "\"projects\"\t3\n",
      "\"put\"\t3\n",
      "\"ran\"\t3\n",
      "\"regelialia.\"\t4\n",
      "\"rewritten,\"\t3\n",
      "\"right\"\t4\n",
      "\"roasted\"\t3\n",
      "\"said\"\t3\n",
      "\"sanctuary,\"\t3\n",
      "\"seem\"\t3\n",
      "\"sense\"\t3\n",
      "\"serenity\"\t4\n",
      "\"seven\"\t3\n",
      "\"should\"\t6\n",
      "\"skyline\"\t3\n",
      "\"so,\"\t3\n",
      "\"souls\"\t4\n",
      "\"splendour\"\t3\n",
      "\"stray\"\t3\n",
      "\"stroke\"\t3\n",
      "\"subline\"\t3\n",
      "\"sun\"\t3\n",
      "\"supplies\"\t4\n",
      "\"surface\"\t3\n",
      "\"sustains\"\t3\n",
      "\"talents.\"\t3\n",
      "\"tall\"\t3\n",
      "\"teems\"\t3\n",
      "\"text\"\t3\n",
      "\"the\"\t155\n",
      "\"there\"\t7\n",
      "\"think\"\t3\n",
      "\"thousands\"\t3\n",
      "\"trees,\"\t3\n",
      "\"under\"\t3\n",
      "\"upon\"\t3\n",
      "\"us,\"\t3\n",
      "\"vapour\"\t3\n",
      "\"warned\"\t3\n",
      "\"were\"\t3\n",
      "\"where\"\t6\n",
      "\"wild\"\t3\n",
      "\"with\"\t20\n",
      "\"within\"\t3\n",
      "\"wonderful\"\t4\n",
      "\"word\"\t7\n",
      "\"Almighty,\"\t3\n",
      "\"Big\"\t3\n",
      "\"Bookmarksgrove,\"\t3\n",
      "\"Commas,\"\t3\n",
      "\"Duden\"\t4\n",
      "\"Grammar.\"\t3\n",
      "\"I\"\t44\n",
      "\"Ipsum\"\t3\n",
      "\"Italic\"\t3\n",
      "\"Lane.\"\t3\n",
      "\"Lorem\"\t3\n",
      "\"O\"\t3\n",
      "\"Pityful\"\t3\n",
      "\"Semantics,\"\t4\n",
      "\"Semikoli,\"\t3\n",
      "\"Text\"\t6\n",
      "\"Text,\"\t3\n",
      "\"The\"\t6\n",
      "\"Writers\"\t3\n",
      "\"\\\"and\\\"\"\t3\n",
      "\"\\u001b[200~Far\"\t1\n",
      "\"a\"\t40\n",
      "\"absorb\"\t3\n",
      "\"absorbed\"\t3\n",
      "\"again.\"\t3\n",
      "\"all\"\t3\n",
      "\"all-powerful\"\t3\n",
      "\"and\"\t78\n",
      "\"artist\"\t3\n",
      "\"bad\"\t3\n",
      "\"be\"\t9\n",
      "\"bears\"\t3\n",
      "\"because\"\t3\n",
      "\"behind\"\t4\n",
      "\"bliss;\"\t3\n",
      "\"breath\"\t3\n",
      "\"but\"\t9\n",
      "\"came\"\t3\n",
      "\"close\"\t3\n",
      "\"conceptions,\"\t3\n",
      "\"continued\"\t3\n",
      "\"copy.\"\t3\n",
      "\"countless\"\t3\n",
      "\"darkness\"\t3\n",
      "\"dear\"\t3\n",
      "\"didn\\u2019t\"\t6\n",
      "\"do\"\t3\n",
      "\"dragged\"\t3\n",
      "\"drawing\"\t3\n",
      "\"enjoy\"\t4\n",
      "\"eternity\"\t3\n",
      "\"far\"\t11\n",
      "\"feel\"\t10\n",
      "\"few\"\t6\n",
      "\"first\"\t3\n",
      "\"floats\"\t3\n",
      "\"flows\"\t4\n",
      "\"fly\"\t3\n",
      "\"form\"\t3\n",
      "\"formed\"\t3\n",
      "\"from\"\t10\n",
      "\"had\"\t3\n",
      "\"has\"\t7\n",
      "\"headline\"\t3\n",
      "\"her\"\t36\n",
      "\"hills\"\t3\n",
      "\"hometown\"\t3\n",
      "\"incapable\"\t3\n",
      "\"infinite\"\t3\n",
      "\"inner\"\t3\n",
      "\"insidious\"\t3\n",
      "\"into\"\t12\n",
      "\"is\"\t15\n",
      "\"last\"\t3\n",
      "\"leave\"\t3\n",
      "\"left\"\t3\n",
      "\"like\"\t10\n",
      "\"line\"\t3\n",
      "\"me:\"\t3\n",
      "\"might\"\t3\n",
      "\"mine.\"\t3\n",
      "\"mirror\"\t6\n",
      "\"not\"\t3\n",
      "\"often\"\t3\n",
      "\"origin\"\t3\n",
      "\"own\"\t6\n",
      "\"rewritten\"\t3\n",
      "\"river\"\t4\n",
      "\"safe\"\t3\n",
      "\"small\"\t7\n",
      "\"so\"\t12\n",
      "\"soul\"\t6\n",
      "\"soul,\"\t7\n",
      "\"spot,\"\t4\n",
      "\"spring\"\t4\n",
      "\"stalks,\"\t3\n",
      "\"steal\"\t3\n",
      "\"still\"\t3\n",
      "\"strikes\"\t3\n",
      "\"sweet\"\t4\n",
      "\"take\"\t3\n",
      "\"taken\"\t4\n",
      "\"than\"\t3\n",
      "\"that\"\t21\n",
      "\"their\"\t10\n",
      "\"these\"\t10\n",
      "\"they\"\t10\n",
      "\"throw\"\t3\n",
      "\"times\"\t3\n",
      "\"turn\"\t3\n",
      "\"us\"\t6\n",
      "\"warm\"\t3\n",
      "\"was\"\t10\n",
      "\"way\"\t3\n",
      "\"weight\"\t3\n",
      "\"which\"\t14\n",
      "\"while\"\t3\n",
      "\"who\"\t3\n",
      "\"whole\"\t4\n",
      "\"world\"\t3\n",
      "\"would\"\t9\n",
      "\"your\"\t3\n",
      "\"A\"\t7\n",
      "\"And\"\t3\n",
      "\"But\"\t3\n",
      "\"Even\"\t3\n",
      "\"Far\"\t2\n",
      "\"Line\"\t3\n",
      "\"Little\"\t9\n",
      "\"Mountains,\"\t3\n",
      "\"Oh,\"\t3\n",
      "\"One\"\t3\n",
      "\"Oxmox\"\t3\n",
      "\"Parole\"\t3\n",
      "\"Pointing\"\t3\n",
      "\"Question\"\t3\n",
      "\"World\"\t3\n",
      "\"abused\"\t3\n",
      "\"advised\"\t3\n",
      "\"agency,\"\t3\n",
      "\"alone,\"\t4\n",
      "\"am\"\t7\n",
      "\"around\"\t9\n",
      "\"as\"\t9\n",
      "\"away,\"\t4\n",
      "\"back\"\t3\n",
      "\"been\"\t6\n",
      "\"beloved\"\t3\n",
      "\"bliss\"\t4\n",
      "\"charm\"\t4\n",
      "\"coast\"\t4\n",
      "\"copy\"\t6\n",
      "\"could\"\t9\n",
      "\"country.\"\t3\n",
      "\"created\"\t4\n",
      "\"decided\"\t3\n",
      "\"describe\"\t3\n",
      "\"down\"\t3\n",
      "\"earth\"\t3\n",
      "\"earth,\"\t3\n",
      "\"entire\"\t4\n",
      "\"everything\"\t3\n",
      "\"existence,\"\t3\n",
      "\"exquisite\"\t3\n",
      "\"flies,\"\t3\n",
      "\"foliage\"\t3\n",
      "\"friend\"\t3\n",
      "\"grass\"\t3\n",
      "\"greater\"\t3\n",
      "\"hasn\\u2019t\"\t3\n",
      "\"hear\"\t3\n",
      "\"heaven\"\t3\n",
      "\"her.\"\t2\n",
      "\"her.Far\"\t1\n",
      "\"image,\"\t3\n",
      "\"in\"\t23\n",
      "\"initial\"\t3\n",
      "\"insects\"\t3\n",
      "\"it\"\t25\n",
      "\"large\"\t4\n",
      "\"lie\"\t3\n",
      "\"life\"\t3\n",
      "\"listen.\"\t3\n",
      "\"little\"\t3\n",
      "\"made\"\t6\n",
      "\"me,\"\t6\n",
      "\"mere\"\t3\n",
      "\"meridian\"\t3\n",
      "\"met\"\t3\n",
      "\"mistress,\"\t3\n",
      "\"moment;\"\t3\n",
      "\"mornings\"\t4\n",
      "\"my\"\t38\n",
      "\"myself\"\t3\n",
      "\"named\"\t4\n",
      "\"necessary\"\t4\n",
      "\"never\"\t3\n",
      "\"no\"\t3\n",
      "\"nothing\"\t3\n",
      "\"now.\"\t3\n",
      "\"on\"\t6\n",
      "\"overspreads\"\t3\n",
      "\"packed\"\t3\n",
      "\"place\"\t4\n",
      "\"plants\"\t3\n",
      "\"power,\"\t3\n",
      "\"presence\"\t3\n",
      "\"present\"\t3\n",
      "\"question\"\t3\n",
      "\"reached\"\t3\n",
      "\"rethoric\"\t3\n",
      "\"return\"\t3\n",
      "\"road,\"\t3\n",
      "\"sentences\"\t3\n",
      "\"she\"\t15\n",
      "\"single\"\t3\n",
      "\"sink\"\t3\n",
      "\"stream;\"\t3\n",
      "\"strength\"\t3\n",
      "\"texts\"\t3\n",
      "\"texts.\"\t4\n",
      "\"then\"\t12\n",
      "\"then,\"\t3\n",
      "\"this\"\t4\n",
      "\"thousand\"\t6\n",
      "\"to\"\t15\n",
      "\"too\"\t3\n",
      "\"tranquil\"\t3\n",
      "\"trickling\"\t3\n",
      "\"universal\"\t3\n",
      "\"unknown\"\t3\n",
      "\"unorthographic\"\t3\n",
      "\"until\"\t3\n",
      "\"upper\"\t3\n",
      "\"using\"\t3\n",
      "\"valley\"\t3\n",
      "\"versalia,\"\t3\n",
      "\"view\"\t3\n",
      "\"visions!\"\t2\n",
      "\"visions!A\"\t1\n",
      "\"way.\"\t6\n",
      "\"when\"\t6\n",
      "\"yet\"\t3\n",
      "Removing s3 temp directory s3://mrjob-42e7145df80ebe94/tmp/wordcount.keuper.20200611.214401.921119/...\n",
      "Removing temp directory /tmp/wordcount.keuper.20200611.214401.921119...\n",
      "Removing log files in s3://mrjob-42e7145df80ebe94/tmp/logs/j-APC543JKKD6H/...\n",
      "Terminating cluster: j-APC543JKKD6H\n"
     ]
    }
   ],
   "source": [
    "! python wordcount.py -r emr --bootstrap-mrjob text1.rst text2.rst -c mrjob.conf \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Case 3: connect to existing cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mrjob_cluster.conf\n"
     ]
    }
   ],
   "source": [
    "%%file mrjob_cluster.conf\n",
    "runners:\n",
    "  emr:\n",
    "    aws_access_key_id: YOUR_ID\n",
    "    aws_secret_access_key: YOUR_KEY\n",
    "    region: eu-west-1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need the **ID** of the cluster we want to connect to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using s3://mrjob-42e7145df80ebe94/tmp/ as our temp dir on S3\n",
      "Creating temp directory /tmp/wordcount.keuper.20200611.211946.917705\n",
      "uploading working dir files to s3://mrjob-42e7145df80ebe94/tmp/wordcount.keuper.20200611.211946.917705/files/wd...\n",
      "Copying other local files to s3://mrjob-42e7145df80ebe94/tmp/wordcount.keuper.20200611.211946.917705/files/\n",
      "Adding our job to existing cluster j-13JDFN62TGS41\n",
      "  master node is ec2-54-75-7-6.eu-west-1.compute.amazonaws.com\n",
      "Waiting for Step 1 of 1 (s-FM2DU2UTBWWQ) to complete...\n",
      "  PENDING (cluster is RUNNING: Running step)\n",
      "  RUNNING for 0:00:45\n",
      "  RUNNING for 0:01:15\n",
      "  FAILED\n",
      "Cluster j-13JDFN62TGS41 is WAITING: Cluster ready after last step failed.\n",
      "Attempting to fetch counters from logs...\n",
      "Waiting 10 minutes for logs to transfer to S3... (ctrl-c to skip)\n",
      "\n",
      "To fetch logs immediately next time, set up SSH. See:\n",
      "https://pythonhosted.org/mrjob/guides/emr-quickstart.html#configuring-ssh-credentials\n",
      "\n",
      "Looking for step log in s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/steps/s-FM2DU2UTBWWQ...\n",
      "  Parsing step log: s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/steps/s-FM2DU2UTBWWQ/syslog.gz\n",
      "Counters: 17\n",
      "\tJob Counters \n",
      "\t\tData-local map tasks=8\n",
      "\t\tFailed map tasks=25\n",
      "\t\tKilled map tasks=8\n",
      "\t\tKilled reduce tasks=3\n",
      "\t\tLaunched map tasks=30\n",
      "\t\tOther local map tasks=22\n",
      "\t\tTotal megabyte-milliseconds taken by all map tasks=783605760\n",
      "\t\tTotal megabyte-milliseconds taken by all reduce tasks=0\n",
      "\t\tTotal time spent by all map tasks (ms)=255080\n",
      "\t\tTotal time spent by all maps in occupied slots (ms)=24487680\n",
      "\t\tTotal time spent by all reduce tasks (ms)=0\n",
      "\t\tTotal time spent by all reduces in occupied slots (ms)=0\n",
      "\t\tTotal vcore-milliseconds taken by all map tasks=255080\n",
      "\t\tTotal vcore-milliseconds taken by all reduce tasks=0\n",
      "\tMap-Reduce Framework\n",
      "\t\tCPU time spent (ms)=0\n",
      "\t\tPhysical memory (bytes) snapshot=0\n",
      "\t\tVirtual memory (bytes) snapshot=0\n",
      "Scanning logs for probable cause of failure...\n",
      "Looking for history log in s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/hadoop-mapreduce/history...\n",
      "  Parsing history log: s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/hadoop-mapreduce/history/2020/06/11/000000/job_1591908888075_0002-1591910413821-hadoop-streamjob315392557014343417.jar-1591910468843-0-0-FAILED-default-1591910417803.jhist.gz\n",
      "Looking for task logs in s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/containers/application_1591908888075_0002...\n",
      "  Parsing task log: s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/containers/application_1591908888075_0002/container_1591908888075_0002_01_000031/stderr.gz\n",
      "  Parsing task log: s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/containers/application_1591908888075_0002/container_1591908888075_0002_01_000031/syslog.gz\n",
      "\n",
      "Probable cause of failure:\n",
      "\n",
      "PipeMapRed failed!\n",
      "java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1\n",
      "\tat org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)\n",
      "\tat org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538)\n",
      "\tat org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)\n",
      "\tat org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)\n",
      "\tat org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)\n",
      "\tat org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)\n",
      "\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)\n",
      "\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)\n",
      "\tat java.security.AccessController.doPrivileged(Native Method)\n",
      "\tat javax.security.auth.Subject.doAs(Subject.java:422)\n",
      "\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)\n",
      "\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)\n",
      "\n",
      "(from lines 52-65 of s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/containers/application_1591908888075_0002/container_1591908888075_0002_01_000031/syslog.gz)\n",
      "\n",
      "caused by:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"wordcount.py\", line 3, in <module>\n",
      "    from mrjob.job import MRJob\n",
      "ModuleNotFoundError: No module named 'mrjob'\n",
      "\n",
      "(from lines 1-4 of s3://aws-logs-846657657993-eu-west-1/elasticmapreduce/j-13JDFN62TGS41/containers/application_1591908888075_0002/container_1591908888075_0002_01_000031/stderr.gz)\n",
      "\n",
      "while reading input from s3://mrjob-42e7145df80ebe94/tmp/wordcount.keuper.20200611.211946.917705/files/text1.rst\n",
      "\n",
      "\n",
      "Step 1 of 1 failed\n"
     ]
    }
   ],
   "source": [
    "! python wordcount.py -r emr --cluster-id=j-CLUSTERID text1.rst text2.rst -c mrjob_cluster.conf  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "Use  *mrjob*  to  compute  employee  **top  annual  salaries** and  **gross pay** in the *CSV* table ``Baltimore_City_employee_Salaries_FY2014.csv``.\n",
    "\n",
    "* use  ``import csv`` to read the data -> [API docs](https://docs.python.org/3/library/csv.html)\n",
    "* use ``yield`` to return *producers* from *map* and *reduce* functions\n",
    "* return top entries in both categories "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}