Lesson3_AWS_Big_Data_Storage.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lesson3-AWS-Big-Data-Storage.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "c_Id55m6Jsbu",
        "_WgVemLat0d6",
        "qZkC2QC-MRiS",
        "uE_KQNBWuDeZ"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/paiml/awsbigdata/blob/master/Lesson3_AWS_Big_Data_Storage.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "Wy6nhdcaVsCH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Lesson 3 Storage"
      ]
    },
    {
      "metadata": {
        "id": "c_Id55m6Jsbu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Pragmatic AI Labs\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "e5p96AqpSDZa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![alt text](https://paiml.com/images/logo_with_slogan_white_background.png)\n",
        "\n",
        "This notebook was produced by [Pragmatic AI Labs](https://paiml.com/).  You can continue learning about these topics by:\n",
        "\n",
        "*   Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863917)\n",
        "*   Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
        "* Watching video [AWS Certified Machine Learning-Speciality](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597)\n",
        "* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*   Viewing more content at [noahgift.com](https://noahgift.com/)\n"
      ]
    },
    {
      "metadata": {
        "id": "r_krRv5Neq45",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.1 Determine and optimize the operational characteristics of the storage solution"
      ]
    },
    {
      "metadata": {
        "id": "lDU5UzzJTUb-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### The Three \"Vs\" of Big Data:  Variety, Velocity and Volume\n",
        "\n",
        "![Big Data Challenges](https://user-images.githubusercontent.com/58792/54285644-98d7cd00-455f-11e9-823d-ee6fe85af26e.png)"
      ]
    },
    {
      "metadata": {
        "id": "YI6eo-tXTVxI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Variety\n"
      ]
    },
    {
      "metadata": {
        "id": "NAnZKI1gUEeO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "LbKgvoELT_tw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Velocity"
      ]
    },
    {
      "metadata": {
        "id": "R5IemZ6QUFId",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "jjzxWHz6UCkS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Volume"
      ]
    },
    {
      "metadata": {
        "id": "XsQc7VhlUFrd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "_t6aCbsNfK9-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.2 Determine data access and retrieval patterns "
      ]
    },
    {
      "metadata": {
        "id": "6fhL7WGJ7Scv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Batch vs Streaming Data"
      ]
    },
    {
      "metadata": {
        "id": "5U8QWB0mEH7m",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Impact on ML Pipeline"
      ]
    },
    {
      "metadata": {
        "id": "qxGzIx74EK0W",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* More control of model training in batch (can decide when to retrain)\n",
        "* Continuously retraining model could provide better prediction results or worse results\n",
        " - Did input stream suddenly get more users or less users?\n",
        " - Is there an A/B testing scenario?"
      ]
    },
    {
      "metadata": {
        "id": "iG_FBgbY6vqH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Batch"
      ]
    },
    {
      "metadata": {
        "id": "wdJonIVdAfaw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "*   Data is batched at intervals\n",
        "*   Simplest approach to create predictions\n",
        "*   Many Services on AWS Capable of Batch Processing\n",
        " - AWS Glue\n",
        " - AWS Data Pipeline\n",
        " - AWS Batch\n",
        " - EMR\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "M22l9fGmApsu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Streaming\n"
      ]
    },
    {
      "metadata": {
        "id": "UsVFF-9NAr-H",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* Continously polled or pushed\n",
        "* More complex method of prediction\n",
        "* Many Services on AWS Capable of Streaming\n",
        " - Kinesis\n",
        " - IoT"
      ]
    },
    {
      "metadata": {
        "id": "xQZatdK8fO0b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "cQWaXVDmfO3s",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.3 Evaluate mechanisms for capture, update, and retrieval of catalog entries"
      ]
    },
    {
      "metadata": {
        "id": "GEkBM0-ff1id",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[Omnigraffle or Whiteboard Demo]"
      ]
    },
    {
      "metadata": {
        "id": "ZzZvEDG_f1lv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.4 Determine appropriate data structure and storage format "
      ]
    },
    {
      "metadata": {
        "id": "zDWHKEfWf1oi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[Omnigraffle or Whiteboard Demo]"
      ]
    },
    {
      "metadata": {
        "id": "d7xUnqTBjjjD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.5 Understand Storage & Database Fundamentals "
      ]
    },
    {
      "metadata": {
        "id": "pNkmKQ6m627N",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Data Storage Concepts"
      ]
    },
    {
      "metadata": {
        "id": "fYH_66i97RoN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Database Overview\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "DHXy69aAJbDn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![Database Styles](https://user-images.githubusercontent.com/58792/48925585-2214a800-ee7a-11e8-8546-767177679328.png)\n",
        "\n",
        "* [One size database doesn't fit anyone](https://www.allthingsdistributed.com/2018/06/purpose-built-databases-in-aws.html)"
      ]
    },
    {
      "metadata": {
        "id": "hssoSYvFjnx-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.6 Learn S3 - storage"
      ]
    },
    {
      "metadata": {
        "id": "ZkpCT3vajq_v",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[Demo]"
      ]
    },
    {
      "metadata": {
        "id": "Zb9m-Jg8jrHk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.7 Understand Glacier - backup & archive"
      ]
    },
    {
      "metadata": {
        "id": "ls4EPhdMjrOz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[Demo]"
      ]
    },
    {
      "metadata": {
        "id": "YrdH2MLEjrTU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.8 Create AWS Glue - data catalog"
      ]
    },
    {
      "metadata": {
        "id": "6t1wD_-57Q3t",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### AWS Glue"
      ]
    },
    {
      "metadata": {
        "id": "_WgVemLat0d6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### AWS Glue is fully managed ETL Service"
      ]
    },
    {
      "metadata": {
        "id": "u8jPN7MZLIsn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "![AWS Glue Screen](https://user-images.githubusercontent.com/58792/49441953-dff23d00-f77c-11e8-9065-dab53c47c345.png)"
      ]
    },
    {
      "metadata": {
        "id": "qZkC2QC-MRiS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### AWS Glue Workflow"
      ]
    },
    {
      "metadata": {
        "id": "GuJTGk5GMbZn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "*   Build Data Catalog\n",
        "*   Generate and Edit Transformations\n",
        "*   Schedule and Run Jobs\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "VdheVbrOeWxk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### [DEMO] AWS Glue"
      ]
    },
    {
      "metadata": {
        "id": "XGLxtZlrebR0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "9bE8Z_G2jw-6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3.9 Use Dynamodb "
      ]
    },
    {
      "metadata": {
        "id": "q7B3KpSfnCYo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Using AWS DynamoDB\n",
        "\n",
        "* *[Watch Video Lesson 11.4:  Use AWS DynamoDB](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_11_04)*"
      ]
    },
    {
      "metadata": {
        "id": "2cheDvcB2x02",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "https://aws.amazon.com/dynamodb/\n",
        "\n",
        "![alt text](https://d1.awsstatic.com/video-thumbs/dynamodb/AWS-online-games-wide.ada4247744e9be9a6d857b2e13b7eb78b18bf3a5.png)\n",
        "\n",
        "Query Example:\n",
        "\n",
        "```python\n",
        "def query_police_department_record_by_guid(guid):\n",
        "    \"\"\"Gets one record in the PD table by guid\n",
        "    \n",
        "    In [5]: rec = query_police_department_record_by_guid(\n",
        "        \"7e607b82-9e18-49dc-a9d7-e9628a9147ad\"\n",
        "        )\n",
        "    \n",
        "    In [7]: rec\n",
        "    Out[7]: \n",
        "    {'PoliceDepartmentName': 'Hollister',\n",
        "     'UpdateTime': 'Fri Mar  2 12:43:43 2018',\n",
        "     'guid': '7e607b82-9e18-49dc-a9d7-e9628a9147ad'}\n",
        "    \"\"\"\n",
        "    \n",
        "    db = dynamodb_resource()\n",
        "    extra_msg = {\"region_name\": REGION, \"aws_service\": \"dynamodb\", \n",
        "        \"police_department_table\":POLICE_DEPARTMENTS_TABLE,\n",
        "        \"guid\":guid}\n",
        "    log.info(f\"Get PD record by GUID\", extra=extra_msg)\n",
        "    pd_table = db.Table(POLICE_DEPARTMENTS_TABLE)\n",
        "    response = pd_table.get_item(\n",
        "        Key={\n",
        "            'guid': guid\n",
        "            }\n",
        "    )\n",
        "    return response['Item']\n",
        "```\n"
      ]
    },
    {
      "metadata": {
        "id": "21TwDX8Kjw8V",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "uE_KQNBWuDeZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Case Study DynamoDB"
      ]
    },
    {
      "metadata": {
        "id": "7b0c3LDzuPGE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![casestudy](https://user-images.githubusercontent.com/58792/55354483-bae7af80-547a-11e9-9909-a5621251065b.png)"
      ]
    },
    {
      "metadata": {
        "id": "gUMlly9JFYQ9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### [Demo] DynamoDB"
      ]
    },
    {
      "metadata": {
        "id": "G1SgzvGTFcCY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    }
  ]
}