Introduction to Big Data analysis with Spark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Big Data analysis with Spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Understanding SparkContext\n",
    "`SparkContext` is an entry point to interact with underlying Spark functionality. <br>\n",
    "An entry point is where control is transferred from the Operating system to the provided program. In simpler terms, it's like a key to your house. Without the key you cannot enter the house, similarly, without an entry point, you cannot run any PySpark jobs. **You can access the SparkContext in the PySpark shell as a variable named `sc`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyspark as sp\n",
    "sc = sp.SparkContext.getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The version of Spark Context in the PySpark shell is 2.2.1\n"
     ]
    }
   ],
   "source": [
    "# Print the version of SparkContext\n",
    "print(\"The version of Spark Context in the PySpark shell is\", sc.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python version shows the version of Python that Spark is currently using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Python version of Spark Context in the PySpark shell is 3.7\n"
     ]
    }
   ],
   "source": [
    "# Print the Python version of SparkContext\n",
    "print(\"The Python version of Spark Context in the PySpark shell is\", sc.pythonVer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Master is the URL of the cluster or “local” string to run in local mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The master of Spark Context in the PySpark shell is local[*]\n"
     ]
    }
   ],
   "source": [
    "# Print the master of SparkContext\n",
    "print(\"The master of Spark Context in the PySpark shell is\", sc.master)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " sc dot master returns local meaning the SparkContext acts as a master on a local node using all available threads on the computer where it is running. You can load your raw data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interactive Use of PySpark\n",
    "Spark comes with an interactive Python shell in which PySpark is already installed in it. PySpark shell is useful for basic testing and debugging and it is quite powerful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data in PySpark shell\n",
    "into PySpark using SparkContext by two different methods. \n",
    "* The first is the SparkContext’s `parallelize()` method on a list. \n",
    "* The second is the SparkContext’s `textFile()` method on a file. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Python list of numbers from 1 to 100 \n",
    "numb = range(1, 100)\n",
    "\n",
    "# Load the list into PySpark  \n",
    "spark_data = sc.parallelize(numb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# file path\n",
    "file_path = \"5000_points.txt\"\n",
    "# Load a local file into PySpark shell\n",
    "lines = sc.textFile(file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use of lambda() with map()\n",
    "The `map()` function in Python returns a list of the results after applying the given function to each item of a given iterable (list, tuple etc.). The general syntax of `map()` function is `map(fun, iter)`. We can also use lambda functions with `map()`. The general syntax of `map()` function with lambda() is `map(lambda <argument>:<expression>, iter)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input list is [3, 5, 6, 8, 10, 12, 27, 31]\n"
     ]
    }
   ],
   "source": [
    "my_list = [3, 5, 6, 8, 10, 12, 27, 31]\n",
    "# Print my_list in the console\n",
    "print(\"Input list is\", my_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The squared numbers are [9, 25, 36, 64, 100, 144, 729, 961]\n"
     ]
    }
   ],
   "source": [
    "# Square all numbers in my_list\n",
    "squared_list_lambda = list(map(lambda x: x**2, my_list))\n",
    "\n",
    "# Print the result of the map function\n",
    "print(\"The squared numbers are\", squared_list_lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use of lambda() with filter()\n",
    "The `filter()` function in Python takes in a function and a list as arguments. The general syntax of the `filter()` function is `filter(function, list_of_input)`. The general syntax of the `filter()` function with `lambda()` is `filter(lambda <argument>:<expression>, list)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input list is: [1, 10, 2, 100, 3, 1000]\n"
     ]
    }
   ],
   "source": [
    "my_list2 = [1, 10, 2, 100, 3, 1000]\n",
    "# Print my_list2 in the console\n",
    "print(\"Input list is:\", my_list2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numbers divisible by 10 are: [10, 100, 1000]\n"
     ]
    }
   ],
   "source": [
    "# Filter numbers divisible by 10\n",
    "filtered_list = list(filter(lambda x: (x%10 == 0), my_list2))\n",
    "\n",
    "# Print the numbers divisible by 10\n",
    "print(\"Numbers divisible by 10 are:\", filtered_list)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}