12.groupBy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# groupBy():\n",
    "                Syntax: RDD.groupBy(<function>, numPartitions=None)\n",
    "\n",
    "The groupBy() transformation returns an RDD of items grouped by a specified function. The\n",
    "<function> argument is an anonymous or named function used to nominate a key by which to\n",
    "group all elements or to specify an expression to evaluate against elements to determine a group,\n",
    "such as when grouping elements by odd or even numbers of a numeric field in the data.\n",
    "    \n",
    "    \n",
    "You can use the numPartitions argument to create a specified number of partitions automatically by computing hashes of the key space from the output of the grouping function. For instance, if you want to group an RDD by the days in a week and process each day separately,\n",
    "specify numPartitions=7 . You will see numPartitions specified in numerous Spark transformations, where its behavior is analogous.\n",
    "    \n",
    "    \n",
    "Listing 4.15 demonstrates the use of the groupBy() function. Notice that groupBy() returns an\n",
    "iterable object; we will look at how to handle this type of object later in this chapter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.context import SparkContext\n",
    "from pyspark.sql.session  import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SparkContext('local')\n",
    "spark = SparkSession(sc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('s', <pyspark.resultiterable.ResultIterable at 0x7fbfa920c9d0>)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "licenses = sc.textFile('file:///opt/Spark/licenses')\n",
    "words = licenses.flatMap(lambda x: x.split(' ')).filter(lambda x: len(x) > 0)\n",
    "groupedbyfirstletter = words.groupBy(lambda x: x[0].lower())\n",
    "groupedbyfirstletter.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Consider Other Functions for Grouping Data\n",
    "If your ultimate intention in using groupBy() is to aggregate values, such as when perform-\n",
    "ing a sum() or count() operation, you should opt for more efficient operators for this pur-\n",
    "pose in Spark, including aggregateByKey() and reduceByKey() , which we will discuss\n",
    "shortly. The groupBy() transformation does not perform any aggregation prior to shuffling\n",
    "data, resulting in more data being shuffled. Furthermore, groupBy() requires that all values\n",
    "for a given key fit into memory. The groupBy() transformation is useful in some cases, but\n",
    "you should consider these factors before deciding to use this function."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}