ch_03/4-reshaping_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reshaping Data\n",
    "\n",
    "## About the data\n",
    "In this notebook, we will using daily temperature data from the [National Centers for Environmental Information (NCEI) API](https://www.ncdc.noaa.gov/cdo-web/webservices/v2). We will use the Global Historical Climatology Network - Daily (GHCND) dataset; see the documentation [here](https://www1.ncdc.noaa.gov/pub/data/cdo/documentation/GHCND_documentation.pdf).\n",
    "\n",
    "This data was collected for New York City for October 2018, using the Boonton 1 station (GHCND:USC00280907). It contains:\n",
    "- the daily minimum temperature (`TMIN`)\n",
    "- the daily maximum temperature (`TMAX`)\n",
    "- the daily temperature at time of observation (`TOBS`)\n",
    "\n",
    "*Note: The NCEI is part of the National Oceanic and Atmospheric Administration (NOAA) and, as you can see from the URL for the API, this resource was created when the NCEI was called the NCDC. Should the URL for this resource change in the future, you can search for \"NCEI weather API\" to find the updated one.*\n",
    "\n",
    "## Setup\n",
    "We need to import `pandas` and read in the long format data to get started:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datatype</th>\n",
       "      <th>date</th>\n",
       "      <th>temp_C</th>\n",
       "      <th>temp_F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TMAX</td>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>21.1</td>\n",
       "      <td>69.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TMIN</td>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>8.9</td>\n",
       "      <td>48.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TOBS</td>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>13.9</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TMAX</td>\n",
       "      <td>2018-10-02</td>\n",
       "      <td>23.9</td>\n",
       "      <td>75.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>TMIN</td>\n",
       "      <td>2018-10-02</td>\n",
       "      <td>13.9</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  datatype       date  temp_C  temp_F\n",
       "0     TMAX 2018-10-01    21.1   69.98\n",
       "1     TMIN 2018-10-01     8.9   48.02\n",
       "2     TOBS 2018-10-01    13.9   57.02\n",
       "3     TMAX 2018-10-02    23.9   75.02\n",
       "4     TMIN 2018-10-02    13.9   57.02"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "long_df = pd.read_csv(\n",
    "    'data/long_data.csv', usecols=['date', 'datatype', 'value']\n",
    ").rename(\n",
    "    columns={'value': 'temp_C'}\n",
    ").assign(\n",
    "    date=lambda x: pd.to_datetime(x.date),\n",
    "    temp_F=lambda x: (x.temp_C * 9/5) + 32\n",
    ")\n",
    "long_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transposing\n",
    "Transposing swaps the rows and the columns. We use the `T` attribute to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>date</th>\n",
       "      <th>2018-10-01</th>\n",
       "      <th>2018-10-01</th>\n",
       "      <th>2018-10-01</th>\n",
       "      <th>2018-10-02</th>\n",
       "      <th>2018-10-02</th>\n",
       "      <th>2018-10-02</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>datatype</th>\n",
       "      <td>TMAX</td>\n",
       "      <td>TMIN</td>\n",
       "      <td>TOBS</td>\n",
       "      <td>TMAX</td>\n",
       "      <td>TMIN</td>\n",
       "      <td>TOBS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>temp_C</th>\n",
       "      <td>21.10</td>\n",
       "      <td>8.90</td>\n",
       "      <td>13.90</td>\n",
       "      <td>23.90</td>\n",
       "      <td>13.90</td>\n",
       "      <td>17.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>temp_F</th>\n",
       "      <td>69.98</td>\n",
       "      <td>48.02</td>\n",
       "      <td>57.02</td>\n",
       "      <td>75.02</td>\n",
       "      <td>57.02</td>\n",
       "      <td>62.96</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "date     2018-10-01 2018-10-01 2018-10-01 2018-10-02 2018-10-02 2018-10-02\n",
       "datatype       TMAX       TMIN       TOBS       TMAX       TMIN       TOBS\n",
       "temp_C        21.10       8.90      13.90      23.90      13.90      17.20\n",
       "temp_F        69.98      48.02      57.02      75.02      57.02      62.96"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "long_df.set_index('date').head(6).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pivoting\n",
    "Going from long to wide format.\n",
    "\n",
    "### `pivot()`\n",
    "We can restructure our data by picking a column to go in the index (`index`), a column whose unique values will become column names (`columns`), and the values to place in those columns (`values`). The `pivot()` method can be used when we don't need to perform any aggregation in addition to our restructuring (when our index is unique); if this is not the case, we need the `pivot_table()` method which we will cover in chapter 4. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>datatype</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2018-10-01</th>\n",
       "      <td>21.1</td>\n",
       "      <td>8.9</td>\n",
       "      <td>13.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-02</th>\n",
       "      <td>23.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>17.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-03</th>\n",
       "      <td>25.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>16.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-04</th>\n",
       "      <td>22.8</td>\n",
       "      <td>11.7</td>\n",
       "      <td>11.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-05</th>\n",
       "      <td>23.3</td>\n",
       "      <td>11.7</td>\n",
       "      <td>18.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "datatype    TMAX  TMIN  TOBS\n",
       "date                        \n",
       "2018-10-01  21.1   8.9  13.9\n",
       "2018-10-02  23.9  13.9  17.2\n",
       "2018-10-03  25.0  15.6  16.1\n",
       "2018-10-04  22.8  11.7  11.7\n",
       "2018-10-05  23.3  11.7  18.9"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivoted_df = long_df.pivot(\n",
    "    index='date', columns='datatype', values='temp_C'\n",
    ")\n",
    "pivoted_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data is pivoted, we have wide format data that we can grab summary statistics with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>datatype</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>31.000000</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>31.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>16.829032</td>\n",
       "      <td>7.561290</td>\n",
       "      <td>10.022581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>5.714962</td>\n",
       "      <td>6.513252</td>\n",
       "      <td>6.596550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>7.800000</td>\n",
       "      <td>-1.100000</td>\n",
       "      <td>-1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>12.750000</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>5.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>16.100000</td>\n",
       "      <td>6.700000</td>\n",
       "      <td>8.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>21.950000</td>\n",
       "      <td>13.600000</td>\n",
       "      <td>16.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>26.700000</td>\n",
       "      <td>17.800000</td>\n",
       "      <td>21.700000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "datatype       TMAX       TMIN       TOBS\n",
       "count     31.000000  31.000000  31.000000\n",
       "mean      16.829032   7.561290  10.022581\n",
       "std        5.714962   6.513252   6.596550\n",
       "min        7.800000  -1.100000  -1.100000\n",
       "25%       12.750000   2.500000   5.550000\n",
       "50%       16.100000   6.700000   8.300000\n",
       "75%       21.950000  13.600000  16.100000\n",
       "max       26.700000  17.800000  21.700000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivoted_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also provide multiple values to pivot on, which will result in a hierarchical index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">temp_C</th>\n",
       "      <th colspan=\"3\" halign=\"left\">temp_F</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datatype</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2018-10-01</th>\n",
       "      <td>21.1</td>\n",
       "      <td>8.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>69.98</td>\n",
       "      <td>48.02</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-02</th>\n",
       "      <td>23.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>17.2</td>\n",
       "      <td>75.02</td>\n",
       "      <td>57.02</td>\n",
       "      <td>62.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-03</th>\n",
       "      <td>25.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>16.1</td>\n",
       "      <td>77.00</td>\n",
       "      <td>60.08</td>\n",
       "      <td>60.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-04</th>\n",
       "      <td>22.8</td>\n",
       "      <td>11.7</td>\n",
       "      <td>11.7</td>\n",
       "      <td>73.04</td>\n",
       "      <td>53.06</td>\n",
       "      <td>53.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-05</th>\n",
       "      <td>23.3</td>\n",
       "      <td>11.7</td>\n",
       "      <td>18.9</td>\n",
       "      <td>73.94</td>\n",
       "      <td>53.06</td>\n",
       "      <td>66.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           temp_C             temp_F              \n",
       "datatype     TMAX  TMIN  TOBS   TMAX   TMIN   TOBS\n",
       "date                                              \n",
       "2018-10-01   21.1   8.9  13.9  69.98  48.02  57.02\n",
       "2018-10-02   23.9  13.9  17.2  75.02  57.02  62.96\n",
       "2018-10-03   25.0  15.6  16.1  77.00  60.08  60.98\n",
       "2018-10-04   22.8  11.7  11.7  73.04  53.06  53.06\n",
       "2018-10-05   23.3  11.7  18.9  73.94  53.06  66.02"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivoted_df = long_df.pivot(\n",
    "    index='date', columns='datatype', values=['temp_C', 'temp_F']\n",
    ")\n",
    "pivoted_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the hierarchical index, if we want to select `TMIN` in Fahrenheit, we will first need to select `temp_F` and then `TMIN`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date\n",
       "2018-10-01    48.02\n",
       "2018-10-02    57.02\n",
       "2018-10-03    60.08\n",
       "2018-10-04    53.06\n",
       "2018-10-05    53.06\n",
       "Name: TMIN, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivoted_df['temp_F']['TMIN'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `unstack()`\n",
    "\n",
    "We have been working with a single index throughout this chapter; however, we can create an index from any number of columns with `set_index()`. This gives us an index of type `MultiIndex`, where the outermost level corresponds to the first element in the list provided to `set_index()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultiIndex([('2018-10-01', 'TMAX'),\n",
       "            ('2018-10-01', 'TMIN'),\n",
       "            ('2018-10-01', 'TOBS'),\n",
       "            ('2018-10-02', 'TMAX'),\n",
       "            ('2018-10-02', 'TMIN')],\n",
       "           names=['date', 'datatype'])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multi_index_df = long_df.set_index(['date', 'datatype'])\n",
    "multi_index_df.head().index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice there are now 2 index sections of the dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>temp_C</th>\n",
       "      <th>temp_F</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th>datatype</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">2018-10-01</th>\n",
       "      <th>TMAX</th>\n",
       "      <td>21.1</td>\n",
       "      <td>69.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TMIN</th>\n",
       "      <td>8.9</td>\n",
       "      <td>48.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TOBS</th>\n",
       "      <td>13.9</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2018-10-02</th>\n",
       "      <th>TMAX</th>\n",
       "      <td>23.9</td>\n",
       "      <td>75.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TMIN</th>\n",
       "      <td>13.9</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     temp_C  temp_F\n",
       "date       datatype                \n",
       "2018-10-01 TMAX        21.1   69.98\n",
       "           TMIN         8.9   48.02\n",
       "           TOBS        13.9   57.02\n",
       "2018-10-02 TMAX        23.9   75.02\n",
       "           TMIN        13.9   57.02"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multi_index_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With an index of type `MultiIndex`, we can no longer use `pivot()`. We must now use `unstack()`, which by default moves the innermost index onto the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">temp_C</th>\n",
       "      <th colspan=\"3\" halign=\"left\">temp_F</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datatype</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2018-10-01</th>\n",
       "      <td>21.1</td>\n",
       "      <td>8.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>69.98</td>\n",
       "      <td>48.02</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-02</th>\n",
       "      <td>23.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>17.2</td>\n",
       "      <td>75.02</td>\n",
       "      <td>57.02</td>\n",
       "      <td>62.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-03</th>\n",
       "      <td>25.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>16.1</td>\n",
       "      <td>77.00</td>\n",
       "      <td>60.08</td>\n",
       "      <td>60.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-04</th>\n",
       "      <td>22.8</td>\n",
       "      <td>11.7</td>\n",
       "      <td>11.7</td>\n",
       "      <td>73.04</td>\n",
       "      <td>53.06</td>\n",
       "      <td>53.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-05</th>\n",
       "      <td>23.3</td>\n",
       "      <td>11.7</td>\n",
       "      <td>18.9</td>\n",
       "      <td>73.94</td>\n",
       "      <td>53.06</td>\n",
       "      <td>66.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           temp_C             temp_F              \n",
       "datatype     TMAX  TMIN  TOBS   TMAX   TMIN   TOBS\n",
       "date                                              \n",
       "2018-10-01   21.1   8.9  13.9  69.98  48.02  57.02\n",
       "2018-10-02   23.9  13.9  17.2  75.02  57.02  62.96\n",
       "2018-10-03   25.0  15.6  16.1  77.00  60.08  60.98\n",
       "2018-10-04   22.8  11.7  11.7  73.04  53.06  53.06\n",
       "2018-10-05   23.3  11.7  18.9  73.94  53.06  66.02"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unstacked_df = multi_index_df.unstack()\n",
    "unstacked_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `unstack()` method also provides the `fill_value` parameter, which let's us fill-in any `NaN` values that might arise from this restructuring of the data. Consider the case that we have data for the average temperature on October 1, 2018, but no other date:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>temp_C</th>\n",
       "      <th>temp_F</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th>datatype</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">2018-10-01</th>\n",
       "      <th>TAVG</th>\n",
       "      <td>10.0</td>\n",
       "      <td>50.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TMAX</th>\n",
       "      <td>21.1</td>\n",
       "      <td>69.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TMIN</th>\n",
       "      <td>8.9</td>\n",
       "      <td>48.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TOBS</th>\n",
       "      <td>13.9</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">2018-10-02</th>\n",
       "      <th>TMAX</th>\n",
       "      <td>23.9</td>\n",
       "      <td>75.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TMIN</th>\n",
       "      <td>13.9</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TOBS</th>\n",
       "      <td>17.2</td>\n",
       "      <td>62.96</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     temp_C  temp_F\n",
       "date       datatype                \n",
       "2018-10-01 TAVG        10.0   50.00\n",
       "           TMAX        21.1   69.98\n",
       "           TMIN         8.9   48.02\n",
       "           TOBS        13.9   57.02\n",
       "2018-10-02 TMAX        23.9   75.02\n",
       "           TMIN        13.9   57.02\n",
       "           TOBS        17.2   62.96"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extra_data = long_df.append([{\n",
    "    'datatype': 'TAVG', \n",
    "    'date': '2018-10-01', \n",
    "    'temp_C': 10, \n",
    "    'temp_F': 50\n",
    "}]).set_index(['date', 'datatype']).sort_index()\n",
    "\n",
    "extra_data['2018-10-01':'2018-10-02']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we use `unstack()` in this case, we will have `NaN` for the `TAVG` columns every day but October 1, 2018:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"4\" halign=\"left\">temp_C</th>\n",
       "      <th colspan=\"4\" halign=\"left\">temp_F</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datatype</th>\n",
       "      <th>TAVG</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "      <th>TAVG</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2018-10-01</th>\n",
       "      <td>10.0</td>\n",
       "      <td>21.1</td>\n",
       "      <td>8.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>50.0</td>\n",
       "      <td>69.98</td>\n",
       "      <td>48.02</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-02</th>\n",
       "      <td>NaN</td>\n",
       "      <td>23.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>17.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>75.02</td>\n",
       "      <td>57.02</td>\n",
       "      <td>62.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-03</th>\n",
       "      <td>NaN</td>\n",
       "      <td>25.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>16.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>77.00</td>\n",
       "      <td>60.08</td>\n",
       "      <td>60.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-04</th>\n",
       "      <td>NaN</td>\n",
       "      <td>22.8</td>\n",
       "      <td>11.7</td>\n",
       "      <td>11.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>73.04</td>\n",
       "      <td>53.06</td>\n",
       "      <td>53.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-05</th>\n",
       "      <td>NaN</td>\n",
       "      <td>23.3</td>\n",
       "      <td>11.7</td>\n",
       "      <td>18.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>73.94</td>\n",
       "      <td>53.06</td>\n",
       "      <td>66.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           temp_C                   temp_F                     \n",
       "datatype     TAVG  TMAX  TMIN  TOBS   TAVG   TMAX   TMIN   TOBS\n",
       "date                                                           \n",
       "2018-10-01   10.0  21.1   8.9  13.9   50.0  69.98  48.02  57.02\n",
       "2018-10-02    NaN  23.9  13.9  17.2    NaN  75.02  57.02  62.96\n",
       "2018-10-03    NaN  25.0  15.6  16.1    NaN  77.00  60.08  60.98\n",
       "2018-10-04    NaN  22.8  11.7  11.7    NaN  73.04  53.06  53.06\n",
       "2018-10-05    NaN  23.3  11.7  18.9    NaN  73.94  53.06  66.02"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extra_data.unstack().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To address this, we can pass in an appropriate `fill_value`. However, we are restricted to passing in a value for this, not a strategy (like we saw with `fillna()`), so while `-40` is definitely not be the best value, we can use it to illustrate how this works, since this is the temperature at which Fahrenheit and Celsius are equal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"4\" halign=\"left\">temp_C</th>\n",
       "      <th colspan=\"4\" halign=\"left\">temp_F</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datatype</th>\n",
       "      <th>TAVG</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "      <th>TAVG</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2018-10-01</th>\n",
       "      <td>10.0</td>\n",
       "      <td>21.1</td>\n",
       "      <td>8.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>50.0</td>\n",
       "      <td>69.98</td>\n",
       "      <td>48.02</td>\n",
       "      <td>57.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-02</th>\n",
       "      <td>-40.0</td>\n",
       "      <td>23.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>17.2</td>\n",
       "      <td>-40.0</td>\n",
       "      <td>75.02</td>\n",
       "      <td>57.02</td>\n",
       "      <td>62.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-03</th>\n",
       "      <td>-40.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>16.1</td>\n",
       "      <td>-40.0</td>\n",
       "      <td>77.00</td>\n",
       "      <td>60.08</td>\n",
       "      <td>60.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-04</th>\n",
       "      <td>-40.0</td>\n",
       "      <td>22.8</td>\n",
       "      <td>11.7</td>\n",
       "      <td>11.7</td>\n",
       "      <td>-40.0</td>\n",
       "      <td>73.04</td>\n",
       "      <td>53.06</td>\n",
       "      <td>53.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-05</th>\n",
       "      <td>-40.0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>11.7</td>\n",
       "      <td>18.9</td>\n",
       "      <td>-40.0</td>\n",
       "      <td>73.94</td>\n",
       "      <td>53.06</td>\n",
       "      <td>66.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           temp_C                   temp_F                     \n",
       "datatype     TAVG  TMAX  TMIN  TOBS   TAVG   TMAX   TMIN   TOBS\n",
       "date                                                           \n",
       "2018-10-01   10.0  21.1   8.9  13.9   50.0  69.98  48.02  57.02\n",
       "2018-10-02  -40.0  23.9  13.9  17.2  -40.0  75.02  57.02  62.96\n",
       "2018-10-03  -40.0  25.0  15.6  16.1  -40.0  77.00  60.08  60.98\n",
       "2018-10-04  -40.0  22.8  11.7  11.7  -40.0  73.04  53.06  53.06\n",
       "2018-10-05  -40.0  23.3  11.7  18.9  -40.0  73.94  53.06  66.02"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extra_data.unstack(fill_value=-40).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Melting\n",
    "Going from wide to long format.\n",
    "\n",
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>21.1</td>\n",
       "      <td>8.9</td>\n",
       "      <td>13.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-10-02</td>\n",
       "      <td>23.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>17.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-10-03</td>\n",
       "      <td>25.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>16.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-10-04</td>\n",
       "      <td>22.8</td>\n",
       "      <td>11.7</td>\n",
       "      <td>11.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-10-05</td>\n",
       "      <td>23.3</td>\n",
       "      <td>11.7</td>\n",
       "      <td>18.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date  TMAX  TMIN  TOBS\n",
       "0  2018-10-01  21.1   8.9  13.9\n",
       "1  2018-10-02  23.9  13.9  17.2\n",
       "2  2018-10-03  25.0  15.6  16.1\n",
       "3  2018-10-04  22.8  11.7  11.7\n",
       "4  2018-10-05  23.3  11.7  18.9"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wide_df = pd.read_csv('data/wide_data.csv')\n",
    "wide_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `melt()`\n",
    "In order to go from wide format to long format, we use the `melt()` method. We have to specify:\n",
    "- `id_vars`: which column(s) uniquely identify a row in the wide format (`date`, here)\n",
    "- `value_vars`: the column(s) that contain(s) the values (`TMAX`, `TMIN`, and `TOBS`, here)\n",
    "\n",
    "Optionally, we can also provide:\n",
    "- `value_name`: what to call the column that will contain all the values once melted\n",
    "- `var_name`: what to call the column that will contain the names of the variables being measured"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>measurement</th>\n",
       "      <th>temp_C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>TMAX</td>\n",
       "      <td>21.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-10-02</td>\n",
       "      <td>TMAX</td>\n",
       "      <td>23.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-10-03</td>\n",
       "      <td>TMAX</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-10-04</td>\n",
       "      <td>TMAX</td>\n",
       "      <td>22.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-10-05</td>\n",
       "      <td>TMAX</td>\n",
       "      <td>23.3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date measurement  temp_C\n",
       "0  2018-10-01        TMAX    21.1\n",
       "1  2018-10-02        TMAX    23.9\n",
       "2  2018-10-03        TMAX    25.0\n",
       "3  2018-10-04        TMAX    22.8\n",
       "4  2018-10-05        TMAX    23.3"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "melted_df = wide_df.melt(\n",
    "    id_vars='date',\n",
    "    value_vars=['TMAX', 'TMIN', 'TOBS'],\n",
    "    value_name='temp_C',\n",
    "    var_name='measurement'\n",
    ")\n",
    "melted_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `stack()`\n",
    "Another option is `stack()`, which will pivot the columns of the dataframe into the innermost level of the index (resulting in an index of type `MultiIndex`). To illustrate this, let's set our index to be the `date` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TMAX</th>\n",
       "      <th>TMIN</th>\n",
       "      <th>TOBS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2018-10-01</th>\n",
       "      <td>21.1</td>\n",
       "      <td>8.9</td>\n",
       "      <td>13.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-02</th>\n",
       "      <td>23.9</td>\n",
       "      <td>13.9</td>\n",
       "      <td>17.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-03</th>\n",
       "      <td>25.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>16.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-04</th>\n",
       "      <td>22.8</td>\n",
       "      <td>11.7</td>\n",
       "      <td>11.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-10-05</th>\n",
       "      <td>23.3</td>\n",
       "      <td>11.7</td>\n",
       "      <td>18.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            TMAX  TMIN  TOBS\n",
       "date                        \n",
       "2018-10-01  21.1   8.9  13.9\n",
       "2018-10-02  23.9  13.9  17.2\n",
       "2018-10-03  25.0  15.6  16.1\n",
       "2018-10-04  22.8  11.7  11.7\n",
       "2018-10-05  23.3  11.7  18.9"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wide_df.set_index('date', inplace=True)\n",
    "wide_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By running `stack()` now, we will create a second level in our index which will contain the column names of our dataframe (`TMAX`, `TMIN`, `TOBS`). This will leave us with a `Series` object containing the values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date            \n",
       "2018-10-01  TMAX    21.1\n",
       "            TMIN     8.9\n",
       "            TOBS    13.9\n",
       "2018-10-02  TMAX    23.9\n",
       "            TMIN    13.9\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_series = wide_df.stack()\n",
    "stacked_series.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `to_frame()` method on our `Series` object to turn it into a `DataFrame` object. Since the series doesn't have a name at the moment, we will pass in the name as an argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>values</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">2018-10-01</th>\n",
       "      <th>TMAX</th>\n",
       "      <td>21.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TMIN</th>\n",
       "      <td>8.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TOBS</th>\n",
       "      <td>13.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2018-10-02</th>\n",
       "      <th>TMAX</th>\n",
       "      <td>23.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TMIN</th>\n",
       "      <td>13.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 values\n",
       "date                   \n",
       "2018-10-01 TMAX    21.1\n",
       "           TMIN     8.9\n",
       "           TOBS    13.9\n",
       "2018-10-02 TMAX    23.9\n",
       "           TMIN    13.9"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_df = stacked_series.to_frame('values')\n",
    "stacked_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we have an index of type `MultiIndex`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultiIndex([('2018-10-01', 'TMAX'),\n",
       "            ('2018-10-01', 'TMIN'),\n",
       "            ('2018-10-01', 'TOBS'),\n",
       "            ('2018-10-02', 'TMAX'),\n",
       "            ('2018-10-02', 'TMIN')],\n",
       "           names=['date', None])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_df.head().index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, we don't have a name for the `datatype` level:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FrozenList(['date', None])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_df.index.names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `set_names()` to address this though:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FrozenList(['date', 'datatype'])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_df.index.set_names(['date', 'datatype'], inplace=True)\n",
    "stacked_df.index.names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<div>\n",
    "    <a href=\"./3-cleaning_data.ipynb\">\n",
    "        <button>&#8592; Previous Notebook</button>\n",
    "    </a>\n",
    "    <a href=\"./5-handling_data_issues.ipynb\">\n",
    "        <button style=\"float: right;\">Next Notebook &#8594;</button>\n",
    "    </a>\n",
    "</div>\n",
    "<hr>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}