Tutorial-week3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div class=\"alert alert-block alert-success\"><h3>IFN619 - Data Analytics for Strategic Decision Makers</h4></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tutorial 03 :: Decision Analysis / Case Studies in Data Analytics\n",
    "\n",
    "In this tutorial, you will be applying the data analytics cycle using the Global Financial Crisis of 2008 as a case study.\n",
    "\n",
    "If you would like to know a little bit more about it, please check this video.\n",
    "\n",
    "[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/0aE7___9ZtQ/0.jpg)](https://www.youtube.com/watch?v=0aE7___9ZtQ)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    " ### Motivation: The Global Financial Crisis of 2008\n",
    "\n",
    "You have been hired to be the data analyst of the company *Hesiod*, which analyses real estate indicators of different countries. \n",
    "\n",
    "Marcel, the CEO of the company, provides you some datasets with financial and economic indicators and asks your opinion about what to expect in the next couple of years in the housing market. Can you help Marcel?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Analytics Process\n",
    "\n",
    "<img src=\"images/Data Analytics Decision Process.png\" />\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 1. Collecting / Loading Data\n",
    "\n",
    "The dataset that you need to analyse is in the folder 'data/financial_data_v2.csv'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the required Python libraries to process the data:\n",
    "import pandas as pd                # used for data manipulation and data analysis\n",
    "import matplotlib.pyplot as plt    # used for visualisation/\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "General size of the dataset: (62, 17)\n",
      "The dataset has 62 rows and 17 columns!\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# specify the location and the filenae of your dataset\n",
    "file_path =   \n",
    "\n",
    "# load the .csv dataset\n",
    "data = \n",
    "\n",
    "# take a look at the dataset\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the dimensions of your dataset\n",
    "dimensions = \n",
    "\n",
    "print( 'General size of the dataset: ' + str( dimensions ))\n",
    "\n",
    "# extract the number of rows and columns from your data\n",
    "num_rows = \n",
    "num_col  = \n",
    "print('The dataset has ' + str( num_rows ) + ' rows and ' + str( num_col ) + ' columns!' )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**QUESTION:** \n",
    "- What do the **rows** and **columns** of this dataset **represent**?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**ANSWER HERE**\n",
    "- **Rows** represent the **house price index** for each **quarter (date)**.\n",
    "- **Columns** represent the **house price index** for each **country**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 2. CLEAN / PROCESS DATA\n",
    "\n",
    "An important step in the data analytics cyle is to ensure the **quality** of our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**QUESTIONS**\n",
    "\n",
    "Answer the following questions (can be done in group)\n",
    "\n",
    "- Q1: What is this dataset about?\n",
    "- Q2: What are the main problems with this dataset? \n",
    "- Q3: Is the data relevant to our initial question (our Business concern)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### EXPLORE THE DATASET\n",
    "\n",
    "This dataset describes the **house indexes of different countries in the world**. \n",
    "\n",
    "Remeber our question: *what to expect in the next couple of years in the housing market?*\n",
    "\n",
    "This is a vague question that requires you to **explore** the dataset to get some *insights* about the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# WHAT COUNTRIES ARE IN THIS DATASET AND HOW MANY?\n",
    "# DO THE REQUIRED CODE TO LIST ALL THE CONTRIES THAT ARE IN THE DATASET\n",
    "\n",
    "# YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# HOW MANY YEARS OF DATA DO WE HAVE IN OUR DATASET?\n",
    "# PRINT A STRING SAYING: \"This dataset describes house priceS from XXXX to XXXX\"\n",
    "# WHERE XXXX CORRESPONDS TO THE YEARS THAT SPAN THIS DATASET\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**QUESTION:** For how many years has this information been collected?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dealing with Missing Data\n",
    "\n",
    "In Pandas missing data is represented by two value:\n",
    "\n",
    "- *None*\n",
    "- *NaN (Not a Number)*\n",
    "\n",
    "Pandas treats *None* and *NaN* for indicating **missing or null values**. To facilitate this convention, there are several useful functions for **detecting** null values in Pandas DataFrame:\n",
    "\n",
    "- isnull()\n",
    "- notnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to fill null values in a datasets, we use the functions *fillna()*, *dropna()* or *interpolate()*. \n",
    "\n",
    "These functions replace NaN values with some value of their own.\n",
    "\n",
    "- **fillna()** fucntion basically hard-codes a value to replace the NaN entries. It is used for simpler datasets with few missing entries (like in this lecture).\n",
    "\n",
    "- **interpolate()** function is basically used to fill NA values in the dataframe but it uses various interpolation techniques to fill the missing values rather than hard-coding the value. This method uses **statistical estimations** based on the distribution of the data. This is an advanced method that you might be interested in exploring in your tutorials or in your assignments if you are dealing with a complex dataset\n",
    "\n",
    "- **dropna()** function basically drops either a row or a column. This is usually applied for datasets where there are variables that have **too many missing entries** and it is not possible to estimate them or to hand-code them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**QUESTION**\n",
    "\n",
    "What is the percentage of missing house prices in \"Australia\"?\n",
    "What about in \"United Kingdom\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE TO COMPUTE YOUR ANSWER HERE (hint: pandas info() function can be useful)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CREATE A NEW COLUMN IN YOUR DATAFRAME CALLED \"Australia_Mean\" WITH THE SAME INFORMATION AS IN COLUMN \"Australia\"\n",
    "# FILL THE MISSING VALUES OF \"Australia_Mean\" BY COMPUTING THE AVERAGE OF THE HOUSE PRICES IN THAT COLUMN\n",
    "\n",
    "\n",
    "# PLOT THE HOUSE PRICE INDICES IN THE COLUMNS \"AUSTRALIA_MEAN\" AND \"AUSTRALIA_FULL\" AND COMPARE THEM\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CREATE A NEW COLUMN IN YOUR DATAFRAME CALLED \"Australia_Interpolate\" WITH THE SAME INFORMATION AS IN COLUMN \"Australia\"\n",
    "\n",
    "\n",
    "\n",
    "# FILL THE MISSING VALUES OF Australia_Interpolate BY USING THE INTERPOLATE FUNCTION (HINT: interpolate(method ='linear', inplace=True))\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# PLOT THE HOUSE PRICE INDICES IN THE COLUMNS \"AUSTRALIA_MEAN\" AND \"AUSTRALIA_FULL\" AND COMPARE THEM\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question**\n",
    "\n",
    "Discuss the impact of how different methods to deal with missing data can impact your analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For curious students (optional)**\n",
    "\n",
    "Interpolation is a way to explore different methods to estimate the distrbution of your data.\n",
    "\n",
    "The column \"United Kingdom\" has 48% of the data missing. Explore the different price estimations in that country when you use different interpolation methods. Which one approaximates the most to the real values (\"United Kingdom_Full\")?\n",
    "\n",
    "method : {‘linear’, ‘time’, ‘index’, ‘values’, ‘nearest’, ‘zero’, ‘slinear’, ‘quadratic’, ‘cubic’, ‘barycentric’, ‘krogh’, ‘polynomial’, ‘spline’, ‘piecewise_polynomial’, ‘from_derivatives’, ‘pchip’, ‘akima’}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Visualization\n",
    "\n",
    "We can try to visualize our data to get some more insights. We can plot the house price indexes of Australia from 2003 to 2018. But for that... we need to process data again...\n",
    "\n",
    "We need to eliminate the *Quarter* dimension of our data by grouping the data by Date (which represents the year). This is performed by making the **average** of the **house price indexes** of each **country** for **ALL QUARTERS** and **aggregating** the results **by year**. In Python, one can do this using the function *.groupby and the aggregation of the data is done by computing the **mean**.\n",
    "The following figure shows an example.\n",
    "\n",
    "<img src='images/groupBy.png' />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Group data records from \"Australia_Interpolate\" and \"United Kingdom_Full\" by DATE in order to have data in the format:\n",
    "AU_house_price_indx = \n",
    "UK_house_price_indx = \n",
    "\n",
    "# Compute the overall average of the house prices in Australia \n",
    "AU_house_price_indx_avg = \n",
    "print( 'The avarage house price index in Australia is ' + str(round(AU_house_price_indx_avg,4)) )\n",
    "\n",
    "# Compute the overall average of the house prices in UK\n",
    "UK_house_price_indx_avg = \n",
    "print( 'The avarage house price index in United Kingdom is ' + str(round(UK_house_price_indx_avg,4)) )\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PLOT THE HOUSE PRICES IN  AUSTRALIA AND IN UK IN A SINGLE GRAPH\n",
    "# DON'T FORGET TO PUT LABELS IN ALL AXIS\n",
    "# ADD A TITLE TO THE VISUALISATION\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**QUESTION** \n",
    "\n",
    "Are there similarities between the house price indexes in UK with Australia?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SPLIT YOUR ANALYSIS BEFORE THE CRISIS AND AFTER THE CRISIS\n",
    "\n",
    "#Select UK house price indexes from 2003 - 2008\n",
    "UK_house_price_03_08 = \n",
    "\n",
    "#Select UK house price indexes from 2003 - 2018\n",
    "UK_house_price_08_18 =  \n",
    "\n",
    "#Select AU house price indexes from 2003 - 2008\n",
    "AU_house_price_03_08 =   \n",
    "\n",
    "#Select AU house price indexes from 2008 - 2018\n",
    "AU_house_price_08_18 =\n",
    "\n",
    "\n",
    "# VISUALISE THE DIFFERENT GRAPHS: COMPARE AUSTRALIA AND UK BETWEEN 2003 - 2008\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n",
    "\n",
    "# VISUALISE THE DIFFERENT GRAPHS: COMPARE AUSTRALIA AND UK BETWEEN 2008 - 2018\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**QUESTION**\n",
    "- Q1: Can you explain the exponential growth of house prices in UK between 2003 - 2007?\n",
    "- Q2: Why house prices started to drop in UK between 2007 - 2014?\n",
    "- Q3: What happened to the housing sector in Australia\n",
    "    - Q3.1. Before 2007:\n",
    "    - Q3.2. Between 2007 - 2010\n",
    "    - Q3.3. Between 2010 - 2013\n",
    "    - Q3.4. After 2014\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### At Home Self Study###\n",
    "\n",
    "It is interesting to analyse the housing market in different sets of countries: \n",
    "- Southern European countries: Portugal, Spain, Italy, Greece\n",
    "- Northen European countries: Sweden, Finland, Norway \n",
    "- Major European powers: Germany, France, UK\n",
    "- USA vs UK\n",
    "\n",
    "Plot three graphs, showing these three sets of countries in different plots. Interpret the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"700\"\n",
       "            height=\"600\"\n",
       "            src=\"https://www.economist.com/finance-and-economics/2019/03/09/prices-of-prime-properties-around-the-world-are-falling\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x11ae484e0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# SOMETHING TO. SUPPORT OUR CONCLUSIONS :-)\n",
    "\n",
    "IFrame(src='https://www.economist.com/finance-and-economics/2019/03/09/prices-of-prime-properties-around-the-world-are-falling', width=700, height=600)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For Self Study (optional)\n",
    "\n",
    "### The Impact of Interest Rates\n",
    "\n",
    "Correlate the change of interest rates sith the financial crisis and the house price indexes.\n",
    "File can be found in: \"data/interest_rates.csv\"\n",
    "\n",
    "Make your own analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify the location and the filenae of your dataset\n",
    "file_path = 'data/interest_rates.csv'   \n",
    "\n",
    "# load the .csv dataset using the function read_csv, which is part of the pandas library\n",
    "int_rates = pd.read_csv(file_path)\n",
    "int_rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}