Forecast_DeepAR.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon Forecast: predicting time-series at scale\n",
    "\n",
    "Forecasting is used in a variety of applications and business use cases: For example, retailers need to forecast the sales of their products to decide how much stock they need by location, Manufacturers need to estimate the number of parts required at their factories to optimize their supply chain, Businesses need to estimate their flexible workforce needs, Utilities need to forecast electricity consumption needs in order to attain an efficient energy network, and \n",
    "enterprises need to estimate their cloud infrastructure needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"BlogImages/amazon_forecast.png\">\n",
    "\n",
    "\n",
    "# Table of Contents\n",
    "\n",
    "* Step 0: [Setting up](#setup)\n",
    "* Step 1: [Preparing the Datasets](#prepare)\n",
    "* Step 2: [Importing the Data](#import)\n",
    " * Step 2a: [Creating a Dataset Group](#create)\n",
    " * Step 2b: [Creating a Target Dataset](#target)\n",
    " * Step 2c: [Creating a Related Dataset](#related)\n",
    " * Step 2d: [Update the Dataset Group](#update)\n",
    " * Step 2e: [Creating a Target Time Series Dataset Import Job](#targetImport)\n",
    " * Step 2f: [Creating a Related Time Series Dataset Import Job](#relatedImport)\n",
    "* Step 3: [Choosing an Algorithm and Evaluating its Performance](#algo)\n",
    " * Step 3a: [Choosing DeepAR+](#DeepAR)\n",
    " * Step 3b: [Choosing Prophet](#prophet)\n",
    "* Step 4: [Computing Error Metrics from Backtesting](#error)\n",
    "* Step 5: [Creating a Forecast](#forecast)\n",
    "* Step 6: [Querying the Forecasts](#query)\n",
    "* Step 7: [Exporting the Forecasts](#export)\n",
    "* Step 8: [Clearning up your Resources](#cleanup)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  First let us setup Amazon Forecast<a class=\"anchor\" id=\"setup\">\n",
    "\n",
    "This section sets up the permissions and relevant endpoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from util.fcst_utils import *\n",
    "import warnings\n",
    "import boto3\n",
    "import s3fs\n",
    "plt.rcParams['figure.figsize'] = (15.0, 5.0)\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although, we have set the region to us-west-2 below, you can choose any of the 6 regions that the service is available in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region = 'us-east-1'\n",
    "bucket = 'bike-demo'\n",
    "version = 'prod'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = boto3.Session(region_name=region) \n",
    "forecast = session.client(service_name='forecast') \n",
    "forecast_query = session.client(service_name='forecastquery')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "role_arn = get_or_create_role_arn()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "<img src=\"BlogImages/outline.png\">\n",
    "\n",
    "<img src=\"BlogImages/forecast_workflow.png\">\n",
    "\n",
    "The above figure summarizes the key workflow of using Forecast. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Preparing the Datasets<a class=\"anchor\" id=\"prepare\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bike_df = pd.read_csv(\"./data/train.csv\", dtype = object)\n",
    "bike_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bike_df['count'] = bike_df['count'].astype('float')\n",
    "bike_df['workingday'] = bike_df['workingday'].astype('float')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take about two and a half week's of hourly data for demonstration, just for the purpose that there's no missing data in the whole range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bike_df_small = bike_df[-2*7*24-24*3:]\n",
    "bike_df_small['item_id'] = \"bike_12\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us plot the time series first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bike_df_small.plot(x='datetime', y='count', figsize=(15, 8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the target time series seem to have a drop over weekends. Next let's plot both the target time series and the related time series that indicates whether today is a `workday` or not. More precisely, $r_t = 1$ if $t$ is a work day and 0 if not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 8))\n",
    "ax = plt.gca()\n",
    "bike_df_small.plot(x='datetime', y='count', ax=ax);\n",
    "ax2 = ax.twinx()\n",
    "bike_df_small.plot(x='datetime', y='workingday', color='red', ax=ax2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that to use the related time series, we need to ensure that the related time series covers the whole target time series, as well as the future values as specified by the forecast horizon. More precisely, we need to make sure:\n",
    "```\n",
    "len(related time series) >= len(target time series) + forecast horizon\n",
    "```\n",
    "Basically, all items need to have data start at or before the item start date, and have data until the forecast horizon (i.e. the latest end date across all items + forecast horizon).  Additionally, there should be no missing values in the related time series. The following picture illustrates the desired logic. \n",
    "\n",
    "<img src=\"BlogImages/rts_viz.png\">\n",
    "\n",
    "For more details regarding how to prepare your Related Time Series dataset, please refer to the public documentation <a href=\"https://docs.aws.amazon.com/forecast/latest/dg/related-time-series-datasets.html\">here</a>. \n",
    "\n",
    "Suppose in this particular example, we wish to forecast for the next 24 hours, and thus we generate the following dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_df = bike_df_small[['item_id', 'datetime', 'count']][:-24]\n",
    "rts_df = bike_df_small[['item_id', 'datetime', 'workingday']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the length of the related time series is equal to the length of the target time series plus the forecast horizon. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(target_df), len(rts_df))\n",
    "assert len(target_df) + 24 == len(rts_df), \"length doesn't match\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we check whether there are \"holes\" in the related time series.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(rts_df) == len(pd.date_range(\n",
    "    start=list(rts_df['datetime'])[0],\n",
    "    end=list(rts_df['datetime'])[-1],\n",
    "    freq='H'\n",
    ")), \"missing entries in the related time series\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything looks fine, and we plot both time series again. As it can be seen, the related time series (indicator of whether the current day is a workday or not) is longer than the target time series.  The binary working day indicator feature is a good example of a related time series, since it is known at all future time points.  Other examples of related time series include holiday and promotion features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 10))\n",
    "ax = plt.gca()\n",
    "target_df.plot(x='datetime', y='count', ax=ax);\n",
    "ax2 = ax.twinx()\n",
    "rts_df.plot(x='datetime', y='workingday', color='red', ax=ax2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_df.to_csv(\"./data/bike_small.csv\", index= False, header = False)\n",
    "rts_df.to_csv(\"./data/bike_small_rts.csv\", index= False, header = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = session.client('s3')\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't have this bucket `amazon-forecast-data-{account_id}`, create it first on S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket_name = f\"amazon-forecast-data-{account_id}\"\n",
    "key = \"bike_small\"\n",
    "\n",
    "s3.upload_file(Filename=\"./data/bike_small.csv\", Bucket = bucket_name, Key = f\"{key}/bike.csv\")\n",
    "s3.upload_file(Filename=\"./data/bike_small_rts.csv\", Bucket = bucket_name, Key = f\"{key}/bike_rts.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2. Importing the Data<a class=\"anchor\" id=\"import\">\n",
    "\n",
    "\n",
    "Now we are ready to import the datasets into the Forecast service. Starting from the raw data, Amazon Forecast automatically extracts the dataset that is suitable for forecasting. As an example, a retailer normally records the transaction record such as\n",
    "<img src=\"BlogImages/data_format.png\">\n",
    "<img src=\"BlogImages/timestamp.png\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project = \"bike_rts_demo\"\n",
    "idx = 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_data_path = f\"s3://{bucket_name}/{key}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we specify key input data and forecast parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = \"H\"\n",
    "forecast_horizon = 24\n",
    "timestamp_format = \"yyyy-MM-dd HH:mm:ss\"\n",
    "delimiter = ','"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2a. Creating a Dataset Group<a class=\"anchor\" id=\"create\">\n",
    "First let's create a dataset group and then update it later to add our datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_group = f\"{project}_gp_{idx}\"\n",
    "dataset_arns = []\n",
    "create_dataset_group_response = forecast.create_dataset_group(Domain=\"RETAIL\",\n",
    "                                                          DatasetGroupName=dataset_group,\n",
    "                                                          DatasetArns=dataset_arns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(f'Creating dataset group {dataset_group}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_group_arn = create_dataset_group_response['DatasetGroupArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset_group(DatasetGroupArn=dataset_group_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2b. Creating a Target Dataset<a class=\"anchor\" id=\"target\">\n",
    "In this example, we will define a target time series. This is a required dataset to use the service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we specify the target time series name af_demo_ts_4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_name = f\"{project}_ts_{idx}\"\n",
    "print(ts_dataset_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we specify the schema of our dataset below. Make sure the order of the attributes (columns) matches the raw data in the files. We follow the same three attribute format as the above example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_schema_val = [{\"AttributeName\": \"item_id\", \"AttributeType\": \"string\"},\n",
    "              {\"AttributeName\": \"timestamp\", \"AttributeType\": \"timestamp\"},\n",
    "              {\"AttributeName\": \"demand\", \"AttributeType\": \"float\"}]\n",
    "ts_schema = {\"Attributes\": ts_schema_val}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(f'Creating target dataset {ts_dataset_name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = forecast.create_dataset(Domain=\"RETAIL\",\n",
    "                               DatasetType='TARGET_TIME_SERIES',\n",
    "                               DatasetName=ts_dataset_name,\n",
    "                               DataFrequency=freq,\n",
    "                               Schema=ts_schema\n",
    "                              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_arn = response['DatasetArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset(DatasetArn=ts_dataset_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2c. Creating a Related Dataset<a class=\"anchor\" id=\"related\">\n",
    "In this example, we will define a related time series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the related time series name af_demo_rts_4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rts_dataset_name = f\"{project}_rts_{idx}\"\n",
    "print(rts_dataset_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the schema of your dataset here. Make sure the order of columns matches the raw data files. We follow the same three column format as the above example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rts_schema_val = [{\"AttributeName\": \"item_id\", \"AttributeType\": \"string\"},\n",
    "              {\"AttributeName\": \"timestamp\", \"AttributeType\": \"timestamp\"},\n",
    "              {\"AttributeName\": \"price\", \"AttributeType\": \"float\"}]\n",
    "rts_schema = {\"Attributes\": rts_schema_val}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(f'Creating related dataset {rts_dataset_name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = forecast.create_dataset(Domain=\"RETAIL\",\n",
    "                               DatasetType='RELATED_TIME_SERIES',\n",
    "                               DatasetName=rts_dataset_name,\n",
    "                               DataFrequency=freq,\n",
    "                               Schema=rts_schema\n",
    "                              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rts_dataset_arn = response['DatasetArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset(DatasetArn=rts_dataset_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2d. Updating the dataset group with the datasets we created<a class=\"anchor\" id=\"update\">\n",
    "You can have multiple datasets under the same dataset group. Update it with the datasets we created before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_arns = []\n",
    "dataset_arns.append(ts_dataset_arn)\n",
    "dataset_arns.append(rts_dataset_arn)\n",
    "forecast.update_dataset_group(DatasetGroupArn=dataset_group_arn, DatasetArns=dataset_arns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset_group(DatasetGroupArn=dataset_group_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2e. Creating a Target Time Series Dataset Import Job<a class=\"anchor\" id=\"targetImport\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_s3_data_path = f\"{s3_data_path}/bike.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_import_job_response = forecast.create_dataset_import_job(DatasetImportJobName=dataset_group,\n",
    "                                                             DatasetArn=ts_dataset_arn,\n",
    "                                                             DataSource= {\n",
    "                                                                 \"S3Config\" : {\n",
    "                                                                     \"Path\": ts_s3_data_path,\n",
    "                                                                     \"RoleArn\": role_arn\n",
    "                                                                 } \n",
    "                                                             },\n",
    "                                                             TimestampFormat=timestamp_format)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_import_job_arn=ts_dataset_import_job_response['DatasetImportJobArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status = wait(lambda: forecast.describe_dataset_import_job(DatasetImportJobArn=ts_dataset_import_job_arn))\n",
    "assert status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2f. Creating a Related Time Series Dataset Import Job<a class=\"anchor\" id=\"relatedImport\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rts_s3_data_path = f\"{s3_data_path}/bike_rts.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rts_dataset_import_job_response = forecast.create_dataset_import_job(DatasetImportJobName=dataset_group,\n",
    "                                                             DatasetArn=rts_dataset_arn,\n",
    "                                                             DataSource= {\n",
    "                                                                 \"S3Config\" : {\n",
    "                                                                     \"Path\": rts_s3_data_path,\n",
    "                                                                     \"RoleArn\": role_arn\n",
    "                                                                 } \n",
    "                                                             },\n",
    "                                                             TimestampFormat=timestamp_format)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rts_dataset_import_job_arn=rts_dataset_import_job_response['DatasetImportJobArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status = wait(lambda: forecast.describe_dataset_import_job(DatasetImportJobArn=rts_dataset_import_job_arn))\n",
    "assert status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3. Choosing an algorithm and evaluating its performance<a class=\"anchor\" id=\"algo\">\n",
    "\n",
    "Once the datasets are specified with the corresponding schema, Amazon Forecast will automatically aggregate all the relevant pieces of information for each item, such as sales, price, promotions, as well as categorical attributes, and generate the desired dataset. Next, one can choose an algorithm (forecasting model) and evaluate how well this particular algorithm works on this dataset. The following graph gives a high-level overview of the forecasting models.\n",
    "<img src=\"BlogImages/recipes.png\">\n",
    "<img src=\"BlogImages/pred_details.png\">\n",
    "\n",
    "\n",
    "Amazon Forecast provides several state-of-the-art forecasting algorithms including classic forecasting methods such as ETS, ARIMA, Prophet and deep learning approaches such as DeepAR+. Classical forecasting methods, such as Autoregressive Integrated Moving Average (ARIMA) or Exponential Smoothing (ETS), fit a single model to each individual time series, and then use that model to extrapolate the time series into the future. Amazon's Non-Parametric Time Series (NPTS) forecaster also fits a single model to each individual time series.  Unlike the naive or seasonal naive forecasters that use a fixed time index (the previous index $T-1$ or the past season $T - \\tau$) as the prediction for time step $T$, NPTS randomly samples a time index $t \\in \\{0, \\dots T-1\\}$ in the past to generate a sample for the current time step $T$.\n",
    "\n",
    "In many applications, you may encounter many similar time series across a set of cross-sectional units. Examples of such time series groupings are demand for different products, server loads, and requests for web pages. In this case, it can be beneficial to train a single model jointly over all of these time series. DeepAR+ takes this approach, outperforming the standard ARIMA and ETS methods when your dataset contains hundreds of related time series. The trained model can also be used for generating forecasts for new time series that are similar to the ones it has been trained on. While deep learning approaches can outperform standard methods, this is only possible when there is sufficient data available for training. It is not true for example when one trains a neural network with a time-series contains only a few dozens of observations. Amazon Forecast provides the best of two worlds allowing users to either choose a specific algorithm or let Amazon Forecast automatically perform model selection. \n",
    "\n",
    "## How to evaluate a forecasting model?\n",
    "\n",
    "Before moving forward, let's first introduce the notion of *backtest* when evaluating forecasting models. The key difference between evaluating forecasting algorithms and standard ML applications is that we need to make sure there is no future information gets used in the past. In other words, the procedure needs to be causal. \n",
    "\n",
    "<img src=\"BlogImages/backtest.png\">\n",
    "\n",
    "\n",
    "In this notebook, let's compare the neural network based method, DeepAR+ with Facebook's open-source Bayesian method Prophet. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm_arn = 'arn:aws:forecast:::algorithm/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 Choosing DeepAR+<a class=\"anchor\" id=\"DeepAR\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'Deep_AR_Plus'\n",
    "algorithm_arn_deep_ar_plus = algorithm_arn + algorithm\n",
    "predictor_name_deep_ar = f'{project}_{algorithm.lower()}_{idx}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(f'[{predictor_name_deep_ar}] Creating predictor {predictor_name_deep_ar} ...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_predictor_response = forecast.create_predictor(PredictorName=predictor_name_deep_ar,\n",
    "                                                  AlgorithmArn=algorithm_arn_deep_ar_plus,\n",
    "                                                  ForecastHorizon=forecast_horizon,\n",
    "                                                  PerformAutoML=False,\n",
    "                                                  PerformHPO=False,\n",
    "                                                  InputDataConfig= {\"DatasetGroupArn\": dataset_group_arn},\n",
    "                                                  FeaturizationConfig= {\"ForecastFrequency\": freq}\n",
    "                                                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor_arn_deep_ar = create_predictor_response['PredictorArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status = wait(lambda: forecast.describe_predictor(PredictorArn=predictor_arn_deep_ar))\n",
    "assert status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_predictor(PredictorArn=predictor_arn_deep_ar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4. Computing Error Metrics from Backtesting<a class=\"anchor\" id=\"error\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating the predictors, we can query the forecast accuracy given by the backtest scenario and have a quantitative understanding of the performance of the algorithm. Such a process is iterative in nature during model development. When an algorithm with satisfying performance is found, the customer can deploy the predictor into a production environment, and query the forecasts for a particular item to make business decisions. The figure below shows a sample plot of different quantile forecasts of a predictor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info('Done creating predictor. Getting accuracy numbers for DeepAR+ ...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "error_metrics_deep_ar_plus = forecast.get_accuracy_metrics(PredictorArn=predictor_arn_deep_ar)\n",
    "error_metrics_deep_ar_plus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info('Done creating predictor. Getting accuracy numbers for Prophet ...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_summary_metrics(metric_response, predictor_name):\n",
    "    df = pd.DataFrame(metric_response['PredictorEvaluationResults']\n",
    "                 [0]['TestWindows'][0]['Metrics']['WeightedQuantileLosses'])\n",
    "    df['Predictor'] = predictor_name\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deep_ar_metrics = extract_summary_metrics(error_metrics_deep_ar_plus, \"DeepAR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deep_ar_metrics.pivot(index='Quantile', columns='Predictor', values='LossValue').plot.bar();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we mentioned before, if you only have a handful of time series (in this case, only 1) with a small number of examples, the neural network models (DeepAR+) are not the best choice. Here, we clearly see that DeepAR+ behaves worse than Prophet in the case of a single time series. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 5. Creating a Forecast<a class=\"anchor\" id=\"forecast\">\n",
    "\n",
    "Next we re-train with the full dataset, and create the forecast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(f\"Done fetching accuracy numbers. Creating forecaster for DeepAR+ ...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_name_deep_ar = f'{project}_deep_ar_plus_{idx}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_forecast_response_deep_ar = forecast.create_forecast(ForecastName=forecast_name_deep_ar,\n",
    "                                                        PredictorArn=predictor_arn_deep_ar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_arn_deep_ar = create_forecast_response_deep_ar['ForecastArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status = wait(lambda: forecast.describe_forecast(ForecastArn=forecast_arn_deep_ar))\n",
    "assert status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_forecast(ForecastArn=forecast_arn_deep_ar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(f\"Done fetching accuracy numbers. Creating forecaster for Prophet ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 6. Querying the Forecasts<a class=\"anchor\" id=\"query\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_id = 'bike_12'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_response_deep = forecast_query.query_forecast(\n",
    "    ForecastArn=forecast_arn_deep_ar,\n",
    "    Filters={\"item_id\": item_id})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = f'./data/bike_small.csv'\n",
    "exact = load_exact_sol(fname, item_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_forecasts(forecast_response_deep, exact)\n",
    "plt.title(\"DeepAR Forecast\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 7. Exporting your Forecasts<a class=\"anchor\" id=\"export\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_export_name_deep_ar = f'{project}_forecast_export_deep_ar_plus_{idx}'\n",
    "forecast_export_name_deep_ar_path = f\"{s3_data_path}/{forecast_export_name_deep_ar}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_forecast_export_response_deep_ar = forecast.create_forecast_export_job(ForecastExportJobName=forecast_export_name_deep_ar,\n",
    "                                                        ForecastArn=forecast_arn_deep_ar,\n",
    "                                                        Destination={\n",
    "                                                            \"S3Config\" : {\n",
    "                                                                \"Path\": forecast_export_name_deep_ar_path,\n",
    "                                                                \"RoleArn\": role_arn\n",
    "                                                            }\n",
    "                                                        })\n",
    "forecast_export_arn_deep_ar = create_forecast_export_response_deep_ar['ForecastExportJobArn']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 8. Cleaning up your Resources<a class=\"anchor\" id=\"cleanup\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have completed the above steps, we can start to cleanup the resources we created. All delete jobs, except for `delete_dataset_group` are asynchronous, so we have added the helpful `wait_till_delete` function. \n",
    "Resource Limits documented <a href=\"https://docs.aws.amazon.com/forecast/latest/dg/limits.html\">here</a>. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete forecast export for both algorithms\n",
    "wait_till_delete(lambda: forecast.delete_forecast_export_job(ForecastExportJobArn = forecast_export_arn_deep_ar))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete forecast for both algorithms\n",
    "wait_till_delete(lambda: forecast.delete_forecast(ForecastArn = forecast_arn_deep_ar))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete predictor for both algorithms\n",
    "wait_till_delete(lambda: forecast.delete_predictor(PredictorArn = predictor_arn_deep_ar))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the target time series and related time series dataset import jobs\n",
    "wait_till_delete(lambda: forecast.delete_dataset_import_job(DatasetImportJobArn=ts_dataset_import_job_arn))\n",
    "wait_till_delete(lambda: forecast.delete_dataset_import_job(DatasetImportJobArn=rts_dataset_import_job_arn))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the target time series and related time series datasets\n",
    "wait_till_delete(lambda: forecast.delete_dataset(DatasetArn=ts_dataset_arn))\n",
    "wait_till_delete(lambda: forecast.delete_dataset(DatasetArn=rts_dataset_arn))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete dataset group\n",
    "forecast.delete_dataset_group(DatasetGroupArn=dataset_group_arn)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}