04_summary_statistics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04-summary-statistics",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MIT-LCP/2019_hack_aotearoa_eicu/blob/master/04_summary_statistics.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "1G_TVh1ybQkl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# eICU Collaborative Research Database\n",
        "\n",
        "# Notebook 4: Summary statistics\n",
        "\n",
        "This notebook shows how summary statistics can be computed for a patient cohort using the `tableone` package. Usage instructions for tableone are at: https://pypi.org/project/tableone/\n"
      ]
    },
    {
      "metadata": {
        "id": "L9XF77F2bnee",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load libraries and connect to the database"
      ]
    },
    {
      "metadata": {
        "id": "wXiSE558bn_w",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Import libraries\n",
        "import numpy as np\n",
        "import os\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.patches as patches\n",
        "import matplotlib.path as path\n",
        "\n",
        "# Make pandas dataframes prettier\n",
        "from IPython.display import display, HTML\n",
        "\n",
        "# Access data using Google BigQuery.\n",
        "from google.colab import auth\n",
        "from google.cloud import bigquery"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pLGnLAy-bsKb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# authenticate\n",
        "auth.authenticate_user()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PUjFDFdobszs",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Set up environment variables\n",
        "project_id='new-zealand-2018-datathon'\n",
        "os.environ[\"GOOGLE_CLOUD_PROJECT\"]=project_id"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bkJUF8HBbvWe",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Helper function to read data from BigQuery into a DataFrame.\n",
        "def run_query(query):\n",
        "    return pd.io.gbq.read_gbq(query, project_id=project_id, verbose=False, \n",
        "                            configuration={'query':{'useLegacySql': False}})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iWDUCA5Nb5BK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Install and load the `tableone` package\n",
        "\n",
        "The tableone package can be used to compute summary statistics for a patient cohort. Unlike the previous packages, it is not installed by default in Colab, so will need to install it first."
      ]
    },
    {
      "metadata": {
        "id": "F9doCgtscOJd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install tableone"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SDI_Q7W0b4Le",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Import the tableone class\n",
        "from tableone import TableOne"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "14TU4lcrdD7I",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load the patient cohort\n",
        "\n",
        "In this example, we will load all data from the patient data, and link it to APACHE data to provide richer summary information."
      ]
    },
    {
      "metadata": {
        "id": "HF5WF5EObwfw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Link the patient and apachepatientresult tables on patientunitstayid\n",
        "# using an inner join.\n",
        "query = \"\"\"\n",
        "SELECT p.unitadmitsource, p.gender, p.age, p.ethnicity, p.admissionweight, \n",
        "    p.unittype, p.unitstaytype, a.acutephysiologyscore,\n",
        "    a.apachescore, a.actualiculos, a.actualhospitalmortality,\n",
        "    a.unabridgedunitlos, a.unabridgedhosplos\n",
        "FROM `physionet-data.eicu_crd_demo.patient` p\n",
        "INNER JOIN `physionet-data.eicu_crd_demo.apachepatientresult` a\n",
        "ON p.patientunitstayid = a.patientunitstayid\n",
        "WHERE apacheversion LIKE 'IVa'\n",
        "\"\"\"\n",
        "\n",
        "cohort = run_query(query)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "k3hURHFihHNA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cohort.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qnG8dVb2iHSn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Calculate summary statistics\n",
        "\n",
        "Before summarizing the data, we will need to convert the ages to numerical values."
      ]
    },
    {
      "metadata": {
        "id": "oKHpqwAPkx6U",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cohort['agenum'] = pd.to_numeric(cohort['age'], errors='coerce')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FQT-u8EXhXRG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "columns = ['unitadmitsource', 'gender', 'agenum', 'ethnicity',\n",
        "          'admissionweight','unittype','unitstaytype',\n",
        "          'acutephysiologyscore','apachescore','actualiculos',\n",
        "          'unabridgedunitlos','unabridgedhosplos']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3ETr3NCzielL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "TableOne(cohort, columns=columns, labels={'agenum': 'age'}, \n",
        "         groupby='actualhospitalmortality',\n",
        "         label_suffix=True, limit=4)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LCBcpJ9bZpDp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Questions\n",
        "\n",
        "- Are the severity of illness measures higher in the survival or non-survival group?\n",
        "- What issues suggest that some of the summary statistics might be misleading?\n",
        "- How might you address these issues?"
      ]
    },
    {
      "metadata": {
        "id": "2_8z1CIVahWg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Visualizing the data\n",
        "\n",
        "Plotting the distribution of each variable by group level via histograms, kernel density estimates and boxplots is a crucial component to data analysis pipelines. Vizualisation is often is the only way to detect problematic variables in many real-life scenarios. We'll review a couple of the variables."
      ]
    },
    {
      "metadata": {
        "id": "81yp2bSUigzh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Plot distributions to review possible multimodality\n",
        "cohort[['acutephysiologyscore','agenum']].dropna().plot.kde(figsize=[12,8])\n",
        "plt.legend(['APS Score', 'Age (years)'])\n",
        "plt.xlim([-30,250])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kZDUZB5sdhhU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Questions\n",
        "\n",
        "- Do the plots change your view on how these variable should be reported?"
      ]
    }
  ]
}