notebook1_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dancing with censored data: How to survive with explainable survival analysis?\n",
    "**ML in PL Conference 2023**\n",
    "\n",
    "\n",
    "Mateusz Krzyziński, Mikołaj Spytek (**MI2.AI**)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "- 9:00 - 9:15 - Introduction, technicalities\n",
    "- 9:15 - 10:00 - ML in survival analysis, creating models `(Python and R)`\n",
    "- 10:00 - 10:10 - *break*\n",
    "\n",
    "- 10:10 - 10:20 - XAI in survival analysis\n",
    "- 10:20 - 10:45 - SurvLIME `(Python and R)`\n",
    "- 10:45 - 11:30 - SurvSHAP(t) and its aggregations `(Python and R)`\n",
    "- 11:30 - 11:40 - *break*\n",
    "\n",
    "- 11:40 - 11:50 - creating explainers for any models `(R with Python via reticulate)`\n",
    "- 11:50 - 12:15 - global performance & global variable importance explanations `(R)`\n",
    "- 12:15 - 12:30 - local variable dependence explanations `(R)`\n",
    "- 12:30 - 12:45 - global variable dependence explanations `(R)`\n",
    "- 12:45 - 13:00 - summary and Q&A\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Technicalities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing and loading packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib as mpl\n",
    "\n",
    "mpl.rcParams[\"axes.spines.right\"] = False\n",
    "mpl.rcParams[\"axes.spines.top\"] = False\n",
    "\n",
    "def plot(survival_prob):\n",
    "    fig, ax = plt.subplots()\n",
    "    ax.plot(survival_prob[0].x, survival_prob[0].y, label=\"Patient A\")\n",
    "    ax.plot(survival_prob[1].x, survival_prob[1].y, label=\"Patient B\")\n",
    "    ax.set_xlabel(\"Time\")\n",
    "    ax.set_ylabel(\"Survival probability\")\n",
    "    ax.set_ylim(0, 1)\n",
    "    ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro to survival analysis\n",
    "\n",
    "### What is survival analysis?\n",
    "- type of supervised learning task\n",
    "- also known as time-to-event analysis, reliability analysis, duration analysis, ... \n",
    "- **data modality:** mostly tabular, censored data (the most popular case is right censoring)\n",
    "\n",
    "<img src=\"images/censoring.png\" width=\"600\">\n",
    "\n",
    "- **output:** survival probability distribution (can be represented by survival function)\n",
    "\n",
    "<img src=\"images/survival_probs.png\" width=\"600\">\n",
    "\n",
    "\n",
    "### Mathematical background\n",
    "\n",
    "#### Data format\n",
    "- instance $i$: $(\\textbf{x}_i, y_i, \\delta_i)$\n",
    "- $\\textbf{x}_i$ is a vector of features\n",
    "-  $\\delta_i$ is the indicator of event of interest's occurrence (usually 0 is for censored data, 1 for event of interest's occurrence)\n",
    "-  $y_i$ stands for the observed time \n",
    "    - it can be either survival time $T_i$ when $\\delta_i = 1$ \n",
    "    - or censoring time $C_i$ when $\\delta_i = 0$\n",
    "- we want to model the conditional distribution of $T$, not $Y$ (but we don't observe $T$ directly!)\n",
    "\n",
    "\n",
    "#### Key quantities  \n",
    "- **survival function:** (probability of surviving past a given time point)\n",
    " $$S(t|\\mathbf{x}) = \\mathbb{P}(T > t|\\mathbf{X}=\\mathbf{x}) = 1 - \\mathbb{P}(T \\leq t|\\mathbf{X}=\\mathbf{x})$$\n",
    "\n",
    "\n",
    "- **hazard function:** (potential for an event to occur at infinitesimally small time interval $[t, \\Delta t]$)\n",
    "$$h(t|\\mathbf{x}) = \\lim_{\\Delta t \\to 0} \\frac{\\mathbb{P}(t \\leq T < t + \\Delta t|T \\geq t, \\mathbf{X}=\\mathbf{x})}{\\Delta t}$$\n",
    "\n",
    "- **cumulative hazard function:**  (total accumulated risk of an event up to time $t$)\n",
    "$$H(t|\\mathbf{x}) = \\int_0^t h(s|\\mathbf{x})ds$$\n",
    "\n",
    "- they are related by: $S(t|\\mathbf{x}) = \\exp(-H(t|\\mathbf{x}))$\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First look at the data & statistical models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('datasets/lung_dataset.csv')\n",
    "df = df.dropna()\n",
    "df[\"status\"] = df[\"status\"].astype('bool')\n",
    "\n",
    "from sksurv.datasets import get_x_y\n",
    "\n",
    "X, y = get_x_y(df, attr_labels=[\"status\",\"time\"], pos_label=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data contain subjects with advanced lung cancer.\n",
    "\n",
    "- time: survival time (in days)\n",
    "- status: event status, 0=censored, 1=deceased\n",
    "- age: age (in years)\n",
    "- sex: 1=male, 2=female\n",
    "- ph.ecog: ECOG performance score as rated by the physician: \n",
    "    - 0=asymptomatic\n",
    "    - 1=symptomatic but completely ambulatory\n",
    "    - 2=in bed <50% of the day\n",
    "    - 3=in bed > 50% of the day but not bedbound\n",
    "    - 4=bedbound\n",
    "- ph.karno: Karnofsky performance score (0=bad, 100=good) rated by physician\n",
    "- pat.karno: Karnofsky performance score (0=bad, 100=good) rated by patient\n",
    "- meal.cal: calories consumed at meals\n",
    "- wt.loss: weight loss in last six months"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kaplan-Meier estimator \n",
    "- non-parametric estimator for survival function \n",
    "- estimates $S(t)$ only looking at the $(y_i, \\delta_i)$ pairs - do not take into account features $\\textbf{x}_i$\n",
    "- separate estimators can be fitted for each group of data instances with the same value of categorical feature (stratifying)\n",
    "\n",
    "Let's check how it looks like for our data, first for the small subset, then for the whole dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_df = df.loc[0:19, [\"time\", \"status\"]]\n",
    "tmp_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sksurv.nonparametric import kaplan_meier_estimator\n",
    "\n",
    "time, survival_prob = kaplan_meier_estimator(tmp_df[\"status\"], tmp_df[\"time\"])\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.step(time, survival_prob, where=\"post\")\n",
    "ax.set_xlabel(\"Time\")\n",
    "ax.set_ylabel(\"Survival probability\")\n",
    "ax.set_ylim(0, 1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time, survival_prob, conf_interval = kaplan_meier_estimator(\n",
    "    df[\"status\"], df[\"time\"], conf_type=\"log-log\"\n",
    ")\n",
    "\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.step(time, survival_prob, where=\"post\")\n",
    "ax.fill_between(\n",
    "    time, conf_interval[0], conf_interval[1], step=\"post\", alpha=0.2\n",
    ")\n",
    "ax.set_xlabel(\"Time\")\n",
    "ax.set_ylabel(\"Survival probability\")\n",
    "ax.set_ylim(0, 1)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cox proportional hazards model \n",
    "- semi-parametric model for hazard function\n",
    "- assumes that the hazard function is a product of two parts: \n",
    "    - baseline hazard function $h_0(t)$ - does not depend on features $\\textbf{x}_i$\n",
    "    - exponential function of linear combination of features $\\textbf{x}_i$\n",
    "$$ h(t|\\mathbf{x}) = h_0(t) \\exp(\\beta_1 x_1 + \\beta_2 x_2 + \\dots + \\beta_p x_p)$$\n",
    "- good baseline model (~*linear regression of survival analysis*)\n",
    "- considered interpretable (exponentiated coefficients $\\exp(\\beta_j)$ can be interpreted as the hazard ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sksurv.linear_model import CoxPHSurvivalAnalysis\n",
    "\n",
    "cph_model = CoxPHSurvivalAnalysis(ties=\"efron\")\n",
    "cph_model.fit(X, y)\n",
    "cph_model.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.exp(cph_model.coef_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patientA = df.iloc[41, 2:9]\n",
    "patientB = patientA.copy()\n",
    "patientB[\"ph.ecog\"] = 2\n",
    "\n",
    "patients = pd.DataFrame([patientA, patientB])\n",
    "patients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_X = X.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risks = np.exp(np.sum((patients - mean_X) * cph_model.coef_, axis=1)).values\n",
    "risks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risks[1]/risks[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "survival_prob = cph_model.predict_survival_function(patients)\n",
    "\n",
    "plot(survival_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine learning models\n",
    "\n",
    "### Survival trees\n",
    "- extension of decision trees for survival analysis\n",
    "- for each leaf node, based on the data instances in the node, one of the key quantities is calculated (e.g., cumulative hazard function)\n",
    "- there are many splitting rules but the most common is the log-rank splitting rule (based on the log-rank test comparing survival distributions of two samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sksurv.tree import SurvivalTree\n",
    "\n",
    "tree_model = SurvivalTree()\n",
    "tree_model.fit(X, y)\n",
    "\n",
    "survival_prob = tree_model.predict_survival_function(patients)\n",
    "\n",
    "plot(survival_prob)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random survival forest\n",
    "- works similarly to classical random forest but uses survival trees instead of classification or regression decision trees\n",
    "- predictions are based on the average of predictions of all trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sksurv.ensemble import RandomSurvivalForest\n",
    "\n",
    "rsf_model = RandomSurvivalForest(n_estimators=100)\n",
    "rsf_model.fit(X, y)\n",
    "\n",
    "survival_prob = rsf_model.predict_survival_function(patients)\n",
    "\n",
    "plot(survival_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/models.png\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Find more here:**\n",
    "- Wang, P., Li, Y., & Reddy, C. K. (2019). Machine learning for survival analysis: A survey. ACM Computing Surveys (CSUR), 51(6), 1-36. https://doi.org/10.1145/3214306\n",
    "- Wiegrebe, S., Kopper, P., Sonabend, R., & Bender, A. (2023). Deep Learning for Survival Analysis: A Review. arXiv preprint arXiv:2305.14961. https://arxiv.org/abs/2305.14961"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}