04_python_stat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2,
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href=\"#Descriptive-statistics\" data-toc-modified-id=\"Descriptive-statistics-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Descriptive statistics</a></span></li><li><span><a href=\"#Classical-hypothesis-testing\" data-toc-modified-id=\"Classical-hypothesis-testing-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Classical hypothesis testing</a></span></li><li><span><a href=\"#Simulation-and-inference\" data-toc-modified-id=\"Simulation-and-inference-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Simulation and inference</a></span><ul class=\"toc-item\"><li><span><a href=\"#Simulation-and-hypothesis-testing\" data-toc-modified-id=\"Simulation-and-hypothesis-testing-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>Simulation and hypothesis testing</a></span></li><li><span><a href=\"#A-permutation-test\" data-toc-modified-id=\"A-permutation-test-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>A permutation test</a></span></li><li><span><a href=\"#Testing-many-proteins\" data-toc-modified-id=\"Testing-many-proteins-4.3\"><span class=\"toc-item-num\">4.3&nbsp;&nbsp;</span>Testing many proteins</a></span></li><li><span><a href=\"#Getting-a-confidence-interval-using-the-bootstrap\" data-toc-modified-id=\"Getting-a-confidence-interval-using-the-bootstrap-4.4\"><span class=\"toc-item-num\">4.4&nbsp;&nbsp;</span>Getting a confidence interval using the bootstrap</a></span></li></ul></li><li><span><a href=\"#Regression-analysis\" data-toc-modified-id=\"Regression-analysis-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Regression analysis</a></span><ul class=\"toc-item\"><li><span><a href=\"#Ordinary-least-squares-(linear)-regression\" data-toc-modified-id=\"Ordinary-least-squares-(linear)-regression-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>Ordinary least squares (linear) regression</a></span></li><li><span><a href=\"#Logistic-regression\" data-toc-modified-id=\"Logistic-regression-5.2\"><span class=\"toc-item-num\">5.2&nbsp;&nbsp;</span>Logistic regression</a></span></li><li><span><a href=\"#Survival-analysis\" data-toc-modified-id=\"Survival-analysis-5.3\"><span class=\"toc-item-num\">5.3&nbsp;&nbsp;</span>Survival analysis</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "lines_to_next_cell": 2,
    "tags": [
     "active-Rmd"
    ]
   },
   "source": [
    "%%R\n",
    "reticulate::use_condaenv('ds', required=T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistical analysis\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Statistical analysis usually encompasses 3 activities in a data science workflow. These are (a) descriptive analysis, (b) hypothesis testing and (c) statistical modeling. Descriptive analysis refers to a description of the data, which includes computing summary statistics and drawing plots. Hypothesis testing usually refers to statistically seeing if two (or more) groups are different from each other based on some metrics. Modeling refers to fitting a curve to the data to describe the relationship patterns of different variables in a data set.\n",
    "\n",
    "In terms of Python packages that can address these three tasks:\n",
    "\n",
    "| Task                   | Packages               |\n",
    "| :---------------------- | ---------------------- |\n",
    "| Descriptive statistics | pandas, numpy, matplotlib, seaborn          |\n",
    "| Hypothesis testing     | scipy, statsmodels     |\n",
    "| Modeling               | statsmodels, lifelines, scikit-learn |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Descriptive statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Descriptive statistics that are often computed are the mean, median, standard deviation, inter-quartile range, pairwise correlations, and the like. Most of these functions are available in `numpy`, and hence are available in `pandas`. We have already seen how we can compute these statistics and have even computed grouped statistics. For example, we will compute these using the diamonds dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "name": "04-python-stat-1"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy as sc\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set_context('paper')\n",
    "sns.set_style('white', {'font.family': 'Future Medium'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "name": "04-python-stat-2"
   },
   "outputs": [],
   "source": [
    "diamonds = pd.read_csv('data/diamonds.csv.gz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "name": "04-python-stat-3"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>median</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>color</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>D</th>\n",
       "      <td>3169.954096</td>\n",
       "      <td>1838.0</td>\n",
       "      <td>3356.590935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>E</th>\n",
       "      <td>3076.752475</td>\n",
       "      <td>1739.0</td>\n",
       "      <td>3344.158685</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F</th>\n",
       "      <td>3724.886397</td>\n",
       "      <td>2343.5</td>\n",
       "      <td>3784.992007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>G</th>\n",
       "      <td>3999.135671</td>\n",
       "      <td>2242.0</td>\n",
       "      <td>4051.102846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>H</th>\n",
       "      <td>4486.669196</td>\n",
       "      <td>3460.0</td>\n",
       "      <td>4215.944171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I</th>\n",
       "      <td>5091.874954</td>\n",
       "      <td>3730.0</td>\n",
       "      <td>4722.387604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>J</th>\n",
       "      <td>5323.818020</td>\n",
       "      <td>4234.0</td>\n",
       "      <td>4438.187251</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              mean  median          std\n",
       "color                                  \n",
       "D      3169.954096  1838.0  3356.590935\n",
       "E      3076.752475  1739.0  3344.158685\n",
       "F      3724.886397  2343.5  3784.992007\n",
       "G      3999.135671  2242.0  4051.102846\n",
       "H      4486.669196  3460.0  4215.944171\n",
       "I      5091.874954  3730.0  4722.387604\n",
       "J      5323.818020  4234.0  4438.187251"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.groupby('color')['price'].agg([np.mean, np.median, np.std])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There were other examples we saw yesterday along these lines. Refer to both the `python_tools_ds` and `python_pandas` documents\n",
    "\n",
    "##  Classical hypothesis testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has the tools to do classic hypothesis testing. Several functions are available in the `scipy.stats` module. The commonly used tests that are available are as follows:\n",
    "\n",
    "| Function           | Test                                                    |\n",
    "| :----------------- | ------------------------------------------------------- |\n",
    "| `ttest_1samp`      | One-sample t-test                                       |\n",
    "| `ttest_ind`        | Two-sample t-test                                       |\n",
    "| `ttest_rel`        | Paired t-test                                           |\n",
    "| `wilcoxon`         | Wilcoxon signed-rank test (nonparametric paired t-test) |\n",
    "| `mannwhitneyu`         | Wilcoxon rank-sum test (nonparametric 2-sample t-test)  |\n",
    "| `chi2_contingency` | Chi-square test for independence                        |\n",
    "| `fisher_exact`     | Fisher's exact test on a 2x2 contingency table          |\n",
    "| `f_oneway`         | One-way ANOVA                                           |\n",
    "| `pearsonr`         | Testing for correlation                                 |\n",
    "|                    |                                                         |\n",
    "\n",
    "There are also several tests in `statsmodels.stats`\n",
    "\n",
    "| Functions           | Tests                                 |\n",
    "| :------------------ | ------------------------------------- |\n",
    "| `proportions_ztest` | Test for difference in proportions    |\n",
    "| `mcnemar`           | McNemar's test                        |\n",
    "| `sign_test`         | Sign test                             |\n",
    "| `multipletests`     | p-value correction for multiple tests |\n",
    "| `fdrcorrection`     | p-value correction by FDR             |\n",
    "|                     |                                       |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at a breast cancer proteomics experiment to illustrate this. The experimental data contains protein expression for over 12 thousand proteins, along with clinical data. We can ask, for example, whether a particular protein expression differs by ER status. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "lines_to_next_cell": 0,
    "name": "04-python-stat-22"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Complete TCGA ID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age at Initial Pathologic Diagnosis</th>\n",
       "      <th>ER Status</th>\n",
       "      <th>PR Status</th>\n",
       "      <th>HER2 Final Status</th>\n",
       "      <th>Tumor</th>\n",
       "      <th>Tumor--T1 Coded</th>\n",
       "      <th>Node</th>\n",
       "      <th>...</th>\n",
       "      <th>NP_001193600</th>\n",
       "      <th>NP_061134</th>\n",
       "      <th>NP_932347</th>\n",
       "      <th>NP_003593</th>\n",
       "      <th>NP_997203</th>\n",
       "      <th>NP_001191293</th>\n",
       "      <th>NP_775791</th>\n",
       "      <th>NP_004065</th>\n",
       "      <th>NP_068752</th>\n",
       "      <th>NP_219494</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>TCGA-A2-A0CM</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>40</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>T2</td>\n",
       "      <td>T_Other</td>\n",
       "      <td>N0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.153614</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>TCGA-BH-A18Q</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>56</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>T2</td>\n",
       "      <td>T_Other</td>\n",
       "      <td>N1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.048144</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.881872</td>\n",
       "      <td>2.527072</td>\n",
       "      <td>-8.111243</td>\n",
       "      <td>-16.029761</td>\n",
       "      <td>-2.046065</td>\n",
       "      <td>-1.778435</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-3.069752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>TCGA-A7-A0CE</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>57</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>T2</td>\n",
       "      <td>T_Other</td>\n",
       "      <td>N0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.644347</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.625952</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.306238</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>TCGA-D8-A142</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>74</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>T3</td>\n",
       "      <td>T_Other</td>\n",
       "      <td>N0</td>\n",
       "      <td>...</td>\n",
       "      <td>-5.107629</td>\n",
       "      <td>-0.97598</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.508629</td>\n",
       "      <td>-12.337110</td>\n",
       "      <td>-9.546530</td>\n",
       "      <td>-4.066584</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>TCGA-AO-A0J6</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>61</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>Negative</td>\n",
       "      <td>T2</td>\n",
       "      <td>T_Other</td>\n",
       "      <td>N0</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.043420</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-3.231339</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-3.753616</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 12585 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0 Complete TCGA ID  Gender  Age at Initial Pathologic Diagnosis  \\\n",
       "0           0     TCGA-A2-A0CM  FEMALE                                   40   \n",
       "1           1     TCGA-BH-A18Q  FEMALE                                   56   \n",
       "2           2     TCGA-A7-A0CE  FEMALE                                   57   \n",
       "3           3     TCGA-D8-A142  FEMALE                                   74   \n",
       "4           4     TCGA-AO-A0J6  FEMALE                                   61   \n",
       "\n",
       "  ER Status PR Status HER2 Final Status Tumor Tumor--T1 Coded Node  ...  \\\n",
       "0  Negative  Negative          Negative    T2         T_Other   N0  ...   \n",
       "1  Negative  Negative          Negative    T2         T_Other   N1  ...   \n",
       "2  Negative  Negative          Negative    T2         T_Other   N0  ...   \n",
       "3  Negative  Negative          Negative    T3         T_Other   N0  ...   \n",
       "4  Negative  Negative          Negative    T2         T_Other   N0  ...   \n",
       "\n",
       "  NP_001193600 NP_061134 NP_932347 NP_003593  NP_997203 NP_001191293  \\\n",
       "0          NaN       NaN  1.153614       NaN        NaN          NaN   \n",
       "1     0.048144       NaN -0.881872  2.527072  -8.111243   -16.029761   \n",
       "2     0.644347       NaN  1.625952       NaN        NaN          NaN   \n",
       "3    -5.107629  -0.97598       NaN  2.508629 -12.337110    -9.546530   \n",
       "4    -1.043420       NaN       NaN       NaN  -3.231339          NaN   \n",
       "\n",
       "  NP_775791  NP_004065  NP_068752  NP_219494  \n",
       "0       NaN        NaN        NaN        NaN  \n",
       "1 -2.046065  -1.778435        NaN  -3.069752  \n",
       "2 -1.306238        NaN        NaN        NaN  \n",
       "3 -4.066584        NaN        NaN        NaN  \n",
       "4       NaN        NaN        NaN  -3.753616  \n",
       "\n",
       "[5 rows x 12585 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brca = pd.read_csv('data/brca.csv')\n",
    "brca.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use both the t-test and the Wilcoxon rank-sum test, the nonparametric equivalent. \n",
    "\n",
    "We will first do the classical t-test, that is available in the `scipy` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "name": "04-python-stat-23"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.277"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import scipy as sc\n",
    "import statsmodels as sm\n",
    "test_probe = 'NP_001193600'\n",
    "\n",
    "tst = sc.stats.ttest_ind(brca[brca['ER Status']=='Positive'][test_probe], # Need [] since names have spaces\n",
    "                   brca[brca['ER Status']=='Negative'][test_probe], \n",
    "                  nan_policy = 'omit')\n",
    "np.round(tst.pvalue, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now do the Wilcoxon test, also known as the Mann-Whitney U test. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.996"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tst = sc.stats.mannwhitneyu(brca[brca['ER Status']=='Positive'][test_probe], # Need [] since names have spaces\n",
    "                   brca[brca['ER Status']=='Negative'][test_probe], \n",
    "                  alternative = 'two-sided')\n",
    "np.round(tst.pvalue, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will come back to this when we look at permutation tests below. \n",
    "\n",
    "## Simulation and inference\n",
    "\n",
    "Hypothesis testing is one of the areas where statistics is often used. There are functions for a lot of the standard statistical tests in `scipy` and `statsmodels`. However, I'm going to take a little detour to see if we can get some understanding of hypothesis tests using the powerful simulation capabilities of Python. We'll visit the in-built functions available in `scipy` and `statsmodels` as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulation and hypothesis testing\n",
    "\n",
    "**Question:** You have a coin and you flip it 100 times. You get 54 heads. How likely is it that you have a fair coin?\n",
    "\n",
    "We can simulate this process, which is random, using Python. The process of heads and tails from coin tosses can be modeled as a [**binomial** distribution](https://en.wikipedia.org/wiki/Binomial_distribution). We can repeat this experiment many many times on our computer, making the assumption that we have a fair coin, and then seeing how likely what we observed is under that assumption. \n",
    "\n",
    "> Simulation under reasonable assumptions is a great way to understand our data and the underlying data generating processes. In the modern era, it has most famously been used by Nate Silver of ESPN to simulate national elections in the US. There are many examples in engineering where simulations are done to understand a technology and figure out its tolerances and weaknesses, like in aircraft testing. It is also commonly used in epidemic modeling to help understand how an epidemic would spread under different conditions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "name": "04-python-stat-4"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEHCAYAAABLKzaMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXxU9b038M9smTUzkz0khAkkQICEUBCQsin4VMRyDaWtxID0ivZ2EXt5qFduUR9qxcfbmkeL2tpre2sraKpGaFFEa0VBBVmkiQaSkJAMkH2bzJqZzPL8ERgStiRmOTNnPu/Xi5c5c84k34Phk19+57dIAoFAAEREJEpSoQsgIqKRw5AnIhIxhjwRkYgx5ImIRIwhT0QkYgx5IiIRkwtdQG+TJ08WugQiorBUUVFx1ddDKuSBaxdKFFHq64HUVKCuDkhJEboaCnHXayCzu4aISMQY8kREIsaQJyISMYY8EZGIMeSJiESMIU9EJGIMeSIiEWPIExGJWMhNhiIKVR1OD6zO7j6v6TUKxGiiBKqIqH8MeaIBsjq78Ul1W5/X5mfEMeQppLG7hohIxBjyREQixpAnIhIx9skTXeZqD1gBwNntE6AaoqFhyBNd5moPWAFg6phoAaohGhp21xARiRhDnohIxNhdQzQEHp8f5lbHFa9zkhSFCoY80RA43F4cbbBd8TonSVGoYHcNEZGIMeSJiESMIU80SJVNNhw83SJ0GUQDwpAnGqTS8534rKZd6DKIBoQhTzRIDZ0utDs8cHk4A5ZCH0OeaBC8fj+arW4APWFPFOo4hJIi1vXWqOlweKCQS6FT9v0n0mx1wx8IIFmvQr2FIU+hb0AhX1RUhN27d0Mul2Pbtm0wmUzBc2azGVu2bIHX60VeXh5Wr14NAHjsscfw5Zdfwu/344EHHsCiRYtG5g6IvqLrrVGz+591MGoUWPm1sX3ONXR2IU6nhClOg/rOrmt+bk6SolDRb8hbLBYUFxejqKgIJ0+eRGFhIbZv3x48X1hYiM2bNyMrKwv5+flYtmwZLBYLqqqq8Nprr6GtrQ3f//73GfIUVho7u2C5Siu/vtOFMQYVUo1qHKxqveb7OUmKQkW/ffIlJSWYO3cuZDIZcnJyUFtb2+d8bW0tsrOzIZfLMWfOHJSWliI+Ph4ajQZerxc2mw0xMTEjVT/RsOt0dcPm9qLF7obd7e1zrsHShRSDCilGNVptbnRx+WEKcf225K1WK/R6ffA4EAj0Od/72GAwoLOzE1qtFsnJyVi2bBmcTmeflj9RqDO3OaGQSaCQSWFuc2BaigFAz/d6Q6cLN01OQKJeCalEgtpWBwCJsAUTXUe/LXm9Xg+b7dKvnVJp37f0PrZarTAYDPjkk0/Q2dmJ9957D7t378YTTzwxjCUTjSxzmwNJehXS47QwtzmDrzdZ3XB7/RhjUEEulSLJoET1VfrdiUJJvyGfm5uLI0eOwOfzoaysrM9DVwAwmUwoKyuDz+fD0aNHMX36dPj9fhgMBkilUuh0Ojidzmt8dqLQU9vmRLJehfQ4DWrbLoV4TasD0Uo5olUKAECKQY0zLQx5Cm39dtcYjUbk5eWhoKAgOLrmwIEDsNvtWL58OTZt2hQcXbNy5UoYjUbMnz8fe/bswV133QW3240f/ehHo3EvRMPibLsTk5OjkRajwb6yRri9PijlMpxpdWCMURW8LsWoRul5C5ZOSRKwWqLrG9AQyvz8fOTn5wePe7fmTSYTduzY0ed6mUyGX/3qV8NUItHo8QcCMLc7sWhSAlKMasikEpxrdyEzUYeaVgfGGNTBa1OMarz9RQO6fX4oZJxXSKGJ35lEvbQ7PPB4/UjWqyCTSjAutqfL5mS9FV/UdSItRhO8doxBBW2UDKXnLdf9nF6/Hx6vf6RLJ7oqhjxRL42dXYjRKKC9MNPVFKfFoeo2vHLEjIK54zCl12beCpkUeTNSsL+iBT5/4FqfEu+fbMZrx86NeO1EV8OQJ+ql0dqF9Dht8HhKsh46lRzrF0zAv+SmQCLpO1zytuwxcHl8KLlGaz4QCKC0zoIm67VnxxKNJIY8US+NnV0wxV3qkkmNUWPjLZMwPl571evVUTIsnBiP/eXNV23N11lcsDi70eH0XLe1TzRSGPJEvTRa+4b8QNw4IQ5Ojw//PHdla/7LOiuS9Sr4Az0zaYlGG0Oe6AKXp2f1SVPc1Vvt16JSyHDL1CS8VVqPNrs7+HogEEBZfSfmjI+FUi5Fu8Mz3CUT9YshT3TBF3WdMKgVSB9kSx4Abhwfi8xEHV45chbdvp6RNI3WLrQ7PJiWokesNgptDnc/n4Vo+DHkiS44bm7HTFMMpJLBr0UjkUiwauZYeLx+vHH8PGpbHfiirhOmOC2iVQrEaqPQbmdLnkYfQ54IQJO1C+c6XJg57quvmKpSyHDX3HHocHqw8bUSfFjRguzUnsX94rRKtLG7hgTAnaGIABw3d2BCghax2qGt9T7GoMaPbsrEWKMK751sRmaiDgAQp41CZdOV68sTjTS25Cni+fwBnDjbgVlDaMVfTq9WYGqKHlHynn9isbqePvnLl+omGmlsyZPoXW8vVwAob7TC6w8E140fCXHaKHT7Ami0dl0xoYpbAtJIYsiT6F1vL1cA+KSqDV8bFxNsdY8EvVoBuVSCM60OnGntu/Q2twSkkcTuGopoZ9udONvuwMLM+BH9OlKJBDGaKDRcZ/NvopHAkKeIdqCyBdmpBsQM8YHrQMRqo9DIkKdRxpCniFXX4cKpBisWTUwYla8Xp2PI0+hjyFPE2v3POmQm6pBiVPd/8TCI1UahkatR0ihjyFNE6nB6sL+iBYsmjU4rHuiZEMWWPI02hjxFhI8qW/BhRXPw+MOKFkxOjkZGgm7UaojTRsHm9sLp9o7a1yRiyFNEONVgxXsnm1DeaEWHw4Pj5nbkz04b1RpidVGIVspR2+YY1a9LkY0hTxGhzeHB+Hgt3jh+Hm+V1iM9Xovs1JGb/HQ1UokEOWMNON1sH9WvS5GNIU+i5/R44XB7sWJ6ClKNapxqtOGWrCRBaskda0B1C0OeRg9DnkSv3tLzsDNOF4Xv3pCG78wai/RrbOc30makGdFq98Di5IqUNDoY8iR69RYX9Co5FDIptEo5vjaMC5ENVpJehVhtFKoudNl0urqxp7ResHpI/BjyJHp1FhditUqhywjKTNThdLMdgUAAbxw/h+3/OA3vhd2kiIYbQ55Er87iQtwoLFswUJkJOlS32HGstgPnO1zw+AIccUMjhiFPoldvcSFWFzohn5Ggg8vjw99K67E8ewwSo5U41cANRWhkMORJ9OosXUPe8Wk4qaNkGBujhilWgxvSYzAhXouKRoY8jQyuJ0+i5vH60WzrCqnuGgAomGuCUi6FRCJBRoIO5Y1WoUsikWJLnkStzuKCP4CQaskDPZuIKBUyAMD4BC3K2ZKnEcKQJ1EztzmgU8qhiQrdX1oz4rU43+GCrevKLQqJhip0v/OJhsHZdidSjSqhy7iucbEaKGQSVDbZMMsU2+fc+XYnfFF9R95wT1gaDIY8icrlm3Z/cb4TSYbQDnm5THqhX/7KkD9a2wGXo2+gc09YGgyGPInK5Zt2f1nXiWkpegErGpis5GiUcxgljQD2yZOotTk8SA7xljwATE7WcxgljQiGPImWPxBAu8ODZH3oh3zWmGiUN1oRCASELoVEhiFPotVqd8PrD2BszOjs4ToUWcnRsHZ50cDtAWmYMeRJtGpaHUiIVsIYBg8pk/UqxGgU+KKuU+hSSGQY8iRaNa0OTBBo3fjBkkgkWDgxAfvLm/u/mGgQBhTyRUVFWL16NdasWQOz2dznnNlsxpo1a7B69WoUFRUFXy8tLcU999yDNWvW4MUXXxzeqon6EQgEUNPiwPgwCXkAuGVqEt4/1Qy/n/3yNHz6HUJpsVhQXFyMoqIinDx5EoWFhdi+fXvwfGFhITZv3oysrCzk5+dj2bJl0Gg0ePbZZ/H8889DrQ79/lASnza7Bza3N6xCfvGkBFicHpSct+BrHNxMw6TflnxJSQnmzp0LmUyGnJwc1NbW9jlfW1uL7OxsyOVyzJkzB6WlpThx4gTUajUeeOABrF+/HpWVlSNVP9FVnWl1IEGnRLRKIXQpA2ZQKzB3Qiz+frJJ6FJIRPoNeavVCr3+0mSSy4d49T42GAzo7OxES0sLKioq8Otf/xo/+9nPsHXr1uGrmGgAzrTaMT4hfFrxF90yJQnvn2LI0/DpN+T1ej1stkuTNKTSvm/pfWy1WmEwGKDX6zFz5kxoNBpkZGT0eT/RSPnDxzU4WtuOQCCA2tbw6o+/6JYpSahssuN8u1PoUkgk+g353NxcHDlyBD6fD2VlZTCZTH3Om0wmlJWVwefz4ejRo5g+fTpyc3NRU1MDn8+H5uZmqFShPxmFwpvfH0DhexV48I1S/K2kHtau8OqPvygtVoOs5GgcrGoVuhQSiX4f7xiNRuTl5aGgoAByuRzbtm3DgQMHYLfbsXz5cmzatAlbtmyB1+vFypUrYTQaAQDf+c53sHbtWvh8Pjz00EMjfiMU2c53uOD0+PDw8in4f+9XIl4XBX0Y9cf3dsuUJBwsqUS+0IWQKAzoGX5+fj7y8y99y/VuzZtMJuzYseOK96xatQqrVq0ahhKJ+neq0Yp4XRT+19QkdPsDcHl8Qpf0ld2clYhdbx8RugwSCQ7UIlGoaLRhcnI0gJ5RKgZ1eLbiAWBGmhHaEN7khMILZ7ySKFQ02jA5KfSXFB4ImVSCGyfECV0GiQRDnkThVKMVWWOihS5j2MzP7Al5rkpJQ8WQp7DX1e1DbasDWcniCfkbM3pCvsnGVSlpaBjyFPaqmu0IAJiYKJ6QN6h7Vs6sbnH0cyXR9THkKeyVN9qQHqeFOkomdCnDrrrFLnQJFOYY8hT2yhusouqq6a2+wwW72yt0GRTGGPIU9iqaLg2fFBuDWoHyBqvQZVAYY8hTWOpwemBudcDc6kBZvRVxmiiYWx1wdofvJKirmZQcjbJ6hjx9dQx5CktWZzc+qW7Duyeb0O7woMPVc+wWWchPTopGVbMdXSK7Lxo9nFZHYa3e4kKUTIpYbejv4/pVpBjV0ER1obzRhhlpPetCeXx+mFuvHHWj1ygQEwb72dLoYshTWKtqtmN8vBZSiUToUkaEVCLBlBQ9yuo7gyHvcHtxtOHK5bvnZ8Qx5OkK7K6hsFbVbMfEJJ3QZYyo7BQDKpts8Hj9QpdCYYgteQpb1q5uNFq7RDUJ6mrGx2shl0rx7slGmGI1MKj5z5YGji15CltVzXYY1QrE68TdRSGTSrAsOxn1HS7s/aIBj/y1TOiSKIywSUBhq6rZjsxEHSQi7Y/vbXZ6LGanx6LZ1oVn3j+Nrm4fVArxzfCl4ceWPIUlfyCA0812TEwSd1fN5WK1UZBKgBabW+hSKEww5CksVTXb4XR7kZEQfvu4DoVcKkWyQYUWO0OeBoYhT2HpWG0HUmPU0ETgDkqpRjVa2ZKnAWLIU1g6am4X/aiaa0k1qtmSpwFjyFPYcXq8+KKuExMTxT0+/lpSjWr2ydOAMeQp7HxW0w6FTIq0WI3QpQhibIwabQ4P/NwakAaAIU9h52BlK2aOM0ImFf/QyatJNarh8wfQ4fAIXQqFAYY8hZ2Dp1twgylW6DIEo1croImSsV+eBoQhT2Gl3uLC6WY7ZqfHCF2KoOJ1So6woQFhyFNY+fh0K8bFajA2JjL74y9K0CnZkqcBYchTWDlwugULJ8YLXYbgEqKVaLGxT576x5CnsOHzB/BxVSsWTkwQuhTBxbMlTwPEkKewUVbfCVuXF/My4oQuRXAJ0Uo43F64PNwWkK6PIU9h4+DpVsxIM8KgVghdiuAuLVTWJXQpFOIY8hQ2Pqpkf/xFMqkEyQYVznW4hC6FQhxDnsKC3e3F5+YOLJrE/viLxsdpUdt2aUPvixt89/7T4eTD2UgXeUv4UVg6XN0GTZQM01MNQpcSMtLjtdh1og6BQAASieSqG3xzc29iS57CwsHTLZifGQ+5jN+yF5nitHB6fFysjK6L/2IoLBw8zaGTl9Mp5UiIVqKmV5cN0eUY8hTyzrU7cabVwYeuVzE+TovaVoY8XRtDnkLeq0fOYny8NmKXFr6e9HgtatucCHDZYboGPnilkNDh9MDq7L7i9d3/rMPvP67B/6ybLUBVoS89ToNOVzcsV/m7IwIG2JIvKirC6tWrsWbNGpjN5j7nzGYz1qxZg9WrV6OoqKjPuWPHjmHy5Mlob28fvopJlKzObnxS3dbnz1N/r8Tz+6vw32tnYQG7aq7KqIlCjEbBfnm6pn5D3mKxoLi4GDt37sRDDz2EwsLCPucLCwuxefNm7NixA8XFxbBYLMFzf/rTn5CdnT38VVNE+LCiGRuWTMRNkxOFLiWkjY/X4lhtBzpdbM3TlfoN+ZKSEsydOxcymQw5OTmora3tc762thbZ2dmQy+WYM2cOSktLAQAffPABZs2aBY2G/ag0eE63F7YuL2aOMwpdSsi7eXIifH4/7n/lBD4/2yF0ORRi+g15q9UKvV4fPL78AU/vY4PBgM7OTvj9frz66qvIz88fxlIpkjTauqCQSTDGoBa6lJAXp1Pi3xZnYPWcNLxx/DwsnOVKvfQb8nq9HjbbpVl0Umnft/Q+tlqtMBgM2LNnD5YsWQKlUjmMpVIkaersQmK0KmL3cR0sqUSC5dnJ0ETJUG/homV0Sb8hn5ubiyNHjsDn86GsrAwmk6nPeZPJhLKyMvh8Phw9ehTTp09HZWUl3n33Xaxfvx4VFRXYuHHjiN0AiVOT1Y0kPRsJgyGRSJBqVKO+k4uW0SX9DqE0Go3Iy8tDQUEB5HI5tm3bhgMHDsBut2P58uXYtGkTtmzZAq/Xi5UrV8JoNOLBBx8Mvn/t2rV4+umnR/QmSHwarV2YlqLv/0LqI8WoRr2FIU+XDGicfH5+fp/+9d6teZPJhB07dlzzvS+//PIQyqNIFAgE0GTtwtIsjqoZrBSjGif48JV64YxXCjmdrm64vX4k6VVClxJ2UgwqWLu8sLu9QpdCIYIhTyGn0doFtUKGaBUnZA9WjDYKSrmUXTYUxJCnkNPz0FUFiYQjawZLKpEgxahGA0OeLmDIU8hpsnYh2cCRNV9VikGFuk4Oo6QeDHkKOU3WLvbHDwFH2FBvDHkKKT5/AM02N5IZ8l9ZilGNdocHXd0+oUuhEMCQp5DSanfD5w8gMZoh/1XF65RQyCScFEUAGPIUYs62ORGvi4I6SiZ0KWFLJpUgWa9CXQdDnhjyFGKqW+3ISNAJXUbYS4/ToobbAhIY8hRCAoEAqlscmMCQH7IJCVrUtjng83NbwEjHkKeQ0Wxzw+H2YkK8VuhSwl56nBYerx+nm239X0yiximFNOqutp+rs9uH6hY7xhhU0Cr5bTlUSoUMqUY1Tpy14NbsMUKXQwLivyYadRf3c+1t6pjonq4atuKHzYQEHU6cs/R/IYkaQ55Cgs8fQE2rHbNnpQldimiMj9fiL0fPobrJBrmsb8+sXqNAjCZKoMpoNDHkKSTUtDrg8fqRzpb8sDHFaeDx+fH653UYF9t3r+X5GXEM+QjBB68UEv55zoJUoxoqBcfHDxelXIaJiTqcabELXQoJiCFPgjta246io+fwtXExQpciOjmpBpzhePmIxu4aEkwgEMCe0nocN3dgw5JMJHApg2E3I82AXSfq4Pb6oJTzt6RIxJY8Cabd4cHhM+24d8EELJ6UIHQ5opSVrEeUXIrTTeyyiVQMeRJMdYsD8Tol0i57KEjDRyaVICtZj5MNVqFLIYEw5Ekw1S12ZCRwNM1ImzpGj/JGK5c4iFAMeRJEIBDAmRY716kZBZmJugvzEPgANhIx5EkQTTY3HB4fZ7iOgii5FBMTo9llE6EY8iSIM1ynZlRNHaPHqQYrAgF22UQahjwJguvUjK6s5GjYurpxnhuJRByGPI26i+vUcHOQ0aNRypGVrMfhM239X0yiwpCnUXe62cZ1agTw9Yw4lNZ1wtbV3f/FJBoMeRp1n9W0Y2yMhuvUjLLx8Vok6JQ4UtMudCk0ihjyNOr2lzcjO9UgdBkRRyKR4OsZcfisph0er1/ocmiUMORpVFU02lDb5kQOQ14QuWlG+AMBvH+qSehSaJRw/BqNqj0l9cgZa4BBrRC6lIikkEmxeFICnvp7Jc53uHD3PFOfbjNuJiI+DHkaMZfv5RoIBLDrRB3yZqQIWBUtnJiAueNj8ewHVdhX1ogHlk6E4sLOUdxMRHwY8jRiLt/Lta7DhXqLC7PTY1Df6RawMpqRZsSGJRPx+Nsnca7DiQnxHM4qVuyTp1FTet6CjEQdjGwphoQouRRpsRrUck0bUWPI06jwBwIorevEdD5wDSnj47VcuEzkGPI0Kqqb7XB6vBw6GWLGx2txtt0Jr59DKsWKIU+j4vjZDmSnGDgBKsSkxWjg9wP1XNNGtBjyNOJcHh9O1lsx08SNukNNlFyK1Bg1u2xEjCFPI660zgKdSo7xXKsmJI2P16KmjSEvVgMK+aKiIqxevRpr1qyB2Wzuc85sNmPNmjVYvXo1ioqKAADV1dXIz89HQUEB1q5di3Pnzg1/5RQ2Pjd3YOa4GEglEqFLoasYH6+Fuc3J7QFFqt+Qt1gsKC4uxs6dO/HQQw+hsLCwz/nCwkJs3rwZO3bsQHFxMSwWC2JiYvC73/0OO3fuxH333YcXXnhhxG6AQluTtQvnOlyYOY5dNaHKFKtBt8+Phk72y4tRvyFfUlKCuXPnQiaTIScnB7W1tX3O19bWIjs7G3K5HHPmzEFpaSliY2Oh1+sBAHK5HDIZH7ZFqsNn2jAxUYdYLcfGhyqlQoYUoxpnWthlI0b9hrzVag0GNoArtg/rfWwwGNDZ2Rk8drlc2L59O9atWzcctVKYsbu9OHHWgnkT4oQuhfoxKSka5Y02ocugEdBvyOv1ethsl/7nS6V939L72Gq1wmDoGQft9XqxceNG3HvvvcjIyBiueimM7CtrhFYpw6TkaKFLoX5MGaOHuc2BThc3FBGbfkM+NzcXR44cgc/nQ1lZGUwmU5/zJpMJZWVl8Pl8OHr0KKZPn45AIIAtW7Zg0aJFuOWWW0aseApdfn8Au0/U4cYJcXzgGgZSDCpEq+TcHlCE+l2gzGg0Ii8vDwUFBZDL5di2bRsOHDgAu92O5cuXY9OmTdiyZQu8Xi9WrlwJo9GIAwcOYN++faivr8c777yDrKwsbNmyZTTuh0LEx1WtaLa5MYtj48OCRCJB1hg9Pq1uw/cX8zdvMRnQKpT5+fnIz88PHvduzZtMJuzYsaPP9YsWLUJJSckwlUjh5ly7Ew/v/hLLpiVDE8WFTsPFlORovH78PDxeP6LknEIjFvw/ScPqdJMN337hU0wZE437b84UuhwahAkJOvj9AXxWwy4bMWEzi4bs4uYgXd0+5L94GLPTY/Eft06Gx8fJNeFEIZNiVnoM3vy8DuNiNH3Occeo8MWQpyG7uDlI6XkLurx+LJiYgM9qOjB1DEfVhJvZpli8cKAaplgNEvWq4OvcMSp8sbuGhk3J+U7kpBogk3I0TbhaNCkeU5L1+OOntbA4PUKXQ8OAIU/DwuXxobLJhuljjUKXQkMglUjwrZljMcagwh8/qYXL4xO6JBoihjwNi7L6TuiUcpjiNP1fTCFNJpVg9exxCAA4zIewYY8hT8Pin+ctyB1r4MQnkYiSS7FwYjwOV7fB6+OuUeGMIU9D1mp3o6bFgdw0dtWIyYw0I/wASs5bhC6FhoAhT0P2zpeNSIhWIrnXaAwKfwqZFDdOiMXHVa1XLExI4YMhT0PicHvx+vFzWDQxARJ21YjO3PFxaLN7cMzcIXQp9BUx5GlIXvnsLNQKObtqREqnlGOWKQb/80kt/Nw5Kiwx5Okr6+r24XcHzqBg7jiOjRexJVmJONvuwOvHuY1nOGLI06B0OD0wtzpgbnXgt/urIJEAiyYlCF0WjaBolQL3LpiAJ98pR4eDE6TCDUOeBsXq7MbBqlb817sVeP7Danw9I44P5SLAsuxkJEQr8ehfvwz+kO/gjNiwwLVraFDsbi9+f7AGzbYufOeGNOSkGoQuiUZBV7cPS7OS8LsD1Rgbo0FarIbr2YQJtuRpUP58qBZOjxc/WTqRAR9h0mI1mGWKxV9L6uDnb29hgyFPA2Zuc6D48zrcPn0MolUKocshAdw6NQkWZzc+q2kXuhQaIIY8DdiT75TjBlMMJiZyCeFIpVHKsWxaMv5+shHtfAgbFhjyNCBHatrx3skm/PAm7v8Z6WaaYpAYrcLT71fyoXsYYMhTv1weH/7zzVKsvdGE9Dit0OWQwKQSCb49ayyO1LSj+PM6ocuhfjDkqV9PvnMK/gDwH8smC10KhYh4nRI/vjkTW/9WhnPtTqHLoetgyNN1fVjRjJ2fncXTd86AJoojbumSFdPHYO74WPxgx3GcabELXQ5dA0OerqrD6cHnte3436+VYO2NJsSoFTC3OuDs5k5B1EMikaDwu7kwxWmw7NcH8dsPq7m+TQhi04yuqs3mxsbXSmDUKDAhQYdPqnt2COLm3HSRx+dHp7MbD92ahXnj4/DL9ypg6+rGfYsmcJJUCGFLnq7q+f1V6HR1I382Fx+jq3O4vfikug2fVLdBJpPi9pwxePHgGZTXW4UujXphyNMVXvnsLN7+shEFc03QKvnLHg1MTqoBGQk6/Oq9Cg6tDCEMeepj35eNePSvX+Jnt2UhNUYtdDkURiQSCe6YkYqKRhtePcJliUMFQ56CPq1uxQNFJ/B//mUabp6cKHQ5FIYMagV+snQitu4pw+dnuZtUKGDIE+osLvz09RKs+f1nuP/mTKy90SR0SRTGbp2WjHXzTPj+n4+jzuISupyIx5CPcPsrmnHzUx+irsOFN380Hw8snSh0SSQCm2+bguljDfje/xzBp9wIXFAM+Qh2tLYdP3z5ONbPT8f/XZkdHAvP8fA0FB6fH+fbnXjwG5MwKSka6/54BMueOYgPypuELi0icehEhCqr78Q9Lx3Ft2eNRVqsFp+e6bt0LMfD01flcHtxtMEGAJifGY+csQZ8VNGC7//5OP5z+RTcMxrmguwAAA1xSURBVD8dEgmH5Y4WtuQj0IcVzbjzd4ex8mupuHfBeKHLIZHTqxRYkZuCrSum4pm/V+JHOz9Hs61L6LIiBkM+gnR1+/D7g2dw75+O4f4lmfj5v0xji4pGzcKJCfjbhgVotbuxtPAjvHzYDB+XQRhx7K6JACXnLPjvA2fwQXkzFDIJHrl9Cm6anIizbU72vdOoGh+vxV++Pw9vHD+PJ945hT8fqsW/LZyA2emxfa7TaxRcGmGYMORFrLzRiqf/Xom/n2zCTZMS8N0bxiIjUQe5VMq1aEgwUqkE352dhlunJeOJvSfx0JtfwBSrweLJCchM0EEikXCT8GHEkBcZt9eHD04148+HzDhc04ZbpyZj378vglJ2KdiJQoFBo8CPbspEWqwWB0634OVDZsTrlJiYpIMEASQZVFApZEKXGfYY8iLQ7fPj0+o2vFVSj31ljQgEgNuyk7Hj5jkYG6MBAHbLUMiK1UYhb0YqlmYl4p/nLKhpdWDb3nL8174KrPxaKvJmpMB4oVXPbpzBG1DIFxUVYffu3ZDL5di2bRtMpkszIs1mM7Zs2QKv14u8vDysXr263/fQ0HS6ulFW34nKRhtONljx/qlmuDw+3DI1CYXfyUV6nAbHzBaY210wt/fMOGS3DIW6aJUCCycmYOHEBExM1KLo6Hm8/UUD/nzYjGkpeswaF9MT+GkKDhgYhH5D3mKxoLi4GEVFRTh58iQKCwuxffv24PnCwkJs3rwZWVlZyM/Px7JlywDguu+h/rk8PrQ53Gh3eHCmxYHyBitq2xwob7TB3OZElFyKCQlaTB1jwGN3TMOSrMTgzk3mVofA1RMNjUImxSxTDGaOM6Km1YFj5g68fNiMP35aC22UDGmxGoyN0WBsjBqJeiUSo1VIiFYiMVqJMQYVDGr+ILio35AvKSnB3LlzIZPJkJOTg9ra2j7na2trkZ2dDQCYM2cOSktLEQgErvuekdDu8MDj9SOAniFZF2dRXxygdXFa9dVmV1+6tp/3Xnb9pVeufI/XF4Cr2wunxwenxwfXhf86Pd6ej7t7XnO4vXB29/y32eZGu90Ni7MbXV4/AEACQK9WIEajQEK0CrlpRqzITUGSXoWFmfEwxXNjbRIviUSCCQk6TEjQYdXMsRgXq0ar3Y2Gzi40dHbhfIcTJecs6HB60Gr3wO72AgB0ShnGxmiQqFfBqFbAqFHAqFYgWqWAUiGFQiZFlEyKKHnPx0p5z8dSiQRSCSCTSiCRSCCTSiCTSCC58JpUIoFMigvX9bwOABJILtR7tXu4dC891172OiRQyCSI0ylH5O+w35C3Wq3Q6/XB48vXoOh9bDAY0NnZCQDXfc9wO9fuxE1PfRhyY25lUgnUChlUCilUCtmFj2VQK6RQR8mhkve8Hh0lR6JOiWkperQ7uqFVyqCNkkOrlEMTJUN2ih4nL8wgJIpUMqkEBrUCdZYuxGqViNUqMS3FAKCnO/Jkgw3dPj+srm60OTzQq+RwuL2wdnnRbO1CVbMddrcX3T4/un0BdHv9PR/7L33sDwD+QACBQAC+AOD3BzAaqSKRAO/++yJMShr+btV+Q16v16OysjJ4LJX2nT/V+9hqtSIrKwuBQOC677meyZMnD/ja3uQIzafI3Rf+MKJp0CZNAjatEbqKiCTB6OfJijdH5vP2ex+5ubn4zW9+A5/Ph/Ly8iseoJpMJpSVlSErKwtHjx7FvffeCwDXfc+1VFRUfIVbICKia5EEBtCX8uqrr+Kvf/1rcKSM2WyG3W7H8uXLrzm65vL3cHQNEdHoG1DIExFReOICZUREIsaQJyISMYY8EZGIMeSJiEQsFIeWh5ympibcf//9UCqV8Hq92Lp1K0wmEzZv3oyWlhZkZmZi69atg5oPEI6OHTuGgoICHDp0CGq1OqLuf8aMGcjJyQEA3H333ViwYEHE3H9paSmeeeYZeDweLF68GGvWrImYez916hSeeOIJAIDD4UAgEMArr7wSVvfP0TUD4PP5IJFIIJVKcejQIRQXF2PGjBnweDy455578Nhjj2HRokW46aabhC51RG3YsAH19fV48cUXsXfv3oi6/29+85t46623gsc7duyIiPv3eDz48Y9/jO3bt0OtVgOInHu/3EsvvYSuri7odLqwuv/Q/fETQmQyWfAntcPhwNSpU3Hs2DEsXrwYALB48WIcO3ZMyBJH3AcffIBZs2ZBo+lZujjS7r+hoQFr1qzBpk2b0N7eHjH3f+LECajVajzwwANYv349KisrI+beL7d3717cfvvtYXf/7K4ZoKqqKjz88MNoaGjAs88+i48//ji4Pk/vNXvEyO/349VXX8Vzzz2Hf/zjHwD6rmkk9vsHgPfffx8xMTHYu3cvnnzyyYi5/5aWFlRUVGDXrl1oaGjAI488ApVKFRH33tvZs2chlUqRlpYWdv/v2ZIfoMzMTBQVFeGFF17AL37xC+j1ethsPSvSWK1WGAwGgSscOXv27MGSJUugVF5aJS+S7h8AYmJiAAC33XYbysvLI+b+9Xo9Zs6cCY1Gg4yMDNhstoi5997eeustfPOb3wQQft/7DPkB8Hg8wY/1ej1UKhVmz56Njz76CABw4MAB3HDDDUKVN+IqKyvx7rvvYv369aioqMDGjRsj6v6dTid8vp6dtY4dO4bU1NSIuf/c3FzU1NTA5/Ohubk54r73L9q3bx9uu+02AAi7++eD1wE4duwYnnnmmeB60Js3b8aECRPC6gn7cFm7di1+/etfR9Tomi+//BKPPPIINBoNZDIZHnvsMSQlJUXM/RcXF6O4uBg+nw8PPvggpk2bFjH3DgDl5eX41a9+hT/84Q8AAJfLFVb3z5AnIhKx0P3xQ0REQ8aQJyISMYY8EZGIMeSJiESMIU9EJGIMeQoJkydPxnPPPRc83rx5Mw4cODDkz/vmm2/iqaeeGvLnuZ7Dhw/j9ttvx1133TUqX3v+/PnD/jlJvBjyFBJ0Oh127doFl8sldClBXq93QNft3bsXGzduxCuvvDLCFRENHkOeQoJKpcI3vvENvPHGG1ecW7JkCdxuN4C+reO1a9fil7/8JVauXIlVq1bhiy++wLp167B06VJ8+OGHwfefPXsWBQUFuPXWW/H2228DALq7u/HYY49h1apVyMvLC/7W8Oyzz+LRRx/F3XffjaeffrpPHe3t7bjvvvuwYsUKrF+/Hq2trdi9ezf27duHJ598Eo8//vgVtdfV1WHdunW45ZZbsGvXruDrzz33HFatWoUVK1YEXz979izuuusurFy5Et/+9rdRXl4OoGfG7Y9//GPcfvvt+PnPfx78HJWVlfjWt76FO+64A3l5ebBarYP+eyfx4wJlFDK+973v4e6770Z+fv6A36PX67Fr1y488cQT+MUvfoGXX34ZNTU12Lp1a3D511OnTmH37t1wu9347ne/i5tuugm7d+9Geno6Hn30UXR0dKCgoAALFy4EAFRXV+Oll16CQqHo87W2b9+OBQsWYN26ddi5cyeeeeYZPP744zh8+DCWL1+ORYsWXVFfVVUVXn/9ddhsNtx5551YuXIlPvroI9hsNhQXF8PtduPOO+/E4sWLkZCQgJdeeglRUVEoKyvDU089hd///vfYuXMnxo0bh+effx779+8P/sbwl7/8BQUFBVi1ahVcLtcV9RIBDHkKIUlJSZg1axb27t074PdcDPJJkyZBIpFAqVRi0qRJqK+vD16zYMECaLVaaLVaTJkyBadPn8bhw4dRVVWF4uJiAD2t5dbWVgDA0qVLrxqYJ06cwIYNGwAAd9xxB3bs2NFvffPmzYNKpYJKpYLf70d3dzcOHTqEf/zjHzh8+DAAwGaz4fz580hPT8fDDz+MyspKSKXSYMv8xIkT+MEPfgAAuPnmm4Prus+YMQO//e1vYbFYsGzZMqSmpg74740iB0OeQsr69euxadMmZGVlBV+TSqXw+/0A+i4WByAYxlKptM/HFxcUAxBcc6j3cSAQwLZt2zBz5swrargYopcLBAJXfK7+REVFXXEfgUAAP/nJT7BixYo+127fvh0mkwlPPfUUnE4nvvGNbwS/7tWsWLECOTk52L9/P9auXYsXX3wRGRkZg6qPxI998hRSMjIykJqaikOHDgVfS0lJwalTp+D3+4Or/w3GwYMH4XA40N7ejlOnTiEzMxPz5s3Dq6++GvzhcbH/+3pmzpwZ7NPfs2cPZs2aNehaAODrX/863njjjeAPrNOnT8Pn88FutyMxMRESiaRP//3MmTPxzjvvAAD2798ffDh97tw5mEwm/Ou//ituuOEG1NTUfKV6SNzYkqeQc9999+HOO+8MHv/whz/E5s2bkZSUhPHjxw/6802fPh33338/6uvrsXHjRmi1WqxevRrnzp3DHXfcAb/fjylTpvQ73HHDhg146KGH8NprryEhIQG//OUvB10L0LObUEVFBVatWoVAIICEhAS8+OKLuOuuu7Bhwwbs2rUr+HwAAAoKCvDTn/4UeXl5mDdvHuLj4wH0jOr529/+BrlcDpPJ1Oc9RBdxFUoiIhFjdw0RkYgx5ImIRIwhT0QkYgx5IiIRY8gTEYkYQ56ISMQY8kREIsaQJyISsf8P5FiiVCZLFV8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rng = np.random.RandomState(205) # Seed the random number generator\n",
    "\n",
    "x = rng.binomial(100, 0.5, 100000) # Simulate 100,000 experiments of tossing a fair coin 100 times\n",
    "\n",
    "sns.distplot(x, kde=True, rug=False)\n",
    "plt.axvline(54, color = 'r'); # What we observed\n",
    "plt.xlabel('Number of heads');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "name": "04-python-stat-5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    100000.000000\n",
       "mean         49.995590\n",
       "std           5.011938\n",
       "min          27.000000\n",
       "25%          47.000000\n",
       "50%          50.000000\n",
       "75%          53.000000\n",
       "max          72.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We convert to pd.Series to take advantage of the `describe` function.\n",
    "pd.Series(x).describe() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we see from the histogram and the description of the data above is the patterns in data we would expect if we repeated this random experiment. We can already make some observations. First, we do see that the average number of heads we expect to get is 50, which validates that our experiment is using a fair coin. Second, we can reasonably get as few as 27 heads and as many as 72 heads even with a fair coin. In fact, we could look at what values we would expect to see 95% of the time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "name": "04-python-stat-6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([40., 60.])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.quantile(x, [0.025, 0.975])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This says that 95% of the time we'll see values between 40 and 60. (This is **not** a confidence interval. This is the actual results of a simulation study. A confidence interval would be computed based on a **single** experiment, assuming a binomial distribution. We'll come to that later). \n",
    "\n",
    "So how likely would we be to see the 54 heads in 100 tosses assuming a fair coin? This can be computed as the proportion of experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "name": "04-python-stat-7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18456"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(x > 54) # convince yourself of this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is what would be considered the *p-value* for the test that the coin is fair. \n",
    "\n",
    "> The p-value of a statistical hypothesis test is the likelihood that we would see an outcome at least as extreme as we observed under the assumption that the null hypothesis (H<sub>0</sub>) that we chose is actually true. \n",
    ">\n",
    "> In our case, that null hypothesis is that the coin we're tossing is fair. The p-value **only** gives evidence against the null hypothesis, but does **not** give evidence for the null hypothesis. In other words, if the p-value is small (smaller than some threshold we deem reasonable), then we can claim evidence against the null hypothesis, but if the p-value is large, we cannot say the null hypothesis is true. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we increase the number of tosses, and we look at the proportion of heads. We observe 54% heads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "name": "04-python-stat-8"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEHCAYAAAC3Ph1GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de3RU5aE28GcuSSaTZCY3QkJuQALhHoEgLVpS1CqiiEUsguTod5TTLk+R+lkL3miXFm3rsQqlHu3Rrxc8QKt4qRdABCUaFRyERG4JSWCSkJDLZDIzmUkymZn3+yNkSrjkRjJ79p7ntxaLZDIz+2ERnmzevd/3VQkhBIiISHbUUgcgIqLBYYETEckUC5yISKZY4EREMsUCJyKSKRY4EZFMaQN5sJycnEAejohIMUpLSy96LKAFfrkQRESKVVsLpKYCZ84Ao0YN6i0ud/LLIRQiIpligRMRyRQLnIhIpljgREQyxQInIpIpFjgRkUyxwImIZIoFTkQkUwGfyEOhw+pyw+7q9H9u0IchTh8uYSIiZWGB07CxuzpRVGHxf35NVgILnGgIcQiFiEimWOBERDLFAicikikWOBGRTLHAiYhkigVORCRTLHAiIpligRMRyRQLnIhIpljgREQyxQInIpIpFjgRkUyxwImIZKpfqxE+9dRTOHLkCHw+Hx588EHMmjULa9euRWNjI7Kzs/GrX/0KajV/FhARBVKfrXv69GmUl5fjH//4B1555RVs2LAB27dvR25uLrZs2QKtVovCwsJAZCWZKa5uwT+Lz2DrgSq0uNxSxyFSnD4LPDExEXq9Hh6PBw6HA3FxcTCZTMjPzwcA5Ofnw2QyDXtQkpfi6hb87B+H4Wj3oM3txaZPynGoytrjOVaXG+Ymp/+XlSVPNCB9DqFERUUhOTkZ8+fPh8vlwsaNG/HSSy/BYDAAAIxGI2w227AHJfkQQmD9h8dx46RkzB0/Aj4h8PGxejz+zhFMSYtFdlI0AG74QHSl+jwDLyoqgs1mw0cffYR33nkHzzzzDAwGAxwOBwDAbrfDaDQOe1CSj4+PN6CkpgX3XTsaAKBWqfCDSSMxJysB/7HZBHt7Z+9vQET90meB+3w+GI1GqNVqREdHw+VyYdasWdi3bx8AoLCwEHl5ecMelORj0yfluHfOGCTF6PyPqVQqPHJjDsI1avxu5wkJ0xEpR58Ffs0118DpdGL58uUoKCjAAw88gMWLF+Pw4cNYvnw53G435s6dG4isJANVFheKq1uwdFb6RV+LCNNg/Q+nYuuBahRXt0iQjkhZ+hwD12g0eO655y56fMOGDcMSiOTt/W9rMSXVgDGJUTA3OS/6+szMONw5Mw1PvHMEG++6SoKERMrBm7dpSL1fXIdbp43q9Tm/mD8BlY2t+PxkU4BSESkTC5yGTGVjK47V2XHL1JRenxcfFY6C747G374yQwgRoHREysMCpyHzfkkdrkqPRXq8vs/n3v+9MahqdqG03hGAZETKxAKnIfNecS1undb72Xe3xOgI3JY7Cp+caOBZONEgscBpSJSedeBkQytu6WeBA8Bds9JRa2tHRePFFzuJqG8scBoS75fUYtboOKQYI/v9msToCORlxuGT0oZhTEakXCxwumJCCLxfUoeFub3ffeL2+nqsfeLq9GLu+BEwW5w4dYlbDomod/1aTpaoN8fq7DhtcWL+lORen+fs8ODrun9dtJyUEoM4fTimp8ehsKwRK2ZnDHdUIkXhGThdsZ1HzmLW6PgeU+cH4tpxiSird+A0z8KJBoQFTlds55GzuLmPs+/ejDToMH5kDP5uqh7CVETKxwKnK1Le0IqTDa24afLgCxzoOgvffbweDY72IUpGpHwscLoiu46eRW56LEbF9v/uk0sZmxiFtNhIvHPozBAlI1I+FjhdkZ1HzmL+ubPvC3fYcXV6+/0+KpUK86ck4w1TDSf2EPUTC5wGrbrZhW/P2Px3n3TvsNP9q2MABQ4AP5g0EpVNTpTUcIcnov5ggdOg7Tp6FhOSYzAmMWpI3i8hKgLfHz8CbxzkxUyi/mCBU79dOETyfkndFV+8vNAdM9PwXnEdOr2+IX1fIiXiRB7qt/M3Iba3d6K4ugW/uWPqkB7j+zkj0N7pxdenmjEnO3FI35tIaXgGToNyvM6OUbGRyBkZM6Tvqw/X4nvjRuCjY/VD+r5ESsQCp0E5XmfH97IToVKphvy9b5w8EruP1fNuFKI+sMBpwDq9PlQ2OjF7bPywvP/1E5JQZ2vDV5WWHmPuVpd7WI5HJFccA6cBq2x0Qq1WISc5psfGxQO577s3CdERyMuMx84jZ5GTbPA/fk1WAuL04UNyDCIlYIHTgJXVO5A9Ihpujw+Hqy3+xyelXNl4ePdyswAwIyMWe0sbehQ4EfXEAqcBK6t3YO64EUP+vucvNxuu1eBkfStaOzyIjuC3KdGlcAycBqSptQMWpxvjk4f27pMLjTREIC4qHOUNrcN6HCI5Y4HTgJxsaEVSTASMkWHDehyVSoXp6bE4yV3riS6LBU4DUtHQiuyk6IAca0ZGLMoaWuHj7YREl8QCp37z+HyobGpF9ojAFHhuWixcHR7U2bhGONGlsMCp307Wt8Lt8Q3Z4lV9idZpkR6v5zAK0WWwwKnfTGYr0uP0iAjTBOyY40ZGo6yeFzKJLoUFTv120GxFVoDGv7uNT4pBVbMT7UM0SYhISVjg1C/tnV4cqbUhK0Dj391S4yKhC9OgopFn4UQXYoFTvxyqaoFKpUJ63JXtfTlQapUK2UkcRiG6FBY49cv+UxZMTjFAqwn8t8z4kTEoq3dwdUKiC7DAqV/2VzYjN80oybGzk6Jha+tELW8nJOqBBU59cnt8+KbKitz0WEmOb9CFISEqHMXVLZIcnyhYscCpTyU1LfAJgckp0q0MODoxCsXcrZ6oBxY49Wn/qWbkpsUG9P7vC41OiMK3Z3gGTnQ+Fjj1af+p5mHbfae/xiRG4UxLO85yHJzIjwVOvfL5BA6ZrcgbLW2Bx+nDkBgdjgOnmyXNQRRM+lXgJSUl+Pd//3esWLEC//M//4O2tjasXr0ay5cvx7p16+Dz+YY7J0nkZEMrHB0ezEiPkzSHSqXCtLRYHDhl6fvJRCGizwJ3u934wx/+gD/+8Y94/fXXsXLlSmzfvh25ubnYsmULtFotCgsLA5GVJGAyN2NcUjSM+uFd/7s/pqUacdDMcXCibn0W+KFDhxAZGYkHH3wQ9913H8rKymAymZCfnw8AyM/Ph8lkGvagJI2DZitmZkp79t1tQnIMTtY7uC4K0Tl9bjbY2NiI0tJSvP3226irq8OTTz4JnU4Hg6HrljKj0Qibjbd3KdU3Ziv+c1621DEAAGNHdC1je6zOjhkZwfFDhUhKfZ6BGwwGzJgxA3q9HllZWXA4HDAYDHA4utZottvtMBqlmaFHw6uptQOnLa6gOQOP0GqQkxyDb3k/OBGAfhR4bm4uTp06Ba/Xi4aGBuh0OsyaNQv79u0DABQWFiIvL2/Yg1LgHTRbEacPC9gGDv0xLc2IEhY4EYB+DKEYjUbceeedKCgogNfrxZo1azB58mSsXbsWy5cvR3Z2NubOnRuIrBRg35wb/1apVFJH8ZuaGou/fHFK6hhEQaHPAgeAO+64A3fccUePxzZs2DAsgSh4HDRbcf3EkVLH6GFamhHlDa1wdngQFdGvb18ixeJEHrqkDo8XJWdsQTP+3W38yBhoNWocq7NLHYVIcixwuqQjZ+wQQmCaREvIXk64Vo2JKQauTEgEFjhdxjdmKyaPMkIn4QJWF3J7fTA3OZEZr8eBU82wutxSRyKSFAucLimYJvB0c3Z4UFRhgUalwuHqFthdnVJHIpIUC5wuIoTAwargK/BuKbE6NDo6OCOTQh4v4xMAwOpy+89oa1va0OjowCijDuYmp/85riApzJEGHVQqoLLJiRwJN5kgkhoLnAAAdlcniiq6Vvo7VNU1gQeA/zEAmJQSI0m2C4Vp1EiK0eFkvQM3T02ROg6RZDiEQhepanYhI14vdYxejYqNRFlDq9QxiCTFAqeLmC0uZCQEz/T5SxkVq8PJehY4hTYWOPXQ3ulFvb0dmcF+Bm6MRGVTKzq93EyEQhcLnHqotroQplVjpEEndZRepRh18HgFyjmMQiGMFzGphyqLC+lxkdCog2cBq0uJCNNgVGwk9pU2QH9uspFBH4Y4fbjEyYgChwVOPZibXciID+7x725jEvXYd7IJhsiu0r4mK4EFTiGFQyjk5xMCNdbgvwOl29jEKNS2tEkdg0gyLHDys7S60d7pQ1pcpNRR+mXsiGjU2drhE0LqKESSYIGTX43VhfiocNmssz02MQpujw/NrVzUikITC5z8qq1tsjn7BgBDZBiMkWGotXEYhUITC5z8aqwupMXJY/y72yijjuPgFLJY4AQAcHt8qLO1I11GZ+BA15T6Wlu71DGIJMECJwBARWMrhBBIMcqvwM9Y2yB4IZNCEAucAAAnzjow0qBDuFZe3xLp8Xq0dXph4YVMCkHy+tdKw+bEWbusLmB2i47QIj4qHFVWl9RRiAKOBU4AgON1DtldwOyWHheJ6mYWOIUeFjjB3t6JqmaXLM/Aga5hFBY4hSIWOOFIjQ0R2q5dbuQoI16Ps/Z2tLmDY8s3okBhgROKa2wYNzIm6FcgvJxkow5qlQpl9Q6poxAFFAucUFzdgonJwbHf5WBo1WqMio3E0Tq71FGIAooFTiiuacFEme/unhGvx5EzNqljEAUUCzzE1dvbUWdrl/UZOACMTtDjSK2NE3oopLDAQ1xxdQvio8KRbJTnBcxuGQlRsLV5UNHolDoKUcCwwENcSY0N09KMUKnkeQGzW3SEFhnxkTCdbpY6ClHAsMBDXHFNC3LTYqWOMSSmphrx9Wmr1DGIAoYFHsJ8PoHi6hZcla6cAjeZeQZOoYMFHsJOW5ywt3swLc0odZQhMTXVCLPFhQY7l5el0MACD2HFNS1Ij49EQnSE1FGGRGpsJBKjI2AycxiFQgMLPIQVV9swTSHj3wCgUqkwa3QcvuaFTAoRLPAQVlzTgqsUVOAAkDc6HiZeyKQQwQIPUW6PD0dr7YoZ/+42a3Qcjtba0NrhkToK0bBjgYeoI7U2+HxCUUMoADApxQBdmAaHq1qkjkI07FjgIerzk00YlxSNBns7zE1OuDqVsRSrVqPG9IxYjoNTSOh3gZtMJuTk5KC5uRltbW1YvXo1li9fjnXr1sHn8w1nRhoGptPNiI8KR1GFBUUVFnQopMABIC8zHgd5JwqFgH4X+F//+ldMmTIFALB9+3bk5uZiy5Yt0Gq1KCwsHLaANPSEEDhSa0dGQpTUUYaU2+uDucmJjPhIHDQ3o7GV94OTsvWrwPfu3YuZM2dCr+/aM9FkMiE/Px8AkJ+fD5PJNHwJachVN7eh2elGZrw898C8HGeHB0UVFtjbPOjw+HDoNMfBSdn6LHCfz4etW7di2bJl/sfsdjsMhq71o41GI2w2rsMsJwermpFs0MEQGSZ1lGEREaZBijES33J9cFK4Pgv8vffew3XXXYeIiH/N1jMYDHA4uravstvtMBqVdSua0h00WzElVd4bOPQlM0GPEhY4KVyfBV5WVoZdu3bhvvvuQ2lpKR566CHMmjUL+/btAwAUFhYiLy9v2IPS0DGdtmLKKGX/0M1MiMK3Z7jBAymbtq8nPPLII/6PCwoK8MILLyAyMhJr167F8uXLkZ2djblz5w5rSLpyVpcbdlcnnB0elJ514KfXZcPeptzJLpkJejQ73ahqdiFTYRdribr1WeDn27x5s//jDRs2DHkYGj52VyeKKiw4We9AmFaNFIMO9rZWqWMNG4MuDKOMOnx92soCJ8XiRJ4QY252ISNOD41a3jvw9MfUVCO+PsUJPaRcLPAQU2VxISNBWbcPXs7UNG7wQMrGAg8hPiFQZXUp7v7vy5k6yoiKRiesTrfUUYiGBQs8hNTb29Hp8SE9RAo8I0EPY2QYp9WTYrHAQ4jZ4sJIgw66MI3UUQJCrVIhLzMOX3MYhRSKBR5CqppDZ/y728zRcTjIDR5IoVjgIcRscYbM+He3vMx4lJyxocOjnNUWibqxwENEU2sHrK7OkLsnelqasWv1RU6rJwVigYeII2dsiI7QIk6vzAWsLkcXpum6H5zDKKRALPAQcaTWjswEPVQq5U/guRA3OialYoGHiKO1NmSE2Ph3t5mZcThobubCVqQ4LPAQ0N7pRVl9a8hdwOzeoWdkdASsrk4cquYGD6QsA1rMiuSppMYGtQoYFRspdZSAcnZ48HVd17r1CVHh+KrCghkZcRKnIho6PAMPASZzM3KSY6DVhO5f9+hz64MTKUno/osOId+Ylb+BQ18yE/S8lZAUhwWucEKIc1uohXaBZyToUW1tQ1Nrh9RRiIYMC1zhKpucsLo6MWWUsvfA7MuI6AjERoZxfXBSFBa4wh00WzEmMQqx+nCpo0hKpVIhN82I/SxwUhDehaJwB09beefFOVNSjdh59CzMTU4AgEEfhrgQ/8FG8sYzcIUzmZsxM5MFDgDjkqJR2ejE7uP1KKqwwO7qlDoS0RVhgStYU2sHKhqduHpMvNRRgkJGgh66MA3MFpfUUYiGBAtcwUynm5EYHY6sEaG1AuHlqFUqjE6MwqlzQyhEcscCV7D9p5px9Zj4kFzA6nLGsMBJQVjgCra/shlXj+bwyfnGJEahtqUN7Z3c4IHkjwWuULa2Thw/a8fVYxKkjhJUUow6RISpYbbwLJzkjwWuUAfNzYiJ0CInOUbqKEFFrVIhM57DKKQMLHCF2l/ZjFmj46FRc/z7QhwHJ6VggSvUp2WNmJgSA3OTE+YmJ1wc8/UbkxiFMy1tcLk9UkchuiIscAVqau1A6VkHNGo1iiosKKqwoIMF7jcqNhJajRpHa+1SRyG6IixwBSoqb0JidDiSYiKkjhKUNGoVMuP1OMwdekjmWOAKVFjWhLxM3v/dm6wR0fimihsdk7yxwBVGCIHPTjZi1miuf9KbrKRonDjrgI3roZCMscAV5mRDKxocHZjJFQh7lWLUIUanxZeVFqmjEA0aC1xhPjnRgKmpRsRFcZnU3qhVKszIiMPn5Y1SRyEaNBa4wuw50YB5E5KkjiELMzPjUFTOM3CSLxa4gthcnThotuJ6Fni/5GXG4VSTEzVWLi9L8sQCV5DCk42I04djaohvYNxfKcZIjEmMwqelHEYheWKBK8jeEw34fs4IqDl9vt+un5CEj4/XSx2DaFBY4ArR1NqBvSfqMS3VyKnzA/CDSSPxRbkFzg5Oqyf56bPAKyoqsGzZMtx9990oKChAdXU12trasHr1aixfvhzr1q2Dz+cLRFbqxf5KC1rbvfD4BKfOD8DMzDjoIzT47CSHUUh++izwuLg4vPLKK/jf//1frFy5Ei+//DK2b9+O3NxcbNmyBVqtFoWFhYHISr34ssKCzMSuPR+p/7QaNa6bkITdxxqkjkI0YH0WeHx8PAwGAwBAq9VCo9HAZDIhPz8fAJCfnw+TyTS8KalPX1VaMCHZIHUMWXF7fTA3OXFVWiw+Pn4Wja3tUkciGpB+j4G3tbVh48aNuOeee2C32/2lbjQaYbPZhi0g9a3O1obyRicmjOTmDQPh7PCgqMICnwBcbi8+Pc5hFJKXfhW4x+PBQw89hPvvvx9ZWVkwGAxwOBwAALvdDqORt61J6ZMTjUiNjUQiVx8clHCtGhNTDNhzgnejkLz0WeBCCDz++OOYO3cubrjhBgDArFmzsG/fPgBAYWEh8vLyhjcl9WrP8Xp8dyw3L74SuWmxKDzZxM2OSVb6LPDPPvsMO3fuxI4dO1BQUID169dj8eLFOHz4MJYvXw632425c+cGIitdQmuHB5+VN+F740ZIHUXWxo2MhkatwicneDGT5EPb1xPmzp2L4uLiix7fsGHDsASigfm0tAExEVpMTTXiq1PNUseRLa1ajfzxI/DP4lrcPDVF6jhE/cKJPDJldblhbnLirYM1+M7YBHR4eS/+lfr++BHYc7weR8/YYG5ywupySx2JqFcscJmyuzrxaVkjPq+wIDE6ghN3hsCYxCjowjT4c9FpFFVYYOdmDxTkWOAyVtHQChWArBFRUkdRBI1ahWlpsSiu4V6ZJA8scBkrrmnBxBQDtBr+NQ6VaWlGVDS2opVro5AM8F++TLncHhyrs2N6eqzUURQlNTYScfpwfHuGk9Mo+LHAZerz8ibotBqMHREtdRRFUalUyE2PRXE1h1Eo+LHAZWr3sXpMSzNCw7W/h9y0NCOqml04a+PaKBTcWOAyVG9vh8lsxVXp3Hl+OCTF6DDKqMNeTuqhIMcCl6Et+6uQNSIao2J1UkdRrGlpsfiYa6NQkGOBy4zb48OWA1X44fRUqFQcPhku09KMqGx04sRZu9RRiC6LBS4zu46eRafXhxu48/ywitWHY0ZmHP7xdY3UUYguiwUuI0II/PWL0/hRXjoiuPPOsLt1agreOlSDDg9nuVJwYoHLyJeVFhTXtOCeOaOljhISrs1OBNB1xw9RMGKBy8iGj0/izrx0pMZGSh0lJIRr1fjh9FRsO1AtdRSiS2KBy8RXlRZ8U2XFA9/PkjpKSLl7dia+qGjCsVpezKTgwwKXASEEnv3wOOZPTobXK2BucsLF1QcDIjspGjdOSsZ/76uQOgrRRVjgMrDzyFmU1TswOdWIogoLiiosXD42gB6Yl4UPSmpxuskpdRSiHljgQc7t8eG3O09gaV46DLowqeOEFLfXB3OTE0ZdGGZkxOH53aVSRyLqgQUe5F7/ygyn24uls9KljhJynB0e//94rh4Tjw9L6nCEqxRSEGGBBymry41vq1vwwsdl+D9zRgOcdSmptDg9rp84Eus/OA4hhNRxiACwwIOW3dWJ3+w8gahwLQyRYRzzDgL3zMnEN1VWbN1fxT0zKSiwwINUdbMLX1ZasGBqCtQ8+w4KUeFazMlKxPO7y7DvZCP3zCTJscCD1MuFlRg/MgbZSdywIZjMHZ8It8eHA6eapY5CxAIPRl9WWPBlpQU3T0mROgpdIEKrwQ8mjcSe4w1o4RAKSYwFHmR8PoFff3AMi3JHYURMhNRx6BJmZMYhMTocmz4plzoKhTgWeJB569AZVDe7cC8XrApaapUKP5yRhk9KG/FJKXftIemwwIOIy+3Bc7tO4MHrx8EYyUk7wSzZoMPdszOwdnsJmp0cSiFpsMCDyMY95dCFaVDw3Uypo1A//Nt3MpEaG4mfv1HMe8NJEizwIHGoyopXP6vEbxZPQ4SWmzXIgVajxsZl02E63YzXPj8ldRwKQSzwINDm9uL//qMYC3NHYZRRx9UGZcLt9cHrFfjFTTn4zY4T+Pxko9SRKMSwwCUmhMAjbxbD6xPITYvlaoMy0r1WilqtRt7oeDzyZglsnNxDAcQCl9hLn1agsKwRz/5wCsK1/OuQq5unJMMYGYb/3PINOr0+qeNQiGBjSMTqcmPjx2V4YXcZ1t06CfHRvOdbzsI0ajxz+xRUNLbil/88youaFBAscIls+cqMF/ecxI/y0uH2Cg6ZKEBMZBieXjQZ/zxci59tOwyLs0PqSKRwLPAAE0Lg9x+V+st7SqpR6kg0RJwdHjQ43Lh3zmjsPl6PR94ogbPDI3UsUjAWeADVtrTh3/7fAfz1SzP+a0kupqXFSh2JhsGo2Ej8eG4WTjW2YtEfi/BtDTeBoOHBAg8Ae3snfr+7DD/4/T4AwM6ffQ9XpbO8lWxETAT+++6ZmDU6Dre/VIQn3zmCBnu71LFIYbRSB1CyysZW/O1LM94wVSPJoMMjN+Xg++NHwN3p433eIUCjUeEnc7NwTVYi/vhJOa797V78cEYaHrx+HFJjI6WORwrAAh9i9vZO7DlejzdMNfiywoIZGbF4bMFE5KbHoqTGhi8qu9aRnpQSI3FSGm7ODg++rnMAAFZ8JxPlja04aLYi/3ef4JZpKVjxnUzkZcZBxQ07aJAGXeDbtm3DO++8A61Wi/Xr1yMzMzTX7+jwePGNuQVfVDShqLwJxTU2JESF49rsRKy6fhySDToAgIf3Boc0lUqFcUkxWH51Bk7U2fH2oTNY9qevMDoxCsuuzsC8nBEYkxjFMqcBGVSBt7S0YPv27di2bRuOHTuG559/Hhs3bhzqbEHH6xOobnbhSK0NR87Y8e2ZFhw0WyEEcFV6LPIy4/Af3xuLnOQYtHt8OFTVInVkCjLODg8cHV7cMCkZs8cmoN7ejte/MuPp94/BGBmG7KRoTB5lwLiRMcgZGYOxI6KQEBXOYqdLGlSBFxcXY/bs2dBoNJg6dSpOnz49xLF68nh9aGztgBCAQNeteN3zJLoe6/rcJ8S5rwM495g47zker0B7pxftnT50eLp+b+/0ov28jzs6vWj3nHv83HMbHO2osbbhrK0dHp9AskGH8cnRGDciGktmpGHyKAO8AjhU1YJmVye+rGzmEAn1KUYXhtlj4pGdFAN7eydqrW0I06hQZW3DFxUWVFmccHsFwrVqjDLqkGzUIU4fDoMuDIZILQy6MBj1YTDowhAdoYVWo4JWrYZGrUKYRgWNuufnKpUKalXXeuaqc78DgFrd9bg+XMtljGVmUAVut9thMBj8nw/3rLP//rQCz+8uG5L30qpViNCqERGmQbhGjQitGuFaNXRhaoRr1AjXahChVUMXrkG4RoUIrQa5qbGYPzkZSTERSI2NRKw+HK5OLw5VtaCt0weTuYWFTVfEoAuDISUMk1JicKzOgXk5Xf/jG52gh9XlRoOjA42ODrR2eOBo70Sjo73r4w4PWts9cLq98Pp88Ppw7ncBj0/AN4B/mjE6LQ6vuxEaNc/25WJQBW4wGFBW9q9CVav7fzdiTk7OYA6JoZxo7jn3yzmE70kUTFQABnou7QYwaeswhCFg/Hhg3rwhf9tBFXhubi5eeukleL1enDhxot8XMEtLSwdzOCIiuoRBFXhsbCxuv/123H333f67UIiIKLBUgsumERHJEqfSExHJFAuciEimWOBERDLFAicikinFFPi2bdtw1113YcWKFTCbzRd93ev1Yv78+Xjttdf8jz311FP40Y9+hCVLlqCwsDCQcS8y0Pw+nw+PPqmDFtEAAAn6SURBVPooVqxYgZ/85CewWq2BjtxDb/nXrl2LxYsXo6CgAOvWrevXawJtoPndbjfuuusu5OXlYefOnVJE7mGg+SsqKrBs2TLcfffdKCgoQHV1tRSx/Qaav76+HnfeeSdWrFiBu+66CydOnJAitt9gvv8BwGQyIScnB83NzYM7sFAAq9UqlixZIjwejygpKRGrVq266Dl///vfxcqVK8Wrr74qhBDi1KlToqCgQAghRFNTk1i8eHFAM59vMPl37twpnnnmGSGEELt37xa/+93vApr5fH3lX7NmjSgpKRnQawJpMPm9Xq+or68XGzduFDt27Ahk3IsMJr/FYhE2m00IIcS+ffvEY489FrC8FxpMfo/HI7xerxBCiC+++EI8/PDDAct7ocHk7/bTn/5ULF68WFgslkEdWxFn4H2tzdLR0YG9e/fipptu8j+WmJgIvV4Pj8cDh8OBuLi4AKf+l8HkN5vNmDx5MgBgypQpMJlMgYzcQ3/WxnnqqadQUFCAzz//vN+vCZTB5Fer1UhKSgpw0ksbTP74+Hj/chharRYajSaQkXsYTH6NRuOfAe50OjFp0qRARu5hMPkBYO/evZg5cyb0ev2gj62I9cD7Wptl8+bNWLp0aY9hhqioKCQnJ2P+/PlwuVySrqY4mPzjx4/Hjh07cNttt+Gzzz6DzSbdtl195V+zZg3i4uJgsVhw7733Ijc3N+Dr6fRmMPljYoJn7Zsryd/W1oaNGzdKOhlvsPnLy8vxxBNPoK6uDn/4wx8CHdtvMPmjoqKwdetWbNq0CXv27Bn0sRVxBm4wGOBwOPyfn782i8PhwIEDBzDvgnUIioqKYLPZ8NFHH+Gdd97BM888E7C8FxpM/vz8fCQlJaGgoAA1NTUYOXJkwPJeqLf8APz/u0lISMCUKVNw6tSpPl8TSIPJH0wGm9/j8eChhx7C/fffj6ysrMAFvsBg82dnZ2Pbtm14+eWX8fTTTwcu8AUGk/+9997Dddddh4iIK1vlSREFnpubiwMHDsDr9eLo0aM91maprKyE1WrFfffdhz//+c944403UFRUBJ/PB6PRCLVajejoaLhcLlnlV6lUePjhh7F582ZkZWXhhhtuCMr8APzf3B0dHTh27BhSU1P7fE0gDSZ/MBlMfiEEHn/8ccydO1fS7x1gcPndbrf/6waDATqdLqCZzzeY/GVlZdi1axfuu+8+lJaW4qGHHhrUsRUzlX7r1q149913/WuzmM1mtLa2YsGCBf7nvPXWW/4y9Hq9WLt2Lc6cOYOOjg7cc889uO2222STv7m5GatXr4ZarUZ2djbWrl2LsDDp1nLuLf/KlSvhcDjg8Xhw1113YcmSJZd8jZQlPpj8q1atwrFjx6DX63HNNddg7dq1sslfWFiIVatWYdq0aQCACRMm4PHHH5dNfpPJhBdffNG/0cXatWv914TkkP98BQUF2LBhA+Lj4wd8XMUUOBFRqFHEEAoRUShigRMRyRQLnIhIpljgREQyxQInIpIpFjgNqcmTJ2PRokVYsGABHn/8cfh8voAc126344033vB/vmfPHvztb38btuOtX78et956K1599dUejxcUFKCiomJIj1VYWCjpLYoUvBQxlZ6CR2xsLN599114vV7ce++92L17d481XDweD7Taof2283g8/gK/8847AQDXX3/9kB7jQjt27MBnn33mvw+ZSAoscBoWGo0Gubm5qKqqwltvvYWioiK0tLQgMTERDzzwAB599FE4HA6MHTsWzz77LPR6Pa677josWLAAn376KVJSUvDCCy8gOjoa+/btw/PPPw+fz4ebbroJq1atQk1NDVatWoUJEybg6NGjmDBhAsrKyrBo0SIsXLgQ8fHxqKysxM9//nOUlJTgV7/6FTo7OzFjxgysW7cOGo0G11xzDRYsWIAvvvgCGRkZ2LRp00WLOl3q2A8++CCsVituv/12rFmzBnPmzOnxmrfeegtffvklhBB46aWXkJKSgoaGBqxbtw4NDQ3Q6/V49tlnkZ6ejq1bt+LNN9+E2+3GhAkT8Oyzz0Kr1eLQoUN44oknEBERgenTp/vf+y9/+Qu2bduG8PBwzJ49W9LJNxQEBrWGIdFlzJkzRwghRFtbm1iyZInYu3ev2L59u7jxxhtFa2urEEKI+++/X+zevVsIIcRvf/tb8dJLLwkhhJg3b57YsmWLEEKIDRs2iE2bNom2tjYxb948UVdXJ9xut1i6dKkwmUyiurpaTJw4UZSVlQkhhKiurhZ33nmnP8f27dvFc889J4QQ4pZbbhFHjx4VQgixevVq8e677wohhBg/frz4+uuvhRBC/PjHPxaff/55jz/L5Y59/p/zQitWrBCbNm0SQgjx2muvid///vdCCCF+9rOfiWPHjgkhhDhw4IB/yVGr1ep/7dNPPy0++OADIYQQt956qzh+/Ljw+Xxi1apVYs2aNUIIIWbPni3a2tqEEELY7fa+/jpI4TgGTkOqpaUFixYtwtKlS/Hd737XvwjXtddei6ioKADA8ePH/etv3HbbbTh48KD/9d3DLTfddBO++eYbnDp1CllZWUhOTkZYWBgWLFjgf35WVhbGjRvXax673Q6fz+dfbnThwoX+1xsMBuTl5QEAJk6ciNra2h6v7e3Yvekevpk4cSLOnDkDANi/fz/Wrl2LRYsW4de//jUaGhoAACdOnMCyZcuwcOFCfPzxxygvL/dnnjBhAlQqFW6++Wb/e0+ePBm/+MUv8OGHHyI8PLzPLKRsHEKhIdU9Bn6hyMhI/8f9HTdWqVQQQvR4vjhv5Yfz3/Nyenv9+QWoVqvh8Xj6/dredK9Jo1ar4fV6/R+//fbbF61U9+STT+KVV17B2LFjsXnzZtTX1190nPM//9Of/oT9+/dj586d2Lp1KzZv3tyvTKRMPAOngJswYQL27t0LAHj//fcxc+ZM/9d27drl/3369OkYO3YsysvLUV9fD4/Hg127dmHGjBkXvader4fT6bzo8e4VJ7u33Prwww97HK83/T12f8ycORNvvvkmgK7t8E6ePAmgaz3uxMREuN1ufPDBB5fM3L1lm8/nw9mzZzFnzhw8+uijQ363C8kPz8Ap4J544gk8+uijeOGFF/wXMbtZLBYsXrwY8fHxePHFF6HT6fDLX/4SK1eu9F9IzMvLQ01NTY/3jI+PR3Z2NhYuXIhFixb1WNlt/fr1eOyxx9DZ2Ynp06fjlltu6VfOyx17MJ588kmsW7cOr7/+OjweD5YuXYpx48bhgQcewOLFi5GSkoKcnBz/85966ik8/PDDiImJwfTp02G1WuH1evHwww/7f1ANdglSUg6uRkhB47rrrsOOHTuueJF7olDBIRQiIpniGTgRkUzxDJyISKZY4EREMsUCJyKSKRY4EZFMscCJiGSKBU5EJFP/H/pGtcpvjWlTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rng = np.random.RandomState(205)\n",
    "x = rng.binomial(10000, 0.5, 100000)/10000\n",
    "sns.distplot(x)\n",
    "plt.axvline(0.54, color = 'r')\n",
    "plt.xlabel('Proportion of heads');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "name": "04-python-stat-9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    100000.000000\n",
       "mean          0.499991\n",
       "std           0.004994\n",
       "min           0.478100\n",
       "25%           0.496600\n",
       "50%           0.500000\n",
       "75%           0.503400\n",
       "max           0.520300\n",
       "dtype: float64"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(x).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, that changed the game significantly. If we up the number of coin tosses per experiment to 10,000, so 100-fold increase, then we do not see very much variation in the proportion of tosses that are heads. \n",
    "\n",
    ">This is expected behavior because of a statistical theorem called the *Law of Large Numbers*, which essentially says that if you do larger and larger sized random experiments with the same experimental setup, your estimate of the true population parameter (in this case the true chance of getting a head, or 0.5 for a fair coin) will become more and more precise. \n",
    "\n",
    "Now we see that for a fair coin, we should reasonably see between 47.8% and 52% of tosses should be heads. This is quite an improvement from the 27%-72% range we saw with 100 tosses. \n",
    "\n",
    "We can compute our p-value in the same way as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "name": "04-python-stat-10"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(x > 0.54)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we would never see 54% of our tosses be heads if we tossed a fair coin 10,000 times. Now, with a larger experiment, we would **reject** our null hypothesis H<sub>0</sub> that we have a fair coin. \n",
    "\n",
    "So same observation, but more data, changes our *inference* from not having sufficient evidence to say that the coin isn't fair to saying that it isn't fair quite definitively. This is directly due to the increased precision of our estimates and thus our ability to differentiate between much smaller differences in the truth. \n",
    "\n",
    "Let's see a bit more about what's going on here. Suppose we assume that the coin's true likelihood of getting a head is really 0.55, so a very small bias towards heads.\n",
    "\n",
    "> Food for thought: Is the difference between 0.50 and 0.54 worth worrying about? It probably depends. \n",
    "\n",
    "We're going to compare what we would reasonably see over many repeated experiments given the coin has a 0.50 (fair) and a 0.55 (slightly biased) chance of a head. First, we'll do experiments of 100 tosses of a coin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "lines_to_next_cell": 2,
    "name": "04-python-stat-11"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAEHCAYAAABocGdZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXiV1bnw/++eh+zsDGQmIQQIBBQQUBAHKk4446tStBZtnc7v+L7aWttqj1OP1qqnp9ZOtra2Wm2rrXNbFYfiiIoiCsocCJnn7GTP8/P7Y0MgDBn3Xpnuz3Xlwuzs7Hs9Jrmzsp617lunaZqGEEKIEU0/3AMQQgjRN0nWQggxCkiyFkKIUUCStRBCjAKSrIUQYhSQZC2EEKOAMRUvOmPGjFS8rBBCjHnbt28/7OMpSda9BRRCCHF4vU10ZRlECCFGAUnWQggxCqRsGUQIIforHo/T0NBAJBIZ7qEoYzKZKCoqQq/v35xZkrUQYtg1NDTgdDpxOp3DPRRl3G43DQ0NFBcX9+v5sgwihBh2kUhkXCVqAKfTOaC/JCRZCyHEKCDLIEKIcamuro6LL76Y6dOnA3DeeeexcuXKHs95/vnnKS4uZuHChcMxxB4kWQshRhyXP4zbP7SbjU67iSy7udfnHHPMMTzyyCNH/PhFF110yGOxWAyDwTCksQ2GJGshDsffAcHOno9ZM8GePTzjGWfc/ghrd7UP6TVOnDqhz2R9oFtvvZWGhga8Xi+333478+fP55e//CXl5eUcffTRfPe736WoqIisrCzuuOOOIY1tMCRZC3E4wU7Y/U7Px6Z8RZL1GPP555+zatUqAL797W+zYMECdu/ezb333ssf/vCHHs+tr6/n8ccfx2q1DsdQJVkLIcavfcsgsViMBx54gAcffBC9Xk9nZ+chz50xY8awJWqQ3SBCCMG2bdtoaGjgL3/5C3fddReHa007HOvUB5KZtRBDIWvbY0JZWRmdnZ1cccUVzJs3b7iHc1i6VHQ3nzFjhlTdEylxuF0C/bnr36vDJdxwAOo+7vnYlK9A9pSej3XsPvza9sHPE72qqqqirKys+31Vu0GG28HX3VvulJm1GFUOt0tgoHf9D3G4m4kFswf/emLIsuzmEZ9oVZM1ayGEGAUkWQshxCggyVoIIUYBWbMWI9KRbjD5I7FhGI0Qw0+StRiRjnTceFZh+jCMRojhJ8laCDHyHG475UD1sd/9wKp7fr+fpUuXsnLlSp544gluvvnmocU+KM4999zTa8Go/pBkLUa9cCxOdZuvx2OjYY+t6MXhtlMOVD9quew7bq5pGhdeeCFXXHFFUhN1MkmyFqOeLxTlk0ZPj8eGvPdajCv7OrY0Nzfzve99j0ceeYTHH3+cd955B5fLxVe/+lW+9rWvsXHjRu69914sFgslJSX8+Mc/Ztu2bdx3333E43HKysr47//+b3w+HzfddBPRaJRp06YlZYySrIUQ49a+qnv19fWccsop2Gy27o999atf5Rvf+AbhcJjly5dz6aWX8vbbb3PNNddw5plnEo/HAbj33nt58MEHyc3N5f777+f999+nsrKSxYsXc9VVV/HCCy9QU1Mz5LFKshYi2aLhxDH0A0m9kBFp3zJIPB7nxhtvpLW1tftjr7zyCi+88AI6nY7W1la6urpYtWoVDz/8MK+//jqLFi1ixYoV7Ny5k+985zsA+Hw+ysvL2bNnDxdccAEAc+bMYfXq1UMeqyRrMSp9UtXBtDwHWWkjcKkj7IWaL3o+JrWwRzS9Xk96ejpffLH/6/a73/2Ol19+mXg8zllnnYWmadjtdm6//XYAli1bxvLly5k+fToPPfQQ2dmJr28kEsHtdvPFF1+wYMGCHq85FJKsxaijaRqvfNnI0hl5LJmem9wXX/9HyJsptUGGmzUz8QtuqK/Rh33LINFolKKiIpYsWcLatWsBOPHEE7n00kspLy8nIyMDgL/97W+8+eabRKNRTj75ZMxmM//1X//FzTffTDQaRa/Xc/vtt7NixQpuuukm3n77bVmzFuNXVyBCKBqn2R1MwYvXgtFy6OMhD7iqpZqeKvbslP8lUlxczLp16w55fN8Wu7vuuuuQj1155ZVceeWVPR6rqKjgscceO+S5v//975M00gRJ1mLUaXaHAGgaTLI+UjlUgHgMgl3gaTz08yrfhE1/g/9cO/CYQiSBJGsxYmmaRrsvTI6j50y32R3EbNDT4gkRiw+wHHtv5VCDXaDFwdOcSNwHctdDRxWEvGBxDPBKhBi6fiXru+++my+//LL7jumSJUtSPS4xjhyuDog/EmN3m48/fbCHO86bhcmwv+ZYsztIRWE6m+q6aPOGDvuaRzwo09tAAi7Q6SEeAXfD/sc1LZGstViiIcHUUwd6iUIMWZ/Jes+ePVRWVvL3v/+d9vZ2rrvuOknWIqkOVwdkVmE6TV1BonGNhs4ApRPSuj/W7AmyoDSbmnb/Edetj3hQprc6k4F2sOdANASu3WDbu2YacEEkABPKofoDSdYpYDKZcLvdOJ3O4R6KMm63G5PJ1O/n95msc3JysNvtRKNRPB4PWVm9zk2ESJpWT2LWXNPh707WsbhGiztEvtNCvtNKU1cSbzL6XftvanVUwcS9/+2uB7MDpp8Je2TNOhWKiopoaGigvf3Q4l1jlclkoqioqN/P7zNZp6WlUVBQwFlnnYXf7+cXv/jFkAYoRH+1eIIY9TpqXYHux5rdidl2frqVggzr4G4yHkmgA2xZYLQmDrVMXJB4vKsOMiYm3v/49xAJgsmavLgCvV5PcXHxcA9jROuz+cDatWvp6uri9ddf58UXX+THP/6xinEJQYsnxKwiJ3Ud/u7Hajr8pFuMpFmMFDityd2+F9g7s3YWgatq/+PuenBOhLyjAB3Uf5q8mEL0U5/JOh6Pk5GRgV6vx+Fw4Pf7+/oUIYasKxDBH46xYFIWnYEInmDiBmR1h598Z2JWm59hxeWP4A9HkxPU35FYp04vBHdjYu0a9idroxmKj0usWwuhWJ/J+sQTT8Tn8/G1r32NVatWcf3116sYlxjn6lx+jHodU3Id2M0GajsSSyE17X7ynYmtfLkOCwadjpr2JEwgtHhiZm3LAkcB6HTgaYKwP/F4xt4/0UtPgC0vJh4TQqE+16wNBgM/+clPVIxFiG51rgA5DgsGvY6SLDu1Lj+zipzUdPg5bnLixp9BryM33cKedj+TDtgtMihBd2Jrni0bDMZEcvY0QiwMehOk7T3Wftw1UPkG/G4prHwSzEOMK0Q/ScNcMSLVugLkpidm0CXZNmpdfj7Y1UZ9Z4DibHv386blOXhnZ+uRXgZN02jpz7p2wAU6A1j3bh0rmgebn4et/0isYev2/qik58M3V0PZEvjjWdC4adDXKMRASLIWI1Jth5+8vcsdJVl2qlp9vLa5iVvPqqDAuX8nxuKpE9je5KG24/BLITtbvPxizU4isXjvAfftBNmXlBf/P5h/JZjthxZ1Mlnh/J/DsVfBS/8XOmsHfZ1C9JckazEi1bkC5KUnkvKkbDszC51cd/JUFpb1LO6TZTdzwtQJvF/ZdtjX2dbkJq5B52E6pfewL1nvo9ND/lGw6D9h2umHPl+ngzPuhlkXwMe/hcjeXxaaBk1fJP4VIokkWYsRJxiJ0e4Lk7d3GcRiMvD140uZmGU77POXzy3iy/ouXL5wj8dLbUF2NXUB4Ii04QzU7S/adDB/x8CrvOl0cNJ3wOKEyn8nHmvYAP/6NnibB/ZaQvRBkrUYcVo9IfQ6mODoX2OB8vx0SifYeXtHz7Xrlg4Xrf4YE0wRGqu2EN/1NkSPkKz37QQZKL0BKs6FqncT9UQ2P5943NvS83n7uscc+ObvGHg8MW5J1T0x4jR0BSjKsGHU938ucfbRhTzy7i5mT8xgWl6iKt6njSGm2QNkmaO0hk1AL/uxfa37TywOVN5Rid0jH/wcHPmQVZZ4vQNJ9xgxRDKzFiPOF3Vdh6xN96Uk285Xpufx3IY6AuFEedNP64PMz/SSZ47QEuqlYE7ABf52yCwd3IB1Oph5fmI3yZxLIXPSoclaiCGSZC1GFHcgQlWbj5PKcwb8uUsrckmzGPjTh3tY/WUT29sjzM/wkmuJ7J1ZH0Hz5sR+6bQhtAjLngJn3JPY5pdRLMlaJJ0kazGifFHfRY7DQtkEe99PPohRr+fyRaUUZdp4dkMduXYDk20h8swRWnubWTdvTixd6HRDGDmJ9WtIHE2XZC2STNasxYiysa6TOSUZ6AaZOLPsZi6YW8TMAgfhtirYDbmWCF1RI8HYET6peTNklw1+0AfLKIaQG6LBRAU/IZJAZtZixOjwhalzBZhb3HdX6r7odDpMhkTCzzEn9lg3Bg4zN4lFoW1HchvhphcBOpldi6SSmbVQ6kgtvAA+r3UxMdN2SM/FobIb4jgMMer8emaFYwS8+/dj2/012ND2F2pKBqMZbJmJZJ1RkrzXFeOaJGuh1JFaeMXiGp/scXHKjCHc5OtFriVCrc9Ahz9Mfau3+/GjA1WQUw6G/u3p7re0PPDKzFokjyRrMSJsbXQTisY4pmToSyCHk2uO0LR3GUQfDzN3569xp5VixgslxyY/YFquLIOIpJI1azEifLS7nfmTsrAYDSl5/VxzhMZgIlk7fXvQx8MY4mFMXXugcG7yAx4uWQdciTVyIQZBkrUYdrUdfqrafBxfNiFlMfIs4e5knenZSZdjKjsmXYpr8W0waXHyAx6crLU4rP051H+S/FhiXJBkLYbdy180Mi3PQU56cm8sHqh7GUTTyPRW0pleDkDckjH0/dWHk5abqMQX9iXeb90BwU7wHb46oBB9kWQtlKtq8xHauwPEG4qyZlsrJ04b+InFgci1RHBFDMRd1VgiXXQ6pqY0HvbsRIcZV3Xi/eq1iX8DUrxJDI4ka6HcX9ZV8+Ln9UBirboo00r53uJLqZJnjqBHI1T9CV5rEVFjauOhN0DJcVD1duL96rWJ2bZU2hODJMlaKOUPR/GHY2ys6+KzGhcf7W5n+TFFgz6x2F9Wg8b09DDWxvV0pk9LaaxuU5ZC206o3wCd1TB5iTTaFYMmyVoo1dSV6Id4WkUez35ah8mg5+QUL4Hsc7ZjJ4X+bXQ6FCXrtFwonAObnoYJ5Ykj7SE3xProWiPEYUiyFko1uYOkWYwsrchjRkE6p8/Mw2hI/beh07uL67t+ynPaUjzWiSmP123qaYkO6ZNP3F+7WmbXYhAkWQulmrqCZNlN6HU6rlg8mQWlqS++bwm7mFHzFK7CU7g19E3qQoniSp6onmgffXSHLHMSzLwApp8FJnuisJPcZBSDIMlaKNXkDpJpT/LR7j44/HWEjU5CpV9hSqaBLR4b/pie726ewj/rB16KdcCmnrq/VrYte/9NRmn1JQZAjpsLpfbNrFWyB5vwW/MBmJdvZGuLnZaQmc6okZ0etb84sGfvXwaRVl9iAGRmLZRqcgfJUjyzTgs247cWADA3z8DGrjRWt2QxJ91HlU/xfOXAmbUQAyDJWiiVSNYKZ9aatndmnUjWx+QbCMQNLMzycHpuJ1VetbN87FmyZi0GRZZBhDK+UJSuQFTpmrUp6sUU8+Oz5mMDsqx6rpnUxHGZHrxRA60hA+5QHKeqAdmyJVmLQZGZtVCmvjMAoHQZxB5sIqq3EjZldD92Rm4nmaYYBZYwBp1GpUthJTx7NgTdUn1PDJgka6FMnctPps2E2aju2y6xXp1/2GJNRj2U2KNUdhypOWMK2LIBDYKy11oMjCyDCGXqXAEKMtQ2kLUHm/DtXa8+nDJHlF2uKBywAcMXjqE7qP2X1WwgLRkDMtnBYJGDMWLAJFkLZepcAQqcqpN1M505Rz5eXpYWOWQZJBiOETio/de0XEdykrVOl1gKkR0hYoAkWYuUObg57vYmDzmO1KxXT7aHsMY83e/nRj1EdV6s4fbunSCHU+aI8OYehcsgkDgg421WG1OMepKsRcoc3By3ssXLjPwjJ86hsMY8RHa+1f1+PMOKzWxE0+kJWI7chLcsLUqtO0YwBtbUdBQ7VEYxtO1QFEyMFXKDUSjj8ofJS2E3mINZmz/DaytG0x85C09Oi6IBuz0K5y0ZJdBVl2j1JUQ/SbIWSvhCiTrWhRk2ZTGtTevpdJT3+hybUWNiup5Kj6ppNYlkHQ2Cu15dTDHqSbIWStR0+LGZDBRmqrnBqA97MLt29qvRwPRsI9s6Fc6sLQ6wZSUaEwjRT5KshRI1HX4mZdvRp7gjzD7Wzp1EbTkELHl9PndhkZkPWhUXdMoogdbtamOKUa1fyXrTpk1cddVVfP3rX+f3v/99qsckxqDqdj+lE1JbjtQY9ePw1wBgc20nWHhsvzqXL5lkZlOHkc6wml8kQCJZy01GMQB9/u0XDof55S9/ya9//WtsNnXrjWLsiMU16jv9nD6z71nuUOS5PqWk5S1q85Zi7dxJx4wLoavvz5uZYyTbEuf9ZjPnlYRSOsZumSWwe03iJqNO/sAVfevzu+Szzz7DZrNx4403cvXVV7Njh8wGxMA0dgWIxTWKs1I7s7ZEuvBZCylqW4suHiGUN6dfn6fX6ViSH+bdZoVLIRklEAmAr1VdTDGq9Tmzbm1tZfv27bzwwgs0NjZyxx138Ne//lXF2MQYUdPhpyDDmvKaIJZwJx3pM3A5K5ioa0cz9v8vwSUFYe7/woGmpXCABzKngaMAOmvBka8oqBjN+vzpcTqdzJ8/H7vdztSpU/F4PH19ihA0dQV58I0d+MNRqtv9TMpOymHtXpkjXYTNmQSsefjz5g3oc0/KC9MUMLDTrXALX/4sqPlQKvCJfukzWc+dO5eqqipisRgtLS1YrWprO4jR6d2drby0sYHfvrOLqjYfpdkp7nWoxbFEOgkdUAp1IHKsGkdnRtQuhSz6/yDYCZ89AXHFR97FqNNnss7IyGDFihWsWrWKG264gVtuuUXFuMQoV9niZVFZNgUZNryhKJNSvRMk5EKvxQiZMwf9Gkvyw7yjMlnbJ8Ci/wTXHtj5mrq4YlTq10mAiy++mIsvvjjVYxFjSGWLl+n5DmYUOGmpyEt5wwGzv5k4esLG9EG/xpKCMH/YaSeocpKblpPoft6wAWacozCwGG1kz5BIicoWL6XZaeh1OiVlUS3+JsIm55C2wc2fEMGo19jQoa5+CQDOieBulFoholeSrEXSBSMxal2pPwRzILO/aUhLIABmPSzOjfBRm+L7Ms4iiEdkG5/olSRrkXS7W31oGpSkeF/1gcz+5h59FgdrSX6YD9sVJ2tzGlgzoUsKO4kjk2Qtkq6y1cvETBs2s7ptcGZ/EyHT0GbWkFi33u010eJTvCSRMVGq8IleSbIWSVfZ4mVankNpTEsSlkEAJjtiTLRF+bhR8d5npyRr0TtJ1iLpdqlO1pqGOdA86D3WB1ucExymZN0A0TB07O75Jv0aBdLWS6RAZYuXk8pzUvb6B/dbzPdWo4+FkrIMAolk/cPNUa6dAHpVhficEyHkhq7aRBeZA035SqLJrhjXJFmLpIrG4lS1+ShP4cz64H6Len0rmk6f2LqXBMdOCOGPwC6/lfK0YFJes0/2bDBYoH0XGBVvHRSjgiyDiKRw+cNUt/n4aFc74Vgci0GPP6LmdIkh1EnMlpO0UqMOo8bsXAMbu1Jfz6SbTp/YwtexS11MMapIshZJsa+T+erNzaRZjHzR4CakKFkbQy5i9uTWyl5YZGSjW2GyhsRSSLska3F4kqxFUtV3BijMULtP2eRvJpJenNTXXFhopNJnwxdN/IhEtTguX4TqNl/3m8sfTmpMssug8XPU1WkVo4kka5FUVW1eynLUzkjN3jrC2b13MR+o6dl6HMYYX3oS1xKOxKlq97F2V3v3m9sfSWpMcisg4AJ3Xd/PFeOOJGuRNJFYnFpXgCkKk7U+FsIYaCOSldxkrdfpmO30qV0KMadB/lHQvFldTDFqSLIWSVPT4Uevg4lZ6np1pgUb0fQGIs5JSX/teU4f77Y7uW1bKQ9snUBcxfJEyfHQvCX1ccSoI8laJE1Vm49J2XaMenXfVmmBBiJpRaBP/i7Uk7LdfGdqPQszPfyrMZ02v4IbppOOh64aCPaj068YVyRZi6TZ3eqjLEftMXNHoIGwY2JKXlung/kZPi7I78BmiNPkUZCsM0sTTQlatqY+lhhVJFmLpAhFY9S5/ErXqyExsw47krsT5GA6HUy0RWn0KjiCrtNB3ixZtxaHkGQtkmJLgxuAYoXr1caoH2ukk3BaambWB5poi9DkVdRCZkI5dO5RE0uMGpKsRVJ8XtvJpAl2jAaF69XBBqJ6C1Fb6uqQ7FNsi9KkYmYNiZOMIU/iTYi9JFmLpPi8rkvZEkhm/TuUNr5KYdtH+GyFSTtm3puJ9oiaNWvYXyfE3aAmnhgVJFmLIQtGYmxp6FJ2czF/59OkBRqI6wy0Zs1XErN475q1pmL73r46IVLfWhxAqu6JIfu8thN0OkpUrFfHY9g8VWwv+SrutDIAilIflYm2COEYeIJRnDZT6gM6i2RmLXqQmbUYsnW7Oziq0KlkvdrsqUEfC+G35Kc81oFyLDFMemj3JbkeyJFIshYHkWQthuyj3e0cU5Kcwv99sbm2EbZOIGpU14wXEk0IChxG2r0hNQGdReBthrjijjVixJJkLYYkFI2xocalLll3bCPgnKIk1sEK0g3qZtbpRaDFEglbCCRZiyHaWNuFBswsTFcSz+raRnC4krXDqC5ZGy1gz5GlENFNkrUYko92tzN/UiYWo0FJPJtr+7DNrAsdBjpULYOArFuLHiRZiyH5aHc7x0+ZoCSWIerH7K4exmWQxMxayfY9kGQtepBkLQZt33r1ojI1yTrTuwt0eoLpyS+H2h+FDgOhaBxfWNHhGGdxotO5dI4RSLIWQ7Cprou4BvMmqbm5mOnZQTBjCprBrCTewXLsBkwGHU1dijqeZ0+GiA86dquJJ0Y0SdZi0D7a1c68kkysptSuV0+2h6iwtFHm34SWWUq2SdHM9iAGvY7iLDu1Lj8A4ViiL2O7N9z9ltRZt9kB6YVQtz55rylGLTnBKAZtXVUHixSsV1tjHiI738LatJ5wTgVGTdHM9jBKsuzUdiSStS8UxdPhI9Lq7f74tFwHSa2QMmEa1H+azFcUo5TMrMWghKNx1ld3cPyUbCXxdPFootHAMK1X7zMp20ZNh1/dTcbsqYlkLevW454kazEom+o6icdh/qQsJfEcgXpAI5ReoiTekRRn2/GHY3So2m89YSoEOqBth5p4YsSSZRAxKOuqOjhmUurXq/dJ91fjtU0EvYIiSr1wWk1k2k3d69YpZ0mHrDLY8iLMXrH/cWtmopSqGDckWYsBi8bivL65ia9Mz1UW0+mrxp02WVm83kzKtlPTEVAXsGA2bH8V0vL2PzblK5KsxxlJ1qJXLn8Ytz/S/b6mafx8TSXN7hBfW1SqZAy6eASHv5aG3JNQ24738Eqy7ImysKoUzoVd/07UCXGorTYoRo5+r1mvX7+eGTNm0NHRkcrxiBHG7Y+wdld799s9r2zljS1NPH7VcRRkWJWMwe7ahg4Njy21jXH7a1K2ncauAKGooi2EkxYndoW8+xPY9ZaamGLE6Xey/tOf/sTRRx+dyrGIES6uaby7o5XvL6ugosCpLG5620a8tolow7xevU9hhhWdTsfb21vVBDRaYME34ZjLYetLEFA4qxcjRr+S9Zo1a1iwYAF2u9oawmJk6fRHiMQ05kzMUBrX0fY57jQ1Sy79YTToOW9OIY++X8XPPnThjynaVFV4TCJxS9nUcanP77J4PM5TTz3FZZddpmI8YgRr9QSxmQxk2tXNcI1RP472Td0tvEaKRWUTeHDFXLa3RXijVc1xe3Q6SMuXZD1O9Zms//nPf3LqqadisVhUjEeMYC2eELnpFnQ6nbKYBW0fEDfY8NiH9zDM4ZRk21lQZKE6oPBnw5EH3hZ18cSI0Wey3rFjB6+99hpXX30127dv56abblIxLjECtXpC5KWr/aVd3PIWXQXHJzp+j0CTMozUKk3WMrMer/rcuve9732v+79XrVrFz372s5QOSIxcrZ4Qs4rU3VjUxaMUtbxLwzHfBn9cWdyBKM00UR+0EI2DUcXvE0ce7HlfQSAx0gzo2+vJJ58kO1s24o9Xrd7EMogqOZ0bMUV9uPMXKos5UJMyjMQ0HQ0hRf9fHPkQ6oKQt+/nijFlZP5tKUYcbyiKPxwj16EuWRe3vEVzziLiRpuymANlN+nJNYepUbUUYs9JLAm59qiJJ0YMSdaiX1o9IYx6HVlp6gr/T2x+m7q8pcriDdYkW0hdsjYYwT4BXFVq4okRQ5K16JdWT4gchwW9op0gZk8tTn81DbknKYk3FJNsIfU3GWVmPe5Ishb90uoJKl2vdjR+iMdegt9WpCzmYJWonFlD4iajJOtxR5K16BfVNxcdjR/QPGHk3lg80CRbiLawSd1JRkc+dMgyyHgjyVr0S6tHYbLWNNIb1tI04Xg18Yao0BrGoNPUza4dedBVC7FI388VY4Yka9GnUDRGpz+ibCeI01eFKdBKc/ZxSuINlVEHxVaF69Zp+RCPQnulmnhiRJBkLfpU2xHAaNCR71RTErWg/SMCWRWELKlvxpsspbYQVX5Fydpsh8xSqP1YTTwxIkjzAdGn6nYfJdl2DPrk7gSZbA9hjXnIjXqosIQACBrSyW//GG/h4qTGSrXpjgCrW7IAn5qABbOh8k0oO3n/Y9Lqa0yTZC36VN3hpzQ7Lemva415iOx8i3iGlUhXEADb1JPI6/iE+tmXJj1eKs1wBHi0pgB3RIeSvwdyZ8CGJ2D3O/sfk1ZfY5oka9Ht4BZeAJ5QlJoOPydPy1Eyhgl7XiauM+EpOglao0piJkOxNYTdEGNTpwUlxVzzjwZfK4Q8iaa6YsyTZC267WvhdSCLUUckGqckO/WNJ/TxMIWVT/D5tOsxGW2AJ+Uxk0Wvg+lpATa6zCxXETCjBExpif3WBbNVRBTDTG4wil5tbfRQmGHFajKkPFZB+zriBiu7Si5OeaxUmO4I8LnLjD8co90b7n7zhVPQq1Gng+zJst96HJFkLXq1pdFN6YTkr1cfTB8LU0w3TxUAAB/tSURBVNj2AQ0zv0l8hPRaHKgZjgCbu8y0eENUtnq734KpSNYAWWVSI2QckWQtjkjTNLY2uimdkPolEEewHp0WwzXx1JTHSpVp9gBRTceODkW1t7OnQFeNHI4ZJyRZiyPa2eLF5Y8omVnbA434rQWgT/1yS6pYDRrljjBfqLoxmlECOkNiC5+mqYkpho0ka3FYn9W4ePLDai5fNIkMW+qXJRyBRny2wpTHSbXZmUE+b0nRssfBDCY49mqoegc+/4vMsMc4SdbiELvbvDy/oZ7/M38iKxYUK4mZFmzEZx39yfrYrCAbmqJEVU10c2fASTdB6zbY8ZqioGI4SLIWh9jR5GVanoP5k7KUxNNFg1jDHfhGQTnUvszLChKOwQ6vwu42jnzIrYDGz9XFFMpJshaHqG73MVnBTcV9zL56YjojAYuagzepZDNoHJNvYKM79ev8PWRNhqZNamMKpSRZix4isTh1nQElNxX3MXsbEjcXdWPj2/H4IiOfdznUBs2aDG2VEHSrjSuUGRs/HSJp6l0BACZmqfsz3uytGxM3F/dZVGRkT8CKK6JwZ0t6IZisUL9eXUyhlCRr0UN1u4/iTBsmg7pvDZOvYUzcXNyn1Kkn1xxho8rZtd4A+bOh9hN1MYVSkqxFD3va/UqXQAyxIKZg+5iaWet0OuY6vXyuet26cA7UrlMbUygjyVp0i2sa1R1qby46/HXE9WYCllxlMVU4NtPLhi4HAVV9GSGRrOvWQ1zRCUqhlCRr0W1Pm49gJM4khck6w7ebUMaUMXNzcZ+5Th8OQ4y1HQrLlxbMgVAXtG1XF1MoM7Z+QsSQbKzrIi/dgt2srnKu01tFMHOasniq6HWwNKeLf7dlqgtqzYCcGVD9gbqYQhlJ1qLbB7vamZ6vbiZoDHaQFmommDFVWUyVTpnQSZXfyna3wiqCU06BXWvUxRPKSLIWAHiCET6rdTGr0KksZnrrBsLGdKK2PGUxVcq1RJnr9PFCncIbjeVnJFp9RcPqYgolJFkLAN7Z0Uqa2ah0vdrZ+ildjimJQvpj1Bm5nfyjzs4fP/fS5klxQwKAySdBPCK7QsYgaes1Th3cb/HFz+pZWJaNPkWJ85BO5ppGRut69mSfjCUlEUeGBRlevjujnQfX6fmHLZPvT6vDoINpuQ6SPt+OhsHTCBPnw5fPQsZE6Xg+hkiyHqcO7LcYi2u8X9nGd06fnrJ4B3cytweaMAba6HKUMTYXQRJ0Ojir0Me8GWWseFFjj9/K1LRgaoKFvVDzBaTlJyrwFR4jHc/HEFkGEVS1+YjFNeaWZCiJZwl1ML32afwlS4gaFdfQGCYFDj2lthA7fAqO8efNTMywA52pjyWUkWQt2NzQRXleOhZj6mtZGIIuZu35E15bMR3HfSfl8UaSckeAnT5r6gOl5YJ9ArRsTX0soYwk63EuFtf4ssHN7IlqZtVpe3eAVBZfBPrxtQo3PS3AThV1rnW6xAGZhk9TH0soI8l6nKtq8xGOxqgoVLO/2uKuojN92pg7sdgf09MCtITNdKqoxleyCNoroasu9bGEEuPvJ0b0sKmuk4oCp5IlEF08htlTi8demvJYI1G+JUK6McpOFevW6QWQWQpb/5H6WEIJSdbjWDQe58uGLuYUK1oCCTag0+J47Wr6Oo40Oh2UpwXVtfwqWQRb/ymFncaIPpP1rl27uOyyy7j88stZtWoVtbW1KsYlFKhs9qJpKDti7vRVE3YUE9crPH49wkxPC6jZEQJQNC+xI6TqHTXxREr1mayzsrJ45JFH+Mtf/sK1117Lb3/7WxXjEgpsrOtkVqFTWaOBdH81IWeZklgjVbkjwG6flYiKya7JBlNPhY1PKQgmUq3Pn9Ls7GyczkS9CKPRiMGgsFWRSBlPMMLmBjfzFHUwJx4j3V9LyDlZTbwRapo9QFTTqSvuNH0ZbHsFIik6iCOU6feUKhAI8Itf/IIrr7wyleMRiqzZ1oLDamRKrpoiQ/aunejjEULO8XlzcR+rQaM8LcC6dgX7rQEmLQa9HirfVBNPpEy/knU0GuWmm27immuuYerUsVnOcrx55csmFkzKSlktkIOlt36Gz1aIZhjLlUD6Z47Tx0dtipK1wQQV58Pm59XEEynTZ7LWNI3bbruNJUuWcPrpp6sYk0ixbU1utjd5mF+qaAkEyGh8H1d66mqPjCZzM3xs7DTjCSvapXH0/4HtqyHsVxNPpESfyfq9995j9erVvPrqq6xatYp7771XxbhECv39kzrml2aRZTcriWcLNuNwbcGVPlNJvJFuqj2I3aDxUb2imtNlXwGjBXa+piaeSIk+z/suWbKEjRs3qhiLUOCdHa38+aNq7jp/lrKYxc1vEXSUELDkoG4uP3LpdbBwQpB3a8KcMUNBQIMJZi2HTx+HWReO6frhY5kcihlHPtjVxnVPrOfmM6dz0rSclMWZbA9RYWkjN9pAhaWNGa2vEiw+UZLEARbnBHmvRmE3l5NugpqPYMdqdTFFUo2vSjrjlMsfptUd5LonPuXS40o466gC/JEUdSqhZ+1qrb2D9NbPaJn9dWiWkp37HDchwI82x9jYEqPYvv9r4YhpqWnGkFUKJ9wIq3+QqHMdDfT8uDQpGPFkZj0OuP0RnvyohkgsztQ8B2t3tRNKYbI+UKZnB2FjOpGsciXxRotsY4SKbD0/3pTGjhYvla2Jt1gshTcdT7oJYhH46NeJPo0HvgXlF+lIJ8l6nNjS4GZmoROjXt2XXB/xU9LyFq1Z82QJ5DD++2Q7mz1pPNuYuiWpHsx2OOUW2Pg0aFIvZLSRZD0ORONxtjS6lXYuR9PI2vUcYZOThtyT1MUdRSam6/nO1HpebJrAp52KOqDPWp6oF9JRpSaeSBpJ1uPAprouIrG4soJNALmuDVi7dlNZfBGaTkoUHMlR6X7OyevgpaYJagJaM6D0BGj4TE08kTSSrMeBd3e0Mj0/HbNRzZdbF49Q3PouXZOWETLLZr2+nJHbyQ6fjZqAotOd05dB00ZZChllJFmPcfG9ncuPKlK3BJJV+290Wgxf3gJlMUezfEuEOU4fb7RmqglYtiRR2Km9Uk08kRSSrMe4d3e24g5GqShQlKw1jfzKv9GUvRDNMH7rVg/UGbmdvNfuxBfRUh/MnJbogN7weepjiaSRZD3G/eH9KpYdlY/NrGbduKj1Pcz+Jlqyj1USb6yYn+HFZojzr52KSplOXAANGyAaUhNPDJkk6zFsR7OH9yvbuGS+mjZaOi3G7Mrf0D7pbKJGu5KYY4VBB2flufj95wFCKrbA5x+daE5Qu05BMJEMkqzHGJc/THWbj+o2Hz9/cyeLJmeT61RTjnP6nr+S5q+nsUJqng/GWXkuYho8sUtB2y+9IVHgqeodudE4SkiyHmPc/ghrd7Xz0ucNvLa5iVlFGak9rejvgI7dZLas45idv6BxzvVkOBypizeGWfQaNx5r55db0+gMKzhEVHI8hL3Q9GXqY4khk2Q9Bm2q6+TXb1cyq8jJ1FR3ggl2wu53sL15K25bMS0BPUZNWkgN1nnlFvKtUX6yyUq7N0y7N0wolqKbjiZropNM5RsQ9qUmhkgaSdZjzJtbm3nm0zqWHVXAymNL0Kk45t2yFZOrkurCs+VY+ZBprCho4m/VaXxQF0p9vZApp0I8Ck9/HXa+AR2797/5O1IXVwyYJOsxJBKL84f3qzi9Io/jp0xQk6jjMdj6D4LFJ8oBmCSZ6/QzK93P0/UKaoZYnYkCT7nl8LevJ5rrSnGnEUmS9Rjy3Kd1+MMxjp+q6OgywNZ/QLCLwKRT1cUcB74+sZUPXU52+RTcHDaYYcn3E0fRq9emPp4YFEnWY0QoGuOXayr52sJJWIyKanH42uHDX0P5MjSTbNVLplJ7iJMndPFEXR6apuCgjE4PU5ZC1buJMqpixJFkPUb8dV0N4Vic5ccUqQmoafDydyC9CCZLVb1UuLSojWq/hRd2KDq4MvHYRNKuW68mnhgQSdZjQGNXgJ++voNbzqrAalI0q/7yOdjxGpx5d2LPrki6bHOUy4tb+Z+PfLQFFdx/MBih7GTY/ZbsvR6BJFmPcpqmcceLmzmmJJOL509UEDCO44s/wT+/BWf8N2RNTn3Mcey0nE6mZhq46/N04gpWQ5h0IoQ9MrsegSRZj1L7Tio+8cEe3tvZyvWnTKWm3Z/S3oplFjfnf7KKrA/vg5NvhmmnQzjQ9yeKQdPr4O4lDj5oMbPqvUyaAyn+kTXboXwZbHsZIvK1HUkkWY9Sbn+EV75s4v7V2zm1Io897f6U91Ys2flnzF176Dz2W2DLStyMOrjxqki6qVlGVp+R2PN89hvZ1PtT/GM7+SQwmGDDE6mNIwZEkvUoFYtrPPNpLRMzbZw4LfX7cU0RNwU7/kJd3iloZnUdZwTEAWMsyEPzWpiWHuaHG+ypO9UIoDfCzPPh08dh+6tyUGaEkGQ9Sj31SQ1NXUEuObYYvYLDL7N2P0bEmk1b5pyUxxI9xeJxKlu97G7zsjK/gTVNNj6oTfGR/oI5UH4m/P1K+Oi3clBmBJBkPQq9ta2FP67dw4oFxTitqS/wbw80MWPPn2mYdW1ia5cYNsW2MMvyXPx4rY9oKjds6HRwwo1QfgZ88ig0b0lhMNEf8pM3yuxs9nDjU59x7UllzFDU/WXB1vtpzTqGroLFSuKJ3q0oasMVjPPzLSku0qXTJW4izzgbPnsCfK2pjSd6Jcl6lNA0jZc3NXL5o+s4Y1Y+lx5XoiSus+ZNilrfY/1Rt0uRphHCbojzk9PS+c12O+82mVMfcOppkFsB6/8IYX/q44nDkmQ9CvjDUa7448d895mNXHnCZO6/eI6SIk2WsIvij+5i85Rr8KSVpjye6L9FRWa+NcvHtz92sqMrxYeSdDqYexmgg3/8Pwi6UxtPHJYk6xEuHtf4f3/9jNoOP09cdRznzS6ksTOQ0v3UaBrHtr3IBe+dj2bLJjZzORWWNrJNKvpNif6IA5cWd7I4J8BZb2Tzn2vTqOqMpi6g0QKLr0/svX5iuewKGQaSrEe4n7y+nU+rXVyyoISdLT7W7mpP6X5qXTzCwi9/yNQN91KffTwt01YS3r2WyM63pKnACBKLx9nd5uWKwlruqaim1gsXP9vBhnZj6oKaHXDBr0CLwe+XJnaHyHY+ZSRZj1CapvGrNTv5w/tV/Gj5UWSnpX5tUh/28JVPb6Cw7QO2f+Vhmicskt0fo8C0tCC3ldeyfLqVy97J4sUaS+qC6XRwzOVgzYSnL4OPH5XtfIrIT+IIFItr3PfqNh5+exePfeM45hRnpjxmbsenzHjxbKyhdl5f/GeCzrKUxxTJo9fB945P44fHeLhlvZP/+5GTjlCK7msYzDD/Cig9Edb9Bt6+H0Le1MQS3VL4N5MYqBZ3kEfe3c0/NjYQisT430vmUJxpS+n6tC3YzOI9D5O/5yW8sy6lesoVlBoMZJtiNKcsqkiVy6YEWZgb4eZPnJzx+gS+G3KxeLa3xw3p3GicIVcf1+lhxjmQNwu2vAg/mQZTlyZm3RXnys6hFJBkPQIEIzH+8H4VD79VSUm2nZOn5TB7YgadgShrd7UzqzD5x7sNgXbmbf0502ueJpA1gy2Tr8CZUU54d6JTiHHmwqTHFGpMTY/x7Ckuntxl40dvNZP7qZszZxUwOSexL/vsotjQk/U+WZPhq3+Bli8TpVWfuxqKj4NlP4ZCOe2aTJKsh1FXIMJf1lXzx/f3YDHquf/iOcwucvLB7tTdsMl0b6es/h9Mr3uOLvsk3p3/czImziBS+TZqjtgIFYx6+GZ5gBPmzea2t7p49P3dlOWkMavQySy7iaR2y4wFE7tESo6H7Gnw5bPw+1Ph2KvgxG9BhoLSveOAJGvFYnGND3a18cz6Ol7b3EReuoVvnlDKGbPysRgNKVnyMPmaqNj9DGUN/yTTs5PmCQupWfK/fGg+EXQ6MnRtSY8pRoa8NCMXzS/mK9NzWVfVwSd7XPxzU5BTCjL41kwf8yYkebtfWg4s/I9EQ4qPfw8fPwKZk6BoPsxZmTgNKUskg9KvZP3000/z4osvYjQauffeeyktlQMS/aVpGlVtPjY3uPmivot/bmzA5Q9zztGF3H/RbHzhGHqdjvXViTvqQ13yMARdFLV8gMNfS1GkmsyWddjduwmkT6aj5Ex2lf6MHUxKxGn0JOMSxSgwwWHhnNmFAEy3uXjs7e1c8nYWczPDrCz1cnpGjKS1WdbpIP+oxMza1wKuPdCyFf5+BWQUw7zLE4dsMoqTFXFc6DNZd3Z28txzz/H000+zZcsWfvrTn/KLX/xCxdhGlWgsToc/TJsnTENngO3NHr6s7+KTPR20ecNMzLQxszCdb51WzrlzCkm3mqhuS+ybHhQtji3YTFqgkXR/DWWe9WS0bcDmqSZqSifkKIa0PFrs5VimXEBNOB10OkwWOyhq6SdGpkKHkSsKazkz28TrLVn8cFMmt3zeRo4lh8mOKKWOGCVpMQra/eSHzRTYYhTZ42SYB1iWVacDR37irWQRFM6Fuk9gw5Ow5keQWZp4LKccssrAWZR4Sy9MdFqXGXgPfSbrjRs3smjRIgwGA7Nnz2bPnj0pHVCHL0w4Gkcj8Y2xr7Hzvm+TfZ2eD274vP95fXzeQc/noOdH4xr+cBRfKIYvFMUXjuEPR/GGogQDIfC3EAhFCEaiuANhOrwhXN4QvlAYAD1xnBYjJVlWpmbbOPu4NKbnZVKUacFpMYLWBs31EPFj63BR2tCKIRbCGAtgiAUpaIxj7vJ0v2+MB8ncHCXf58UYC2KIh7BH3ViCzRwTj6LpDITs+UQnzKTJcTTWqRdSE3aCTsfEmQtp3voxRXYrRORAi+ipwBLhipIWLpvYSvq0RezaVUmNz0it38jHLUbamvw0e9LpiiSOs2eaYkzObifXkEGWOZ54s2jYW/yYfVYseg2zAcx6DUs4hLHThEEHBh3odRp6kx5D+knoTzkZm7cae8dWrO49mGo3Ydr8L4zeRvThxFH2uNFKzJ5PNL2IuDWbuC0bzZ6DZkknbnaimR2JrjZ6I+iM6A1GMh020BkSj+n1ez+29/3uxL/334N/EfR4/+DnDOB9gzmxFJQCfSZrt9uN07n/1pN2cJZMotoOP6f879vElDSbOzKDXofdbMBuMmAzG7Dt/febgT9xvudvh/+kg88huPa+7TrMc41WMFrJ0Ztx6szEDRbiRit6kxWd34pFMxE3WImbzRgsWejMVuIOI0GDmbjBStRqRfO1Ux+yEjE6QKdn4syFtEhiFoNg0mtMztAT1LUx0QGLHYnHK+adwLbPPiAc19EWNtIYNKPLq6CtuYPOiJ7KLj2dEQOR5gDBgJVQXEckriMc1xGhi0g0g5gGcU1HHIhr9fT8yZ6x920/G0EKdC4Kwh0U+dsp6WghGw8OXRXpbCZdFyAdPw4C2HVBDMS732JGDYMWSzT7HbaGvzq4/kPIm5n8V9b6yL7vvPMO69ev5+abbwZg+fLlvPTSS72+6IwZM3r9uBBCiMPbvn37YR/vc2Y9d+5cHn74YWKxGNu2bevXzcUjBRNCCDE4fSbrzMxMLrzwQi6//PLu3SBCCCHU6nMZRAghxPCTQk5CCDEKSLIWQohRQJK1EEKMApKshRBiFBjzhZx6q2ty5513UllZSTAY5IILLuAb3/jG8A10kPqq2xKLxTj33HNZsWIFV1999TCNcmh6u8Zbb72VHTt2kJaWRllZGXffffcwjnTwertGn8/Hj370IxoaGojH4zz55JPDONLB6e36brrpJtraEsXEvvjiC55++mkqKiqGa6iD1ts1vvvuuzz00EOYzWZmzZrFnXfeOfAA2hjmcrm0Sy65RItGo9qmTZu0G264ocfHQ6GQpmmaFolEtDPPPLP7/dGir+vTNE3729/+pl177bXao48+OgwjHLq+rvGWW27RNm3aNEyjS46+rvH+++/XPvnkk2Ea3dD15/tU0zStvb1dO+eccxSPLjn6usaVK1dq9fX1mqZp2nXXXadt3bp1wDHG9DJIX3VNzOZEX8NQKERpaSkmk2kYRjl4fV1fKBRizZo1LFu2bHgGmAT9qU1z9913s2rVKt5//331A0yCvq5xw4YNvPLKK6xatYo///nPwzPIIehvfaHVq1dz1llnqR1ckvR1jdOnT8ftdhOLxQiFQmRmDrxV35hO1v2pa/L973+fM888k9mzZ/dofTQa9HV9Tz75JCtXrhx113Wgvq7xlltu4ZlnnuGhhx7igQcewOMZfWVf+7rGzZs3c9ppp/HYY4/x+uuvU1lZqXqIQ9Lf+kIvv/wy5513nqphJVVf17hs2TKuueYazjrrLKZMmUJBQcGAY4zpZO10Onv88Or1h17u//zP/7BmzRrWrl076n4Iers+j8fDxx9/zNKlS4djaEnT19cwKyvR82TChAkcffTRVFVVKR1fMvTnGk844QSMRiPHH388O3bsUD3EIenPz2FjYyOhUIiystHZqLmva7znnnt4/vnnee2113C5XHz66acDjjGmk/XcuXP5+OOPicVibN68+ZCbb+Fwoqyp2WzGZrNhsRxcOm9k6+36du/ejcvl4uqrr+axxx7jmWeeYe3atcM42sHp62u47wckFAqxZcsWJk4cfS2k+rrGBQsWsGXLFgC+/PLLUdf8o6/rA/jXv/7FueeeOwyjS46+rtFoNJKeno5er8fpdOJ2uwccY8wfN3/qqad46aWXuu/QVldX4/V6Oeecc7juuusIBAKEw2GWLVvGVVddNdzDHbDerm+f559/vjtxj0a9XeO1116Lx+MhGo1y6aWXcskllwz3cAelt2usra3ljjvuIBgMsnDhQr7zne8M93AHrK/v04suuojf/OY35OfnD/NIB6+3a3z55Zd57LHHMJlMFBUV8cADD2A0Dmwz3phP1kIIMRaM6WUQIYQYKyRZCyHEKCDJWgghRgFJ1kIIMQpIshZCiFFAkrUYtKOOOorly5dzzjnncNtttxGPq+ko7Xa7eeaZZ7rf//e//80TTzyRsnj33nsv5513Ho8++miPx1etWsWuXYdrXz947777LrfeemtSX1OMDWO+6p5InczMTF566SVisRjf+MY3eOONN3rUIYlGowPeS9qXaDTanaxXrFgBwGmnnZbUGAd79dVXee+990b1sX0x+kmyFkNmMBiYO3cuNTU1PP/886xdu5bOzk5ycnK4/vrr+cEPfoDH42HKlCncd9992O12Tj31VM455xzefvttCgsL+dnPfobD4eCdd97hpz/9KfF4nGXLlnHDDTdQV1fHDTfcQEVFBZs3b6aiooIdO3awfPlyzj//fLKzs9m9ezff/e532bRpEz/84Q+JRCLMnz+fO++8E4PBwIknnsg555zDBx98wKRJk/jVr36FwWDocR2Hi33jjTficrm48MILueWWWzjhhBN6fM7zzz/Phx9+iKZpPPzwwxQWFtLS0sKdd95JS0sLdrud++67j5KSEp566imeffZZwuEwFRUV3HfffRiNRj777DNuv/12LBYL8+bN637txx9/nKeffhqz2cyiRYu47bbblHw9xQg1+KKAYrw74YQTNE3TtEAgoF1yySXamjVrtOeee04788wzNa/Xq2mapl1zzTXaG2+8oWmapj3wwAPaww8/rGmapi1dulT761//qmmapv385z/XfvWrX2mBQEBbunSp1tjYqIXDYW3lypXa+vXrtdraWm3mzJnajh07NE3TtNraWm3FihXd43juuee0n/zkJ5qmadq5556rbd68WdM0TfvWt76lvfTSS5qmadr06dO7y4z+x3/8h/b+++/3uJYjxT7wOg/29a9/XfvVr36laZqm/eEPf9AefPBBTdM07dvf/ra2ZcsWTdM07eOPP+4ul+lyubo/95577tFefvllTdM07bzzztO2bt2qxeNx7YYbbtBuueUWTdM0bdGiRVogENA0TdPcbndfXw4xxsmatRi0zs5Oli9fzsqVK1m8eHF30aiTTjqJtLQ0ALZu3crpp58OwAUXXNCjgM2+JZNly5axYcMGqqqqmDp1KgUFBZhMJs4555zu50+dOpXy8vJex+N2u4nH48yaNQuA888/v/vznU4nxx57LAAzZ86koaGhx+f2Frs3+5ZgZs6cSX19PQDr1q3j1ltvZfny5fzoRz+ipaUFgG3btnHZZZdx/vnn8+abb1JZWdk95oqKCnQ6HWeffXb3ax911FF8//vf55VXXuku5yvGL1kGEYO2b836YDabrfu/+7vOq9Pp0DStx/O1AyohHPiaR9Lb5x+Y7PR6PdFotN+f25t9NdD1ej2xWKz7v1944YVDKq/dcccdPPLII0yZMoUnn3yS5ubmQ+Ic+P7vfvc71q1bx+rVq3nqqadGZYcYkTwysxYpVVFRwZo1a4BEZbUFCxZ0f+y1117r/nfevHlMmTKFyspKmpubiUajvPbaa8yfP/+Q17Tb7fh8vkMez8jIQK/Xs23bNgBeeeWVHvF609/Y/bFgwQKeffZZAOLxODt37gQgEAiQk5NDOBzm5ZdfPuyYV69e3f15TU1NnHDCCfzgBz9I+q4TMfrIzFqk1O23384PfvADfvazn3XfYNynvb2diy66iOzsbB566CGsVit33XUX1157bfdNvmOPPZa6uroer5mdnc20adM4//zzWb58OdnZ2d0fu/fee/mv//ovIpEI8+bN63fZzSPFHow77riDO++8kz//+c9Eo1FWrlxJeXk5119/PRdddBGFhYXMmDGj+/l33303N998M+np6cybNw+Xy0UsFuPmm2/u/qV00003DWosYuyQqntiWJx66qm8+uqro66GuBDDRZZBhBBiFJCZtRBCjAIysxZCiFFAkrUQQowCkqyFEGIUkGQthBCjgCRrIYQYBSRZCyHEKPD/A/hFY6igUDtqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rng = np.random.RandomState(205)\n",
    "x11 = rng.binomial(100, 0.5, 100000)/100 # Getting proportion of heads\n",
    "x12 = rng.binomial(100, 0.55, 100000)/100 \n",
    "\n",
    "sns.distplot(x11, label = 'Fair')\n",
    "sns.distplot(x12, label = 'Biased')\n",
    "plt.xlabel('Proportion of heads')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that there is a great deal of overlap in the potential outcomes over 100,000 repetitions of these experiments, so we have a lot of uncertainty about which model (fair or biased) is the truth. \n",
    "\n",
    "Now, if we up our experiment to 10,000 tosses of each coin, and again repeat the experiment 100,000 times, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "name": "04-python-stat-12"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEHCAYAAACk6V2yAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXRb9Z3//6cWy7ZkS953Z19JiMkGA7QJMBQYloYJpYFACgfIzBxOQ5sCQygBOnRS2tNvCwHKlLb8Sps20CVQhr0sQwKBkgSykT22432TLUuyZFuWdH9/3NgkIY5tWdK9kt+Pc3KcONK970vMyx9/7ue+PwZFURSEEEIkHaPWBQghhIgNCXghhEhSEvBCCJGkJOCFECJJScALIUSSkoAXQogkZY7nyaZPnx7P0wkhRNI4dOjQiN8T14CHyIoUQoixLNLBsUzRCCFEkpKAF0KIJBX3KRohhBiucDhMY2MjfX19WpcSFykpKZSUlGA0RmfsLQEvhNCtxsZG7HY7drtd61LiwuPx0NjYSFlZWVSOJ1M0Qgjd6uvrGzPhDmC326P604oEvBBCJCmZohFCiBPU19dz3XXXMW3aNACuvvpqli1bdtJrXnzxRcrKyjj33HO1KHHYJOCFEAnB5Q/g8Y9++sJuTSHbajnja8455xyeeeaZQf9+6dKlX/pcKBTCZDKNur5okoBPcP1f9MP5ohUikXn8fWytbB/1cS6cnDui/1fWrFlDY2MjXV1drF27lnnz5vHkk08ydepUZs+ezT333ENJSQnZ2dk8+OCDo64vmiTgE1z/F/1Iv2iFEIPbtWsXK1asAOC73/0u8+fPp6qqinXr1vHss8+e9NqGhgaee+450tLStCj1jCTghRDiFP1TNKFQiJ/85Cf8/Oc/x2g00tnZ+aXXTp8+XZfhDrKKRgghBnXw4EEaGxv54x//yMMPP8zptrDW27z7iWQEL4QQg5g4cSKdnZ1861vfYu7cuVqXM2IG5XTfkmJk+vTp0k0yymqcvoE5+PF5Nq3LESKqqqurmThxIhDfVTRaOvGa+0WanTKCF0IkhGyrRdfBrEcyBy+EEElKAl4IIZKUBLwQQiQpCXghhEhSEvBCCJGkZBWNECJy/g7oOf50Z1oWWHPic67RGKLOE7tJ+v1+Lr74YpYtW8bvf/977r777tGf/4Tz/PCHPzxjU7PRkoAXQkSupxOqNqu/n7Q4tgF/4rlGYxh19rcqUBSFa6+9lm9961tRDfd4kYAXQohB9O+u1NLSwr333sszzzzDc889x+bNm3G5XHzzm99k+fLl7N69m3Xr1pGamkp5eTk/+tGPOHjwII8++ijhcJiJEyfyX//1X/h8PlavXk0wGGTKlCkxr39YAf/II4/w+eefEw6Hueuuu1i4cCFr1qyhra2NKVOm8IMf/CBqm8QKIYTW+rtJNjQ0cNFFF5Genj7wd9/85je59dZbCQQCLFmyhBtuuIH333+fO+64g8suu4xwOAzAunXr+PnPf05+fj4//vGP+fDDDzl69Cjnn38+t912Gy+99BK1tbUxvY4hU/nYsWMcPXqUP//5zzzzzDOsX7+eTZs2UVFRwcaNGzGbzWzZsiWmRYrT6+kLseEfNTy/rZbuQEjrcoRIGueccw4bNmzgnXfeobW1lba2toG/e/3117npppu47bbbaGtrw+12s2LFCrZt28Y999zDpk2bADhy5Ajf+973Bv6utbWVY8eOUVFRAcCcOXNifh1DjuDz8vKwWq0Eg0G8Xi/Z2dns2LGDVatWAbB48WK2b9/ORRddFOtaxSl+80EVL+1sAODRNw7w3G3nYjAYZBMQETvhMHTWgBIGgwEC3VpXFFNGo5HMzEz27t078Llf/epXvPbaa4TDYa644goURcFqtbJ27VoALr/8cpYsWcK0adN4/PHHyclR5/v7+vrweDzs3buX+fPnn3TMWBky4G02G0VFRVxxxRX4/X6eeOIJnn766YGdzh0OB263O+aFipMpisJLOxu48dxyUs0mnvy/o7x/uI2LpxfIJiAiNvp64NmvQfMeKP8nqLgBis6O3/nTstQbpNE4zhD6p2iCwSAlJSUsWrSIrVu3AnDhhRdyww03MHXqVBwOBwB/+tOfeOeddwgGg3z1q1/FYrHw/e9/n7vvvptgMIjRaGTt2rVcf/31rF69mvfff18fc/Bbt27F7Xbz97//HafTyX/8x38wbtw4vF4v+fn5eDyegYsU8bOv0UO108clMwo42NzFtRUl/Ozvh7hoWr7WpYlk9cHPoLsTrn4MXv0eTLoovgFvzYntKp3jysrK+OSTT770+f7ljA8//PCX/u6WW27hlltuOelzM2bM4Le//e2XXvvrX/86SpUObcg5+HA4jMPhwGg0kpGRgd/vZ+HChWzerC5X2rJlCwsWLIh5oeJkb+1r5p8m5ZJrSwVg+bnjONzSxcdVo9+zUogvaT0AHz6mhvuki6BwFhx9W+uqxBCGDPgLL7wQn8/H8uXLWbFiBXfeeSdLly5l165dLF++nEAgwKJFi+JRqzjBZ7Uuzp34xWgm22bhillF/Hl7nYZViaQUDsH/3gWz/hWmXqp+btz50Lpf/TuhW0NO0ZhMJn76059+6fPr16+PSUFiaKGwwu46N/+2aPJJn79hYTm3PredO74ycZB3CjFCigKvrgZXNVzxKHRUqTdWcyZDsBfaj2hdoTgDWbyegCrbuujqDXJO2ck3i9QpGwtbj8o0jYgCRYG3H4L9L8OSX0DzXvVJ0mA3pKRB1jho3BnTElJSUvB4PDE9h554PB5SUlKidjx5kjUB7ax1MSnfhsOaQqc/MPB5o9HAZWcV8sFRJ1eeXaxhhSIp7NwA25+Fb70MtlzwNJ7897lToeEzKJwdsxJKSkpobGykvX1sDFpSUlIoKSmJ2vEk4BPQztpO5pZnn/bvLptVxG3PbefSmYVxrkokFUWBj56Ei9ZA+UJ1auZUeVNh+5bja+JjMxlgNBopKyuLybHHApmiSUA7azuZO+70a3nPnZhDislIZVtXnKsSSaX2Y3AdgwkXfjHvfip7GYR6oas17uWJ4ZERfILx9vRxuNU7aMCnmIycPymH/Y1jZ95SxMDOP6rLIZv2qH8+3Xp3ixUyi8FdD5lF8axODJOM4BPMnno3aWYT0wszB33NV6bkcaDZQ/B40yMhRqzmQ5jw1aFflzsFPPWxr0dERAI+weyq6+TsMgdm08n/dIFQmBqnjxqnj9llDgLBMPsaZBQvItDVpk7PDOcp1dyp4G6IeUkiMhLwCWZnrYu55V+envH1Btla2c7WynZMBgOT8m18VDk2Vh6IKGvYofZryRo39GvzpoC7Tr0pK3RHAj6BKIrC7no3FacJ+FNNL7Lzj2oJeBGB+u1QtlDtFjmU3KkQ7AG/fK3pkQR8Amn29NDm7WVO2dDN3aYXZnKs3U9dhz8OlYmkUvMx5E4eXitgaw6k2sEj0zR6JAGfQPbUu8m1WSjNSh/ytTk2C+NzrLx/SJawiRFQFLUdcDisPrE6HI4ydSWN0B0J+ASyp76TOWUODMP50RlYMCFbukuKkemsgUAXOEqH/x5HmToPL3RHAj6B7Kl3M6csC5c/MLBixt83eDe/OaUOtlW7UOQGmBiu5r2QfnzaZbjsx0fw8nWmOxLwCUJRlOMB7xjYsWlrZTu9Zwr4siycXb1UO31xrFQktOa9kD99eDdY+zlK1VG/r23o14q4koBPELUdftzdfcwpG3oFTb8cm4VJ+Ta2VXfEsDKRVPoDfiTScyDFCm0HY1OTiJgEvM71T8d8VNlOiSON/MzUEb1/4fgcdtS4YlSdSDrNeyFv2sjeYzCAvRSch2NTk4iYBLzO9U/H7KxxjWj03m92mYN90pdGDIe/Q71ZOtIRPIC9BNoORb8mMSoS8AniYIuXOeUj39x8VomdIy1eeoOytZoYQsvnYE6HrPEjf6+jVAJehyTgE4CiKBxt7WJ2ycgDfmaRnbCicKRF2geLITTvVTfTNppG/l778aWSvd7o1yUiJgGfADr9ffgDIWYUD95BcjDpFhOT8jP4pLp9YGml64RdoIQY0Lx3eA3GTiejAIwp0LIvujWJUZGATwDNnh4c6Sn4e4JDrn0/nVkldvbWuweWVnr8fTGqVCS00QS80ay2N2jeG92axKhIwCeAFk8P43LS+aiqY8i17yfqbyFc4kjjUIv86CzOINirLnMsmhP5MfKnq20OhG5IwCeAZk8P43NsI35ffwvhQEihqs1HWJ40FINpOwjhEBSeFfkxciZDw6fqahyhCxLwCaDZ3cP4XGvE7y92pNEbDOPs6o1iVSKpNO9Vd2eyjHwgMcBRDm2Hwe+MXl1iVCTgdS5wPJhHE/BWi5n8zFSaOnuiWJlIKqOZf++XOwnCfeCqiU5NYtQk4HWutsOPosC4nMgDHmBSno1G9zDbv4qxJxoBb8kAa6480aojEvA6V+XsIttmIS0lgrXJJ5iUZ6OxUwJenIaiQNMesOVDR9XwNvoYjL1UetLoiAS8zlW1+Siyp436OBPybDS7ZYpGnEZnDQS86s3Rqs3D3+jjdDKLoL0yerWJUZGA17kqp4/CKAT8+BwrvkAIb4+sgRenaN6rTq2kjaAH/GAyCsF1bPTHEVEhAa9zVW1dFDlGH/AF9lQsJiMtHllJI07RvBfyImgwdjoZheBphD6ZDtQDCXgdc3f30dYVoNA+shbBp2M0GCi0p9LskWkacYrmvZA/whbBg7HlAwq0H43O8cSoSMDrWLXTh8kAubbRBzxAoT2NFpmHF6eKZJOPwZhTIbNYOkvqhAS8jlU7uyjOSsdkHMH2aWdQ5EiTEbw42UAP+BnRO2bORHAeid7xRMQk4HWsus1HeXZ61I5XaE+j1dsjLQvEF1o+V7fbc5RH75jZE2UtvE5IwOtYldNHWfboHnA6UX5GKn0hBadXbrSK40bTA34w2RNkBK8TEvA6Vu2M7gg+M82MxWyk1uWP2jFFgmvdDwWjaDB2Oo4ydW29/KSoOQl4nVIURQ34UbYoOJHBYCA/I5Xadgl4cZzzSPRusPazl0KvB7pls3etScDrVKu3F38gRFkUR/AA+Zmp1LpkjbJAHWG3HYK8KC2R7JdZBAYjuKqje1wxYhLwOlXV5iM9xUReRnSWSPbLy7BQ1yEjeAH426GnE/KmRve4phR1mkaeaNWceTgv2rNnD48//jiBQIDFixdz8803s2bNGtra2pgyZQo/+MEPMBrle0U0VTm7mJBnw2iIzhLJfvmZaeypd0f1mCJB1W8HUyqE+iAY5X16sydIwOvAkKkcCAR48skn+cUvfsEf/vAHVq5cyaZNm6ioqGDjxo2YzWa2bNkSj1rHlOo2H5PyRrH5wiBybRZavb30jHBfV5GEWj5Xe9Ac+3B0DcZOFQyox23eK7s7aWzIgN+5cyfp6encdddd3H777Rw+fJgdO3awePFiABYvXsyOHTtiXuhYU+30MTEGAZ9ttQDQIK2DRUc1ZBRE/7iBLggFoPlzdQpIaGbIKZq2tjYOHTrESy+9RFNTEw8++CBpaWnY7WrnOYfDgdstP/JHW7XTx5VnF0f9uOkWExmpZupd3UzOz4j68UUC6ayJTcCDOoLvltG71oYcwdvtdubNm4fVamXy5Ml4vV7sdjterxcAj8eDw+GIeaFjSV8oTG2Hn4n50R/Bg9qyoF7Wwgt3A1jzYnNsa566TDIk7am1NGTAV1RUUF1dTSgUorW1lbS0NBYuXMjmzZsB2LJlCwsWLIh5oWNJvaubYFiJyRw8qJtw18tSybFNUcDbBOnZsTm+NReUMHS1xOb4YliGnKJxOBxcf/31rFixglAoxH333cesWbNYs2YNy5cvZ8qUKSxatCgetY4Z1c4usq0pZFktuP3RHwEV2dNkqeRY53NCsAesObE5fooVzGngro/N8cWwDGuZ5HXXXcd111130ufWr18fk4KEugY+FjdY+xU50vjgiDNmxxcJwF2rPoyUlhWb4xsM6jcPT0Nsji+GRRav65C6giZ2N0CL7TJFM+Z11qo3WKPZZOxU1jwZwWtMAl6Hqp0+JsXoBiuoI3hnl6yFH9M66yCzJLbnsOaqN3KFZiTgdShWa+D75djUtfD7mzwxO4fQuc5asEd/Ge5JrLngkRG8liTgdcYfCNLk7olpwAOkp5g40uyN6TmEjrnrwB6HEbynMbbnEGckAa8zx5zq6pYJubEN+GxbimzfN5Z11qp7p8aSNRd63NAtT7NqZViraER8uPwBtlW3U5CZSuvx8PXHaJ4822qhWTbgHpsU5fgUTQn0xfBrID0HMKhPzKbHaLWOOCMZweuIx9/H1sp2ihxpbK1sZ2tlO72xDHgZwY9N3S61X0ysb7KazGDLB1dNbM8jBiUBrzNOby8ljrSYnyfbmkKTjODHps5awKBuzBFrjlJ1BC80IQGvM86uXkqyoruL0+nIFM0Y5q5T599NKbE/l71ERvAakoDXGWdXIC4Bn2Wz0Nndhz8QjPm5hM501kJWeXzOZS+VjT80JAGvI+7uPrr7QpRmxWeKBpAnWseizjrIGhefc9llikZLEvA6UufyYzRAQWbsAz7VbCIrPYVPj3VQ4/Th8kd5yzahX5214IjXCL5EPZ+ixOd84iQS8DpS39FNji0VkzG6+7AOpsCeOrBaxxODrpVCp9y18RvBO0rVrpXSNlgTEvA6Ut/pJ/d4G4F4KLSn0eGTkfuY0xnHgLcVgDFFbrRqRAJeRxo7e8jJiF/AF0nAjz09bvVXvAI+HFKXY9Zvlw24NSABryONnd1xHsGnSsCPNZ116kdHWXzOF+iClHSo2SobcGtAAl5HGt095FjjO0Xj8gdQ5AbY2NGyT+0R422CQJxWUFlzwd8en3OJk0jA64S3pw93d99AK994KLKn0RdS8PbKWvgxo+MoWDKgajME4xXwOTI9oxEJeJ2oPb5HanYcAz7HZsFkNOCSaZqxw9MYu422B5MuI3itSMDrRG27n/wMCymm+P2TmIwGsq0pMg8/lniaYrfR9mCsOer8e1h+Uow3CXidqO3wUxyHFgWnyrFZJODHEm/j8Ta+cWTNBSUMXlkLH28S8DpR0+GnxBH/gM+2SsCPKZ6m+Ae8JQNMFvDI/qzxJgGvE3Udfkri0IPmVDk2Cx3SpmBs6O1Sp0riPUVjMKjnlICPOwl4nahp98eli+SpcmwWuck6VriPr4GP901WUKdp3LIBd7xJwOtAMBSmobM7Lht9nCrHZsHTE4zZzlFCRzprIS0LzKnxP7ct//hGIyKeJOB1oLGzh1BYoVSDEXz28QermmT7vuTnOha/LpKnkoDXhAS8DtR2+LFZTDjS47DDzinSUkxYLSaaOiXgk15HtdrdUQu2AjXgw2Ftzj9GScDrQE2Hj/IcKwZDfNoEnyrXZqG+06/JuUUcuY7FrwfNqWz5attgb5M25x+jJOB1oLbDz/hcq2bnz8tIpbZDdnZKeloGfJodzGnQUanN+ccoCXgdqG33My5Hu4DPz0yl3iUj+KSmKGrA2zWaojEY1RbF7Ue1Of8YJQGvA7Udfsbl2jQ7f15GKnUdEvBJratFbS6m1Qgejge8jODjSQJeY4qiaD6Cz8tMpa0rgE+6SiYv1zH1aVJbvnY1ZI2XgI8zCXiNdfr78PYGGa9hwOfaLBiAaqdPsxpEjLmOqQFrNGlXg0zRxJ0EvMZqOvwYDWjyFGu/FJORQnsalW1dmtUgYqyjGrInaFtD9jj1G01IflKMFwl4jdV2qC0KLGZt/ynG5aTLCD6ZuY5pH/BZ4yHcB2554CleJOA1Vtvu03T+vV95jpWqNgn4pKWHgDdbITUTjm2VHZ7iRAJeY1qvge9Xnm2lyilTNEnJ36Fu1ZeSHr99WE+nz6c2Oqt8VzbgjhMJeI3VtPsp18EIflyOleo2n2zAnYy8TeBzgrc5fvuwDsZWoNYi4kICXmPH2n1M1HANfL/y7HR8gRAtnl6tSxHR5mlUP1pzta0DwJYHvlatqxgzJOA15OsN0uLpJdVspMbpw69hy968zFTSU0xUyUqa5OOuV+e+tWgTfCpbAXS1aV3FmDHsgN+xYwfTp0+no6OD7u5uvvOd77B8+XIeeughwtIhLiLH2tWbmnWubrZWtmvak91oMDAxz0aVrKRJPu56fYzeATLyodsFQdlkJh6GHfC/+93vmD17NgCbNm2ioqKCjRs3Yjab2bJlS8wKTGbVTh8FmamkmLT/QSoQClNkT+VAk0frUkS0eXQU8LZ8QPlidykRU8NKlvfee4/58+djtao3A3fs2MHixYsBWLx4MTt27IhdhUnsmNNHebZ2DzidyNcbBIOBo60yRZN03A36CXhzGqTaZfOPOBky4MPhMM8//zw33njjwOc8Hg92ux0Ah8OB2+2OXYVJrMrpozRb+xU0/aTpWJJy14M1T+sqvmDLk4CPkyED/pVXXuGSSy4hNfWLGzR2ux2v1wuoYe9wOGJXYRLT0wgeIC/DQrOnh96g7M+aNMJh8OhoBA/Hd3eq0bqKMWHIgD98+DBvvfUWt99+O4cOHWL16tUsXLiQzZs3A7BlyxYWLFgQ80KT0bF2P2W6CvhUwoq6Nl8kia5mCAV0FvCyP2u8mId6wb333jvw+xUrVvDYY4+Rnp7OmjVrWL58OVOmTGHRokUxLTIZ1bT76PAF1Fa9Xn2sKEhLMZFrs1DV5mNaYabW5Yho6G8TnGbXupIv2PKh7mOtqxgThgz4E23YsGHg9+vXr496MWPJgUYPRgNkp6foJuDheE8aaVmQPDqq1V2cDNqv1BqQka8+zdrbBakZWleT1HT0rz621Ln8ZFstmHWwRPJE43LSpelYMtFyH9bBWPMAg+zPGgf6SpcxpN7VTW6GResyvqQ82ypPsyYT1zFwaLQP62BMKZBZJJt/xIEEvEbqXN3kZejg0fFTqF0lZQSfNFzHwK6zETwc376vSusqkp4EvEYaXH5y9RjwOel0+vtw+fRzX0CMgqtaf1M0ANnjZQQfBxLwGlAU5fgIXn9TNEWONFJMBrnRmgx6u8DXps+Azxonc/BxIAGvAWdXAH8gRJ5NfyN4s9HI+FwblXKjNfH1P0xkL9G2jtPJLAHnYdnZKcYk4DVQ7fRhMRlwWFO0LuW0JuXZZCVNMuiohowidScnvbHmQo9bvUcgYkYCXgPHnD5Ks9IxGgxal3JaE/NtspImGehhH9bBZBaBwSQtC2JMAl4D1e36ajJ2qsl5GbKSJhnoOeCNJrDlgksCPpYk4DVQ3aavJmOnmpRvo6bdRygs+7MmNFe1fgMepOlYHEjAa+BYu09XTcZONSk/g76QQr1Lmo4ltPZKsNggoPFG24OxFcgcfIxJwMdZOKxQ7fRRrtMpmkAojLe7D3uamd31nVqXIyIVDql94H1tENRpwGcUyBRNjEnAx1mTp4feYFi3I3hfb5Ctle1kWS0cavJqXY6IVGcNhPvUENWrjAJ1676w7D8QKxLwcVbd5iMj1UyOTX8POZ0oL8NCnUzRJC7nUXVrvBSb1pUMzpav9qqX3vAxIwEfZ1XOLibm2TDodIlkP3X7Pp3+aC+G1n5UbQeg568zSwakZkrLghiSgI+zqjYfk/J1PKo6Li8jVUbwiaz9iL5X0ID6zSdrAjiPaF1J0pKAj7Mqp4+JefoP+PzMVJxdAbp6g1qXIiLRflTt2Kh30nQspiTg46za2cWkfP3vYpNrs2A0qPcMRAJyHp+i0bvs8epPGyImJODjqKcvRL2rm0kJMII3m4wU2tP4pLqdGqcPl1/aByeM3i7wNup/igbUnzKcMoKPFQn4OKpp96MoJMQUDUCxI42PKtvZWtmOx9+ndTliuJyH1D1YHeVaVzK07AnqN6Ne6X0UCxLwcVTt7KLQnootdUR7nWumJCsdZ1ev1mWIkWrZD7lTway/dtRfklUOGGQePkYk4OOoss3HpDz9z7/3K81Kx+mVgE84Lfug8Cytqxgec5r6k4YEfExIwMdRtdPHxARYItmvNCsdZ1cARZGmYwmldR8UztK6iuHLmyJLJWNEAj6Oqtq6EuIGa7/SrDQCoTCeHlkqmTAURR3BFyRQwOfPhLYDWleRlCTg46jamRgPOfXLsVmwmIy0yTRN4uhqBX974ozggwHIyIfGXbJ9XwxIwMeJyxfA5e9LqDl4g8FAXoZFbrQmkpbPwZKpbmqdCAJd0ONV+9F4m7SuJulIwMdJlbOLFJNBt10kB5OXmSoBn0ha96s3WPXcg+ZUmYXqx44qbetIQomxXi8JHGz2MiHPRoNLbeDl70uMFql5Gamy8UciafgMHGVqWOp1o49TmSxqZ0lZSRN1MoKPk4NNXibk2th6/MGh3gQJ+PyMVJmDTySt+9WPVZv1u9HH6diLZSVNDEjAx8nBZg+TE2gFTb+8zFQ6/X30BhPjG9KYFgqqI/fMEq0rGbnMEulJEwMS8HGgKAoHm7xMToAmY6fKz1CfhpTe8Amgo1LdQCOzWOtKRs5eLD1pYkACPg4aOrvx9gaZnEBLJPtZzEaybRaqndJVUvdaPoeMIrDoc7/fM8osge4OdZmniBoJ+Dg42OQlx2bR/TZ9gynMTJWATwTNn0PeVK2riIw1B1LS1Ye0RNRIwMfBwWYPM4oydb9N32AK7WlUt0vA617TLihIkB40pzIYIWeKBHyUScDHwYFmLzOK7FqXEbFCe5qM4PVOUaBxJxTM1LqSyOVN/WIVkIgKCfg4ONjkYUZxptZlRKzAnkqTuwefbN+nX5010O1K8ICXEXy0ScDHWE9fiGqnj5kJPILPz0jFbDRwsNmrdSliMI071dUzGQVaVxK53KnQdhDCsiQ3WiTgY+xIi7pTzdTCxFsi2c9sMjIxz8b+RrfWpYjBNO6CkrlaVzE6eVMh2CMtC6JIAj7GPq3toCw7nRZ3T8K0JzidaYUZ7G2QgNetxp1QfI7WVYyOOR1sBVD5nnSWjJIhA76yspIbb7yRm266iRUrVlBXV0d3dzff+c53WL58OQ899BDhcDgetSak3XVusqyWhGpPcDpTCzL5vMGjdRnidBQlOUbwgS51ueTRd6CnU+tqksKQAZ+dnc0zzzzDH//4R1auXMkvfxtkbgAAABfMSURBVPlLNm3aREVFBRs3bsRsNrNly5Z41JqQjrR4Kc1KrA6SpzOtMIPDLV5pWaBHjTuh1w3W7MRpMDYYewl4GrWuImkMGfA5OTnY7eoNQrPZjMlkYseOHSxevBiAxYsXs2PHjthWmaBCYYWjrV2UJEHAj8u1oigK24/Jj866U/cPSMtSH3RKpAZjp5NZIn3ho2jYc/Dd3d088cQT3HLLLXg8noHQdzgcuN0yN3s6lW1d9ATDlDgSP+CDIYUCexrbqyTgdaf1gLpxdTKwl6g7UgXkuYtoGFbAB4NBVq9ezR133MHkyZOx2+14veqSOY/Hg8PhiGmRiWpvvZvSrDTSLSatS4mKcTlW9jXJPLzutOyHrCQJ+IxCMKao37TEqA0Z8Iqi8MADD7Bo0SIuvfRSABYuXMjmzZsB2LJlCwsWLIhtlQlqb4ObaYWJ+4DTqcblWNnXKAGvK+EQtO5LnC36hmI0gaNUHniKkiED/oMPPuDNN9/kjTfeYMWKFaxbt46lS5eya9culi9fTiAQYNGiRfGoNeHsa3QzrSC5Ar7J3UOrt0frUkS/1gPQ1wNZ47WuJHqyxknAR8mQW/YtWrSI3bt3f+nz69evj0lBySIUVtjX6OGGheX0BhWty4mKHJuFrPQUPqvp5IrZRVqXIwDqt0PORLUTY7JwjIPKd7WuIinIg04xsqvOhT8QojwnAXtzD8JgMDCz2M77h1qpcfpw+QNalyTqd0DR2VpXEV1Z49WVNF1tWleS8CTgY+SzGhfZ1hQspuT6TzylwMbHVeq+sh5/n9bliPrtUDRH6yqiy5YHqZnQ+JnWlSS85EofHTnckhzr3081oyiTBlc3QXl6WXvdneA8BMVJFvAGAxTMgoZPta4k4UnAx8i+Rg/l2ckzPdNvSn4GYUWhqVNutGqu4VOwZEL2RK0rib6i2dAgI/jRkoCPgZ6+EIdbvIzPTb6AT00xUexIp7bDr3Upon47lM5TlxYmm4Kz1G9gSnIsUNCKBHwM7Kl3YzCQlFM0oC6XlIDXgfrtULZQ6ypiI2+6ugn3sQ+ls+QoSMDHwKc1LqYXZpKSZDdY+0nA60A4rK6gSdaAT0mDNAfs+bN0lhyF5Ewgje041sHskuRt3zAux4q7u08eeNJSw6dq8NnyEr+D5GCyJ4DrmNZVJDQJ+CjrC4X5pLqDueOytC4lZrKsKWSmmqVtgZaq/k/t29K8N/E7SA4mexK4ZHen0ZCAj7LddZ0EgmEqypI34A0GA+NypS+Npuq3q3uYJrOcSeBugN4urStJWBLwUfbBESfzxmclTQfJwUzItbGrTuZGNaEo6hRN7hStK4ktewmYLOpPKSIiEvBR9sGRNr46NV/rMmJuUr6No61duOVp1vhzHlF7pudO1rqS2DKa1Hl4eaI1YhLwUdTm7WVnXScXTy/QupSYK7SnkZFmlh2etFD1vjp6T02eTqWDyp2srhYSEZGAj6K397dQnm1lZnHy/49nNBioKMvi46p2rUsZe468BRO+onUV8ZE3FVo+l3n4CEnAR9Fb+5q5fFYhBoNB61LiYsH4bN4/1Kp1GWNLwAfVH8D4C7WuJD4c48CUCrUfa11JQpKAjxKXL8DWo07OKc+ixunD3xfSuqSYu2ByLpVtPqqdsn9mXPg7YO9fwJQCudO1riY+jCYona9OS4kRk4CPklf3NpGfmUqnv4+tle30joGAL7SncVaxnXcPtGhdytjQ0wm7X1Dn35UxdHN73HlQ+Z7WVSQkCfgo+dvOBr42c+xMzwAEQmEWTsjm1T1NWpcyNoSD0LwHis/RupL4Kj8fWverU1PSl2ZEJOCjoLbdz6c1Lr52VqHWpcSVrzdIltXCrrpO6qQ3Tew1fArBABTM1LqS+Ep3QGYxbH9W+tKMkAR8FLywvZYZRZnkZaZqXUrcFdrTmFaYwf/ubtS6lOR35G21ja557H2dUThbXU0jRkQCfpQUReHVPY1MKcgYE/Pup3PJjAL+tL2OY21dsk9rrISC6jx0SYXWlWijeA60HwWf7NM6EhLwo7S3wU2Dq5uzS5O3e+RQzpuYQ12Hn7982iD7tMZK7UcQ8Ktb2Y1F9jLILIRDb2hdSUKRgB+ll3Y2MH98DplpKVqXoplsq4UpBRnsqnNpXUry2vc3mHDh2JyeAXWf1rJz4cArssvTCEjAj0IwFOaV3Y1cNsZurp7OOeVZ7Kp30xeSzbijLhiA/S/DlEu1rkRbpfPV/vB127SuJGFIwI/Ch0ed+AMhvjIlT+tSNDe71IGiKLx/SOZIo+7Q6xDug8kXa12JttIcMPUy+PhJrStJGBLwo/DyrkYun1WU9K2BhyPFZGThhBz++lk9ivwIHV2f/Q7m3ADmNK0r0d6cG+Dga1D/qdaVJAQJ+Aj5eoO8+Xkz184t1boU3finSbkcafGyU/rER0/rQfUx/fm3aF2JPjhK1Q25t/xE60oSggR8hF7f24Qt1cyFk3O1LkU3HOkpLJ6Wz2+3HtO6lOSx+Scw+RJISU/evVdHauY16jMBDdInfigS8BFQFIXnPjrG8vPGYTbJf8ITXTevjDf2NtHQKWE0aq0HYP/fYMHtULU5efdeHSl7CUy/Et55WFbUDEHSKQI7alwcavZy0dS8MdM5crimFmYwq8TOj18/IA89jdb7P1aDrGCG1pXoz/l3Qu0ncPRdrSvRNQn4CDzx7hEunVnAwZauMdM5crj8gRDnTszltb1NHJRNuSPXsg8O/C9ctEbrSvQpPRcqlsGba6BLVm4NRgJ+hD6qdPKPqnZuPX+C1qXo1sQ8G5PyMvjdxzVal5KYFAVev1dd926xydz76QS61JutnnrY/Uetq9EtCfgRaPX2sPalz7lmTgkOm0XrcnTt0rMKeXt/M5VtstXaiB34X7Vz5LjzZe79TCw2mPI1+OgX0OvVuhpdkoAfgWc/qKbN28vsUodMywxhXI6V8yfn8b0/7aLd16t1OYnD0wivroZz/w3Ss7WuRv8mLlKD/v0fa12JLknAD9Peejf/39Zqvn5OCWkp8mDTcNx24QQOt3Txmy1VWpeSGPp64K+3QdEcWfc+XEYzLPpP+OSX8Pkm2RDkFBLww1Dv8vNvG3Zw1dnFzCoZu10jRyrHZuH6+WX86oNq/r6vWety9C0cgr/eCp4GuGQt9MlPPcOWPw0mfBVeuxtc1VpXoysS8EPYWeti2TP/YHapg1UXT9G6nIQzo9jOqoun8O2NO3l5V4PW5eiTosCb96vL/uZ+C5r3yrz7SM24CtKy4JXvqm2VBSABP6hQWOHZD6v55jMfc9H0fO67fDqBsDxUEYmr5hTz3Uuncs9fdnPfpj30BuX+xYBwGN64D3ZthCVPgk0a10XEaIaFd6g3Wzf8q0zVHCcBfxrbqju46okPWP/OYR666ixu/qfxbDvmkhurEerfu3XlVyex+VAr1/3PR+xrdGtdlvacR+B316hzx7e+om7HJyJnscG1v1R/+vnFueo3zTH+pGvEAf/CCy9www03cPPNN1NTk/jrnXv6Qry0s57r/ucjbvz1Pzir2M5d/zwVk8kowR4lZdlWnr5pHuXZVpY8tZVHXz9Am3eMzTUrChz7UL2Z+ovz1FC68Xm1Fa6sdx89k1md5ipdoK5G+tViOPAqhMbmTmPmSN7U2dnJpk2beOGFF9i/fz8/+9nPeOKJJ6JdW8z19IX4pLqDd/a38OqeRgwGA1+vKOHer03DYbOws1a6Ikab0WDgq1PzuWh6Pr/aUsVvPqzm7FIHF03P5ytT8phd6ki+VUrdLrUx1tF34fAb4K6HaVfA9b+FnClQf3wDi6Kzta0zWRhNMO1yuOAu2PM8vHgHGFOg/FwomQsl89SP9mKtK425iAJ+9+7dnHfeeZhMJs4++2yOHTsW5bJOFgyFaevqRVEgrCgoCgO/V38BqB/DikJPXxhfbxBvT5Cu3iBdPX109Qbx9gbp6gni6QlypMXL0dYuLGYj50/O5dsXT2HR1HyCisLO2k7Okh7vMTW9MJPbLpxIQ2c33X0h3j/Uxi/+7yjhMEzKtzEhz0ZBZioFmWkU2lPJzUgl1WzEYjaSYjIO/N5kNGA0GDAZDBgMYDIasKenkJEa0Ze2+oXlbYZwEJTw6X+FQ8d/H1J3Wwp2q0sc+z8GutQNolsPQNsh6GqGFBuMPx/mLIOpXwNTmhrsdhm1x4zZooZ54WwwmKDtANR8BNt+DT2dYM0FRxnYSyGzWG1iZsuHNLv6E1WqXZ3bN5rU9w98NJ7w52FOgqTaITUjttd7GhH9X+DxeLDb7QN/jvUGD//zfiU/e/vwsF9vAKwWE7ZU80kfrRYzGcd/f9WcYibn2ZhSkEFIgZ21nWyvcXFWcWbsLkScxGAwUJZt5aziTKYWZNIXKiXXZqGm3Uezp4c2b4AjLV7afQFc/j76QmH6gmH6QmFCZ/iSm1GUyZvfXRRZUQdegT+vGP7rjWZ1Iw5zqvrLlAopVsgqg8JZcNa/Qs4EtY95sE8N9aY9MlqPJ5NF/e9tNEHFjep//+4ONXS7O8DXBl2t0LIXujuhtwsCXvUjUcq20vmw8r3oHGsEIgp4u93O4cNfBK7ROPyp/OnTp0dySka61XAQcB//JZKLmTN/4VYD0/9nNGcoivB9YaD7+K92YPdoihCash3/FS0N8P8iy77RiCjgKyoqePrppwmFQhw8eJDx48cP632HDh2K5HRCCCEiEFHAZ2Vlce2113LTTTdhNptZt25dtOsSQggxSgZFdkgWQoikJA86CSFEkpKAF0KIJCUBL4QQSUoCXgghklSEj/tp54UXXuBvf/vbwOqdU5dohkIhrrrqKq6//npuv/12AB555BE+//xzwuEwd911F4sWRfgQjIbOdN1r1qzh8OHD2Gw2Jk6cyCOPPDLkexLBSK+5srKStWvXYjQaMRqN/OhHP6K8vFzDK4hMJP/WADt27OCmm27i448/JicnR4vSIxbJNe/Zs4fHH3+cQCDA4sWLWblypVblRyyS6x5RnikJxOVyKd/4xjeUYDCo7NmzR1m1atWXXvOnP/1JWblypfKb3/xGURRFqa6uVlasWKEoiqI4nU5l6dKlca05Goa67vvuu0/Zs2fPiN6jd5Fcc3t7u+J2uxVFUZTNmzcr3//+9+NWb7REct39vv3tbytLly5V2tvb41Fq1ERyzb29vcodd9yh+P3+eJYaVZFc90jzLKGmaIbqgdPb28t7773H5ZdfPvC5vLw8rFYrwWAQr9dLdnbi7XM5nN4/jzzyCCtWrODDDz8c9nv0LJJrzsnJGWihYTabMZkSr59QJNcN8N577zF//nysVmscq42OSK55586dpKenc9ddd3H77bef9GR9oojkukeaZwk1RTNUD5wNGzawbNkyXC7XwOdsNhtFRUVcccUV+P3+hOx6OdR133fffWRnZ9Pe3s6tt95KRUVF3PsFRVsk15yZqfYR6u7u5oknnkjIB/AiuW6bzcbzzz/PU089xbvvvhvvkkctkmtua2vj0KFDvPTSSzQ1NfHggw+ycePGeJc+KpFcd0ZGxojyLKFG8Ha7Ha/XO/DnE3vgeL1etm3bxsUXX3zSe7Zu3Yrb7ebvf/87f/vb3/jRj34Ut3qj5UzXDQx8F8/NzWX27NlUV1cP+R69i+SaAYLBIKtXr+aOO+5g8uTJ8Ss4SiK57ldeeYVLLrmE1NSRdmzSh0i/vufNm4fVamXy5MknvT9RRHLdI82zhPq/vqKigm3bthEKhdi3b99JNySqqqpwuVzcfvvt/Pa3v+Uvf/kLW7duJRwO43A4MBqNZGRk4Pcn3n6NZ7puYOCLpLe3l/3791NaWjrke/QukmtWFIUHHniARYsWcemll2pR9qhFct2HDx/mrbfe4vbbb+fQoUOsXr1ai9IjFunXd3V1NaFQiNbWVtLS0rQofVQiue6R5lnCtSp4/vnnefnllwfuOtfU1NDV1cWVV1458JoXX3xxIOxDoRBr1qyhoaGB3t5ebrnlFr7+9a9reAWROdN1r1y5Eq/XSzAY5IYbbuAb3/jGad+TaCE/0mvesmULq1atYs6cOQDMmDGDBx54QOOrGLlI/q37rVixgvXr1yfcKppIrnnTpk1s2rSJUCjEvffey4IFCzS+ipEb6XWPNM8SLuCFEEIMT0JN0QghhBg+CXghhEhSEvBCCJGkJOCFECJJScALIUSSkoAXUTVr1iyWLFnClVdeyQMPPEA4HI7LeT0eD3/5y18G/vzuu+/y+9//PmbnW7duHVdffTW/+c1vTvr8ihUrqKysjOq5tmzZwpo1a6J6TDE2JFSrAqF/WVlZvPzyy4RCIW699Vbefvvtk3oDBYNBzOboftkFg8GBgL/++usB+Od//ueonuNUb7zxBh988AEGgyGm5xFiNCTgRUyYTCYqKiqora3lxRdfZOvWrXR2dpKXl8edd97J/fffj9frZdKkSTz66KNYrVYuueQSrrzySt5//32Ki4t57LHHyMjIYPPmzfzsZz8jHA5z+eWXs2rVKurr61m1ahUzZsxg3759zJgxg8OHD7NkyRKuueYacnJyqKqq4p577mHPnj384Ac/oK+vj3nz5vHQQw9hMpm48MILufLKK/noo48YN24cTz311JcalJ3u3HfddRcul4trr72W++67jwsuuOCk97z44ot8/PHHKIrC008/TXFxMa2trTz00EO0trZitVp59NFHKS8v5/nnn+evf/0rgUCAGTNm8Oijj2I2m9m5cydr164lNTWVuXPnDhz7ueee44UXXsBisXDeeecl5INcIo5G3fNSiBNccMEFiqIoSnd3t/KNb3xDee+995RNmzYpl112mdLV1aUoiqLccccdyttvv60oiqL85Cc/UZ5++mlFURTl4osvVjZu3KgoiqKsX79eeeqpp5Tu7m7l4osvVpqampRAIKAsW7ZM2bFjh1JXV6fMnDlTOXz4sKIoilJXV6dcf/31A3Vs2rRJ+elPf6ooiqJcddVVyr59+xRFUZTvfOc7yssvv6woiqJMmzZN2b59u6IoivLv//7vyocffnjStQx27hOv81Q333yz8tRTTymKoijPPvus8vOf/1xRFEX57ne/q+zfv19RFEXZtm3bQGtYl8s18N4f/vCHymuvvaYoiqJcffXVyoEDB5RwOKysWrVKue+++xRFUZTzzjtP6e7uVhRFUTwez1D/HGKMkzl4EVWdnZ0sWbKEZcuWcf755w80f/vKV76CzWYD4MCBAwO9Yr7+9a/z6aefDry/fzrn8ssv57PPPqO6uprJkydTVFRESkoKV1555cDrJ0+ezNSpU89Yj8fjIRwOc9ZZZwFwzTXXDLzfbrcPPN4+c+ZMGhsbT3rvmc59Jv3TQzNnzqShoQGATz75hDVr1rBkyRL++7//m9bWVgAOHjzIjTfeyDXXXMM777zD0aNHB2qeMWMGBoOBf/mXfxk49qxZs/jP//xPXn/9dSwWy5C1iLFNpmhEVPXPwZ8qPT194PfDnbc2GAwoinLS65UTOmuceMzBnOn9Jwak0WgkGAwO+71nkpKSMnDMUCg08PuXXnrpSx0DH3zwQZ555hkmTZrEhg0baGlp+dJ5Tvzzr371Kz755BPefPNNnn/+eTZs2DCsmsTYJCN4EXczZszgvffeA+DVV19l/vz5A3/31ltvDXycO3cukyZN4ujRo7S0tBAMBnnrrbeYN2/el45ptVrx+Xxf+nx/572DBw8C8Prrr590vjMZ7rmHY/78+fz1r38FIBwOc+TIEUDtXZ+Xl0cgEOC11147bc1vvvnmwPuam5u54IILuP/++6O+WkckHxnBi7hbu3Yt999/P4899tjATdZ+7e3tLF26lJycHB5//HHS0tJ4+OGHWbly5cCNzgULFlBfX3/SMXNycpgyZQrXXHMNS5YsOamb4rp16/j+979PX18fc+fO5aqrrhpWnYOdOxIPPvggDz30EH/4wx8IBoMsW7aMqVOncuedd7J06VKKi4uZPn36wOsfeeQR7r77bjIzM5k7dy4ul4tQKMTdd9898I0s0doCi/iTbpJCNy655BLeeOONhN24Qgi9kSkaIYRIUjKCF0KIJCUjeCGESFIS8EIIkaQk4IUQIklJwAshRJKSgBdCiCQlAS+EEEnq/wcun8djwEIXhgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rng = np.random.RandomState(205)\n",
    "x21 = rng.binomial(10000, 0.5, 100000)/10000\n",
    "x22 = rng.binomial(10000, 0.55, 100000)/10000\n",
    "\n",
    "sns.distplot(x21, label = 'Fair')\n",
    "sns.distplot(x22, label = 'Biased')\n",
    "plt.xlabel('Proportion of heads')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now find almost no overlap between the potential outcomes, so we can very easily distinguish the two models. This is part of what gathering more data (number of tosses) buys you. \n",
    "\n",
    "We typically measure this ability to distinguish between two models using concepts of *statistical power*, which is the likelihood that we would find an observation at least as extreme as what we observed, under the **alternative** model (in this case, the biased coin model). We can calculate the statistical power quite easily for the two sets of simulated experiments. Remember, we observed 54% heads in our one instance of each experiment that we actually observed. By doing simulations, we're \"playing God\" and seeing what could have happened, but in practice we only do the experiment once (how many clinical trials of an expensive drug would you really want to do?). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "name": "04-python-stat-13"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The p-value when n=100 is  0.18\n",
      "The p-value when n=10,000 is  0.0\n",
      "Statistical power when n=100 is  0.54\n",
      "Statistical power when n=10,000 is  0.98\n"
     ]
    }
   ],
   "source": [
    "pval1 = np.mean(x11 > 0.54)\n",
    "pval2 = np.mean(x21 > 0.54)\n",
    "\n",
    "power1 = np.mean(x12 > 0.54)\n",
    "power2 = np.mean(x22 > 0.54)\n",
    "\n",
    "print('The p-value when n=100 is ', np.round(pval1, 2))\n",
    "print('The p-value when n=10,000 is ', np.round(pval2, 2))\n",
    "print('Statistical power when n=100 is ', np.round(power1, 2))\n",
    "print('Statistical power when n=10,000 is ', np.round(power2, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So as *n* goes up, the p-value for the same experimental outcome goes down and the statistical power goes up. This is a general rule with increasing sample size. \n",
    "\n",
    "This idea can be used to design a two-armed experiment. Suppose we are looking at the difference in proportion of mice who gained weight between a wild-type mouse and a knockout variant. Since mice are expensive, let's limit the number of mice we'll use in each arm to 10. We expect 30% of the wild-type mice to gain weight, and expect a higher proportion of the knockouts will gain weight. This is again the setup for a binomial experiment, with the number of \"coin tosses\" being 10 for each of the arms. We're going to do two sets of experiments, one for the WT and one for the KO, and see the difference in proportions of weight gain ('heads') between them, and repeat it 100,000 times. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "name": "04-python-stat-14"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD5CAYAAADBX4k8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAcd0lEQVR4nO3df3DT9eHH8SdpakMKSVsUdBtmEyrFH8fsD9cbru0dDHDi+ePo0ZRfmxb0j9KcekBRcc4hRaretTo26QB33dk6TvwxlR43t0uRGwdtRx1QqMJaf03KbGlSgmlLPt8/OHPfXsH+SH8An9fjv7w/7ySvd679vJJP8knGGIZhICIipmQZ7QAiIjJ6VAIiIiamEhARMTGVgIiIiakERERMTCUgImJi1r4mHD9+nCeffBKLxYLFYmHDhg1cffXVFBYWcurUKaZOncrTTz+NxWKhvr6eoqIiDMNgxYoVzJo1C4CSkhL27dtHbGwsmzZtIiEhgdbWVlavXs2ZM2dIT0/H4/H0K/C0adMiW7GIiAkdO3bswhuMPnz99ddGe3u7YRiG4fV6jccff9woLy83tm7dahiGYfzmN78x/vGPfxiGYRgLFy40vvrqK6Ojo8O49957ja6uLuPYsWPGQw89ZBiGYezatcvYuHGjYRiGsXHjRqOqqsowDMN46KGHjGPHjvUVxTAMw7jxxhv7NU9ERM77rv1mn4eDEhIScDgcAFitVqKioqipqSEzMxOAzMxMampqCAaDhEIhJk2aRGxsLC6Xi+bm5h5zs7KyqKurA6C2tjY8npGRQW1tbcRNJyIiA9Pv9wTOnj1LaWkpy5Ytw+fzhYvB6XTS3t7O6dOnGT9+fHi+w+Ggvb29x1ybzUYgEAjfns1m63EbIiIysvpVAt3d3TzyyCPk5eUxZcoUHA4Hfr8fAJ/Ph9PpJC4ujo6OjvB1/H4/Tqezx9xgMIjdbgdg7NixBIPBHrchIiIjq88SMAyDJ554goyMDGbPng1AWloaXq8XgOrqalJTU4mJicFisdDS0kIgEKCpqQmXy9VjrtfrJTk5GYCUlJQe46mpqcOyQBERubg+Px20Z88eqqqq+PLLL9m1axdJSUk8+uijFBYWkpuby9SpU8nIyACgsLCQgoICQqEQ+fn5WK1WEhMTSUpKwu12Y7fbKS4uBmD58uWsWrWK7du3k56eTmJi4vCuVEREehljGJfXt4hOmzbt4h91EhGRXr5rv6mTxURETEwlICJiYn2+JyAi/dMW6MQX6Oo17rBHE2+/ahQSifRNJSAyRHyBLvYe/7rX+MwpE1QCcsnS4SARERNTCYiImJhKQETExFQCIiImphIQETExlYCIiImpBERETEwlICJiYioBERETUwmIiJiYSkBExMRUAiIiJqYSEBExMZWAiIiJqQREREysz98T6OzsZOnSpXzyySesX7+eefPmsWXLFvbs2QNAU1MTy5cvZ+nSpSxZsoSuri6io6NJS0ujoKAAgJKSEvbt20dsbCybNm0iISGB1tZWVq9ezZkzZ0hPT8fj8QzvSkVEpJc+S8BqtVJaWsrrr78eHluxYgUrVqwA4L777mPOnDnhbZs3byYhISF8ubGxkYaGBioqKqiqqqKsrIw1a9ZQVlZGdnY2c+fO5eGHH6axsZEbb7xxKNcmIiJ96PNwkMViYeLEiRfcdvz4ccaNG8e1114bHsvPz+eBBx7g0KFDANTU1JCZmQlAVlYWdXV1ANTW1obHMzIyqK2tjWwlIiIyYBH9vORf//pX5s+fH75cWlpKfHw8zc3N5Ofn88477+Dz+Zg8eTIANpuNQCAAwNmzZ7HZbAA4nU4+++yzSKKIiMggRPTG8O7du5k7d274cnx8PAAul4v4+Hja2tpwOBz4/X4AgsEgdrsdgLFjxxIMBgHw+Xw4nc5IooiIyCAMugQ++ugjrr/+euLi4sJjHR0dALS1tXHy5Eni4uJIS0vD6/UC4PV6SU5OBiAlJaXHeGpq6qAXISIig9Ovw0ErV67kyJEj2O12Dh48SGFhIe+++y533313eE4oFGLp0qXExMTQ3d3N2rVrsVgsJCYmkpSUhNvtxm63U1xcDMDy5ctZtWoV27dvJz09ncTExOFZochlpi3QiS/Q1WvcYY8m3n7VKCSSK9kYwzCM0Q4xENOmTePYsWOjHUMuQ8O9c23+3xn2Hv+61/jMKRNwXR074rcj8q3v2m9G9MawyOXEF+i66M5Vz7DFrHTGsIiIiakERERMTCUgImJiKgERERNTCYiImJhKQETExFQCIiImphIQETExlYCIiImpBERETEwlICJiYioBERETUwmIiJiYSkBExMRUAiIiJqYSEBExMZWAiIiJqQREREyszxLo7OwkJyeH1NRUqqqqANi5cyezZ89myZIlLFmyhG+++QaA+vp6cnJyWLhwIR988EH4NkpKSnC73eTl5dHa2gpAa2sreXl5uN1uSkpKhmNtIiLShz5LwGq1UlpayrJly3qMu91uysvLKS8vx2azAVBUVERJSQnbtm3j5Zdfpru7m8bGRhoaGqioqGDBggWUlZUBUFZWRnZ2NhUVFTQ0NNDY2DgMyxMRke/SZwlYLBYmTpzYa3zHjh3k5uaybds2AILBIKFQiEmTJhEbG4vL5aK5uZmamhoyMzMByMrKoq6uDoDa2trweEZGBrW1tUO2KBER6Z9BvScwe/Zs3nvvPf70pz9RU1PDP//5T06fPs348ePDcxwOB+3t7fh8PhwOBwA2m41AIADA2bNnw68gnE4n7e3tka5FREQGaFAl4HA4iIqKIjo6mjlz5nD48GHi4uLo6OgIz/H7/TidThwOB36/Hzj/asFutwMwduxYgsEgAD6fD6fTGelaRERkgAZVAt/u1AH279+Py+UiJiYGi8VCS0sLgUCApqYmXC4XaWlpeL1eALxeL8nJyQCkpKT0GE9NTY10LSIiMkDW/kxauXIlR44cwW63c/DgQcaOHcuHH35IVFQUN910E7NnzwagsLCQgoICQqEQ+fn5WK1WEhMTSUpKwu12Y7fbKS4uBmD58uWsWrWK7du3k56eTmJi4vCtUkRELmiMYRjGaIcYiGnTpnHs2LHRjiGXoeb/nWHv8a97jc+cMgHX1bGXzO0Pd04xn+/ab+pkMRERE1MJiIiYmEpARMTEVAIiIiamEhARMTGVgIiIiakERERMTCUgImJiKgERERNTCYiImJhKQETExFQCIiImphIQETExlYCIiImpBERETEwlICJiYioBERETUwmIiJhYn78x3NnZydKlS/nkk09Yv3498+bNo7y8nLfffhur1crNN9/MunXrAFiyZAldXV1ER0eTlpZGQUEBACUlJezbt4/Y2Fg2bdpEQkICra2trF69mjNnzpCeno7H4xnelYqISC99vhKwWq2UlpaybNmy8FhGRgY7duygsrKS1tZWampqwts2b95MeXl5uAAaGxtpaGigoqKCBQsWUFZWBkBZWRnZ2dlUVFTQ0NBAY2PjUK9NRET60GcJWCwWJk6c2GPM5XIxZswYAKKiooiKigpvy8/P54EHHuDQoUMA1NTUkJmZCUBWVhZ1dXUA1NbWhsczMjKora0dguWIiMhA9Hk46LscOHCA1tZWbrvtNgBKS0uJj4+nubmZ/Px83nnnHXw+H5MnTwbAZrMRCAQAOHv2LDabDQCn08lnn30WSRQRERmEQZfAxx9/zPPPP8/vf//78Fh8fDxw/pVCfHw8bW1tOBwO/H4/AMFgELvdDsDYsWMJBoPExMTg8/lwOp2RrENERAZhUJ8O+vLLLyksLOT5558nISEhPN7R0QFAW1sbJ0+eJC4ujrS0NLxeLwBer5fk5GQAUlJSeoynpqZGtBARERm4fr0SWLlyJUeOHMFut3Pw4EFOnjzJ6dOnefzxxwFYvnw5d9xxB0uXLiUmJobu7m7Wrl2LxWIhMTGRpKQk3G43drud4uLi8HVWrVrF9u3bSU9PJzExcfhWKZettkAnvkDXBbc57NHE268a4USXl4s9fnrs5Fv9KoGXXnqpXze2c+fOC457PJ5eHwFNSEhg69at/bpdMS9foIu9x7++4LaZUyZoR9aHiz1+euzkWzpZTETExFQCIiImphIQETExlYCIiImpBERETEwlICJiYioBERETUwmIiJiYSkBExMRUAiIiJqYSEBExMZWAiIiJqQRERExMJSAiYmIqARERE1MJiIiYmEpARMTEVAIiIiamEhARMbE+S6Czs5OcnBxSU1OpqqoC4OzZs3g8HnJzc3nqqacIhUIA1NfXk5OTw8KFC/nggw/Ct1FSUoLb7SYvL4/W1lYAWltbycvLw+12U1JSMhxrExGRPvRZAlarldLSUpYtWxYee+ONN5gxYwavvfYaVquV6upqAIqKiigpKWHbtm28/PLLdHd309jYSENDAxUVFSxYsICysjIAysrKyM7OpqKigoaGBhobG4dpiSIicjF9loDFYmHixIk9xmpqasjMzAQgMzOTmpoagsEgoVCISZMmERsbi8vlorm5ucfcrKws6urqAKitrQ2PZ2RkUFtbO6QLExGRvg3qPQGfz4fD4QDA6XTS3t7O6dOnGT9+fHiOw+Ggvb29x1ybzUYgEADOH1Ky2Ww9bkNEREbWoErA4XDg9/uB84XgdDqJi4ujo6MjPMfv9+N0OnvMDQaD2O12AMaOHUswGOxxGyIiMrIGVQJpaWl4vV4AqqurSU1NJSYmBovFQktLC4FAgKamJlwuV4+5Xq+X5ORkAFJSUnqMp6amDsV6RERkAKz9mbRy5UqOHDmC3W7n4MGDeDweCgsLyc3NZerUqWRkZABQWFhIQUEBoVCI/Px8rFYriYmJJCUl4Xa7sdvtFBcXA7B8+XJWrVrF9u3bSU9PJzExcfhWKSIiF9SvEnjppZd6jV3oY50zZsygsrKy17jH48Hj8fQYS0hIYOvWrf3NKSIiw0Ani4mImJhKQETExFQCIiImphIQETExlYCIiImpBERETEwlICJiYioBERETUwmIiJiYSkBExMRUAiIiJqYSEBExMZWAiIiJqQRERExMJSAiYmIqARERE1MJiIiYmEpARMTEVAIiIibWr98YvpCGhgY2bNgAwJkzZzAMgyVLlrB582auu+46AMrKyrDZbNTX11NUVIRhGKxYsYJZs2YB53+neN++fcTGxrJp0yYSEhKGYEkiItJfgy6B6dOnU15eDsCrr77KN998A4Db7ebBBx/sMbeoqIiSkhLGjRvH4sWLyczM5MSJEzQ0NFBRUUFVVRVlZWWsWbMmgqWIiMhADcnhoPfff5+77roLgB07dpCbm8u2bdsACAaDhEIhJk2aRGxsLC6Xi+bmZmpqasjMzAQgKyuLurq6oYgiIiIDMOhXAt/69NNPsVgsTJ48GafTyT333EMoFMLj8TB9+nRuuOEGxo8fH57vcDhob2/H5/MxefJkAGw2G4FAINIoIiIyQBG/Enj33XeZP38+cH4HHxUVRXR0NHPmzOHw4cPExcXR0dERnu/3+3E6nTgcDvx+P3D+1YLdbo80ioiIDFDEJVBVVcWdd94JEN6pA+zfvx+Xy0VMTAwWi4WWlhYCgQBNTU24XC7S0tLwer0AeL1ekpOTI40iIiIDFNHhoKNHj3LNNdcwYcIEALZt28aHH35IVFQUN910E7NnzwagsLCQgoICQqEQ+fn5WK1WEhMTSUpKwu12Y7fbKS4ujnw1IiIyIBGVQFJSElu3bg1f9ng8eDyeXvNmzJhBZWVlr/GLzRcRkZGhk8VERExMJSAiYmIqARERE4v4PAERufK1BTrxBbp6jTvs0cTbrxqFRDJUVAIi0idfoIu9x7/uNT5zygSVwGVOJSDDQs8cRS4PKgEZFnrmKHJ50BvDIiImphIQETExlYCIiImpBERETEwlICJiYioBERETUwmIiJiYSkBExMRUAiIiJqYSEBExMZWAiIiJqQREREwsoi+Q+/GPf8ytt94KwNKlS7njjjsoLCzk1KlTTJ06laeffhqLxUJ9fT1FRUUYhsGKFSuYNWsWACUlJezbt4/Y2Fg2bdpEQkJC5CsSEZF+i+iVwA9+8APKy8spLy/n5z//OW+88QYzZszgtddew2q1Ul1dDUBRURElJSVs27aNl19+me7ubhobG2loaKCiooIFCxZQVlY2JAsSEZH+i6gE/vvf/7J48WIee+wxWltbqampITMzE4DMzExqamoIBoOEQiEmTZpEbGwsLpeL5ubmHnOzsrKoq6uLfDUiIjIgER0O+tvf/kZ8fDzvv/8+GzduxOfz4XA4AHA6nbS3t3P69GnGjx8fvo7D4aC9vR2fz8fkyZMBsNlsBAKBSKKIiMggRPRKID4+HoA777yTo0eP4nA48Pv9APh8PpxOJ3FxcXR0dISv4/f7cTqdPeYGg0HsdnskUUREZBAGXQKBQIBz584BUFNTw/e//33S0tLwer0AVFdXk5qaSkxMDBaLhZaWFgKBAE1NTbhcrh5zvV4vycnJQ7AcEREZiEEfDjpx4gTr1q3DbrcTFRXFM888w6RJkygsLCQ3N5epU6eSkZEBQGFhIQUFBYRCIfLz87FarSQmJpKUlITb7cZut1NcXDxkixIRkf4ZdAnccsstvPnmm73GS0pKeo3NmDGDysrKXuMejwePxzPYCCIiEiGdLCYiYmIqARERE1MJiIiYmEpARMTEVAIiIiamEhARMTGVgIiIiakERERMTCUgImJiKgERERNTCYiImJhKQETExFQCIiImphIQETExlYCIiImpBERETEwlICJiYoP+ZTG5MrUFOvEFunqNO+zRxNuvGoVEIjKcBl0Cx48f58knn8RisWCxWNiwYQMHDhxg8+bNXHfddQCUlZVhs9mor6+nqKgIwzBYsWIFs2bNAs7/FOW+ffuIjY1l06ZNJCQkDM2qZNB8gS72Hv+61/jMKRNUAhIxPcm49Ay6BOLj43nllVdwOBxUV1fzhz/8gZSUFNxuNw8++GCPuUVFRZSUlDBu3DgWL15MZmYmJ06coKGhgYqKCqqqqigrK2PNmjURL0hELl16knHpGfR7AgkJCTgcDgCsVitRUVEA7Nixg9zcXLZt2wZAMBgkFAoxadIkYmNjcblcNDc3U1NTQ2ZmJgBZWVnU1dVFuhYRERmgiN8TOHv2LKWlpTz77LNcc8013HPPPYRCITweD9OnT+eGG25g/Pjx4fkOh4P29nZ8Ph+TJ08GwGazEQgEIo0iIiIDFNGng7q7u3nkkUfIy8tjypQpOBwOoqKiiI6OZs6cORw+fJi4uDg6OjrC1/H7/TidThwOB36/Hzj/asFut0e2EhERGbBBl4BhGDzxxBNkZGQwe/ZsgPBOHWD//v24XC5iYmKwWCy0tLQQCARoamrC5XKRlpaG1+sFwOv1kpycHOFSRERkoAZ9OGjPnj1UVVXx5ZdfsmvXLpKSkhg3bhwffvghUVFR3HTTTeFyKCwspKCggFAoRH5+PlarlcTERJKSknC73djtdoqLi4dsUSIi0j+DLoGMjAzq6+t7jXs8nl5jM2bMoLKy8oJzLzRfRERGhs4YFhExMZWAiIiJqQRERExMJSAiYmIqARERE1MJiIiYmEpARMTEVAIiIiamEhARMTGVgIiIiakERERMTCUgImJi+qH5K4R+u1XMRH/vQ0clcIXQb7eKmejvfejocJCIiImpBERETEwlICJiYioBERETUwmIiJjYqJZAZWUlOTk5LF68mObm5tGMIiJiSqP2EdHTp0/zxhtvUFlZyZEjR3jhhRcoLS0drTijQp91Fhl9Zv8/HLUSqK+v5yc/+QlRUVHceuutNDU1jVaUUaPPOouMPrP/H45aCfh8PhwOR/iyYRjDfp9D1fhmf+YgIr1drvuXUSsBh8NBY2Nj+LLF0v+3J6ZNmzYckURETGfUSmDGjBls3ryZc+fOcfToUVwuV7+ud+zYsWFOJiJiHqNWAnFxcdx7770sWrQIq9XKs88+O1pRRERMa4wxEgfjRUTkkqSTxURETEwlICJiYioBERETUwmIiJjYFV8Czc3NLF68mJycHCorK3tt9/l8/OpXv2Lx4sUsXryYzz///JLIBbBjxw6WLVvGokWL2Lt37yWTC2Dz5s3Mnz9/RDL1J9euXbvIzs5m0aJFPPLII3R2dg5rnu/63qv+PoYjnWvNmjXk5OSQnZ3Nm2++ecnkAjh37hzz5s1j69atl0yuM2fOsHbtWpYtW8aSJUsumVzV1dXcf//95OTk8Mwzz0R+Z8YVbuXKlca///1vo6ury1iwYIHR1tbWY/vrr79uvPTSS4ZhGMauXbuMDRs2XBK5jh49aqxbt25Esgwkl2EYRmtrq/Hoo48ad9111yWT69NPPzW6uroMwzCM4uJi46233hq2LG1tbcaCBQuM7u5u46OPPjJWrlw5oKyjles///mPYRiGEQwGjblz5xqdnZ2XRC7DOP9/uHz5cuOPf/zjiGTqT66NGzcaBw4cGLE8/c21cOFC44svvjAMwzBWrFhhNDQ0RHR/V/wrgaamJm655RasViu33347H330UY/tU6ZM4cyZMwD4/X4mTJhwSeTavXs3FouFX/7ylzz22GP4fL5LIhfAli1beOCBB0YkT39zTZ48Gav1/GkvUVFRREVFDVuWvr73qj+P4Wjk+uEPfwgQfpwGcpb+cOYKBoP8/e9/Z+7cuSOSp7+56urqeP/991myZAl//vOfL5lcN954Iz6fj3PnzhEMBomLi4vo/q74EjD+32kQTqeT9vb2HtunTp3Kv/71L+6++262bNlCdnb2JZHr1KlTdHR08Oqrr5Kens6WLVsuiVxfffUVp06d4uabbx6RPP3N9a3jx4+zd+9e5syZM2xZ+vreq/5mHelc3yorK+Ouu+4a1qIcSK7y8nIWLlzImDFjRiRPf3MdPnyYWbNmsX37dnbv3s0nn3xySeSaO3cueXl5zJs3jxtuuIFrr702ovsbtTOGh1JLSwv5+fm9xh966KEez3Z8Ph9JSUk95mzdupW7776bRYsW4fV6eeGFF1i/fv2o53I4HNx2220A/OxnP2P37t1DkinSXL/73e94+OGHhyzLUOUCOHnyJIWFhbz44otcddXwfZFfX9971Z+so5EL4J133qGhoYEXX3xxRDL1lcvv97N//37y8vLYuXPniGXqKxdAfHw8P/3pTxkzZgzp6ek0NjYyderUUc/129/+lp07d3L11Vfz2GOPUVtbS0pKyqDv74oogYkTJ/KXv/zlgtvefvttDh8+TFJSEgcOHCAvL6/H9lAoRHx8PHD+qyyG8rBLJLnS0tLYs2cP9913H4cOHer3dysNd67PP/+coqIiAL744gs2bdrE6tWrRz2Xz+fD4/Hw1FNPcf311w9Jnovp63uvXC7Xd2YdrVzV1dW8+eabvPLKKyN2KKivXCdOnKCtrY0HH3yQlpYWurq6SEpKYubMmaOaCyAlJYUjR45w8803c+jQITIzM4c9U39yWa1Wxo8fj8ViweFwRLzPuuK/NqK5uZknnniC7u5u7r33XnJycgBYu3YtRUVFnDx5klWrVmEYBl1dXfz6179m+vTpo54rFAqxfv16Pv74Y6Kjo3nuuee45pprRj3X/zd//nzefffdYc/Un1zPP/88b731Fj/60Y8AuO+++7j//vuHLU9FRQVvv/12+Huvmpub6ejo4Be/+MVFs46E78o1c+ZMJk6cyLhx4wB48cUXR+Rvqq9c39q5c2e4EEbKd+X67LPPWLduHd988w233347jz766CWR67333mP79u1ER0fzve99j+eeey78Ps9gXPElICIiF3fFvzEsIiIXpxIQETExlYCIiImpBERETEwlICJiYioBERETUwmIiJiYSkBExMT+D7aisO9FM2z6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rng = np.random.RandomState(304)\n",
    "N = 10\n",
    "weight_gain_wt0 = rng.binomial(N, 0.3, 100000)/N # Get proportion\n",
    "weight_gain_ko0 = rng.binomial(N, 0.3, 100000)/N # Assume first (null hypothesis) that there is no difference\n",
    "\n",
    "diff_weight_gain0 = weight_gain_ko0 - weight_gain_wt0\n",
    "sns.distplot(diff_weight_gain0, kde=False); # Since we only have 10 mice each, this histogram is not very smooth. \n",
    "                                           # No matter!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We usually design the actual test by choosing a cutoff in the difference in proportions and stating that we will reject the null hypothesis if our observed difference exceeds this cutoff. We choose the cutoff so that the p-value of the cutoff is some pre-determined error rate, typically 0.05 or 5% (This is not golden or set in stone. We'll discuss this later). Let's find that cutoff from this simulation. This will correspond to the 95th percentile of this simulated distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "name": "04-python-stat-15"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(np.quantile(diff_weight_gain0, 0.95), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that at least 5% of the values will be 0.3 or bigger. In fact, this proportion is "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "name": "04-python-stat-16"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.06673"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(diff_weight_gain0 > 0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we'll take 0.3 as the cutoff for our test (It's fine if the Type 1 error is more than 0.05. If we take the next largest value in the simulation, we dip below 0.05). We're basically done specifying the testing rule. \n",
    "\n",
    "What we (and reviewers) like to know at this point is, what is the difference level for which you might get 80% power. The thinking is that if the true difference was, say, *p > 0* rather than 0 (under the null hypothesis), we would reject the null hypothesis, i.e., get our observed difference to be more than 0.3, at least 80% of the time. We want to find out how big that value of *p* is. In other words, what is the level of difference in proportions at which we can be reasonably certain that our test will REJECT H<sub>0</sub>, given our sample size, when the true difference in proportions is *p*. Another way of saying this is how big does the difference in true proportions have to be before we would be fairly confident statistically of distinguishing that we have a difference between the two groups given our chosen sample size, i.e., fairly small overlaps in the two competing distributions.    \n",
    "\n",
    "We can also do this using simulation, by keeping the WT group at 0.3, increasing the KO group gradually, simulating the distribution of the difference in proportion and seeing at what point we get to a statistical power of about 80%. Recall, we've already determined that our test will reject H<sub>0</sub> when the observed difference is greater than 0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "lines_to_next_cell": 2,
    "name": "04-python-stat-17"
   },
   "outputs": [],
   "source": [
    "p1 = np.linspace(0.3, 0.9, 100)\n",
    "power = np.zeros(len(p1))\n",
    "for i, p in enumerate(p1):\n",
    "    weight_gain_wt1 = rng.binomial(N, 0.3, 100000)/N\n",
    "    weight_gain_ko1 = rng.binomial(N, p, 100000)/N\n",
    "    diff_weight_gain1 = weight_gain_ko1 - weight_gain_wt1\n",
    "    power[i] = np.mean(diff_weight_gain1 > 0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "name": "04-python-stat-18"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEKCAYAAAD0Luk/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXhM9+IG8DfJZCUrWawhSGKJJTQRS13LDw2tfZcu1qIorotW6UJxlSqq1SqtNZaocO1KpZYKYs0iIRKJLetksk9m5vv7wzW3KckkmDlJ5v08T5+aczJn3iNj3jnb95gIIQSIiMgomUodgIiIpMMSICIyYiwBIiIjxhIgIjJiLAEiIiPGEiAiMmJ6KwGlUonhw4ejXbt2OHLkyDPzExMTMXr0aAwfPhzBwcH6ikFERKXQWwnIZDKsXr0a77zzznPnr1ixAnPnzsXWrVsREhICuVyuryhERFQCvZWAqakpXFxcSpyfkJCAFi1aQCaTwc/PD9evX9dXFCIiKoFMqhf+64XK9vb2yMrKKtPzvLy89BWJiKjKunXr1nOnS1YCpqb/2whRKBTw9vYu83NLWhkiInpWaV+eJTs7yN3dHZGRkVCr1bh48SJatmwpVRQiIqOl1y2BqVOnIioqCjY2Nrh69So6dOiAnJwcBAYGYtasWfj444+hUqkwYMAAODg46DMKERE9h0llG0XUy8uLu4OIiMqhtM9NXixGRGTEWAJEREaMJUBEZMRYAkRERowlQERUgWXkKvH96Ts4GvlIL8uX7GIxIiJ6Po1GIOJeJoIvJmH/tQeo52iNhW8218trsQSIiCoAIQQi7slx4NoDHL75EBm5SnT1csGmd19Dh0Y1YGJiopfXZQkQEUkou6AI+67cx7YL93A7JQf/8HLGv3p5o0dTV9jbmOv99VkCREQSkOcpsfFsAjadvQs7K3OM9K+PIe3qwsXWyqA5WAJERAb0WFGAX84lYPP5RNR2sMKXA3wQ6FMLZqb62d2jC0uAiMgArifLseGPuzh04yGa1bbDskEt8UYLN5hK9OH/FEuAiEiPkjPzsPRwDA7ffITezd2wc2J7+NZ31NuB3vJiCRAR6UGeUoV1p+7ghz/i4d/QCQendYK3m53UsZ7BEiAieoWEEDh04xEWHYyClbkZvh/ti65eLhXmm//fsQSIiF6RyAdZWHwwGleT5JjarQnGdGoAS5mZ1LFKxRIgInpJyZl5WHksFvuu3sebrWrjt1ldUMveWupYZcISICJ6AUIIhN/NwPbwezh84xHaNXDE/g86oUUde6mjlQtLgIionE7HpmLRf6KQkJ6LN1rUwpaxfvBr6FRh9/uXhiVARFRGWflFWHwwCnsj7mPC6x4Y06khala3lDrWS2EJEBGVwbHIR/gk9CZqVrdE6Acd0bx25drtUxKWABFRKVKzC/HpgUgcj3qM6d2bYMLrHjA3qzq3YmEJEBE9h1ojsPNiEpYdiUETl+o4NK0zGrtUlzrWK8cSICL6mz/j0/HZgSg8ysrH7F5eGOlXX/IxfvSFJUBE9F/ZBUX47EAU9l25j9Ht3fFhjyZwsLGQOpZesQSIiPDk2/+sXddgayXDgamd0LRWxRvnRx9YAkRk1LLyi7Dy2C1su3AP4zp7YMb/NanwQz28SiwBIjJKGo1ASEQylh6OgbOtJXZObI+27k5SxzI4lgARGR15nhLTgq/iSmImZvX0xOj27pBVodM+y4MlQERGJeaRAhM2X0aN6hY4PrML3OwNe0/fisY4q4+IjFLo1fsY8O05dGhUA8ET2ht9AQDcEiAiI6AoKMLC0EgcuvEQC95shpF+9SvlYG/6wBIgoirt6amf9tbmODC1EzxdbaWOVKGwBIioSiooUuPfR27hl/MJGNe5IWb+n6dRnfpZViwBIqpyribJMXPXVWg0AjsntEe7BsZ36mdZsQSIqErZdTEJ8/fdxHC/epj7hjdsLPgxVxr+7RBRlaBSa7DkcAy2/JmIfw9uif5t6kgdqVJgCRBRpacoKMLU7VcQ/VCBXRMD0Lqeg9SRKg29lkBwcDD27dsHmUyGxYsXw93dXTsvLCwMq1atgoWFBZo1a4YFCxboMwoRVVH30vMw9peLsDQ3RegHHVHL3lrqSJWK3i4Wk8vlCAkJwbZt2zBnzhysWLGi2Px169Zh7dq1CA4Oxv379xETE6OvKERURV1MyED/dWfh4VwNuyYGsABegN62BK5duwZ/f3+YmZnBx8cHCQkJxeZ7enpCoVDA1dUVhYWFcHAo++bbgwcPtH82MTFBrVq1oFKpkJKSUuznnJycYGVlhdTUVBQVFWmnW1lZwcnJCbm5ucjKyir2nFq1akEIgUePHhWb7uDgABsbG6Snp6OwsFA73cLCAjVr1kR+fj4yMzOLPcfV1RVmZmbF8gKAnZ0dqlevjszMTOTn52uny2QyuLi4oLCwEOnp6cWe4+zsDHNzczx69AgajUY7vVq1arC3t0dWVhZyc3O1001NTeHm5oaioiKkpqYWW1aNGjVgaWmJlJQUqFQq7XRra2s4OjoiJycHCoWi2HNq164NtVqNx48fF5vu6OgIa2trpKWlQalUaqdbWlqiRo0ayMvLg1wuL/YcNzc3mJiY4OHDh8Wm29vbo1q1asjIyEBBQYF2urm5OZydnVFQUICMjIxiz3FxcYFMJsPDhw8hhNBOt7W1ha2tLeRyOfLy8rTTzczM4OrqCqVSibS0tGLLqlmzJiwsLPD48WOo1WrtdBsbGzg4OCA7OxvZ2dna6XzvSfPeKyhS4/StFByKzcal+7kIau2ECR1rQZ6WAjn43ivpvVcioSf79+8X69ev1z7u27dvsflnzpwRHTt2FD169BCfffZZmZfr6ekpAGj/s7OzE0IIERcXV2w6ALF//34hhBD+/v7Fpg8dOlQIIcSaNWueeY5SqRRyufyZ6Rs3btSux1+nd+vWTQghRHBw8DPPuX//vhBCCJlMVmz6smXLhBBCvPvuu8Wm+/j4CCGEOHHixDPLunbtmhBCiFq1ahWbPnv2bCGEELNnzy42vVatWkIIIa5du/bMsk6cOCGEEMLHx6fY9HfffVcIIcSyZcuKTZfJZEIIIe7fv//MsoKDg4UQQnTr1q3Y9Ke/740bNz7zHLlcLpRK5TPT16xZI4QQYujQocWm+/v7a99Tf39OXFycEEIIOzu7YtMXLlwohBBiypQpxaY3atRICCHE+fPnn1nW+fPnhRBCNGrUqNj0KVOmCCGEWLhwId97Fei9N/RfX4m4xwq+98rw3vP09BQlMRHiLxX2Cp0+fRqXLl3CrFmzAAD9+vVDaGiodn7v3r2xefNm1KxZE7NmzcLo0aPRtm1bncv18vLCqVOntI/5bYxbApXp2xjfey/+3rsRl4jxmy+hdT0HfNSnKdyca/K9V8b3XteuXXHr1i08j95KQC6XY+LEidi+fTtiYmKwfv16rF69Wju/b9++2L17N6ytrbFw4UL84x//QNeuXXUu18vLq8SVIaKqKSuvCEPXn0ctByv8+HY7mBvpsM8vqrTPTb0dE3BwcED//v0xatQo7dlBYWFhyMnJQWBgICZNmoSgoCCYm5ujdu3a6Ny5s76iEFElJs9TYtwvl2BlYYZ1o3xZAK+Y3rYE9IVbAkTG48q9THyw/QpqVrfApvf84FStat/0XV8k2RIgInpRQgj8dOYulh2JQVD7Bpj7hjcsZNwC0AeWABFVOF8fj8WmswlYO9IXvZq7SR2nSmMJEFGFsvHMXawPi8eWsf7wa8jRP/WN21dEVGH8eiUZSw5HY90oXxaAgXBLgIgkp1Rp8OMf8Vh1IhZLB7ZE96auUkcyGiwBIpLUpYQMfPTrDSjyVTwGIAGWABFJQqnSYPnRGGw8m4Cg9u6Y1dMTtlbmUscyOiwBIjK4e+l5mLojAmk5Suya2B5t3bn/Xyo8MExEBiOEwN6IZPRZ/Qdc7axwcFonFoDEuCVARAbxKKsAH/96A+fj0zEvsClG+9eHiYmJ1LGMHkuAiPQuLDYVU7ZHoEVtexz98HXUc7KROhL9F0uAiPTqdko2Jm+LwLjODTGtWxOYmvLbf0XCEiAivcnKK8L4zZfxf81cMb17E+7+qYB4YJiI9EKtEZgafAV2VjIsGejDAqiguCVARK9cSnYBFuyLRPRDBQ580AlW5mZSR6IScEuAiF4ZjUZg24VEdF9xGum5hQie0B5u9lZSx6JScEuAiF4JtUZgyrYInLuTho8Dm2Jou3o8CFwJsASI6KUJIbBw/01cvpeJg9M68xTQSoS7g4jopa37/Q72XXmAn997jQVQybAEiOil7Ai/h29OxOH70W3RvLa91HGonEotAbVajXHjxhkqCxFVImqNwJLD0VgYGokVQ1uhU5OaUkeiF1DqMQEzMzPY2NggIyMDTk4c5ImInsguKML04Ku4nizHjgn+HASuEtN5YDg/Px9vvPEGfH19YWPzv319K1as0GswIqqYHsjz8e6mcMhMTRH6QSfUcbCWOhK9BJ0lMH78eEPkIKJKIOaRAu9uvIimtWyxdqQvqlnyBMPKTudv0M/PD3FxcUhMTESPHj2Qk5MDpVJpiGxEVIGcvZ2G97dcRp+WtbCofwvIzHheSVWgswR++OEHXLhwAcnJyejRowcUCgVmz56Nbdu2GSIfEUnsblouvjp6C4dvPsS07k04EFwVo7PKDx48iB9//BHW1k/2+9WuXRsKhULvwYhIWkIILPpPFP5v5WnkF6lxaHpnfNjDkwVQxejcEjA3N4epqan2F5+dnc03AZER+OnMXey8lISt4/zR3qOG1HFIT3SWwLBhwzBjxgzI5XL88MMPOHjwIMaMGWOIbEQkkcuJGVh2JAbfjvRlAVRxOktgyJAhaN26NS5cuAAhBL766is0adLEENmISALpOYWYsu0K3uvYED2bu0kdh/RMZwn8/PPP8PX1xfDhwyGT8XQwoqosp1CF6cFXUdfRGrN7eUkdhwxA56e6hYUFtmzZgujoaDg4OKB169Zo06YNunfvboh8RGQgV5PkmB58BZYyU/wyxg/mPAXUKJgIIURZfjApKQnHjh3D5s2boVQqcf78eX1ney4vLy/cunVLktcmqoo0GoHvTt/BqhOxGOlXH/MCm/JOYFVMaZ+bOrcEpk+fjrS0NNSvXx9t27bFxo0b0ahRo1cekogML6dQhRk7r+JSQgbWB7VFN29XqSORgeksAWdnZ2RlZSEzMxMpKSlITU1FnTp1YGXFW8YRVWZJGXkY98slmJgA+z/oxPsAGCmdJTB//nwAQE5ODnbt2oU5c+YgPT0dN2/e1Hs4ItKPM3FpmLojAn4NnbByaGuOAWTEdP7m165diytXruDBgwfw9PTEmDFj4OvrW6aFBwcHY9++fZDJZFi8eDHc3d2183Jzc7Fo0SI8ePAAGo0GW7ZsefG1IKIyebr//+vjsZjWvQk+6NqY9wE2cjpLwN3dHQMGDECdOnXKtWC5XI6QkBAEBwcjKioKK1aswOrVq7Xz165di0GDBqFdu3blT01E5Zb739M/LyVm4Kd3X0MXT2epI1EFoLME+vTpg0OHDmHDhg0AgLZt26JPnz46h464du0a/P39YWZmBh8fHyQkJBSbHxERgcLCQnzzzTfo1asXRo8e/eJrQUSlKihSY+wvF5GZW4QD3P9Pf6HzROAvvvgCv/32Gzp37oxOnTrh1KlT+Oyzz3QuWKFQwM7OTvv472eiRkZGonv37ti0aROOHTuG27dvv0B8ItKlUKXGhC2XkZpdiG3j/VkAVIzOLYHLly9j//792sfdu3fHW2+9pXPBdnZ2iI2N1T42NS3eN46OjujQoQNMTEzQvn17xMbGonHjxuXJTkQ6FKk1+GD7FSSk5WLXxADUrG4pdSSqYHRuCZibmyMmJkb7+NatWzA3N9e54FatWiE8PBxqtRqRkZHFDgoDT3YrRUVFAQBu3rz5zHwiejlZ+UV4d1M4oh4osH28P9zseVo3PUvnlsDChQsxe/Zs7e4cU1NTfPnllzoX7ODggP79+2PUqFHas4PCwsKQk5ODwMBAzJo1C5988gkKCgrg5+eH5s2bv/zaEBGAJ9cAvPfzRdhYmOHXKR3gYssCoOcr87AR2dnZEEIU288vBQ4bQVS6y4mZmLD5Eto1cMSqYW1gbcEhIIzdSw0bkZKSgtWrVyMiIgImJiZo06YNpk2bBhcXl1celIhezq6LSZi/7ybe69QAc3p58xoA0knnMYEPP/wQvr6+2L17N3bt2oU2bdpg+vTphshGRGWkUmvw2YFIzA+9iWWDfTDvjaYsACoTnSWQm5uLgQMHolq1aqhWrRoGDRoEtVptiGxEVAZKlQZTtkfg0I2H2D0xAAPa1JU6ElUiOncHtW/fHitXrkTPnj1hYmKCo0ePIiAgAHfv3gUANGzYUO8hiej5nhZA1AMF9rzfgdcAULnpLIGnp3FeuXKl2PSnxwg2b96sn2REVKpClRqTt0YgNiUbOye2R11HFgCVn84S4MBuRBWPoqAIk7ZeRlJGPoInBKCOg7XUkaiS4vixRJXMA3k+3tt0EeYyE+yZFMBrAOil8CaiRJVI1AMFBq47hzqO1tg5gQVAL48lQFRJnIpJwZDvz6FbUxf8ENSWN4KhV6LEd9GKFStKHS565syZeglERM/afD4Bnx+Iwpze3hjXuaHOodyJyqrEEvDw8DBkDiJ6jiK1Bov+E4Wdl5KwdqQverdwkzoSVTEllsCAAQMMmYOI/iZFUYAp2yNwPzMfuyYGoGVdB6kjURWkc6dibGwsli9fjjt37kCpVGqnnzlzRq/BiIzZ5cQMvL81Ap6u1XFgaifU4H0ASE90HhieP38+ZsyYATs7Oxw7dgzvv/8+Ro4caYhsREbp3J00jNpwAf1b18Yv7/mxAEivdJaASqVCs2bNoFarYWNjg9GjRyMsLMwQ2YiMzvk76Rjz80VM6tIYH/dpBpkZT+Aj/dK5O8ja2hpFRUXw8vLC2rVr4eLigry8PENkIzIqTwvg/S6NML1HE6njkJHQ+TVj6dKl0Gg0WLhwIYQQuHPnDtasWWOIbERGQaXWYM1vcXh74wW836URPuzhKXUkMiI6twSsrKxgYmICW1tbTJ06FUqlEllZWYbIRlTlJaTlYuauq7iXkY8fgtqhqzdv1kSGpXNL4P333y/2WAiBSZMm6S0QkbGIfqhAv2/PwtnWEsdmvM4CIEno3BJQqVSwsLDQPra0tCx2qigRlV9CWi6CfgpHoI8bvhzgwyuASTI6twRcXV2xb98+7eN9+/bB1dVVr6GIqrKHWfkYteEC2ns4YVF/FgBJS2cJLFq0CCdPnsTrr7+OLl264NSpU1i8eLEhshFVOefupGHED3+iiWt1rBzaGma8DzBJTOfuIBcXF6xevdoQWYiqrPvyfHx5MBpHIx/h7YAGmN3LCxYyXgNA0iuxBFauXImZM2di5syZz91cXbFihV6DEVUVEfcyEbThAlrXd8Dh6Z3RxNVW6khEWiWWQM+ePQEAw4cPN1gYoqom5pEC7226iFHt3THvDW/u/6cKp8Tt0RYtWgAATp48CT8/v2L/nTx50mABiSqrxPT/nQHEAqCKSudOyQsXLjwzjSOIEpXuRnIWRm24AL+GPAOIKrYSdwdt3LgRGzduhFwuR6dOnbTTTUxM0LdvX4OEI6psClVqrPntNr4/fQdDX6uHT99szjOAqEIzEUKI0n5g9erVmDZtmqHy6OTl5YVbt25JHYPoGQlpuXh/62Uo8ouwbHBLdG7iLHUkIgClf27q3B0UEBCgHTX0119/xZIlS5CUlPRqExJVclfuZWLgd+fg4VwNR2e8zgKgSkNnCXzxxRewsbHBjRs3sHXrVnh6emLOnDmGyEZUKRyPeowRP/6JgW3qYO0IX9hamUsdiajMdJbA0wNaJ06cQFBQEAYNGsT7CRAB0GgE1v1+G5O2Xsac3t6Y37cZTLn/nyoZnVcMOzs7Y968ebhw4QL27t0LpVIJtVptiGxEFVZmrhKzdl9DxL1M/Pg2h4CmykvngeH8/Hz88ccf8PT0RIMGDZCSkoLY2NhiZwwZEg8Mk9SiHyow7pdLcLa1xLejfFHHwVrqSESlKu1zs8QtgatXr6J169a4fPkybGxskJycjOTkZL2FJKoMLidm4r1N4ejTshY+e6sFx/+hSq/EEggLC0Pr1q1x8ODB586XakuASCpn4tIwYcslBAW4Y25vXgFMVUOJJfD02oABAwbAz8+v2Lzw8PAyLTw4OBj79u2DTCbD4sWL4e7uXmy+Wq1Gnz59MGTIEIwdO7a82YkMokitwdY/E7HkcAymd2+CKV0bSx2J6JXRuS27ZMmSMk37O7lcjpCQEGzbtg1z5sx57qijISEhqF+/fhmjEhmWEAInYx6j16owfHvqDpYN8mEBUJVT4pbA+fPnce7cOaSmpmLlypXa6Tk5OWXaDL527Rr8/f1hZmYGHx8fJCQkFJtfWFiIkydPolevXpDL5S++BkR6oNYIzN93AyER9zGuU0NM7toY1S11nkxHVOmU+K52dHSEh4cHzM3N0bBhQ+10GxsbTJ06VeeCFQoF7OzstI//fhLSli1bMGzYMGRmZr5IbiK9Uak1+OfuazhzOx37P+gIbzc73U8iqqRKLAFvb294e3tjwIABL7RgOzs7xMbGah+bmv5vz1N2djbCw8Mxbtw47N2794WWT6QPSpUGH+68gsuJmdg5sT0aOVeXOhKRXuncvo2NjcXy5ctx584dKJVK7XRdw0m3atUK69atg1qtRkxMTLGDwvHx8cjMzMTYsWORkpKCoqIieHt7o2PHji+xKkQvrlClRuiVB9hwJh65hWrsmhgA9xrVpI5FpHc6S2D+/Pn49NNP8dFHH2H79u3Yu3cvFAqFzgU7ODigf//+GDVqlPbsoLCwMOTk5CAwMBC7d+8GAOzduxeZmZksAJLM3ohkfHkoBoDA2wENENTeHY7VLKSORWQQOktApVKhWbNmUKvVsLGxwejRozF8+HBMnjxZ58JHjBiBESNGaB///RRRABg4cGA5IxO9OiGXkzF373XM79MMw16rBytzM6kjERmUzhKwtrZGUVERvLy8sHbtWri4uHAAOaoSDt14iDkh17F8SEsMaFNX6jhEktB5ncDSpUuh0WiwcOFCCCFw584drF271hDZiPTmyM2HmB58BZ/3a8ECIKOmswSOHz8OS0tL2NraYurUqZg3bx5OnDhhiGxEr9xjRQGmbI/A1B1X8HFgU4z058WKZNx0lsCBAweemRYaGqqXMET6IoTA1j8T0X3FaaRlF+LQtM54t2ND3U8kquJKPCawe/du7NmzBwkJCRg+fLh2ek5ODho35qXzVHkUqTVYuD8S+67cxxf9WmCgbx0O/kb0XyWWQO/evREQEIA1a9YUu9G8jY0NHB0dDRKO6GVl5RVh8vbLiE/NxZ73O6BZbV79S/RXJe4OkslkcHV1xbJly1CnTh3k5eXh2LFj+PPPPw2Zj+iFnbuThre+PYOcAhVCp3RkARA9R4klMGbMGDx69AgAcPv2bYwYMQLJycnYvXs3Fi1aZLCAROWVlVeEOXuu4+2fwtG7hRt2TgyAi52V1LGIKqQSdwfl5OSgXr16AJ5c1du/f3/Mnz8fKpUK/fv3N1hAovI4ezsNM3ZehYudJfZN6YgWdeyljkRUoZW4JSCE0I78ee7cOXTp0gXAk91EPKhGFY1KrcFXR2/hnY3hGO5XH/smswCIyqLELYGePXti7NixcHR0RFFREQICAgAAycnJsLGxMVhAIl0eKwrwwfYIJKbnYfNYP3RoVFPqSESVhon4+0D/f3HlyhWkpqaiQ4cOqF79yZC6d+/eRV5eHpo3b26wkH/l5eWFW7duSfLaVPFcTszA+1sj4OlaHauHt0GN6pZSRyKqcEr73Cx17KA2bdo8M+2vN5ghkooQAlsv3MPnByIxplNDzO7pBZmZzmsfiehveL88qnQeyPPx0a83EH43A18Pa42+LWtLHYmo0mIJUKWh0QhsD7+HpYdj0Ka+A47NeB11HXl8iuhlsASoUniYlY/Zu6/jerIcC99shsFt6/IsNaJXgCVAFV7o1fv4ZN9NtKrngGMzusDNnhd+Eb0qLAGqsIrUGnx2IBJ7Lifjo8CmCGrvzm//RK8YS4AqJHmeEpO3ReBu2pOB33jhF5F+sASoQtFoBE7HpeKz/ZFwrGaB0A86wsWWu3+I9IUlQBWCUqVBSEQyfjpzF8mZeRjp545/9fbijd+J9IwlQJKLeqDAzF1XkZajxHsdG2CkX304VrOQOhaRUWAJkGRUag3Wh8Vj1YlYDGhTB5/0bQZbK3OpYxEZFZYASeLm/SzM3Xsdj7IK8d2otujRzFXqSERGiSVABpWnVOHr47HYeDYBg33rYutYbzjYcNcPkVRYAmQwcY+zMWlbBNQaga1j/RHQqIbUkYiMHkuADCL06n3MDbmBN1q4YdGAFrCx4FuPqCLgv0TSq8gHWVh36g6ORz/GZ281x/DX6vGqX6IKhCVAenHlXiZWHo/Fmdtp6NnMFaFTOqJpLTupYxHR37AE6JVSawS++/02Vp2IQ/82dXBiZhc0cq4udSwiKgFLgF6ZFEUBPtx5FbceZePHd9qhq5eL1JGISAeWAL20zFwlfvwjHr+cS0Creg44NL0zXO043g9RZcASoBdWqFLj+9/j8eMf8ajjYI2vhrRCr+ZuMDXlgV+iyoIlQC/kcmIm5oRcR75SjaWDfBDYohY//IkqIZYAlUtKdgG+PXkbWy/cw2j/+pjd2xvVLfk2Iqqs+K+XyiRFUYDvT8dj24VEeLnZInhCe7zWwEnqWET0klgCVCohBHaEJ2HRwSh4utri+6C2+IenMy/4Iqoi9FoCwcHB2LdvH2QyGRYvXgx3d3ftvDlz5iAxMRFqtRojR47EgAED9BmFXkBKdgHmhtxA+N0MfN6vBQb51uGHP1EVo7cSkMvlCAkJQXBwMKKiorBixQqsXr1aO3/SpElo0KABlEol3nrrLfTt2xfm5hxLXmqZuUqExaXiVEwKfotJQVM3Oxye3hn1nGykjkZEeqC3Erh27Rr8/f1hZmYGHx8fJCQkFMAGyMEAAA/wSURBVJvfoEGDJwFkTyKYmprqKwqVgVoj8ENYPFYevwVHGwt09XLB8sGt8H/NXGHGs36Iqiy9lYBCoYCd3f/GihFCPPfnfvzxR/Tp0wdmZryXrFTuy/Mxc+dVxKXkYO1IX/Rs5srdPkRGQm9fv+3s7JCdnf2/F3rON/39+/cjOjoaU6ZM0VcMKoVSpcHGM3fRe1UYrMzNcOTDzujV3I0FQGRE9LYl0KpVK6xbtw5qtRoxMTHFDgoDQFhYGH799VesX7+eu4IMTAiBo5GPsfRwNHKVaszv0xRD23GIZyJjZCJK2k/zCuzYsQOhoaHas4MSExORk5ODwMBAdOzYES4uLqhe/ckIkytXroSzs7POZXp5eeHWrVv6ilzlXUuSY/GhaFxPlmNCZw9M7NII1XixF1GVVtrnpl5LQB9YAi8mKSMPXx27hQPXHmCgb138s6cX3Ow5yBuRMSjtc5NfAau4R1kFWHMyDjsvJqG9Rw0cmNoJzWvbSx2LiCoIlkAVJITAxYRM7LmchNCrD9Cijj228MbuRPQcLIEqpKBIjW0X7mHz+QQ8lBege1MX/Ph2O3RuUpMHfYnouVgCVUBBkRo7wu/hu9/vQGZqgvGve6B/6zpwrGYhdTQiquBYApVUQZEaYbGpOHjjIU5EPYadtTmmdm+Coe3qwlLGC++IqGxYApVMYnoutl24h92XkqDSCPRq7oa1o3zRsVFNWMh4vQURlQ9LoJKIe5yNZUdu4beYx3jN3QmfvtUcvVu48Vs/Eb0UlkAFl5JdgFUnnpzi2bOZK45Mfx1ebrZSxyKiKoIlUEHdTcvFj3/EI+RyMprVtsOuie3R1p138iKiV4slUMFcuZeJ70/fwfGox+jQqCY2vNMOnRrzFE8i0g+WQAWg0Qicjk3F96fv4FJiJvq2rIX9H3RCizq8speI9IslIKE8pQp7I+5j09m7uC/Px7B29fDVkFa8ixcRGQxLwMCSMvIQFpeKsNhUnLudjmqWMrzToQFG+NWDgw0v7iIiw2IJGIBaI/Bb9GNsPp+IM7fT4OFcDa83ccY3I1qjcxNnmJvx/H4ikgZLQI/ylWrsvHgPG87cRWauEoPb1sWnbzVDYxee4klEFQNL4BXSaAQeZxfgblouwu9mYPP5RFjJTDG2sweGtqsLWytzqSMSERXDEngF7qXnYd3vtxF69QHyi9SwMjeFl6stPgpsin6ta3N3DxFVWCyBF1BQpEZCei7iU3NxIvoxQq8+QHsPJ3w7qg2a1bKHq50lz+snokqBJVBGT8/l33j2Ls7cToMQQC17K7Ssa49dEwPQ1t1R6ohEROXGEtAht1CFvRHJ2HTuyY1aBreti1k9vdDEpTpv0E5ElR4/xZ5DCIHYxznYczkJwReTYGspw9sdGmDEa/Vhb8ODu0RUdbAE/isrrwjHox/jdGwqzt9JQ1qOEn4NnLBsUEv0bOYKGQ/uElEVZNQlkKIowG8xKThy8xHO3k6Ds60lunm74PN+LdDeowaceHtGIqrijKYENBqBexl5iH6oQOQDBcLiUnE9OQv1nWzQs5krpvdogtZ1HWBqyrN6iMh4GE0JvPfzRZyOTYWLrSWa1rLDGy1qYcWQVmjsUp2ncxKR0TKaElg+pCXMTExQo7ql1FGIiCoMoykBF1srqSMQEVU4POWFiMiIsQSIiIwYS4CIyIixBIiIjBhLgIjIiLEEiIiMGEuAiMiIsQSIiIxYpbxYzMvLS+oIRERVgokQQkgdgoiIpMHdQURERowlQERkxFgCRERGjCVARGTEWAJEREaMJUBEZMSqZAkEBwdj+PDhGD16NBITE4vNW7BgAUaOHImBAwfi559/liZgOZS2LgCgVqvRu3dv/PTTTxKkK5/S1mXu3LkYOHAggoKCsGDBAokSll1p65Kbm4t58+bhnXfeQVBQkEQJy6a09ZgxYwaCgoIQFBSE1q1bIyYmRqKUZVPauoSFhWHgwIEYPnw4Pv/8c4kSll1p63Ls2DEMGTIEI0aMwOnTp1/+xUQVk5mZKQYPHixUKpW4fv26mDp1arH5hYWFQgghioqKRM+ePbWPKyJd6yKEEDt37hTjx48XGzZskCBh2elalzlz5ojr169LlK58dK3L0qVLxcWLFyVKV3ZleX8JIUR6eroIDAw0cLry0bUuw4YNE/fv3xdCCDFhwgQRHR0tRcwyKW1dVCqV6NOnj8jPzxd5eXli4MCBQqVSvdTrVbktgWvXrsHf3x9mZmbw8fFBQkJCsfkWFhYAgMLCQri7u8Pc3FyClGWja10KCwtx8uRJ9OrVS5qA5aBrXQDg888/R1BQEM6cOWP4gOWga10iIiJw6NAhBAUFYevWrdKELIOy/E4A4MiRI+jdu7dhw5WTrnXx9PSEQqGAWq1GYWEhHBwcpAlaBqWtS2ZmJpydnWFlZQVra2vY2to+dw9BeVS5ElAoFLCzs9M+Fs+5IPpf//oXevbsCR8fH5iYmBgyXrnoWpctW7Zg2LBhFXodntK1LnPmzMHu3buxatUqLFu2DNnZ2YaOWGa61iUyMhLdu3fHpk2bcOzYMdy+fdvQEcukLP9WAODgwYPo27evoWK9EF3r0qtXL4wbNw69e/eGh4cH3NzcDB2xzEpbFycnJ6SkpCAjIwPp6emIjo6GQqF4qderciVgZ2dX7APE1PTZVfz3v/+NkydP4uzZsxX2HyhQ+rpkZ2cjPDwcXbt2lSJauen6vTg6OgIAatSogRYtWuDu3bsGzVceZVmXDh06QCaToX379oiNjTV0xDIpy7+Vhw8forCwEA0bNjRktHLTtS5ffPEF9u7di6NHjyIzMxOXL182dMQyK21dTE1NMX/+fEyfPh2ffvopmjZtCmdn55d6vSpXAq1atUJ4eDjUajUiIyPh7u5ebL5SqQTwZLeQtbU1LC0tpYhZJqWtS3x8PDIzMzF27Fhs2rQJu3fvxtmzZyVMWzpdv5enb/rCwkJERUWhTp06UsQsE13r0rZtW0RFRQEAbt68+cz8ikLXegDAf/7zH/Tp00eCdOWja11kMhlsbW1hamoKOzu7l/72rE+61iUgIABbtmzB559/DktLy5f+t1IlB5DbsWMHQkNDIZPJsHjxYiQmJiInJweBgYGYMGEC8vPzoVQq0atXL4wZM0bquKUqbV2e2rt3r7YQKrLS1mX8+PHIzs6GSqXC8OHDMXjwYKnjlqq0dUlKSsInn3yCgoIC+Pn5YebMmVLHLZGu99fAgQPx3XffwdXVVeKkupW2LgcPHsSmTZtgbm6O2rVrY9myZZDJKu4gyqWty9KlSxEZGQlLS0t89NFH8PDweKnXqpIlQEREZVPldgcREVHZsQSIiIwYS4CIyIixBIiIjBhLgIjIiLEESFLNmzdHv379EBgYiI8//hgajcYgr6tQKLB7927t499++w2bN29+6eWOHz9eey1KWXTs2FH7523btmHkyJEoKCjA+fPn0b9/f/Tu3Rv9+vXD+fPnXzjTjh07cOTIkRd+PlVxLzXyENFL6tChgxDiycBYo0ePFkeOHCk2v6io6JW/ZlFRkUhKShJDhgx55csur6frHxoaKgYMGCAUCoV49OiR6NKli4iLixNCCBEXFye6dOkiHj16JGVUqqIq7tUSZFTMzMzQqlUr3Lt3D3v37sXZs2chl8tRs2ZNTJ48GfPmzUN2djY8PDywZMkS2NjYoFu3bggMDMTvv/+OWrVq4euvv0b16tVx+vRprFixAhqNBr169cLUqVORnJyMqVOnwtvbG5GRkfD29kZsbCz69euHN998E05OToiPj8c///lPXL9+HZ9++imKiorg6+uLBQsWwMzMDB07dkRgYCDOnTuH+vXrY+3atTAzMyu2Ht26dcPhw4dx9epV/PDDDzA3N0d8fDwGDBiASZMmPXfdT506hfXr12Pz5s2wtbXFhg0bMGjQIDRu3BgA0LhxYwwaNAjbt2/HjBkznnm9fv364cSJE3BxccHkyZPx1VdfIT09HStWrICPjw/WrFmDmjVrYsSIEYiIiMCXX34JpVKJ2rVr4/vvv0dKSgoWLFiAlJQU2NjYYMmSJahXr55+ftFU8UjdQmTcnn4Tzs/PF4MHDxYnT54UISEhomfPniInJ0cIIcS4cePE8ePHhRBCLFu2TKxbt04IIUTXrl3F9u3bhRBCfPPNN2Lt2rUiPz9fdO3aVTx8+FAolUoxbNgwcenSJZGUlCSaNm0qYmNjhRDimS2BkJAQsXz5ciGEEH369BGRkZFCCCGmT58uQkNDhRBCeHp6aoeInjhxojhz5swz69O1a1dRUFAg/vzzTxEQECAyMjJEXl6eeP3110Vubu4zP9+iRQvh7+8vkpOTtdMmT54sTpw4Ueznjh8/LqZMmfLc19u/f7/2eVOmTBFqtVocP35czJw5UwghxOrVq8X27dtFYWGh6NGjh7h9+7YQ4smQxUII8eGHH4qoqCghhBDh4eElDilNVROPCZCk5HI5+vXrh2HDhiEgIEA7IF6nTp1QrVo1AEB0dDR69OgBAHjrrbeKDf71dBjtXr16ISIiAnfv3kWjRo3g5uYGc3NzBAYGan++UaNGaNKkSal5FAoFNBoNmjVrBgB48803tc+3s7NDu3btAABNmzbFgwcPSl1WmzZt4OjoCGtra9StWxepqanP/IyNjQ0aN26MQ4cOlf4XVYqnf2eenp7w8/ODqakpvLy8nskXHx+P+vXro1GjRgCgHU75woULmDt3Lvr164dFixYhJSXlhbNQ5cPdQSQpBwcHhIaGPjPd2tpa++eyDpVtYmICIUSxnxd/GRXlr8ssSWnPf3ovCuDJaI4qlarUZZXl52UyGb799luMGDECderUQWBgIBo1aoSoqCh0795d+3PR0dHaD++/e3pPDFNTU+2fTUxMoFarn/nZ5/1dmpqa4tdff33uKKJU9fG3ThWet7c3Tp48CeDJqJZt27bVzjt69Kj2/23atIGHhwdu376Nx48fQ6VS4ejRo/D19X1mmTY2NsjNzX1mur29PUxNTbW3Ujx06FCx19MHe3t7rF+/HsuXL8elS5cwcuRI7NmzRzvM+Z07d7Bnzx6MGDHipV7Hw8MDiYmJiI+PB/BkKwx4Murpnj17AAAajQZxcXEv9TpUuXBLgCq8+fPnY968efj666+1B4afSk9Px8CBA+Hk5IRVq1bBysoKCxcuxPjx47UHhtu1a4fk5ORiy3RyckLjxo3x5ptvol+/fnByctLOW7x4MT766CMUFRWhTZs2BhlKuV69evjmm28wbdo0bNq0CV9++SVmzZqFwsJCWFhYYPHixS99IxQLCwssXboU//znP6FSqbQHhj/55BMsWLAAW7duhUqlwrBhw3TuNqOqg6OIUqX19EycinxPCKKKjruDiIiMGLcEiIiMGLcEiIiMGEuAiMiIsQSIiIwYS4CIyIixBIiIjBhLgIjIiP0/tkT49c2dCm4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.lineplot(p1, power)\n",
    "plt.axhline(0.8, color = 'black', linestyle = '--');\n",
    "plt.ylabel('Statistical power')\n",
    "plt.xlabel('Proportion in KO mice');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "name": "04-python-stat-19"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.48"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(p1[np.argmin(np.abs(power - 0.8))] - 0.3, 2) # Find the location in the p1 array where power is closest to 0.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So to get to 80% power, we would need the true difference in proportion to be 0.48, or that at least 78% of KO mice should gain weight on average. This is quite a big difference, and its probably not very interesting scientifically to look for such a big difference, since it's quite unlikely. \n",
    "\n",
    "If we could afford 100 mice per arm, what would this look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "name": "04-python-stat-20"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEHCAYAAABIsPrhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deVxU5f4H8M8MwzbKqoA7BgZokqJeES0NM/Wi5pYJBtV1qcys1Ns1y7QsS6/hLeV6r1bazy1KMZc0t4vFdUncTVAWEcQVBYZhn+35/eF1akIYUGYOMJ/36+VLz3PmnPk8cpjvnO05MiGEABER2SS51AGIiEg6LAJERDaMRYCIyIaxCBAR2TAWASIiG8YiQERkwxRSB6irwMBAqSMQETU6aWlp92xvdEUAqL4zRERUVU1fnnk4iIjIhrEIEBHZMBYBIiIbxiJARGTDWASIiGyYxYqARqNBZGQkevXqhd27d1eZn5OTg+joaERGRiI+Pt5SMYiIqAYWKwIKhQLLli3DCy+8cM/5sbGxePvtt7F+/XokJCRApVJZKgoREVXDYvcJyOVyeHt7Vzs/OzsbXbt2BQD07t0bZ8+eRf/+/S0Vh8iiKrR6qCu0KK7QoUKrR6XOgAqNHiWVOpRqdCip1OPuozuEADQ6g/F1OoOAQQgYDAIGAQgICAEYhLgzzyCgNwiI/y37e3dajRPGKSHuzDEIoKZHhshkMuPrqWEb/mhrDO3aut7XK9nNYr/f6Nzc3FBUVCRVFKJ70ugMuFxQisy8Uly8VYJrqnKUafQo0+hQUqlDQakWhaUaFJRpoNEZTJa1k8vgpJCjuZMCzRwVaOaggJ1cZpzvoJDDyd4Ojgo57O1kkMlkkMtkkMsAGWCcVshlkMtlcLSXQYY7y8v+95q77n6Q/zZ952+5TPa/ddW9739c5x8JIcy+huqXm7ODRdYrWRGQy387EqVWqxEUFCRVFLJhBoPAVVU5Lt0uRW5hGS4XlCH7diky80qQk18GnUHA28UR/l7N0c7DGW7O9mjt5oRmjgp4NnOAZzMHeCgd4OZsD1dnBZo7KuBsbweFHa+5oMZBsiLg6+uLlJQUBAUF4dixY5g8ebJUUciGFFdocfhiPn5Ov4VzV4uQmVeCMo0eLk4KtPdQor2nMzq2aIZBnX3Qybs5/L2bw9XJXurYRBZj0SIwffp0pKamQqlU4vTp0+jbty9KSkoQERGBWbNm4d1334VOp8Po0aPh7u5uyShkw3ILyrAv9Sb2pt7A8exCNHdS4PGHvTAmpC0CfFzwsI8LWjZ34OENskmyxvag+cDAQA4gR2bdVFdgx5lr2Hr6Ks5dVSPQxwWDH/FBeJA3urVzNzk+T9TU1fS52ShHESW6F53egP3n87DhaA4OZd6Gv1dzjAppi7ioHujYspnU8YgaJBYBavTKNXqsPnQJ63/JQUmFDmN7tsPbfw5Cl9auPMRDZAaLADVaQgj8eO4GPvohFQo7OV4b2AmjurdFM0du1kS1xd8WapTOXS3Cx7vO49RlFV4b2AmTHnsITvZ2UscianRYBKhRyS0oQ+zeNGw/cw1Pd2uD/8wagDbuzlLHImq0WASoUdAbBL74bxaW7k1HqJ8ntr/2GLq2dZM6FlGjxyJADV727VLM2nQG2bdLsSwqBEO7tpI6ElGTwSJADdruczcw49vT6B/QEitjeqJlc0epIxE1KSwC1GCtPZKNBTtS8d7wLng+zJeXexJZAIsANThCCCzZk4avDl5C3IQePPxDZEEsAtSgFJVr8damMzh6qQAbJoeiV0dPqSMRNWksAtRgpF5TY+qGE3BxUuCH6Y+hvadS6khETR4HPacGYV/qTYxecQh9/Vti8yt9WQCIrIR7AiS5pPRbmLbhJOZEBOEv/R6SOg6RTeGeAEkq+VIBXlp3HDMHB7AAEEmARYAkc/aKChO/PoaXHvfDKwP8pY5DZJNYBEgS6TeL8fzqZDzbqz1mPBUgdRwim8UiQFZ3Ob8M0V8exVOdfTB3WGfeBEYkIRYBsqo8dQWivzqKHh088MmYYMj5mEciSbEIkNVUaPWY9H/H0cFTic+jukNhx82PSGq8RJSsQgiB2QlnUVqpw/rJoXBU8AEwRA0BiwBZxcqkLCSez8PW1/rBzdle6jhE9D8sAmRxB9Ly8OmeNHzxfC/4ezWXOg4R/Q4PypJFXVWVY8a3pzHjqQCEB3lLHYeI/oBFgCxGozPgtY0nEdLeHVN5MxhRg8TDQWQxS/ZcwM2iCqx+4U+8FJSogWIRIIvYn3oTXx/ORvxLYfBo5iB1HCKqBg8HUb0rrdTh3a2/4s1BAejp6yF1HCKqAYsA1bt//XQRSgcFpjzuJ3UUIjKDRYDqVW5BGVb9NwvvDe8MBwU3L6KGjr+lVK8+3nUeYX4tEB7Iy0GJGgMWAao3hy/exr7Um3hvOEcGJWosWASoXpRr9Hhv6zk8H9YRnbxdpI5DRLVk0SIQHx+PyMhIREdHIycnx2ReUlISxowZg8jISCxYsMCSMcgKPtqZCiGAvw7hA2KIGhOLFQGVSoWEhARs2LABs2fPRmxsrMn8FStWIC4uDvHx8bh69SouXLhgqShkYftSb+K747n4PDIESgfeekLUmFjsN/bMmTMIDQ2FnZ0dgoODkZ2dbTI/ICAAarUaPj4+qKyshLu7e63Xfe3aNeO/ZTIZWrduDZ1Oh7y8PJPXeXp6wsnJCbdu3YJWqzW2Ozk5wdPTE6WlpSgqKjJZpnXr1hBC4MaNGybt7u7uUCqVyM/PR2VlpbHdwcEBLVu2RHl5OQoLC02W8fHxgZ2dnUleAHB1dUXz5s1RWFiI8vJyY7tCoYC3tzcqKyuRn59vsoyXlxfs7e1x48YNGAwGY3uzZs3g5uaGoqIilJaWGtvlcjlatWoFrVaLW7dumayrRYsWcHR0RF5eHnQ6nbHd2dkZHh4eKCkpgVqtNlmmTZs20Ov1uHnzpkm71s4ZsxPO4uXeXmghL8W1a3cyODo6okWLFigrK4NKpTJZplWrVpDJZLh+/bpJu5ubG5o1a4aCggJUVFQY2+3t7eHl5YWKigoUFBSYLOPt7Q2FQoHr169DCGFsd3FxgYuLC1QqFcrKyoztdnZ28PHxgUajwe3bt03W1bJlSzg4OODmzZvQ6/XGdqVSCXd3dxQXF6O4uNjYzm1P2m3Pw8MDzs7OuH37NjQajbGd2969t71qCQvZvn27WLlypXF6+PDhJvMPHjwo+vXrJwYNGiQ++OCDWq83ICBAADD+cXV1FUIIkZGRYdIOQGzfvl0IIURoaKhJ+7PPPiuEEGL58uVVltFoNEKlUlVpX716tbEfv28fOHCgEEKI+Pj4KstcvXpVCCGEQqEwaV+8eLEQQogXX3zRpD04OFgIIcT+/furrOvMmTNCCCFat25t0v7WW28JIYR46623TNpbt24thBDizJkzVda1f/9+IYQQwcHBJu0vvviiEEKIxYsXm7QrFAohhBBXr16tsq4BrywUUauOiIEDB5q03/15r169usoyKpVKaDSaKu3Lly8XQgjx7LPPmrSHhoYat6k/LpORkSGEEMLV1dWkff78+UIIIaZNm2bS7u/vL4QQ4siRI1XWdeTIESGEEP7+/ibt06ZNE0IIMX/+fG57DWjbi4+PF0IIbnu12PYCAgJEdWRC/K6E1aOff/4Zx48fx6xZswAAI0eOxLZt24zzhw4dirVr16Jly5aYNWsWoqOj0bNnT7PrDQwMxIEDB4zT/DYm3bexY9kFmLk1AwdmD4FSlPHb2O9w2+OeQEPa9sLDw5GWloZ7sVgRUKlUePnll7Fx40ZcuHABK1euxLJly4zzhw8fjk2bNsHZ2Rnz58/HE088gfDwcLPrDQwMrLYzZD1CCIxecRiPtnPDgpFdpY5DRDWo6XPTYucE3N3dMWrUKDz33HNQKBRYuHAhkpKSUFJSgoiICEydOhUxMTGwt7dHmzZt8Pjjj1sqClnAvtSbSLtRjFXPm997I6KGy2J7ApbCPQHp6Q0Cf/48CU918cFbQ4KkjkNEZtT0ucmbxajOtp66ipvqSrzUnw+KIWrsWASoTiq0eizdl45XBvjzgfFETQCLANXJF0lZkMmAv/TrKHUUIqoHLAJUa9eLyrHip4t4N6IznOztpI5DRPWARYBqbdGPF9CtvRuGdm0ldRQiqicc6IVq5Xh2AX44ex0/TH+Mw0QTNSHcEyCzDAaBD3akIqp3e3Ru7Sp1HCKqRywCZNZ/LuQhM68EMwZxmGiipoZFgGokhEBcYgai+3RAi+aOUschonrGIkA1+m/GbZy/UYwpj/tJHYWILIBFgGoUl5iJyD+1h7erk9RRiMgCWASoWr9k5eNUbiFeHsDhIYiaKhYBqlZcYibG9miHtu7OUkchIgthEaB7OpZdgCNZ+Zj6BPcCiJqyGouAXq/H5MmTrZWFGpDYvWkY17MdfFs0kzoKEVlQjUXAzs4OSqWyymPVqGk7nHkbJ3IK8drATlJHISILMztsRHl5Of785z+jR48eUCqVxvbY2FiLBiNpCCHw6d40RPXugHYeSvMLEFGjZrYITJkyxRo5qIH4Kf0WUq6p8a9oPjaSyBaYLQK9e/dGRkYGcnJyMGjQIJSUlECj0VgjG1mZEAJL96Yjpo8vfHhfAJFNMHt10KpVq7Bo0SIsWbIEAKBWqzF9+nSLByPr+zn9FjLyivEKrwgishlmi8DOnTvxxRdfwNn5zrXibdq0gVqttngwsr5VSVkY17M9WnKMICKbYbYI2NvbQy6XG8eQLy4u5njyTdC5q0X4JSsfkx9/SOooRGRFZs8JjB8/HjNmzIBKpcKqVauwc+dOTJw40RrZyIpWJmVhaNdWvC+AyMaYLQLjxo1D9+7dcfTo0TuXD376KR5++GFrZCMryS0ow65fr2PzK2FSRyEiKzNbBL7++mv06NEDkZGRUCj4NMqmaPWhS+jZwQMhHTykjkJEVmb2U93BwQHr1q3D+fPn4e7uju7duyMkJARPPvmkNfKRhRWVafHtsVwsiwyROgoRSUAmhBC1eWFubi727t2LtWvXQqPR4MiRI5bOdk+BgYFIS0uT5L2bos/3Z2DH2WvY+2Z/yOU84U/UFNX0uWl2T+CNN97A7du30aFDB/Ts2ROrV6+Gvz+vI28KSit1WHP4EuaP6MICQGSjzF4i6uXlBXt7exQWFiIvLw+3bt1CRUWFNbKRhW08ehkuTgqMeLSN1FGISCJm9wTmzp0LACgpKcF3332H2bNnIz8/H+fOnbN4OLKcCq0eq/6bhTcHPQyFHR8rQWSrzBaBuLg4nDp1CteuXUNAQAAmTpyIHj16WCMbWdCmE1cgA/BMz3ZSRyEiCZktAr6+vhg9ejTatm1rjTxkBVq9ASt/voiX+vvBUWEndRwikpDZIjBs2DDs2rULX375JQCgZ8+eGDZsGIeOaMR2n7uB4godonp3kDoKEUnMbBH48MMPoVKpMGLECAghsGvXLhw/fhzvv/++2ZXHx8dj69atUCgUWLhwIXx9fY3zSktL8dFHH+HatWswGAxYt27dA3WEam/D0Rw826sdmjny5j8iW2f2U+DEiRPYvn27cfrJJ5/E008/bXbFKpUKCQkJiI+PR2pqKmJjY7Fs2TLj/Li4OIwdOxa9evW6z+h0PzLzSnD0UgE+Hh0sdRQiagBqNYrohQsXjNNpaWmwt7c3u+IzZ84gNDQUdnZ2CA4ORnZ2tsn8kydPYteuXYiJicH69evrnpzuy8ajl9HXvwX8vJpLHYWIGgCzewLz58/HW2+9hbs3Fsvlcnz88cdmV6xWq+Hq6mqc/uONySkpKXj99dfxzjvvYOLEiejTpw86deKDzS2pXKPH5hO5WDT2UamjEFEDYbYIPProo9ixYweKi4shhDD5YK+Jq6sr0tPTjdNyuelOh4eHB/r27QuZTIY+ffogPT2dRcDCfjh7DY72dniqi4/UUYiogTB7OCgvLw9z587F+PHjERUVhblz5yIvL8/sirt164bk5GTo9XqkpKSYnBQG7lxllJqaCgA4d+5clflU/zYcvYzxvdrDnjeHEdH/mN0TePPNN/HMM89gzpw5AIDdu3fjjTfewDfffFPjcu7u7hg1ahSee+4549VBSUlJKCkpQUREBGbNmoX33nsPFRUV6N27Nx555JH66RHdU8q1Ipy9osI/n+ONfkT0G7OjiI4cORLbtm0zaXv22Wfx3XffWTRYdTiK6P2Zvfks8ksr8eULf5I6ChFZ2QONItqnTx8sXboUgwcPhkwmw549exAWFoZLly4BAB56iM+kbegKSzXYevoqvmIBIKI/MFsE7h63P3XqlEn7yZMnIZPJsHbtWssko3oTfywX7T2V6NephdRRiKiBMVsEeCdv46bTG7DuSDamhnfiUB9EVAUvE2ni9qXeRHGlDmNCOAAgEVXFItDEfX04G8/2as9xgojonlgEmrDz19VIzi7A82G8B4OI7q3ar4exsbE1HkOeOXOmRQJR/Vn3Sw7CA73h26KZ1FGIqIGqtgj4+flZMwfVs5JKHbaduoplUSFSRyGiBqzaIjB69Ghr5qB6tu30Vbg52+OJQG+poxBRA2b2bGF6ejqWLFmCixcvQqPRGNsPHjxo0WB0/4QQWP/LZUT17gA7OS8LJaLqmT0xPHfuXMyYMQOurq7Yu3cvXnnlFUyYMMEa2eg+nc5VIf1mMZ79U3upoxBRA2e2COh0OnTp0gV6vR5KpRLR0dFISkqyRja6TxuPXsZTnX3g4+okdRQiauDMHg5ydnaGVqtFYGAg4uLi4O3tjbKyMmtko/tQVKbFjrPX8MXzfGwnEZlndk9g0aJFMBgMmD9/PoQQuHjxIpYvX26NbHQfNp+8Am8XJ/Tzbyl1FCJqBMzuCTg5OUEmk8HFxQXTp0+HRqNBUVGRNbJRHen0Bqw5dAkT+z0EOU8IE1EtmN0TeOWVV0ymhRCYOnWqxQLR/fvx3A2oy7UYzxPCRFRLtTox7ODgYJx2dHQ0uVSUGgYhBFYlZSG6jy/HCSKiWjNbBHx8fLB161bj9NatW+HjwweVNzRHsvKRdqMYL/btKHUUImpEzBaBjz76CImJiejfvz8GDBiAAwcOYOHChdbIRnXwRVIWRoe0hTcvCyWiOjB73MDb2xvLli2zRha6T+k3i/FT+i3sm9Ff6ihE1MhUWwSWLl2KmTNnYubMmfccTTQ2Ntaiwaj2vj6cjfBAb3TydpE6ChE1MtUWgcGDBwMAIiMjrRaG6q5co8eO09ewdHx3qaMQUSNU7TmBrl27AgASExPRu3dvkz+JiYlWC0g125NyA472cjwR6CV1FCJqhMyeGD569GiVNo4g2nBsOpGLUd3bwt6OD4kjorqr9nDQ6tWrsXr1aqhUKjz22GPGdplMhuHDh1slHNXsSmEZDl/Mx7zhj0gdhYgaqWqLwMSJEzFx4kQsW7YMr7/+ujUzUS0lnLiK4LZuCGzFE8JEdH/MHkMICwszjhr6/fff45NPPkFubq7Fg1HNDAaBzSdzMa5nO6mjEFEjZrYIfPjhh1Aqlfj111+xfv16BAQEYPbs2dbIRjVIzi7AzaJKPN2trdRRiKgRM1sE7t4jsH//fsTExGDs2LF8nkADEJ98GU894gM3pb3UUYioETNbBLy8vDBnzhzs2LEDTzzxBDQaDfR6vTWyUTWuqsqx4+x1TOz3kNRRiKiRM1sEli9fjvDwcKxevRru7u5QqVQ8HCSx1QcvoUcHd/T09ZA6ChE1ctVeHXT69Gl0794dJ06cgFKpxJUrV3DlyhVrZqN7KCrT4pvky/g8MkTqKETUBFRbBJKSktC9e3fs3LnznvN/f+8AWc/6ozlo7eaEJ4O8pY5CRE1AtUXg7r0Bo0ePRu/evU3mJScnWzYV3VOFVo81hy7hb0OC+PhIIqoXZs8JfPLJJ7Vqu5f4+HhERkYiOjoaOTk5Vebr9XoMHToUX331Va3WZ+u2nLwKuUyGkSFtpI5CRE1EtXsCR44cweHDh3Hr1i0sXbrU2F5SUnLPoaX/SKVSISEhAfHx8UhNTUVsbGyV5xIkJCSgQ4cODxDfdgghsObQJbzYryMcFXZSxyGiJqLaIuDh4QE/Pz/Y29vjoYd+uxRRqVRi+vTpZld85swZhIaGws7ODsHBwcjOzjaZX1lZicTERAwZMgQqler+e2Ajki8VIDu/FON78SHyRFR/qi0CQUFBCAoKwujRo+9rxWq1Gq6ursZpIYTJ/HXr1mH8+PEoLCy8r/Xbmg1HL+PPXVujRXNHqaMQURNi9pxAeno6pkyZgoEDB+Kxxx4z/jHH1dUVxcXFv72R/Le3Ki4uRnJyMsLDw+8ztm25XVKJH89dR3QfX6mjEFETY/YZw3PnzsX777+Pd955Bxs3bsSWLVugVqvNrrhbt25YsWIF9Ho9Lly4AF/f3z7AsrKyUFhYiEmTJiEvLw9arRZBQUHo16/fg/Wmidp84go6tmiGP3XkzWFEVL/MFgGdTocuXbpAr9dDqVQiOjoakZGRePXVV2tczt3dHaNGjcJzzz0HhUKBhQsXIikpCSUlJYiIiMCmTZsAAFu2bEFhYSELQDUMBoGNRy9jYr+OtTohT0RUF2aLgLOzM7RaLQIDAxEXFwdvb+9aDyAXFRWFqKgo4/Tv9wbuGjNmTB3i2p6DmbeRV1yB0T04ZDQR1T+z5wQWLVoEg8GA+fPnQwiBixcvIi4uzhrZCMD6X3LwdLc2cHPmaKFEVP/MFoF9+/bB0dERLi4umD59OubMmYP9+/dbI5vNyy0ow/7zNxHTp6PUUYioiTJbBHbs2FGlbdu2bRYJQ6b+73A2evl6Iridm9RRiKiJqvacwKZNm7B582ZkZ2cjMjLS2F5SUoJOnTpZJZwtK6nU4dtjuVgy7lGpoxBRE1ZtERg6dCjCwsKwfPlykwfNK5VKeHjwUkVL23w8F25KezzVpZXUUYioCav2cJBCoYCPjw8WL16Mtm3boqysDHv37sUvv/xizXw2yWAQWHM4Gy/27Qg7jhZKRBZUbRGYOHEibty4AQDIzMxEVFQUrly5gk2bNuGjjz6yWkBblHghD7eKKzGO4wQRkYVVeziopKQE7dvf+RDasmULRo0ahblz50Kn02HUqFFWC2iLVh+6hHE92/GyUCKyuGr3BIQQxkHfDh8+jAEDBgC4c5iId65aTtqNYhzJyscLfTtKHYWIbEC1ewKDBw/GpEmT4OHhAa1Wi7CwMADAlStXoFQqrRbQ1vzfkWz0f9gLfl7NpY5CRDagxsdLnjp1Crdu3cIHH3wAheLOS7VaLebNm2e1gLakqEyL709exYrnekgdhYhsRI1jB4WEhFRp+/0DZqh+bTqRCx9XRwwI8JI6ChHZCLN3DJN16A0Ca4/kICasIx8iT0RWwyLQQPyUlofbJZUY14ujhRKR9bAINBBfH87GmB5t4erEy0KJyHpYBBqAk5cLcfhiPv7Sj+dbiMi6WAQkJoTA33dfwNgebeHPy0KJyMpYBCT234zbOJmjwhuDAqSOQkQ2iEVAQkIILNmThug+vmjr7ix1HCKyQSwCEvrx3A1k3SrBtHB/qaMQkY1iEZCI3iDw6d40THrcDy2aO0odh4hsFIuARH44ew23iysx+XFeEURE0mERkIDBIBCXmIm/9HuI9wUQkaRYBCSwO+UGrhdVYCLvCyAiibEIWJkQAssTM/F8mC/clNwLICJpsQhY2f7zeci+XYpJj3EvgIikxyJgRUIIxCVmILpPB14RREQNAouAFf2UfgsXbhRjSn8/qaMQEQFgEbAaIQQ+25eO6D6+8HZxkjoOEREAFgGr+Sntzl7AywO4F0BEDQeLgBUIIfCP/dwLIKKGh0XACg6k5SH9JvcCiKjhYRGwMCEEPtufgehQ7gUQUcOjsOTK4+PjsXXrVigUCixcuBC+vr7GebNnz0ZOTg70ej0mTJiA0aNHWzKKZLafuYbMvBJ8+UIvqaMQEVVhsSKgUqmQkJCA+Ph4pKamIjY2FsuWLTPOnzp1Kjp27AiNRoOnn34aw4cPh71907qDtqhciw9/OI8ZgwK4F0BEDZLFDgedOXMGoaGhsLOzQ3BwMLKzs03md+zYEQCgUNypQ3J50zsyFbs3DS2bO+DFfh2ljkJEdE8W2xNQq9VwdXU1Tgsh7vm6L774AsOGDYOdnZ2lokji7BUVNhy9jO9e7gN7u6ZX4IioabDYp5OrqyuKi4t/e6N7fNPfvn07zp8/j2nTplkqhiT0BoF3vz+HcT3boaevp9RxiIiqZbEi0K1bNyQnJ0Ov1yMlJcXkpDAAJCUl4fvvv8ff//73Jnco6Jvky7hSWIbZQ4OkjkJEVCOLHQ5yd3fHqFGj8NxzzxmvDkpKSkJJSQkiIiIwZ84ceHt7Y9KkSQCApUuXwsvLy1JxrKaoTIvYvWmYNTgQHs0cpI5DRFQjmajuYH0DFRgYiLS0NKljVOuDHSk4cjEfO19/HHZymdRxiIhq/Ny06H0CtibjZjHWHcnB+smhLABE1Cg0rYPxEhJCYMEPqRjySCv08WshdRwiolphEagnu369geRLBZgTwZPBRNR4sAjUg+tF5Xjn+1/xt6FBaOehlDoOEVGtsQg8IINBYNZ3Z/BoOzf8pW9HqeMQEdUJTww/oC8PZuH8dTV2v9kfcp4MJqJGhkXgAZy7WoRP96Rj+YQQ+LhygDgianx4OOg+FZVpMXXDCYzr1Q5DHmkldRwiovvCInAfDAaBGd+dhqfSAfNGdJE6DhHRfePhoPsQdyATpy4X4ofXH4ejommNfkpEtoVFoI4OpOVh2X8ysOYvf0Jbd2ep4xARPRAeDqqDlGtFeG3DSbw1JBCPP9z4B7sjImIRqKVrqnJM/PoYRoW0xUv9/aSOQ0RUL1gEakFdocXEr4/hkTZu+ODpRyCT8X4AImoaWATMqNDq8dLa41DYybA8KgQKPiqSiJoQnhiugUZnwNT1J5BXXInvXg5DM0f+dxFR08KvtdXQ6Q2Y8e1pZOSVYMPkULRs7ih1JCKiescicA9CCMzbnoJj2QXYMDkUrd14KSgRNU08vnEP637JwdZTV7H5lb7wbdFM6jhERBbDPYE/OJx5Gx/+kIqlzzA9vH4AAAy7SURBVHZDlzauUschIrIoFoHfyckvxasbT+K18IcxtGtrqeMQEVkci8D/HL54G+P+fQR9/Vtg+sBOUschIrIKmz8noNMbsCwxEysOZGLqE/5448mH+XAYIrIZNl0EMvNKMDvhLC4XlOH/JvZGv04tpY5ERGRVNlkEtHoDViVl4fP/ZOCpLj5YGdOT9wEQkU2yuSJwOleFOVt+RUFpJeKiQjCYTwUjIhtmM0VAXaHFkt1p2Jh8GZF/ao+/DQ2Cm7O91LGIiCRlM0Xg1fUncbukEpteCUOPDh5SxyEiahBspgh8Ftkdbs72sOcooERERjZTBHjil4ioKn4tJiKyYSwCREQ2jEWAiMiGWbQIxMfHIzIyEtHR0cjJyTGZl5OTg+joaERGRiI+Pt6SMYiIqBoWKwIqlQoJCQnYsGEDZs+ejdjYWJP5sbGxePvtt7F+/XokJCRApVJZKgoREVXDYkXgzJkzCA0NhZ2dHYKDg5GdnW0yPzs7G127doVCoUDv3r1x9uxZS0UhIqJqWKwIqNVquLr+9lAWIYTJ/N9Pu7m5oaioyFJRiIioGhYrAq6uriguLv7tjeSmb/X7abVaDTc3N0tFISKialjsZrFu3bphxYoV0Ov1uHDhAnx9fU3m+/r6IiUlBUFBQTh27BgmT55c63UHBgbWd1wiIpskE388TlOPvvnmG2zbtg0KhQILFy5ETk4OSkpKEBERgZycHLz77rvQ6XQYNWoUIiMjLRWDiIiqYdEiQEREDRtvFiMismEsAkRENoxFgIjIhrEIEBHZMBYBIiIb1iSLQE0D182bNw8TJkzAmDFj8PXXX0sTsA5q6gsA6PV6DB06FF999ZUE6eqmpr68/fbbGDNmDGJiYjBv3jyJEtZeTX0pLS3FnDlz8MILLyAmJkaihLVTUz9mzJiBmJgYxMTEoHv37rhw4YJEKWunpr4kJSVhzJgxiIyMxIIFCyRKWHs19WXv3r0YN24coqKi8PPPPz/4m4kmprCwUDzzzDNCp9OJs2fPiunTp5vMr6ysFEIIodVqxeDBg43TDZG5vgghxLfffiumTJkivvzySwkS1p65vsyePVucPXtWonR1Y64vixYtEseOHZMoXe3VZvsSQoj8/HwRERFh5XR1Y64v48ePF1evXhVCCPHSSy+J8+fPSxGzVmrqi06nE8OGDRPl5eWirKxMjBkzRuh0ugd6vya3J2Bu4DoHBwcAQGVlJXx9fWFvby9Bytox15fKykokJiZiyJAh0gSsA3N9AYAFCxYgJiYGBw8etH7AOjDXl5MnT2LXrl2IiYnB+vXrpQlZC7X5mQDA7t27MXToUOuGqyNzfQkICIBarYZer0dlZSXc3d2lCVoLNfWlsLAQXl5ecHJygrOzM1xcXO55hKAumlwRMDdwHQD87W9/w+DBgxEcHAyZTGbNeHViri/r1q3D+PHjG3Qf7jLXl9mzZ2PTpk347LPPsHjxYpNxpxoac31JSUnBk08+iTVr1mDv3r3IzMy0dsRaqc3vCgDs3LkTw4cPt1as+2KuL0OGDMHkyZMxdOhQ+Pn5oVWrVtaOWGs19cXT0xN5eXkoKChAfn4+zp8/D7Va/UDv1+SKgLmB6wDg73//OxITE3Ho0KEG+wsK1NyX4uJiJCcnIzw8XIpodWbu5+Lh4QEAaNGiBbp27YpLly5ZNV9d1KYvffv2hUKhQJ8+fZCenm7tiLVSm9+V69evo7KyEg899JA1o9WZub58+OGH2LJlC/bs2YPCwkKcOHHC2hFrraa+yOVyzJ07F2+88Qbef/99dO7cGV5eXg/0fk2uCHTr1g3JycnQ6/VISUmpMnCdRqMBcOewkLOzMxwdHaWIWSs19SUrKwuFhYWYNGkS1qxZg02bNuHQoUMSpq2ZuZ/L3Y2+srISqampaNu2rRQxa8VcX3r27InU1FQAwLlz56rMbyjM9QMAfvjhBwwbNkyCdHVjri8KhQIuLi6Qy+VwdXV94G/PlmSuL2FhYVi3bh0WLFgAR0fHB/5daZJjB9U0cN1LL72E8vJyaDQaDBkyBBMnTpQ6bo1q6stdW7ZsMRaEhqymvkyZMgXFxcXQ6XSIjIzEM888I3XcGtXUl9zcXLz33nuoqKhA7969MXPmTKnjVsvc9jVmzBj861//go+Pj8RJzaupLzt37sSaNWtgb2+PNm3aYPHixVAoLDaI8gOrqS+LFi1CSkoKHB0d8c4778DPz++B3qtJFgEiIqqdJnc4iIiIao9FgIjIhrEIEBHZMBYBIiIbxiJARGTDWARIUo888ghGjhyJiIgIvPvuuzAYDFZ5X7VajU2bNhmn//Of/2Dt2rUPvN4pU6YY70WpjX79+hn/vWHDBkyYMAEVFRU4cuQIRo0ahaFDh2LkyJE4cuTIfWf65ptvsHv37vtenpq4Bxp5iOgB9e3bVwhxZ2Cs6OhosXv3bpP5Wq223t9Tq9WK3NxcMW7cuHpfd13d7f+2bdvE6NGjhVqtFjdu3BADBgwQGRkZQgghMjIyxIABA8SNGzekjEpNVMO9W4Jsip2dHbp164bLly9jy5YtOHToEFQqFVq2bIlXX30Vc+bMQXFxMfz8/PDJJ59AqVRi4MCBiIiIwE8//YTWrVvjH//4B5o3b46ff/4ZsbGxMBgMGDJkCKZPn44rV65g+vTpCAoKQkpKCoKCgpCeno6RI0dixIgR8PT0RFZWFv7617/i7NmzeP/996HVatGjRw/MmzcPdnZ26NevHyIiInD48GF06NABcXFxsLOzM+nHwIED8eOPP+L06dNYtWoV7O3tkZWVhdGjR2Pq1Kn37PuBAwewcuVKrF27Fi4uLvjyyy8xduxYdOrUCQDQqVMnjB07Fhs3bsSMGTOqvN/IkSOxf/9+eHt749VXX8Wnn36K/Px8xMbGIjg4GMuXL0fLli0RFRWFkydP4uOPP4ZGo0GbNm3w73//G3l5eZg3bx7y8vKgVCrxySefoH379pb5QVPDI3UVItt295tweXm5eOaZZ0RiYqJISEgQgwcPFiUlJUIIISZPniz27dsnhBBi8eLFYsWKFUIIIcLDw8XGjRuFEEJ8/vnnIi4uTpSXl4vw8HBx/fp1odFoxPjx48Xx48dFbm6u6Ny5s0hPTxdCiCp7AgkJCWLJkiVCCCGGDRsmUlJShBBCvPHGG2Lbtm1CCCECAgKMQ0S//PLL4uDBg1X6Ex4eLioqKsQvv/wiwsLCREFBgSgrKxP9+/cXpaWlVV7ftWtXERoaKq5cuWJse/XVV8X+/ftNXrdv3z4xbdq0e77f9u3bjctNmzZN6PV6sW/fPjFz5kwhhBDLli0TGzduFJWVlWLQoEEiMzNTCHFnyGIhhHjzzTdFamqqEEKI5OTkaoeUpqaJ5wRIUiqVCiNHjsT48eMRFhZmHBDvscceQ7NmzQAA58+fx6BBgwAATz/9tMngX3eH0R4yZAhOnjyJS5cuwd/fH61atYK9vT0iIiKMr/f398fDDz9cYx61Wg2DwYAuXboAAEaMGGFc3tXVFb169QIAdO7cGdeuXatxXSEhIfDw8ICzszPatWuHW7duVXmNUqlEp06dsGvXrpr/o2pw9/8sICAAvXv3hlwuR2BgYJV8WVlZ6NChA/z9/QHAOJzy0aNH8fbbb2PkyJH46KOPkJeXd99ZqPHh4SCSlLu7O7Zt21al3dnZ2fjv2g6VLZPJIIQweb343agov19ndWpa/u6zKIA7oznqdLoa11Wb1ysUCvzzn/9EVFQU2rZti4iICPj7+yM1NRVPPvmk8XXnz583fnj/0d1nYsjlcuO/ZTIZ9Hp9ldfe6/9SLpfj+++/v+cootT08adODV5QUBASExMB3BnVsmfPnsZ5e/bsMf4dEhICPz8/ZGZm4ubNm9DpdNizZw969OhRZZ1KpRKlpaVV2t3c3CCXy42PUty1a5fJ+1mCm5sbVq5ciSVLluD48eOYMGECNm/ebBzm/OLFi9i8eTOioqIe6H38/PyQk5ODrKwsAHf2woA7o55u3rwZAGAwGJCRkfFA70ONC/cEqMGbO3cu5syZg3/84x/GE8N35efnY8yYMfD09MRnn30GJycnzJ8/H1OmTDGeGO7VqxeuXLlisk5PT0906tQJI0aMwMiRI+Hp6Wmct3DhQrzzzjvQarUICQmxylDK7du3x+eff47XX38da9aswccff4xZs2ahsrISDg4OWLhw4QM/CMXBwQGLFi3CX//6V+h0OuOJ4ffeew/z5s3D+vXrodPpMH78eLOHzajp4Cii1GjdvRKnIT8Tgqih4+EgIiIbxj0BIiIbxj0BIiIbxiJARGTDWASIiGwYiwARkQ1jESAismEsAkRENuz/AcqzhFX+G+R9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rng = np.random.RandomState(304)\n",
    "N = 100\n",
    "weight_gain_wt0 = rng.binomial(N, 0.3, 100000)/N # Get proportion\n",
    "weight_gain_ko0 = rng.binomial(N, 0.3, 100000)/N # Assume first (null hypothesis) that there is no difference\n",
    "\n",
    "diff_weight_gain0 = weight_gain_ko0 - weight_gain_wt0\n",
    "cutoff = np.quantile(diff_weight_gain0, 0.95)\n",
    "\n",
    "p1 = np.linspace(0.3, 0.9, 100)\n",
    "power = np.zeros(len(p1))\n",
    "for i, p in enumerate(p1):\n",
    "    weight_gain_wt1 = rng.binomial(N, 0.3, 100000)/N\n",
    "    weight_gain_ko1 = rng.binomial(N, p, 100000)/N\n",
    "    diff_weight_gain1 = weight_gain_ko1 - weight_gain_wt1\n",
    "    power[i] = np.mean(diff_weight_gain1 > cutoff)\n",
    "\n",
    "sns.lineplot(p1, power)\n",
    "plt.axhline(0.8, color = 'black', linestyle = '--');\n",
    "plt.ylabel('Statistical power')\n",
    "plt.xlabel('Proportion in KO mice');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "name": "04-python-stat-21"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.17"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(p1[np.argmin(np.abs(power - 0.8))] - 0.3, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The minimum detectable difference for 80% power is now down to 0.17, so we'd need the KO mice in truth to show weight gain 47% of the time, compared to 30% in WT mice. This is more reasonable scientifically as a query."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A permutation test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A permutation test is a 2-group test that asks whether two groups are different with respect to some metric. We'll use the same proteomic data set as before. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea about a permutation test is that, if there is truly no difference then it shouldn't make a difference if we shuffled the labels of ER status over the study individuals. That's literally what we will do. We will do this several times, and look at the average difference in expression each time. This will form the null distribution under our assumption of no differences by ER status. We'll then see where our observed data falls, and then be able to compute a p-value.\n",
    "\n",
    "The difference between the simulations we just did and a permutation test is that the permutation test is based only on the observed data. No particular models are assumed and no new data is simulated. All we're doing is shuffling the labels among the subjects, but keeping their actual data intact. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "lines_to_next_cell": 2,
    "name": "04-python-stat-24"
   },
   "outputs": [],
   "source": [
    "nsim = 10000\n",
    "\n",
    "rng = np.random.RandomState(294)\n",
    "x = np.where(brca['ER Status']=='Positive', -1, 1)\n",
    "y = brca[test_probe].to_numpy()\n",
    "\n",
    "obs_diff = np.nanmean(y[x==1]) - np.nanmean(y[x==-1])\n",
    "\n",
    "diffs = np.zeros(nsim)\n",
    "for i in range(nsim):\n",
    "    x1 = rng.permutation(x)\n",
    "    diffs[i] = np.nanmean(y[x1==1]) - np.nanmean(y[x1 == -1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "name": "04-python-stat-25"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAD5CAYAAADV5tWYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dfViUZaI/8O+8MMAAw8DAAIKCorwlUmlhmZpZWtppPa0WqB07WyfPns1zfrvtpm17Omfbn9eeU2u763Xq/Nq21mrbcM2XrE3bshcKXN+FAAHlZXgHgYEZZmCGmXl+f6AoYswAMzzz8v1cF5cNzwzzjQu+Pt7Pc9+3RBAEAURE5NOkYgcgIqLJY5kTEfkBljkRkR9gmRMR+QGWORGRH2CZExH5AblYb5yeni7WWxMR+ayqqqrrfl60Mge+PRT5kJYWIDERaG4Gpk0TO43v4veRXDDWSTCHWYiI/ADLnIjID7DMiYj8AMuciMgPsMyJiPwAy5yIyA+wzImI/ADLnIjID4g6aYhoqujNVhjMg6M+r1IGIUqpECERkXuxzCkgGMyDKKrpGvX5Rakaljn5BQ6zEBH5AZY5EZEfYJkTEfkBljkRkR9gmRMR+QGWORGRH3Dp1sSCggIcOHAAcrkc27dvR3Jy8vCxbdu2obq6GmFhYZg5cyaef/55j4UlIqLrc1rmPT092Lt3LwoKClBRUYEdO3Zg586dI57z85//HNnZ2R4LSUREY3M6zFJSUoLc3FzIZDJkZ2ejvr5+1HOef/55PPLII/j66689kZGIiJxwemZuMBigUqmGHwuCMOL41q1bERUVha6uLjz66KPIyclBRESE+5MSEdG3cnpmrlKpYDQar7xAOvIlUVFRAACNRoO5c+eirq7OzRGJJk4QBFzo6ENZcy+sNofYcYg8xumZeU5ODl555RXY7XZUVlaOuPgJAEajEREREbBYLKioqEBiYqLHwhK5Sm+y4rdHzuOTinY09/RDLpXAIQhIiAzFAznTMD1aKXZEIrdyWuZqtRpr1qzBhg0bhu9mKSwsRF9fH1atWoUf/ehHMBqNsNlseOSRR6DRaKYiN9F1V0IUBAFFtV3Y8ddqJEWF4qkVabhjdgx6zYPYf7YZpxv0+F1hLVbNS8DCmdEiJSdyP5duTczPz0d+fv7w46vPzl977TX3pyJywbUrIdodAg6WNKOsuRc/WZmBTbenQCaVAAD6rXYkRSmRFKVEiiYM+840o9dsxR2zY8SKT+RWXAKX/MKg3YHdJxrR2G3Gy+tvxl2Zcd/63HlJakSGBuH3X9Xhi6oObIqZOYVJiTyDM0DJ59kdAv74Nx3aDQPYvDQVqbHhTl+TrAnDqux4/NfhKlzo6JuClESexTInnyYIAj4obUGH0YJ/WjwL0WGubzSxcJYGi2Zr8C/vnOKdLuTzWObk04prunC2oQePLEyGKjRoXK+VSCR46p40GAds+P3XtR5KSDQ1WObks8paenGorBUP3zId09ShE/oaSoUcz92fhZ1HzqNJb3ZzQqKpwzInnzQwaMd/H65E7kwNMhNUzl/wLax2BzLjIzAvMRJb3yuFrtMEvdnqxqREU4NlTj7p159Uw2YXsPKG+El9HZPFhuLabiyeE4tjdd14o6hu1L3rRL6AZU4+52xjD37/dR2eXpkOhXz0j7DV7oCu0zTiwzxoH/NrasKDcXuqBofL22B3CGM+l8gbsczJpwiCgOc/KMe6+Um4aUbUdZ9jsthQVNM14sPipMwBYGmaFr39g/ikot3dsYk8jmVOPuXTcx2oaDXgh/ekuf1rhypkuDNNi9eL6jDgQvkTeROWOXkdvdk6aphEb7bC7hDw4seV+MdFMxGnCvHIe9+WOrS20NtHdR75+kSewun85HWuXXMFABalanDkXAfaDRb889JUj713kEyK7y1Kwf98fgEPLZiOSOX47l0nEgvPzMkn9A/a8auPq5C3YDp6TFaXLmpO1IqseMSpgvG/X9Z45OsTeQLLnHzCkXPt0JutiI8MGddFzYmQSSXYem8G/lBUh9befo+8B5G7sczJJxwsacWtKdEIDpJNyfvdlaFFznQ1fvPJ+Sl5P6LJYpmT12voNuN8h3H44qSnWe0ONHSZ8ehtyXjvVCO+rOrgrFDyeixz8npFFzpxe6oGaqXrKyJOxuX71Hv6bUiLi8Cv/lrNWaHk9Vjm5NX0ZivKW3rxQM40Ud7/7qw4lDX3cs1z8nosc/JqJ+v1mKYORVpchCjvnxAZirmJkXijqE6U9ydyFe8zJ6/lEAScbtDjrnStqDmWZ2ix87PzOPxN66gVGlXKIERN0fAP0VhY5uS1zrf3od9qx7ykSFFzaFUhWDInFr/+9Dw23Z4y4tiiVA3LnLwCh1nIa53UdSM7MXLKbkccy9r5iahuN6JZz/vOyTuxzMkr9VlsONdqwIKU66+MONWSopSYmxiJz6s6xI5CdF0sc/JKZxr00IQFY0a0Uuwow5ala3Gu1YC23gGxoxCNwjInr3SmoQfzk6MgkUjEjjIsPjIEmQkqnp2TV2KZk9epudiHdsMAcqarxY4yyp3psShr7kW3iTNCybuwzMnrfFLRjpkxYYgM9b7lZ5OilEjWhKG4plPsKEQjsMzJqzgcAo5UduBGLzwrv+yO2TE4qdOj38rdiMh7sMzJqxyv74bebMUN08S9t3wsGQkRiAiW46SuW+woRMNY5uRV3j/bjNtmaRCqEP/e8m8jlUhw++wYFNd0wWZ3iB2HCADLnLyIxWbHX0pbcU9mnNhRnJo/IwpWmwNfXeDYOXkHl8q8oKAAeXl52LhxI3S60Rvd2u123HvvvXj99dfdHpACxxdVFyEAyJ0VLXYUpxRyKW6aocYHpa1iRyEC4EKZ9/T0YO/evXjnnXewdetW7NixY9Rz9u7dixkzZngkIAWO9882Y9XcBATLvXeI5Wq3pETjtE6P+k6T2FGInJd5SUkJcnNzIZPJkJ2djfr6+hHHLRYLPvvsM6xcudJTGSkAGAYG8em5DnznJnHWLZ+IOFUI5iZGouBEo9hRiJyXucFggEp1ZdlPQRBGHH/77bfx8MMPe9VMPfI9h8vaoAlTYOHMqdkazl0eyEnAe6caYbXxQiiJy2mZq1QqGI3GKy+QXnmJ0WjE8ePHsWzZMs+ko4Bx4EwzHsiZBqnUt04Kls6JxaBdwCcV7WJHoQDntMxzcnJw/Phx2O12lJeXIzk5efhYbW0t9Ho9HnvsMfzhD3/Anj17UFRU5NHA5H/aegdwtLYL37kxUewo4xYcJMODNyei4ESD2FEowDndnEKtVmPNmjXYsGED5HI5tm/fjsLCQvT19WHVqlXYs2cPAGDfvn3Q6/VYtGiRx0OTf/mgpAVztOHITBBna7jJ+u7NSXjgf75Gh2EAWlWI2HEoQLm001B+fj7y8/OHH199dn7Zgw8+6L5UFFD2nWnGmpsSffa6yw3TVEiNDcfBkhY8vniW2HEoQHHSEImqss2AyjYD1vjgEAsAWO0ONHSZcWd6LHafaISu0wS9mSsq0tRjmZOo9p8emr4/TR0qdpQJMVlsKKrpglqpwPmOPuw70wyDeVDsWBSAWOYkGrtDwIGzzfj7m3zzrPxqUUoFUjRhONvYI3YUClAscxJNcU0nevsHcV92gthR3OKmGWqcbeyB45q5GERTgWVOotl/uhkrsuIRHuzSdXivN3daJPoGbKhoMYgdhQIQy5xEYbbacLi8DX9/s+8PsVwWqpBhtjYcX1RfFDsKBSD/OCUin/NxeRtCg2SYERUK3TULVZkHfXcHn7mJKnxZfRGCIPjsrZbkm1jmJIp9p5txV4YWx+r0o45l+ejkIQDITFDhwNkWlDT1evXWd+R/OMxCU67dMICiC51YkeX9m1CMl1Ihx80z1Dj0Ddc5p6nFMqcp9/7ZZszWhmOONlzsKB6xNC0WH5W1jlphlMiTWOY0JfRmK3SdJug6Tdh9ohHL0rTo99NlY3NnRqNZ349PK9qH/585K5Q8jWPmNCUM5kEU1XShtbcftRdNUM9XwOLDFzrHEiSTIkUThneONWDFDfEAgEWpGkQpFSInI3/GM3OaUqd0eqTGhiMyNEjsKB6VmaBCZZvR+ROJ3IRlTlPGZnfgTEMP5qdEiR3F4zLiI9BmGODwCk0ZljlNmXOXzlSzElROnun7NOHB0EYEo7KVs0FparDMacqcrO/GjdPVCJIFxo9dRjyHWmjqBMZvFYmu3TCACx19WBAAQyyXZSZEoLbT5LcXesm7sMxpShwqa0OCOgQJkb65bvlETI9WIlguxfmOPrGjUABgmZPHORwCDpW1YUFytNhRppRUIkF6XAQq2zhuTp7HMiePO1rbhW6zFTlJgbdWScalWxTtDs4GJc9imZPH7T7RiKVzYhCqkIkdZcrN0YbDMuhABe9qIQ9jmZNH9ZoHcbi8Dav8ZDeh8QoJkmFmbBiKazrFjkJ+jmVOHvV+STPiVMEBvRxsRnwEimu6xI5Bfo5lTh61+0QjHpo/HdIA3qghM16F+i4zGrrMYkchP8YyJ48pa+5FRasBaxckiR1FVFFhCsyMCcOn59rFjkJ+jGVOHrPnZCOWzIkNqHvLv83tqRocqWSZk+ewzMkjBgbtOHC2BQ/fMl3sKF5hUaoGx2q7YRgYFDsK+SmWOXnEx+VtkEkluDvT/7aGm4jMBBXUyiAUVl8UOwr5KZY5ecSek034+5sSoZDzRwwYmg26LF2LI+c6xI5Cfoq/aeR2jd1mFNV04qEFHGK52vJMLT6v6oDN7p/b5ZG4WObkdu+dasK8JDXS4yPEjuJVFs+Jhdlix+mGHrGjkB9yqcwLCgqQl5eHjRs3QqfTjTj23HPPYf369XjwwQexa9cuT2QkH+JwCNh7ugnr5gf27YjXExYsx8JUDY7wFkXyAKcbOvf09GDv3r0oKChARUUFduzYgZ07dw4f/9nPfgaFQgGbzYbVq1dj/fr1UCi4cW2gOlbXjQ6DBTcmRULXaRr+vJlregMA7s7U4s3iejyzKlPsKORnnJZ5SUkJcnNzIZPJkJ2djfr6+hHHLxe3xWJBcnIygoL8e6NeGtt7p5pwx2wNSptHLiyVlcAhFwC4K0OL594vR32nCSkxYWLHIT/idJjFYDBApbqyZ6MgjF7K8+mnn8aKFSuQnZ0NSQBP2w50JosNh8pace/ceLGjeB2r3QFdpwl2u4DU2DC8d6oRuk4TN3wmt3Fa5iqVCkbjlX0MpdLRL3nhhRfw2WefoaioCBcuXHBvQvJaerMVTd1D6400dZvx9tF6KBUyZE2LFDeYFzJZbCiq6UJRTRemRyvxUVkbimq6YDBzEhG5h9Myz8nJwfHjx2G321FeXo7k5OQRx63WoTMLhUKB0NBQBAcHeyYpeR2DeRAn6vUAgBP1euw+2YSshEjeeudEZrwK9Z0mDPA6ArmR0zFztVqNNWvWYMOGDZDL5di+fTsKCwvR19eHVatW4cknn0R/fz+sVitWrlyJ6dN5b3Eg6jFbUddpwndypokdxeslRoVCqZCjut2I5RlaseOQn3Ba5gCQn5+P/Pz84cdXn53/7ne/c38q8jnfNPdielQotKoQsaN4PalEgvT4CFS2GZ0/mchFnDREblHW3Iubk6PEjuEzMuMjUNVmhM3BISlyD5Y5uUWfxYZ5iYG7m9B4pWrDMWh3oLyZe4OSe7DMyS3mxEUE5IbNExUsl2FWbBi3kyO3YZnTpFjsQ3dkzEvk7YjjlRGvQnEtN3om92CZ06Scv3QRL1nD2YzjlREfgYbuftRc7BM7CvkBljlNSnnr0JivTMqZv+OlViqQGR+Bw2VtYkchP8AypwnrMVtRf9ViWjR+S9JicaisVewY5AdY5jRhX1ZfhCqUK2ROxtK0WJQ1G9Dc3S92FPJxLHOasCOVHcjiBhSTkqgORVaCCp9Xc41zmhyWOU1Ia28/Spt6kZmocv5kGtN9c+Nx5Bw3eqbJYZnThHxY0oqUmDBowzl9f7Luy45HeUuv2DHIx7HMaUIOlrRwkSg3ma2NwCxuVEGTxDKncavrNOGb5l6WuRsty4gVOwL5OJY5jdvBsy3Ima7GNHWo2FH8xrL0OABAZ9+AyEnIV7HMaVwEQcDBkmY8wHXL3WpOXDgA4MsqTu+niWGZ07hUtBpQ22nC/fMSxI7iVy7vnftZFW9RpIlhmdO4HCxpwcKZGsRxEwq3uLzR8+W9VE/X61HXybVaaPxY5uQyQRDwYUkrHriRQyzucnmj58t7qYYHB+HQN1yrhcaPZU4uO92gR4dxAPfNjRc7it9Kiw/Hl9WcQETjxzInl31Q0oolc2KhVnI9Fk/JiFfhlE6P3v5BsaOQj2GZk0tsdgc+LG3F3/EuFo+apg5FlDIIn1bwQiiND8ucXHKsrht9lkHckxUndhS/JpVIsCQtFh99w2VxaXxY5uSSg2dbsDwjDmHBcrGj+L2labH46nwnDAMcaiHXsczJKavNgY/KWrFwVjR0nabhD/OgXexofik9PgIRIXLsPt4w/L3Wm61ixyIvx9Mscqqw+iIcDgEOASi6ajf5rASuZe4J/VY75sSFY9+ZFoQFBwEAFqVqEMULzzQGnpmTUx+UtmDxnBgEyfjjMlXmJkbifLsRA/zXD7mIv500pn6rHZ9UtOMurpA4pVI0YQgJkqGqzSh2FPIRLHMa05HKdgTLpZg/I0rsKAFFKpHghmkqfNPMTSvINSxzGtMHJS1YlZ0AOYdYptzcxEhUtxthsXGohZzjbyh9K8PAID6vusiJQiJJ0YQhWC7lUAu5hGVO3+qv5e2IUgbh1pRosaMEJJlUgqxpkSjjUAu5wKUyLygoQF5eHjZu3AidTjfi2NatW5GXl4d169Zh//79HglJ4jhY0oL7502DVCoRO0rAyk6MRFW7Ef1WDrXQ2JzeZ97T04O9e/eioKAAFRUV2LFjB3bu3Dl8/Pvf/z5SUlJgtVrxwAMP4P7770dQUJBHQ5Nn6c1WNHSaUHT+ItbfMp0ThEQ0MyYMQTIpjtV1IWOaSuw45MWcnpmXlJQgNzcXMpkM2dnZqK+vH3E8JSUFACCXD/29IJVy5MbXGcyDeL2oHpFKBbpMVhTVdMHCMheFTCpBVoIKX3BZXHLCafMaDAaoVFfOCARBuO7zXnvtNaxevRoymcx96Ug0pU29mJcYObydGYlnbmIk/lbbxaEWGpPTMlepVDAar1xNv96Z98GDB3Hu3Dn84Ac/cG86EkWHcQC6LhPmTVeLHYUApMaGQy6TctMKGpPTMs/JycHx48dht9tRXl6O5OTkEccLCwuxf/9+vPDCCxxi8ROfV11EbEQw4rnPp1eQSSW4Y3YMl8WlMTm9AKpWq7FmzRps2LABcrkc27dvR2FhIfr6+rBq1So888wz0Gq1eOyxxwAAL730EmJjYz0enDznyLkO5PCs3KvcmRaL5z+swMCgHSFBHMqk0VxaNTE/Px/5+fnDj68+Oy8qKnJ/KhJNfacJVe1G3D8vQewodJX5yVGQSSUorL6IFTdwD1YajeMiNML+M83IjI+AJjxY7Ch0lSCZFPdkxeNQWZvYUchLscxpmCAI2H+mGSt55ud1rHYHFiSr8dfyNlS3GbhpBY3CMqdhp3R6tPT0Y1k6r3l4G5PFhkG7ALsg4O2/NaCopgtFNV0wmLm1HA1hmdOwfWeasSxDCzV3tPFKcpkUWQlcFpeuj2VOAACLzY6/lLbiwZsSxY5CY8hOVKOi1YBBu0PsKORlWOYEAPi8sgOCIOCuTO4o5M1StWGQSSSobueyuDQSy5wAAHtPN+P+nGkIlvMeZm8ml0qRxR2I6DpY5gS9yYovqjo4xOIj5iVGorLVCKuNQy10Bcs8gOnNVug6TXizuB6x4cGICVNwuVsfMCs2HHKZBFUcaqGruDQDlPyTwTyIopou7D3dhIwEFYpruwEAWQkRIiejscik3OyZRuOZeYDrNFrQqO/HTVyLxadkJ6pR1WaA2WoTOwp5CZZ5gDvT2IMZ0UpO3/cxM2PCoJBJ8bdL/5oiYpkHMIcg4GyjHjfN4Fm5r5FJJbghMRKfV3WIHYW8BMs8gJ1t7IFxwIbsxEixo9AEzEuMxN/qutFn4VALscwD2oelrZibGAmlgtfBfVFKTBjCFTIcOdcudhTyAizzAKU3WVF4/iIWJEeJHYUmSCqRYGlaLD4o4Q5ExDIPWAfONkMbEYKZMWFiR6FJuDszDl9Wd6DbxKVwAx3LPAAJgoCC441YnR0PiUQidhyahBumqTA9SokDZ5rFjkIiY5kHoLONPbhwsQ/3chMKnyeRSPDd+UnYc6pJ7CgkMpZ5APrTsQYsz9Dy3nI/8d2bk1DVZkAZZ4QGNJZ5gNGbrDhY0oJ/uC1F7CjkJvGRIViSFos9JxvFjkIiYpkHmPdONSFRHYrbUzViRyE3Wjd/Ot4vaYHFxkXSAhXLPIA4HAL+eEyHDQuTIZXywqc/uTtLCwmAw2VtYkchkbDMA8hXFzrRbhjA2puTxI5CbhYslyH/1hl4o6he7CgkEpZ5AHn7aD3W3JiISGWQ2FHITax2B3SdJug6TViWHouyph58Wc31WgIR53EHiLpOE45UduAvWxaLHYXcyGSx4UTrlU0qsqZF4s2ieixN416ugYZn5gHija/rcHuqBlnTVGJHIQ9alKrBl9UX0drbL3YUmmIs8wDQY7Ziz6lGPL54lthRyMOmRysxJy4CbxbrxI5CU4xlHgDeOdaAhMhQpEQrh8dXudenf5JIJFh/6wy8fbSe67UEGJa5n7PaHHizuB7fvTkRR2u7UVTTNfxhYZn7pcVzYjBDE4ZXC2vEjkJTiGXu5z4oaYHdIWBFZpzYUWiKSCUS/OieNLxVrMNFo0XsODRFWOZ+TBAE/P7rOmxcmIzgIJnYcWgK3Z2pRVpcOP73C56dBwqXyrygoAB5eXnYuHEjdLqRF1ZefvllLFu2DJs3b/ZIQBo/vdkKXacJB840o6bDiDvTYjk+HmAkEgl+eE8a/nhMh4Yus9hxaAo4LfOenh7s3bsX77zzDrZu3YodO3aMOP7QQw/hrbfe8lhAGj+DeRBFNV34f1/WYl6SGufajBwfD0BL02KxKFWD/zhYBkEQxI5DHua0zEtKSpCbmwuZTIbs7GzU19ePOB4bG8sNDrxQh3EA1e1GLJodI3YUmmKXZ4U2dJmxefEsFNV0Ye9prnfu75zOADUYDFCprkw04d/wvqHoQidma8MRpwoROwpNsWtnhd6ZFov/PlyF++YmICyYk779ldMzc5VKBaPxyg+GVMprpt6uwziA07oeLE2LFTsKeYE75sRAGSTD9o/OiR2FPMhpM+fk5OD48eOw2+0oLy9HcnLyVOSiSdh9ohGJUaHcrJkAAHKpFD9bnYn3TjbhL6WtYschD3Fa5mq1GmvWrMGGDRvwy1/+Ek899RQKCwvx0UcfAQB2796Nn/zkJygpKcGjjz6Kzs5Oj4emb9fZZ8EHpa1Yls5rGXRFWlwEnlmVgW37StHYzbtb/JFLA2j5+fnIz88ffnz12fnDDz+Mhx9+2P3JaELe+LoO06OVSIuLEDsKeZlHb09B0YUuPPmn09i9+TaEcO6BX+EAuB/pMVvx9lEdNubO4Fk5jSKRSLBjXQ6MAzb8eE8JHA7ezOBPWOZ+5NXCWiRGhfLCJ32rSGUQ3nj0Fnx9oRO/+bRa7DjkRrxPyU90GAewq6geO/NvgpRn5TSGlJgwvLpxPh55/Tg04cG485q//FXKIEQpFSKlo4limfuJVz6vQVp8BO7O1HL6No1yeSLRZfGqEDx9bzp+8WEFanNnID3+ylySRakalrkPYpn7geaefvzpWAPeePQWjpXTdV07kQgAlAo5HlmYjD8e0+HxO2ZherRSpHTkDhwz9wMvHK7EgpQoLJqtETsK+Zjv3DgNuTM1ePNoPTq5XK5PY5n7uJP13fiwtBX/fn8Wz8ppQu6dG4852nD8obgOxoFBsePQBLHMfZjdIeA/DpZj/a0zkJnAjZppYqQSCb47PwmasGDsKq6HyWITOxJNAMvch/35ZCOa9P340T1pYkchHyeXSrEhdwYkEuDf3y+DxcYlk30Ny9xHdRgG8F+HKvHjlemICuOdBzR5wUEybLotBS29A/jxnlJOKvIxLHMfJAgCfrr/G2QmRGDDrTPEjkN+JCIkCL9aOw/FFzrxy0NcZdGXsMx90L7TzSiu6cKLa3MglfKiJ7lXUpQSv9+0AG8d1eHto/VixyEX8T5zH9Pc04///KAcP12VifAQ+YiJIJdxv0+arJtmROG3eTfiyT+dQWJUKO7KiBM7EjnBMvcRerMVFw0D2PLuWWQnRuKOVA3aegdwpqFn1HOzErhiIk3evXMTsO2+fjz5pzP48+bbMDcxUuxINAYOs/gIg3kQP91fhot9FizPiENxbTc3aSaPuDz1X9dpwt0ZWqzMisOmN46jorVX7Gg0Bpa5jzhY0oKSph5syJ2BUAXXoSbPMVlsKKrpQlFNF4pru3FzcjRiI4Lxg3fOoLefk4q8FcvcB/y1vA2/PXIea+dPR0JkqNhxKMDIpBI8fMt0BMul+N6uEzBbOanIG7HMvdzxum5sefcM/uXOVGRzzJJEEiyXYfuauejus+AfXj+O821G6DpN0JutYkejS1jmXuyUrhuP7TqBf1o8C9+9OUnsOBTgZFIJHr51Bmo7TdhScAZfVF+EwcxhF2/BMvdSX52/iI2/P44NC5Px1ApO1yfvoAoJwuN3zMRFowVvHa3nkIsXYZl7offPNuOxN09iy/LZ2HZfBldDJK+iVirwxJJZ6LPY8OP3StFt4lCLN2CZexGb3YH/+2EFfrKnFL/4zg34lztnix2J6LoiQoLwT3fMglQiwXde/hrV7UbnLyKPYpl7CV2XCetfO4YPSlvw2qb5WDhTM3yvr67TxFmd5HWUwXK8tC4HuTM1ePCVYhwuaxM7UkDjDFCR2ewOvHlUhxc/rsTiObH4cMNimC/d53s1zuokb6SQS/Hi2nnISlDhX989g4duScLPVmchJIhzIaYay1wkgiDgs8oO/PJQJbpNVrywNgd/Ny8BEokEOm4OQD7CanegocuM5RlaJKlD8fMPK1Bc8xVeXJuD+W9RDroAAAi2SURBVMlRYscLKCzzKSYIAr6ouoj/+fwCypp78dgdM/HPd6ZCFRIkdjSicbt2o+jHF8/EN029eOjVo3hkYTJ+tCKNP9tThGU+RdoNAzhwuhl7TjWiUd+Pv5uXgGdXZUATFgx9nxX6vit3BHB8nHxVsFyGf1s+BxsWzsAz+77BwRdb8MN70pB/y3TIZbxE50kscw+7aLTgnWM6vFlcD6vNgdxZGjx0ywyEB8shlUhGjY0DHB8n3zc/ORof/etivHuiEb/+pBp/+LoOT941Gw/kTGOpewjL3AMEQcCJej0KTjTgw5JWzIoNw+YlsxCqkCOIP8jk5y6vuggAS2bH4Obpauw73YT/PFiO3x45j023pWDtgiQOv7gZy9yNOgwDeO90E/acbEKT3owVWfHY9Y+34LZUDRq6zNc9CyfyN9eOowNAqjYCb33vVvy1vB2vfVWLFz+uwp3psVhzUyLuzoyDjDtmTRrLfBIEQUBjdz8+q2zHkcoOFNd0YWZMGFZnx+OezDiolUMbLTd0mTkOTgQgKVqJHyybjeo2I0439mDLu2cQGRqEu9K1WJahRe7MaG5QPkEulXlBQQEOHDgAuVyO7du3Izk5efiYTqfDs88+C5vNhjVr1iAvL89jYcVkdwho0ptR3d6H8x1GlLcYcKpejzbDAOZow7E8Mw4/XpGOyBA5imu7UX7NmQnHwYmGSCUSZCSokJGgQk5SJGo6TfjsXDu27StFj3kQqbFhWJAcjfkpUZifHIUUTRjP3F3gtMx7enqwd+9eFBQUoKKiAjt27MDOnTuHj+/YsQPbtm1DRkYG8vPzce+990KtVns0tKeYLDY06fvRpDejSd+Pxu6hPxu6zai52AeLzQF1aBBSNErMig3H9++chawEFbQRIcNfg2fgRK5TyKXISYxETmIk/m35HDR0mVHWakBpYw9++2k1mnsGoJBLkaJRIi0uAvOSIjErJhwJ6hDEq0IQHabg2kWXOC3zkpIS5ObmQiaTITs7G/X19SOO19fXY+7cuQCAW2+9FaWlpViyZIlHwgJAv9WOnn4rHALgcAgQBMAhCBBw6U9BgEMAbHYB/YM2mK129Fvt6B+0w2y99GGxodtsRWefFV19FnT1WdFhHID+0nKemjAFkqJCkRSlRLJGiUVzYhChkKGjz4rw4CvfsqyECFS0GnG+wzTic0TkmuuNry9K1SBKqcDSdC1MFhvaDQNoNwxAIgE+Lm9HfWctui4t7qWQSxGvCkGUMghKhRxKhQyhChnCFPKhP4NlCJHLIJdJESSTQC6VXPXfUshlEgTJpJBLJZBJJZBKJJBIhv71MPQBSL7lz8vPGX6+FNe85srzAEAqHXocEx7skRshnJa5wWCASqUafiwIwojjVz+OjIxEb69n9wl8/K0TKLrg2oXEIJkEIUEyKBUyKBXyq/5bBk2YAqoQORJUwVArFYhWKhAfGYLEqFAorvONNg/aYb7O5slE5DlhwXLMig3HrNhw3JISNfy7abU5hk7EzFa09Q7AOGBD/6AdA4N2DNgc6LcOnbD1W+2w2uywOYZO+IChE71BuwM2hwCLzY5B29B/O4ShPrt8gnj58fDngauODX1+IjYvnYVn7st0y/fnak7LXKVSobq6evixVDqy6K5+bDAYkJGR4fKbp6enu/zcqwWP47nWSx+sYc/ZlpYGPLVR7Bg+j99H7ycBILv0MVG79gO7/o+bAl3FaZnn5OTglVdegd1uR2Vl5YiLnwCQnJyM8vJyZGRk4MSJE3j88cddeuOqqqqJJSYiolEkwrXjJtfx7rvv4v333x++m0Wn06Gvrw+rVq0KmLtZiIi8mUtlTkRE3o1zy4mI/ADLnIjID7DMiYj8AMuciMgP+FWZP/fcc1i/fj0efPBB7Nq1S+w4Ltu6dSvy8vKwbt067N+/X+w4Lnn55ZexbNkybN68WewoYyooKEBeXh42btwInU4ndhyXWK1W5OXlYcGCBTh8+LDYcVxSU1OD/Px8bNiwAY888ggaGxvFjuRUe3s71q1bh40bNyIvLw+VlZViR5ocwY9YLBZBEARhcHBQWLFixfBjb1dXVycIwlD+lStXClarVdxALujo6BAaGhqEJ554Quwo30qv1wtr164VbDabUFpaKmzZskXsSC6x2+1Ce3u7sHPnTuHQoUNix3FJV1eX0NvbKwiCIHz55ZfCT3/6U5ETOWez2QS73S4IgiAUFxcLTz31lMiJJsevzswViqGlMy0WC5KTkxEU5BuL36ekpAAA5PKhOVzXzrL1RrGxsV6/wJGzdYW8lVQqhVarFTvGuERHRw8v+yGXyyGTTWaO5NSQyWTDv2smkwlZWVkiJ5oc72+NcXr66aexYsUKZGdne33ZXOu1117D6tWrfeIXwRc4W1eI3K+/vx87d+7Epk2bxI7ikgsXLiAvLw+/+MUvsGDBArHjTIrPbU7R0dGBJ598ctTnN2/ejOXLl+OFF16AxWLBpk2bcN9992H27NkipBzNWe6DBw/i3LlzeOmll0RId33OMns7Z+sKkXvZbDb88Ic/xOOPP47U1FSx47hk9uzZKCgowLlz5/Dcc89hz549YkeaMJ8rc61Wiz//+c/XPWa1WqFQKKBQKBAaGorg4PEsyeVZY+UuLCzE/v378eqrr3pV4YyV2Rc4W1eI3EcQBDz77LNYsmQJ7r77brHjuORyXwBDf/GHhIQ4eYV386vp/E888QT6+/thtVqxcuVKfO973xM7kksWLVoErVaL8PBwAMBLL72E2NhYkVONbffu3Thw4ADq6uqQkZGBX/3qV4iJiRE71ijXrivkK4W+ZcsWVFRUQKlUYtGiRdi2bZvYkcZUWFiILVu2YN68eQCAjIwMPPvssyKnGtvJkyfxm9/8Zng4dtu2bbjhhhtETjVxflXmRESBynv+TU9ERBPGMici8gMscyIiP8AyJyLyAyxzIiI/wDInIvIDLHMiIj/AMici8gP/H2S7frZ/F5t9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(diffs)\n",
    "plt.axvline(x = obs_diff, color ='r');\n",
    "plt.axvline(x = -obs_diff, color = 'r');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "name": "04-python-stat-26"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'P-value from permutation test is 0.2606'"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pval = np.mean(np.abs(diffs) > np.abs(obs_diff))\n",
    "f\"P-value from permutation test is {pval}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is pretty close to what we got from the t-test. \n",
    "\n",
    "Note that what we've done here is the two-sided test to see how extreme our observation would be in either direction. That is why we've taken the absolute values above, and drawn both the \n",
    "observed value and it's negative on the graph. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing many proteins \n",
    "\n",
    "We could do the permutation test all the proteins using the array operations in `numpy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "name": "04-python-stat-27"
   },
   "outputs": [],
   "source": [
    "expr_names = [u for u in list(brca.columns) if u.find('NP') > -1] \n",
    "            # Find all column names with NP\n",
    "\n",
    "exprs = brca[expr_names] # Extract the protein data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "name": "04-python-stat-28"
   },
   "outputs": [],
   "source": [
    "x = np.where(brca['ER Status']=='Positive', -1, 1)\n",
    "obs_diffs = exprs[x==1].mean(axis=0)-exprs[x==-1].mean(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "lines_to_next_cell": 2,
    "name": "04-python-stat-29"
   },
   "outputs": [],
   "source": [
    "nsim = 1000\n",
    "diffs = np.zeros((nsim, exprs.shape[1]))\n",
    "for i in range(nsim):\n",
    "    x1 = rng.permutation(x)\n",
    "    diffs[i,:] =exprs[x1==1].mean(axis=0) - exprs[x1==-1].mean(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "name": "04-python-stat-30"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12395"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pvals = np.zeros(exprs.shape[1])\n",
    "len(pvals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "name": "04-python-stat-31"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-76-dee677ecb26c>:2: RuntimeWarning: invalid value encountered in greater\n",
      "  pvals[i] = np.mean(np.abs(diffs[:,i]) > np.abs(obs_diffs.iloc[i]))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Results of permutation test')"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAEHCAYAAACOWawdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXhTZcI28Dsna9M23XcKpUUKWBZZBAqCyCgKYkFEywsVZxB1dNBR5/0EFxwXRkdRsc74ufM5bhWHzQ1QFEVZKptsLYXuLV3oStM1TfJ8fxQitUvCkqSnuX/X1YvEnOTcJ165+/Q5SxRCCAEiIpINyd0BiIjo/LC4iYhkhsVNRCQzLG4iIplhcRMRyQyLm4hIZljcHubyyy9HUlISpk+fjocffhitra2X7LVTUlKQk5ODuro6fPbZZxf9eidOnMBNN92EWbNmob6+/hIkvDTeffddu8sUFxdj8+bNtvuffPJJu/vOXHdnMjMzsXPnzoteP/UMLG4P4+/vj40bN+LLL79EVVUVvv7660u+jktV3Nu2bcOcOXOwYcMG+Pj4XIJkgNlsvujXeO+99+wuc/LkSWzZssV2f968ebj++utdsu7OsLh7F5W7A5B7SJKEhIQElJeXAwByc3Px9NNPo66uDkFBQXjhhRdgMBiwdOlSZGRkQJIk3HXXXZg5cyauueYabNq0CVqtFuvWrUNubi7+9re/2V47NTUVx48fR1JSEmbOnImhQ4fi2WefhSRJ8PLyQlpaWrssTU1NeOyxx3DixAl4e3vjueeeQ2lpKd5//32oVCrs27cPqamptuXT09PxxhtvQJIkFBcXIzk5GX/84x8BAP/617+wbds2mEwm/OlPf8Ls2bOxbt067NixA7W1tQgODkafPn1QVlaG3NxcVFRU4LnnnkNaWhqOHDmCGTNm4P7770dxcTEeeughrFmzBgCwdOlSTJ8+HQcOHEBtbS2SkpIwduxY3H///bj33nthNBphtVqxbNkyjBs3rt17sGjRIhQUFCA4OBjz5s3Djz/+iJdeeglWqxXTpk3DkiVLUFxcjCVLliAuLg5HjhzB2LFj8dRTT7V7n1599dV263700Uc73d709PQO73dqaipMJhN++uknW0aSMUEeJTExUQghRHNzs5g/f77IyMgQQghxxx13iJKSEiGEEOvXrxfPP/+8OHz4sJg/f77tuXV1dUIIIaZMmSKam5uFEEKsXbtWvPjii0IIIRYsWCCys7NFUVGRmDt3ru15d999t9i1a1e71zjXW2+9JVasWCGEEOKHH34Qd9xxhxBCiNTUVPHxxx93WH737t1i5MiRoqKiQjQ1NYkZM2aIwsJC8cMPP4h//OMftu1LSkoSVVVVYu3ateK6664T9fX1ttddtGiRsFgs4ttvvxVjxowRhYWFoqWlRUyePFnU19d32IZHHnlE/Pjjj+3eQyGEMJlMwmg0CiGEKCsrE7NmzbJl/Otf/2pb7uy2NDU1iSlTpojS0lJhMpnEbbfdJvbu3SuKiopEQkKCyM/PF2azWdx0000iLy+vy/9/Z9+rzra3s/f73P9PJH8ccXuYsyO2wsJCjB8/HoMHD0Z9fT0OHDiAe+65BwBgtVoRFxeH6OholJSU4Nlnn8XUqVMxfvz4C1rnFVdcgZUrV2L27NmdThecu+7Jkyfj8ccft/uao0aNQnBwMABg4sSJOHToEA4fPozvvvsOu3fvBgAYjUYUFxfblvH29rY9f9KkSZAkCfHx8YiKikJ0dDQAICoqCpWVlVAqlQ5v38qVK7Fv3z5IkoS8vDxYLJYul83Ly0NcXBzCw8MBANOnT8e+ffswffp0xMXFoV+/fgCA+Ph4lJSUICYmpsvX2rVrV6fba+/9JvljcXuYs3Pc1dXVSElJwdatWzFu3DiEhYVh48aNHZb//PPPsX37drz55pvYu3cvlixZAkmSYLVaAQAmk8nuOu+++25MmjQJ27Ztwy233IJ169YhICDA9ri4gMvlKBSKDveFEHjggQcwc+bMdo9lZ2fDy8ur3X9Tq9W25529DbRNIZnNZmg0Gts2Al1v5+eff46WlhasX78eKpUKY8eO7XYeXQjRLvu5267RaDrk6E5X2zts2LAO7zf1Ltw56aECAwPx8MMP45133oGPjw8MBgN+/vlnAG0llZubi+rqagBto8K7774bmZmZAIDIyEhkZmbCarXixx9/7PDaer0eDQ0NtvtFRUUYPHgw7r33XkRGRqKsrKzd8iNHjrTtJN2+fTsGDBhgN/++fftQWVmJ5uZm/Pzzzxg6dCgSExPx3//+11ayJ06c6Hb0252goCCUlZWhsbERRqMR+/btsz2mUChspV5fX4/g4GCoVCp8//33qK2t7fQ9OCs2NhbZ2dkoLy+H2WzGli1bMHLkSIdznbvurra3s/e7qzwkTxxxe7ApU6bg1VdfxcGDB/Hiiy/iySefxAsvvACr1Yp7770X/fv3x7JlyyCEgFKpxPLlywEAf/7zn7F06VKEhYWhf//+HV43MDAQAwYMwMyZM5GUlISSkhKkp6dDkiSMHDkSgwYNarf8ggUL8Nhjj2HmzJm2nZP2jBgxAsuXL0dubi5uu+02REdHIzo6GllZWZgzZw6EEAgJCcHbb799Qe+NRqPBwoULMXv2bMTGxiI+Pt722I033ogbb7wREydOxD333IN77rkHc+bMwbBhwxAZGQmgbaqjqanJtnPyLJ1OhyeffBKLFy+27ZwcPXq0bUrHnnPX/eijj3a6vatXr+7wfoeFheGtt95CUlISd072AgpxIX+nErlReno60tLS8Morr7g7CpFbcKqEiEhmOOImIpIZjriJiGSGxU1EJDMsbiIimXHK4YDnHjpFRESOy8rKsruM047jdmTlRET0G0cHvZwqISKSGRY3EZHMsLiJiGSGxU1EJDMO7Zw8dOgQVq1aBZPJhMmTJ2Px4sXOzkVERF2wW9wmkwmvvfYa/v3vf3e4pjEREbme3amSAwcOwMvLC/fffz8WLVqE48ePuyIXERF1we6Iu6KiAllZWVi/fj1KS0vxxBNP4OOPP3ZFNiIi6oTd4jYYDBg5ciT0ej3i4uJgNBpdkatHqGk0oa6x1aFlDXo1AvQa+wsSEV0ku8U9fPhwvP7667BYLKiqqoJOp3NFrh6hrrEVO3KqHFp2QlwQi5uIXMJucfv5+WHu3LlISUmBxWLBI4884opcRETUBYcOB5wzZw7mzJnj7CxEROQAnoBDRCQzLG4iIplhcRMRyQyLm4hIZljcREQyw+ImIpIZFjcRkcywuImIZIbFTUQkMyxuIiKZYXETEckMi5uISGZY3EREMsPiJiKSGRY3EZHMsLiJiGSGxU1EJDMsbiIimWFxExHJDIubiEhmWNxERDLD4iYikhkWNxGRzLC4iYhkhsVNRCQzLG4iIplhcRMRyYzKkYVGjBiBoUOHAgBuv/12XHvttU4NRUREXXOouPv06YMPPvjA2VmIiMgBDk2VlJaWYsGCBXj44YdRXV3t7ExERNQNh4p769at+PDDDzF16lQ8//zzzs5ERETdcKi4AwICAAA33HADjh075tRARETUPbvF3djYCIvFAgDYu3cvoqKinB6KiIi6ZnfnZG5uLp544gno9XoolUo8/fTTrshFRERdsFvcCQkJWL9+vSuyEBGRA3gCDhGRzLC4iYhkhsVNRCQzLG4iIplhcRMRyQyLm4hIZljcREQyw+ImIpIZFjcRkcywuImIZIbFTUQkMyxuIiKZYXETEckMi5uISGZY3EREMsPiJiKSGRY3EZHMsLiJiGSGxU1EJDMsbiIimWFxExHJDIubiEhmWNxERDLD4iYikhkWNxGRzLC4iYhkhsVNRCQzDhf33r17ER8fj+rqamfmISIiOxwu7vfffx8JCQnOzEJERA5wqLi///57jBo1Cnq93tl5iIjIDrvFbbVa8cknn2DevHmuyENERHbYLe4vvvgC11xzDbRarSvyEBGRHXaL+/jx49iyZQsWLVqErKwsPPjgg67IRUREXVDZW+B///d/bbdTUlLwyiuvODUQERF1z25xn+uDDz5wVg4iInIQT8AhIpIZFjcRkcywuImIZIbFTUQkMyxuIiKZYXETEckMi5uISGZY3EREMsPiJiKSGRY3EZHMsLiJiGSGxU1EJDMsbiIimWFxExHJDIubiEhmWNxERDLD4iYikhkWNxGRzLC4iYhkhsVNRCQzLG4iIplhcRMRyQyLm4hIZljcREQyw+ImIpIZFjcRkcyo7C1QXl6Ov/zlL9BqtTCbzfj73/+OQYMGuSIbERF1wm5xBwcH49NPP4UkSdi1axfeeecdrFy50hXZiIioE3aLW6lU2m43NDRgyJAhTg1ERETds1vcAJCdnY3HH38cpaWleO2115ydiYiIuuHQzskBAwYgLS0Nb7zxBp555hlnZyIiom7YLW6TyWS7bTAYoNPpnBqIiIi6Z3eq5NChQ1i1ahUUCgUAYOnSpU4PRUREXbNb3KNHj8aHH37oiixEROQAnoBDRCQzLG4iIplhcRMRyQyLm4hIZljcREQyw+ImIpIZFjcRkcywuImIZIbFTUQkMyxuIiKZYXGfJ5PZiiaTxd0xiMiDOXQ9bgLqW8zYmVOJ3blVMFsExvQPxKTLQuDnpXZ3NCLyMBxxO8DY3IpVW48jq8yIpOFRmD+2L07WNOHlb7NQXNPo7nhE5GE44rZDCIH1B04izKDDoon9IZ25vO3AMF98cagUH/9SiL9cPcDNKYnIk3DEbceBwlrkVjRgzsg+ttIGAIVCgelDw2HQqbFmXxEsVuHGlETkSVjc3agwtuDLwyW4YWg4Ar01HR5XSRLmXdk2bfLffcVuSEhEnojF3Y21+4sR7KPFlTGBXS7j56XGDUMj8J/dBTjd2OrCdETkqVjcXWgxW/D1kTKMjw2yfW1bV0ZE+yPER4M3t+e4KB0ReTIWdxc2HykDIJAQ5Wd3WUmhwOKrYvHejjycqmt2fjgi8mgs7i58tLsQN1weAbXSsbcoMS4Il0f64dXvTjg5GRF5OhZ3J7LKjNhTUI2ZwyMcfo5CocDD1w3Emr1FKOeom4iciMXdiY/TCzBxQDD6BOjP63njY4MwKNyA93fmOycYERFY3B1YrQJfHynDLaP6nPdzFQoF7ryqPz5KL0SjyeyEdERELO4ODhbXoqbBhKvjQy/o+dOHRkCvUfK4biJyGhb373ybUY6xsYEXfPEotVLCHyfE4N2f83g2JRE5BYv7d77NKMe1g8Mu6jWSr+yLSmMLtmaWX6JURES/YXGfI7+yASdO1eMPQy6uuA06NW4Z1Yc7KYnIKewWd05ODubNm4f58+cjJSUFRUVFrsjlFlszyzE4wnDeR5N05vbEGOzKrcLxcuMlSEZE9Bu7xR0QEIA333wTH330ERYvXow33njDFbnc4puMclx7kaPts+JCfHDVZSEcdRPRJWe3uAMDA2EwGAAAKpUKSqXS6aHcobbRhL351Rc9v32uOxL7Yd3+kzjdxItPEdGl4/Acd1NTE1JTU7Fw4UJn5nGbXTlVCNBrcHmk4ZK95tUDQxFq0OKzvb13eomIXM+h4jabzXjwwQdx5513Ii4uztmZ3OLn7EqMjwuCJHV/JcDzIUkKpIzrh//sKuChgUR0ydgtbiEEHnvsMUyaNAl/+MMfXJHJLXbmVGHigOBL/rpzR0ejwtiCH7JOXfLXJiLPZLe4f/rpJ2zevBmbNm1CSkoKVqxY4YpcLlVS24S8ygZMcEJx+3mpcfPIKPw/7qQkokvE7pcFT5o0CQcPHnRFFrfZkV2J6EAvRAde/GGAnbkjMQbXrdqOnIp6xIX4OGUdROQ5eAIO2qZJJsRd+tH2WZeF+SIxLgj/4aibiC4Bjy9uIQR2ZFci0QnTJOdaOD4G/91XjLpmHhpIRBfH44s7+1Q9ThlbkBgX5NT1TB0chlCDDh/tLnTqeoio9/P44t6RXYlB4b4I9tE6dT1K6bfvpWwxW5y6LiLq3Ty+uNPzqjEu1rmj7bNuHhkFIYD1+0+6ZH1E1Dt5dHELIfBLXjXGxQa6ZH06tRJ/mhiDt7bn8oQcIrpgHl3c2afqUdVgwpgY1xQ3AMwf2w+njC345miZy9ZJRL2LRxf37rxqXBbqgyAnz2+fy89LjYWJ/bBq6wmOuonognh0cafnVmGsi6ZJznXXVXEoOd2ELw6WuHzdRCR/ds+c7K2EEEjPq8byG4dcktczWawoqGxwaFmDXo17Jsfh5W+PY8awCKiVHv37k4jOk8cWd35VIyqMLZdsxN3QYsaeUse+7WZCXBDuSIzB6h15+GxvMf5nbN9LkoGIPIPHDvXSc6sQG+yNUF+dW9bvrVXh3qsHYNXW4zDybEoiOg+eW9x51W6Z3z7XgnH9YPBSY9XWE27NQUTy4pHFLYRAem4Vruzv3uLWqCQ8k5SA93fm41hZnVuzEJF8eGRxF9c0oeR0M8b2d80Zk90ZHxeEG4dF4PH1R2Dl4YFE5ACPLO70vGpEB3oh0t/L3VEAAI/OGIysMiM+TC9wdxQikgHPLO7cqh4x2j4r1FeHZ2cnYMVXmcgqc+zIFCLyXJ5Z3HnVGOvm+e3fSxoRhRnDInD/JwfQ3MqrBxJR1zyuuEtPN6GwurFHjbjPejopAU2tFjz1RQaE4Hw3EXXO407ASc+tRoSfDtGB7pvf7u4sy+U3DsaSTw4gwEuNxZNjEaDXuDgdEfV0nlfcZ6ZJFAqF2zLYO8vy1tHReGN7DsIMWiyc0N+FyYhIDjxuqiQ9rwpjXfTFCRcqPtyAmcMjseLrTGw7dsrdcYioh/Go4j5lbEZuRUOP2zHZmbH9g7D4qljc/cE+fHWo1N1xiKgH8aipkvTcaoT4atE/2NvdURxy6+ho9AnU44G0A6isb8Ht4/u5dYqHiHoGjyrunTmVmBAXJKvym3dlXwTo1Xh4zUH8WlSLFbMToNd41P82Ivodj5oq+Tm7EokDgt0d47xdnxCBjX+ZgMMnT+Omf+3AvoIad0ciIjeyW9wmkwnJyckYPXo0Nm/e7IpMTlFU3Yii6iZMkGFxA8CAUF9svG8CJsQF4dY3d2H5xiOobTS5OxYRuYHd4lapVEhNTcXChQtdkcdpdmRXon+wN6J6yPVJLoS3VoWnkhKw5u7x+CWvGle9sA2vfXcC9S1md0cjIheyW9ySJCE0NNQVWZzq5+xKTBjQsw8DdNSofgH4+v6r8OysBKzdX4zxz32HZ77MQEGVY1+dRkTy5hF7uaxWgV05VXh2VoK7o1wykqRou77J0Ah8m1GO1TvzMfnFHzAmJgBJI6Jw3eVhbvt2HyJyLo8o7mNlRlQ3mjA+rneMuM+lUkq4YWgEbhgagexTRmw4UII3fszB4xuOYEiEAZMGhmDywBCM6hcAjcqj9kUT9VoeUdw7cyqREOkHf5ld9+N8vjleqVRALUmYO6oPbhkZhYKqRvySX409+dV4b0celApgSIQBQyINGB0TiIkDghHko3XyFhCRMzhU3EuWLEFGRgb0ej1+/fVXLF261Nm5LqmfTlTK8miS8/nm+CERvsj43bJRAXpEBegxY1gk8qsaUFjdiPS8amz4tQTGZjP6BupxeaQBQyIMGHym1CP8dLI6zp3IEzlU3K+99pqzczhNQ4sZu3KqcN+UAe6O4jZqpYTLQn1xWagvACAxNhBWAL8W1SKztA6786rw7o481Da2wl+vbhuZn1PmA0J9oFZymoWop+j1UyU/Z1fCS6PEyL7+7o7SY7RaBTRKCaP6BmBU3wAAbV+gXFHfguxT9cg+VY8Tp+qxJaMMRdVN0CglXBbmg8ERBgwK98XAMF8MCvdFiK+Wo3MiN+j1xf1dZjmmxIdAxRGjTfdTMArEhfoiLtQXY2ICYLZYkVPR0FboFfVYt/8k8iob0NRqgUGnQmywN/oHe2NIlB+uiPbHwHBfGHRql24Pkafp1cVttQp8f6wCy2cOcXcUWWpoMdvmzUMNOoQadEiMC4YQArWNrSiva0ZZXTNyKxtwsPg0Xth8DK0WgUg/HQaG+yI+3BfxYW3/xgb7wEujdPMWEfUOvbq4D508jZpGEyYPDHF3lF5FoVAgwFuDAG8NBkUYAAAT4oIQGeCF/MoGHCsz4ni5EcfKjNh8pAyF1Y0QAoj00yHmzAj93J/oQD3n0InOQ68u7u8zyzEmJgB+XvzT3dlMFitKatrmw4dF+WFYlJ/tsSaTBcW1jSiuaUJxTRNO1jbh16JaFNc04nSTGUoFEO7nhSh/L4T7aRHqq0Oorxb9Q7wxKNyAMIOOx6ATnaNXF/fWzFO4eWSUu2N4BEcOXdSolIgN8cGNwyJsUzCNJjOq6k2orG9BVYMJJ2uakVFiRG2TCcZmM1rMVigUQLCPFiE+WgT5aBDkrUGgd9vtYJ9zbntrEeCtho9WxZ2m1Kv12uIuqm5ERmkd/jXoCndHoW7oNSroA1WIDtR3eCwxNhAGvQYltU0oqW1CVYMJ1Q1tJV9Z34Lj5UZb4VfXm2ARAgCgkhQw6FQweKnhq1PDz0sFg04Ng5cafjo1Qv20iPL3gr9egwC9BgF6Nfz1Go7qSTZ6bXF/frAECVEGxIb4uDsKXaBWq4CxqRW+WhXiw3y7Xbbe1Ha8fkOLBU0mMxpNljM/bbcr600orGlCo8kMCUCDyYKaRhOaW62219BrlAjQa+CvVyNAr4Feq4SXSgk/LzUMZ8rfz0sNX50Kfl5tZe+tUUKhUMBw5jlErtAri1sIgfUHTiJ5TLS7o9BFON8zR/Ua1ZlvB+r+VP4JcUHod+br65pbLahtbEVNowk1jSbb7drGVhRWNeDEqQZklRvRaLKg6cwvgqZWC6xtg3soJQV8tCqEGbQI9/NCsLembTrHR4ugs7e9tQj01sCgU8Nbq5TNoak1jSbUNbY6tCx/cblWryzuoyV1yK2ox03DI90dhXqgzq4Bo1croffzQpTfb9drb2y14EBhbYfnW4VAS6sVDSYzGlrMqG8xI9yggwBQ3WBC6elmHDlZh6qGFlSfmd45W/QA4KVWwkengq9WBR+dCj7aMz86Fbw1KujUEnRqJYRoG4RoVRK0KiU0Kgk6ddttrUqCRiVBrZQgKX4rTkmhgFJSQFIooFAArRYrzBYBs9WKVouw3TZbBUxma9uPxWq73Wr57f6pumbkVDTAbBWwCgFJoTjz+oBSoYBKKcFLrYSXRokxMQGIDzec2eeg4T4GJ+uVxb3hwElMGBCMUAMva0odOTqSHxLR+fSMpFDAS9NWWMFnLtQ1JiYAmi5G0hargLG5FY0mCxrOjNiNzWY0trTdP/tvg8mCCmMLTGYLWsxWNLZaUN1gQqvlbOm2/Xu2jM/O6Z8vSdF2VUmtsq38z/4C0KgkaJQS1Kq2xyxCoL7ZDKXU9svAKgQsZ0rcYhVotQg0t1rQ1GrBp3uKYLK0TTtpVRLCDDqEGbQIM+jQL1CPAaE+GBDigz5Beo7ML4FeV9wWq8DGgyVYev0gd0chD3IhFwTTa1XQa1VAF9P3nV047FznFukV0f5QSQpYrAJCtD0m0LajVikpoJIkqJQKaNQShLXLl2ynq784usp6sPg0jM1m1J6dcmoyobS2GQeLalFW14xWi0CIrxbDovwwKiYAY/sHYVgfPx7DfwF6XXHvzKmEsbkV0xLC3R2FyKkkhQKS8rcpiUMn6+w+x94vg98vez7USgmB3m1TJb9nFQJV9SYYdCqUG1vwXeYpvPLtcejUSkweGIJrh4ThD4PD4K39rZI4x961XlfcH+0uxA0JEfDR9rpNI5ItSaFAiK/WNqU0b0w0mkwWHCiqwc6cKjz1+VEsXXsIEy8LwY1DIzC8jx+azFaHR/wT4oJY3HJVUNWAbzLK8PlfJro7ChF1ouOUkgLj44JxZf8g5FTUY39hDR5acxAhvlrMHRWFUIMXlBJ3dP5eryrud3/Ow7jYICScc7o1EfV8SkmBgWFtlww2Dm3Fnvxq/Gd3ASSFAlcPDMXIfgEs8HP0mr0CNQ0mrNlbhMVXxbo7ChFdBF+dGtcMCsObC0ZhXGwQvsksx2vfn0D2qXp3R+sxes2I+8PdBYgO0PNKgES9hE6txFWXhWBMTCC2HTuF93fmY2C4L6YnhHv896X2ihH36aZWrN6Zj8VXxULin1NEvYpOrcQNQyPwwNTLYLUKrPruBLYcLYPJ7OBxjb1Qrxhxv/xNFkJ9tbwSIFEvFuyrxcLEGGSVGfHloRIcKq5F0ogoDLRzHZveSPbFfbTkND5ML0TaXeNkcw0IIrpw8eG+iA25DNuyTuGDXQUY2scPQ8J9bdef8QSybjqrVWD5xqOYNSIKY2IC3R2HiFxErZRw3ZBw3HfNAFQ3mHD76l+wZm8RxAVeBkBuZF3cb/2Ui+PlRiy9gae3E3micIMOd02KxaKJ/fHMFxmY9/Zu5Fb0/qNPZFvc32aUY+WWLKTOuwIhvp69h5nIk0kKBWaNiMLWhycjQK/B9at+wotbjqGhxezuaE4jy+LOLK3DX9MOYNn0wZgSH+ruOETUA4QZdPi/C0bhzZRR2HS4DFNW/oA1e4pgtvS+o09kV9w7cyox7+3duGlEFP40IcbdcYioh5kyKBSb/zoJd02KxXObMnHtK9ux4cBJtPaiApdNcQsh8MGufCx87xfckRiDFbMSeLF2IuqURiXhzqtisf3/TMHsK6KwfOMRTH5hG978MQfVDSZ3x7toDh0OmJaWhg0bNkClUmHFihXo16+fs3O1syunCs9vPobsciNeTb4C04dGuHT9RCRPvjo17p96GRZN7I+1+4uxekc+Vn6ThavjQ3HT8EhMGhgCPy+1u2OeN7vFXVtbi7Vr1yItLQ0ZGRl46aWXkJqa6tRQQgjkVzXim6Nl+OpwKTJK6jB/bF+8c/to7ogkovPmrVXh9vExSBnXD/sLa7B2/0k89cVR1DS2YmRff4yOCcSIaH9cHmlApJ9Xjz8D225xHzx4EGPHjoVSqcTQoUORn5/vtDBV9S14cM1B/FpYg7pmM4ZEGDBjWAT+/T8jER2od9p6icgzKBQKjOoXiFH9AvFsUgKOlJzG9uMV2F9Yi0/3FKG6wQQvtRL9gvQI8dUixEeLYF8tgqOxkpAAAAWYSURBVH008PNSn/l6N+WZr3tTQIi2L5xuNlvQ0mqFVQBJIyLbfSGEM9h99bq6OhgMBtt9Zx7grlMrcfXAEPwxMQYJUX4cXROR00iSAsP6+GNYH38Abd1WYWxBdkU9CqoaUWFsQWV9C4prGvFrYS3qmlvbf7myxQpJoYBO1fblzlq1Er5aFSYNDHZ/cRsMBhw/ftx2X5Ic258ZHx9/4amIiGSi4Xf3p77i/HXaLe7hw4fj9ddfh8ViwbFjxxzaMZmVlXVJwhERUUd2i9vf3x+zZs3C/PnzbUeVEBGR+yiEp1yVhYiol5DNCThERNSGxU1EJDMsbiIimWFxExHJjNOKu6CgAAsWLEBycjLS0tI6PL5p0ybMnTsX8+fPx4MPPgiTyb0XfklLS0NycjIWLFiAgoKCdo/Z2xZ36y77I488guTkZMydOxfr1693U8KudZcdACwWC66//nq8++67bkjXve6yNzQ0YNmyZVi4cCFSUlLclLBr3WXfvn07br75ZiQnJ+Ppp592U8LOmUwmJCcnY/To0di8eXOHx3vyZ9Ve9vP6rAonWbJkiTh8+LBobW0Vt9xyi6ipqWn3eGFhoWhtbRVCCPHiiy+KDRs2OCuKXTU1NeKWW24RZrNZHDp0SCxZsqTd4/a2xZ3sZc/LyxNCCNHS0iKmTZsmTCaTG1J2zl52IYT49NNPxeLFi8U777zjhoRds5f9+eefF3v27HFTuu7Zy37bbbeJkydPCiGEuOuuu0RmZqY7YnbKYrGI8vJykZqaKjZt2tTh8Z78WbWX/Xw+q04bcefn5yMhIQEqlQpXXnklDh061O7x6OhoqFRth5ErlUoolUpnRbHL3vVY7G2LO9nLHhMTAwC299rRM19dwV72lpYWfP/995g2bZp7AnbDXvb9+/fj66+/RkpKCj788EP3hOyCvewDBw5EXV0dLBYLWlpa4O/v756gnZAkCaGhXX95Sk/+rNrLfj6fVad9isU5h4f7+fnh9OnTnS6Xk5ODHTt24LrrrnNWFLvsXY/F0W1xB0evJfP2229jxowZbv0F+Xv2sn/wwQe47bbbeuR11+1lP3r0KKZOnYrVq1fjm2++QXZ2tqsjdsle9mnTpuHOO+/E9ddfj9jYWISHh7s64gXryZ9VRznyWb2o4j516hRuvfXWDj/fffddu98WdXV18PPz6/D88vJyLF26FC+//DI0Gs3FRLkoBoMBRqPRdv/3v+kc2RZ3sZcdAD7//HNkZmbivvvuc2U0u7rLbjQa8csvv2DKlCnuiGaXvfc9ICAAiYmJUKlUGDduXLvr/bibvezPPPMM1q1bhy1btqCmpgb79u1zdcQL1pM/q45w9LN6UcUdGhqKNWvWdPiZOnUq+vXrh6NHj8JisWDPnj0YNmxYu+fW1dXhgQcewPLly9G3b9+LiXHRhg8fjl9++QUWiwVHjx7tcD0We9viTvayb9++HevXr8cLL7zQo6ZJgO6z5+bmoqamBosWLcLq1avx2WefYceOHW5M2569933UqFHIyMgAABw5csTlXz7SHXvZVSoVfH19IUkSDAYD6urq3JT0/PXkz6o95/NZddop7wUFBXjsscdgNpsxa9YsJCcnAwCWLVuG5557DitXrsSGDRvQv39/AMDs2bNx8803OyOKQz755BNs3LjRdj2WgoIC1NfXY/r06V1uS0/RXfYJEyYgNDQUPj4+AICXX34ZISEhbk78m+6yn7Vu3Tpbifck3WUvKirCE088gebmZlx55ZV46KGH3B23ne6yf/XVV1i9ejXUajUiIyPxz3/+0zbv2hMsWbIEGRkZ0Ov1mDBhAhITE2XzWe0u+/l8VnmtEiIimelZfzsTEZFdLG4iIplhcRMRyQyLm4hIZljcREQyw+ImIpIZFjcRkcywuImIZOb/A9l1tf30DVLUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in range(len(pvals)):\n",
    "    pvals[i] = np.mean(np.abs(diffs[:,i]) > np.abs(obs_diffs.iloc[i]))\n",
    "\n",
    "sns.distplot(pvals);\n",
    "plt.title('Results of permutation test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This plot shows that there is probably some proteins which are differentially expressed between ER+ and ER- patients. (If no proteins had any difference, this histogram would be flat, since the p-values would be uniformly distributed). The ideas around Gene Set Enrichment Analysis (GSEA) can also be applied here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "name": "04-python-stat-32"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "896"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exprs_shortlist = [u for i, u in enumerate(list(exprs.columns)) \n",
    "                   if pvals[i] < 0.0001 ]\n",
    "\n",
    "len(exprs_shortlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that, if we considered a p-value cutoff for screening at 0.0001, we would select 896 of the 12395 proteins for further study. Note that if none of the proteins had any effect, we'd expect 0.0001 x 12395 or 13 proteins to have a p-value smaller than 0.0001. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also do the same thing using both the t-test and the Mann-Whitney test. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAEHCAYAAACOWawdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de1hUdcIH8O/cuAww3EEQQi4qGl7ySpial7JMxU0tepX0fcy1bdda8+lNt63dp7bLpmuF1VtmWq9Wlpuo7eZlNcu8X/IOigIiCqIwwHCb++/9A0WJywzKzHCY7+d5fBhmzpz5Dj7z5fA7v3OOTAghQEREkiF3dQAiImobFjcRkcSwuImIJIbFTUQkMSxuIiKJYXETEUkMi5sc6u6770ZqairGjx+PBQsWwGQytdu609PTkZubC51Oh3Xr1t3x+s6dO4dJkyZh8uTJqK6ubrj/0qVL2LJlS4vP+/TTT2/r9bKzs7F3797bei65NxY3OVRAQAA2btyIf/3rXygrK8P333/f7q/RXsW9c+dOTJkyBRs2bICvr2/D/ZcvX8bWrVtbfN7KlStv6/VY3HS7WNzkFHK5HElJSSgpKQEA5OXlYdasWXj00UcxZ84clJeXw2Kx4IUXXsAjjzyCiRMn4rvvvgMAjB49GgaDAQCwfv16LFmypNG6MzIykJOTg9TUVKxYsQIHDhzAxIkTkZqairS0tCZZ6urq8Pzzz2PixIlIS0tDfn4+9u7di88//xwrV67Es88+22T9u3fvRmpqKjZt2tTosffeew8VFRVITU3FG2+8AQB4//33MWXKFEycOBGZmZkA0GymjIwMrF+/Hqmpqdi/f/+d/ojJnQgiB0pJSRFCCKHX68X06dNFVlaWEEKIWbNmiaKiIiGEEJmZmeKtt94SJ0+eFNOnT294rk6nE0IIMWrUKKHX64UQQnz77bdi8eLFQgghZsyYIc6fPy8KCwvFtGnTGp43d+5csW/fvkbruNXy5cvF66+/LoQQ4scffxSzZs0SQgiRkZEhvvzyyybL79+/X/zxj3+0+R5vrO+NN95oeM+pqamirKys2Uy3vheituAWNznUja3R5ORkaDQa9OrVC9XV1Th69CiefvpppKam4tNPP0VxcTGio6NRVFSEv/3tb9i3bx/8/Pxu6zXvueceLFmyBF988QWMRmOTx48ePYoJEyYAAEaOHInz58/f0Xu81b59+7Bjxw6kpqbiscceg06nw6VLl2xmImoLpasDUOd2Y4xbq9UiPT0d27dvR3JyMsLDw7Fx48Ymy2/atAm7du3Cxx9/jMOHD2PevHmQy+WwWq0AYFfpzZ07FyNGjMDOnTsxdepUrF+/HoGBgQ2Pizs8Pc+sWbNQXl6O0aNH47nnnmv0mBACzz33HCZOnNjo/r59+zbJRHS7uMVNThEUFIQFCxZgxYoV8PX1hUajwe7duwHUl3FeXh60Wi0AYPz48Zg7dy6ys7MBAJGRkcjOzobVasVPP/3UZN1qtRo1NTUN3xcWFqJXr1545plnEBkZiStXrjRafsCAAQ07SXft2oWEhIRWs/96/Z999hk2btzYUNoymazhF0tKSgr++c9/NvyCOXfuHCwWS7OZfr1eIntxi5ucZtSoUXjvvfdw/PhxLF68GH/5y1/w9ttvw2q14plnnkFsbCwWLVoEIQQUCgVeeeUVAMDvfvc7LFy4EOHh4YiNjW2y3qCgICQkJDTs/CsqKsKBAwcgl8sxYMAAJCYmNlp+xowZeOmllzBx4kT4+PjgzTffbDV3z549UVdXh9TUVMyePRuTJk1q9PiECRMwYcIE3HffffjTn/6Es2fPYsqUKRBCIDQ0FJ988glWrVrVJFN4eDiWL1+O1NRULFq0CMnJyXf4EyZ3IRN3+ncjERE5FYdKiIgkhsVNRCQxLG4iIolhcRMRSQyLm4hIYhwyHbBnz56OWC0RUad39uxZm8s4bB63PS9OREQ32bvRy6ESIiKJYXETEUkMi5uISGLsGuM+ceIE3n33XRiNRowcORJz5sxxdC4iImqBzeI2Go1YtmwZPvjgA3h7ezsjExERtcLmUMnRo0fh7e2NZ599FrNnz0ZOTo4zchERUQtsbnFfu3YNZ8+eRWZmJoqLi/Hyyy/jyy+/dEY2IiJqhs0tbo1GgwEDBkCtViM+Ph5VVVXOyEVERC2wucXdr18/fPjhh7BYLCgrK4OXl5czcnUI5bVG6GpNdi2rUasQqPZwcCIiIjuK29/fH9OmTUN6ejosFgtefPFFZ+TqEHS1JuzJLbNr2WHxwSxuInIKu6YDTpkyBVOmTHF0FiIisgMPwCEikhgWNxGRxLC4iYgkhsVNRCQxLG4iIolhcRMRSQyLm4hIYljcREQSw+ImIpIYFjcRkcSwuImIJIbFTUQkMSxuIiKJYXETEUkMi5uISGJY3EREEsPiJiKSGBY3EZHEsLiJiCSGxU1EJDEsbiIiiWFxExFJDIubiEhiWNxERBLD4iYikhgWNxGRxLC4iYgkRmnPQv3790efPn0AAE8++SQeeOABh4YiIqKW2VXcUVFRWL16taOzEBGRHewaKikuLsaMGTOwYMECaLVaR2ciIqJW2FXc27dvx5o1azBmzBi89dZbjs5EREStsKu4AwMDAQAPP/wwzpw549BARETUOpvFXVtbC4vFAgA4fPgwunbt6vBQRETUMps7J/Py8vDyyy9DrVZDoVDg1VdfdUYuIiJqgc3iTkpKQmZmpjOyEBGRHXgADhGRxLC4iYgkhsVNRCQxLG4iIolhcRMRSQyLm4hIYljcREQSw+ImIpIYFjcRkcSwuImIJIbFTUQkMSxuIiKJYXETEUkMi5uISGJY3EREEsPiJiKSGBY3EZHEsLiJiCSGxU1EJDEsbiIiiWFxExFJDIubiEhiWNxERBLD4iYikhgWNxGRxLC4iYgkhsVNRCQxdhf34cOH0bNnT2i1WkfmISIiG+wu7s8//xxJSUmOzEJERHawq7h/+OEHDBw4EGq12tF5iIjIBpvFbbVa8dVXX+GJJ55wRh4iIrLBZnF/9913GD16NDw9PZ2Rh4iIbLBZ3Dk5Odi6dStmz56Ns2fPYv78+c7IRURELVDaWuCFF15ouJ2eno533nnHoYGIiKh1Nov7VqtXr3ZUDiIishMPwCEikhgWNxGRxLC4iYgkhsVNRCQxLG4iIolhcRMRSQyLm4hIYljcREQSw+ImIpIYFjcRkcSwuImIJIbFTUQkMSxuIiKJYXETEUkMi5uISGJY3EREEsPiJiKSGBY3EZHEsLiJiCSGxU1EJDEsbiIiiWFxExFJDIubiEhiWNxERBLD4iYikhgWNxGRxLC4iYgkRmlrgZKSEvzhD3+Ap6cnzGYz/vrXvyIxMdEZ2YiIqBk2izskJARff/015HI59u3bhxUrVmDJkiXOyEZERM2wWdwKhaLhdk1NDXr37u3QQERE1DqbxQ0A58+fx5///GcUFxdj2bJljs5EREStsGvnZEJCAtauXYuPPvoIr732mqMzERFRK2wWt9FobLit0Wjg5eXl0EBERNQ6m0MlJ06cwLvvvguZTAYAWLhwocNDERFRy2wW96BBg7BmzRpnZCEiIjvwABwiIolhcRMRSQyLm4hIYljcREQSw+ImIpIYFjcRkcSwuImIJIbFTUQkMSxuIiKJYXETEUkMi7sNrELg5OVKHCnQwmyxujoOEbkpu87HTcDxwgpszy5BtcEMD4Uc/8kqwcgeoUiOC244ARcRkTOwuO2Qe60a644UYmyvcCTHBUMhl+FIQTm2ZV2B2SowvHuoqyMSkRthcdtgMFmw/pdLGN49FPf3DGu4PzkuGAHeKqw5UICugd4YFh/swpRE5E44xm3D5tNXoFLIMSYxrMljiREajOgRirUHC1FabXBBOiJyRyzuVpy6XInDF7SYOjAKSkXzP6qxvcIR6ueJD3aed3I6InJXLO5W/POXS+gbFYCoQHWLy8hlMkzoG4Efc67h1OVKJ6YjInfF4m7BtSoDdp0rxdDYIJvLRvh7Y3RiGN7eetYJyYjI3bG4W/DN4ULcFaTGXUEtb23favawWOw9X4q9uaUOTkZE7o7F3QyLVeCrgxeR2i/S7jnakQHe+K+hd2Hx1rMQQjg4IRG5MxZ3M3adu4ayaiMe6B3epuf9flQCTl2uxKEL5Q5KRkTE4m7W1wcLMalfJHw92zbNPVzjhYl9I/HJz3kOSkZExOJuQm+y4Mecq5jUP/K2nv/U8Dhszy5BfmlNOycjIqrH4v6V3edKoVLIMcSO2STN6R2pQUp8MFbuzm/nZERE9Vjcv7I9uwT39wyDqoUDbuzx1PA4rDtSiPIaYzsmIyKqx+K+hdUqsD37Ksb2anp4e1vc3yMUUYFqrD1U2E7JiIhuYnHf4vilClTUGhudTOp2yGQyzEzphtX7LvC83UTU7mwWd25uLp544glMnz4d6enpKCzsvFuR27NLMDQuCP7eqjte16P3dEWVwYzt2SXtkIyI6CabxR0YGIiPP/4YX3zxBebMmYOPPvrIGblc4j9ZJRjbq21zt1vi46nEY4OisWrPhXZZHxHRDTaLOygoCBqNBgCgVCqhUCgcHsoVCrW1yCmpbrfiBoAn743BwQtaZBfr2m2dRER2j3HX1dUhIyMDM2fOdGQel9mbW4puwWpE23luEnvEBPtgdM8wfL73Qrutk4jIruI2m82YP38+nnrqKcTHxzs6k0vsOV+GlISQdl/vzJRu2HDsMqcGElG7sVncQgi89NJLGDFiBMaOHeuMTE4nhMDe3DIMi2//4h7ePQRdA7zx9eHOu1OXiJzLZnH//PPP2LJlCzZv3oz09HS8/vrrzsjlVOeuVqO02oDkuNs7WrI1N6cGFnBqIBG1C5tnURoxYgSOHz/ujCwus+d8KXpFaBDs6+mQ9T86IAqLt5zF9uyreCipi0Neg4jcBw/AQf34tiOv0u7rqcTUQVHcSUlE7cLti9tsseJAXhmGOWDH5K1m3tsNB/LLeF1KIrpjbl/cp4p0qDNZMPg2zwZor24hPhh3dxd8vIvn6iaiO+P2xb03txT9ogPafNGE2/H0yHj8+0QRLpbVOvy1iKjzcvvi3p+ndchskub0iw7A0NhgXiGHiO6IWxe32WLFkQtaDIl13I7JX3v6/nh8c7gQpdUGp70mEXUubl3cp4t00JutGBgT6LTXHNE9BAlhvljxM6+QQ0S3x62L+2C+FkmRGqeMb98gk8kwf2wPfLY3H1d1eqe9LhF1Hm5d3AfyyzA0znnDJDeM6RWGXhEavL/zvNNfm4ikz3mbmh2M1SpwMF+LtMF3tcv6jBYrCuy8srtGrcIL43pi5sqDmDM8rl3PSEhEnZ/bFveZK1WoMpgxuFv7zCipMZhxqLjKrmWHxQcjJT4EQ2OD8c5/crD08f7tkoGI3IPbDpUczC9DYhcN/NV3fpmy2/XiQ4nYeLwIv1wsd1kGIpIety3uA/laDHXw0ZK29InyR9rgaPw58xTPHEhEdnPL4haifnzb1cUNAP8zLhElOj3W7C9wdRQikgi3LO7ca9UoqzFiSAcobn+1CovG98I/tuWghNMDicgOblncB/K1SAjzddj5t9tqyoCu6BvtjwXfHIfVKlwdh4g6OPcs7ryOMUxyg0wmw9LH+uN0USWW8zwmRGSD2xX3jfHtjjBMcqtwjRcWT+2Hf2w7i+OFFa6OQ0QdmNsV90VtLa7o9BjqxBNL2Wts73BMHxqD3605wvFuImqR2x2AcyBfi5hgNbr4e7ksQ2tHWaYPvQtZRZWYvuIAPvvvwYgK5FGVRNSY+xV3BxjftnWU5cNJEVj+cx6e//oYVj81FJ5KhRPTEVFH53ZDJQcvlDn1/Nu3w1OlwJP3dkNheR3m/N8R1Bktro5ERB2IWxV3UUUdCrV1Lt/itoe/twrL0u5BUUUdZq48CJ3e5OpIRNRBuFVx788rQ6S/F6ICvV0dxS6hfp74+rfJqDNZMPmDPcgpse8kVkTUublVce85X4aUhBDIZDJXR7FbsK8nvpl7L+6JDkTq+3uQefQShOBBOkTuzG2KWwiBvbmluC8hxNVR2szbQ4El0/rizxN6YdH6k5jzf4dRVFHn6lhE5CI2i9toNCItLQ2DBg3Cli1bnJHJIfJLa1BcqUdKfMfeMdkSmUyG6UNjsO2PI6E3WfHA0p+wbMc51BjMro5GRE5ms7iVSiUyMjIwc+ZMZ+RxmD25Zege5oswjevmb7eHu4LVWD17CN6c0hff/nIJI97eiY9/ykVlHXdeErkLm/O45XI5wsLCnJHFofaeL8UwCQ6TNEcmk2FSv0g8nNQF/zxyCct35eG9HecwZUAUHhsUjaSuGkmN4xNR27jFAThWq8C+vDK8PaWvq6O0iT3XsUyJC0ZybBAOFWjx7ZHLSP1gN2KC1Li/ZxhS4oPRPcy3SYlr1CoEqj0cGZ2IHMgtijurWAddnQnJEhvfbst1LHtH+CG1f1eMSgzDicIK7Dx7FZ/tvQA/LyV6dtEgsYsfYkN84KVSYFh8MIubSMLcorh3ny9F36gAaLxcd31JZ9F4qXBf91Dc1z0UtQYzcq5WIbu4CuuOFMJgsiIiwAtHL5ZjTK9wDO4W2GHOSU5E9rOruOfNm4esrCyo1WocO3YMCxcudHSudrX7XCmGJUhra7s9qD2V6B8diP7RgbBYBa7o9LhQWoMrOj3+lHkS2hojooO80bdrAPpG+aNPlD/6dPWHnxv8giOSMruKe9myZY7O4TDVBjMO5Jdh/gPdXR3FpRRyGboGeKNrgDeGxQfjrmA18kprcOJSBY4XVmJbVgne2Z4DvcmKuFAf9IsKQO8IDRIj/NArQoMQbpkTdRidfqjk55xr8PNSoX90oKujdCgymQzxob6ID/XFb+6JAgCYLVacu1qNE5cqcOJSJb47UYSl/6lCncmKIB8PxIf6XH9O/deYIDWUivoZpdzhSeQ8nb64d5y5ivt7hkIh5/S4G1qbraJWKZAcG4zk2GDUmiw4UlCO8hojiiv1uKLT48Sl+q1zbY0RCpkMYRpPdNF44d64YKQkhCAxwo9b50QO1qmL22oV2HnmKl5NTXJ1lA7F3tkqvSP8IJfJEOzriWBfTyR19W94TG+yoESnry/0Sj1+OncNn+27gFqjBSG+HkgI80X3MD90D/dtuB3i68H55UTtoFMX97FLFaisM2F4j85x4E1H4qVSICbYBzHBPgCAYfHBiA5S46K2FjklVTh3tRq5V6ux7vAlnL9ajTqTBQFqFeJCfNDF3wshPp6IDPBCpL83Ivy9EOLnCTnnmxPZpVMX9w/ZVzE0LsgtpgF2BHK5DN1CfNAtxAcP3n3zfqtV4HJFHc5frUbutWqcLtLhQL4W2lojymuMMFsFFHIZArxV0Hir4OelhMZLhT5dNUiM0CDUzxNhfl4I9vGAv7cKcg57kZvr1MW948xVTB0Y5eoYbsGeozzjQnwQF+KDYQkhOHqx/kr2ViFQrTdDW2NEea0RVXozqvQmVNaZsCe3DJtPXUGJzoA6U/1VgOQyIFDtgUAfDwSpPRDoo0KQjwcC1R6Nv/p4IPj6Vx8PBYdoqFPptMVdqK1FdrEOYxKlf54VKWjrUZ43yGUyaK5vaXeDT6PlBncLhIdCDiEEao0WVNaZGv5V3Hq71oRL5XWoqL15n67OhBtnLVcpZPD3VjX8C/H1RBd/LwRdL/cgH08E+9647YEAtQd3ZlOH1mmLe+Oxy+gb5Y9uIT62F6YOqbVfBn5eKvh5qRB1fZZn7wg/ZN2yrFUI6I0W1BgtqDGYUWs0o8ZoQa3BDH9vFaoNZhSU1UJbY4S2xojSagMMZiuAm1v1/moVNJ5KBKg9EKBWIcBb1eR2oLo+R6CvB8fjyWk6ZXELIZB59DL+a2iMq6OQi8hlMqg9lVB7KhHq13h64o0t+VsJIVBnsqCytn5rvqLWhJIqPbKLq1BjMOOithbZBguqDWbUGM2oMZhhstRv08sAaLxVCPPzRIivJ0L8PBHi64HQ69+H+t643wPBPp7wULrN9UvIQTplcZ8u0iG/tAYT+0W4Ogp1QPYO64zsEYpQv5bP3240W1FzvchjgtSQy2Uora7fer9WZcDZK1UNt8trb54vPUBdP1wT7OMBPy8VNF5K+Hkpr/8VcfOrl0oBL5UcnsqmXxVyGeSy+l9QcpkMcvnN2xYhYLZYYbIImK1WmC0CJosVZuv1r9fvN5rrvzddX/bmbSuMFgFdnQl1RnPDehu9plwGD4Ucag8FfDyVCPf3QmSAN4LUHvD2ULTL/xO1rFMW94ajlzEsIQRhrXzoiO6Uh1IOD2X9DtABMU234m9ltlhRUWeCtsYIvdkKvckCbc3NnbFVejMuV+gbblcbzNCbLNCbLDCYrQ3DOO1FpagvXqVCDpVCDg+FDCpl/e0b31usAlV6M6xCQKB++EmI+q9WAVisoj6fyQrLLddB9fVUItTXA8HX/9KIDvJGbLAPYq9PBQ3w4bDSnep0xW2xCmw6XoQXH0p0dRRyI23ZOdvcUE1zFAoZLNeHY4QQMFqsMF4v8V+XaK3RgtNFOliFaLR1rJDLoLi+hVx/nwxDYwPhqbS9VVxrsjTM/rElIdQHRwsrUGu0QFdngk5vQmWdGdpaI3JKqnBFp0et0QKVov5UC8lxwRgSG4TkuGAE+bDE26rTFfe+3DLo9CaMS+ri6ihEzWrLkatZbZip8+ux/JbUGi04XGC7kG+d/WOLh1LesMM4vIXLA1YbzCjR6eGhkCGruAobj11GZZ0Jd0dqkBIfgjGJYY2eywOwWtbpivuLAwV4OCkCvp6d7q0RSZqvpxK+ob7oHeGH6CAfPNg7HMWVepy5UoXvjhdh+a48dAvxwZDYICRF+iM5Lgi6WvuupepuJd+p2q2grAZbT1/Bpj/c5+ooRGSDTCZDZIA3IgO8MToxDFd0ehwtKMemY0XYfLIYk+/pitgQH7uGddztqk6dqrhX7s7H0NjgRidDIiJp6KLxwsN9IjCmVziOFpZj2+kS6PQmjOgeinvjg6GyY7+Au+g0P4mKWiO+OXwJvx0R5+ooRHQHPJRyDI0NxvtP9MfDSV2wP78M727PQVaRDuKW2SvurNNscX9x4CK6BnpjZI9QV0chonagVMgxMCYIfaMCsOvcNaw9dBGxIT6Y0DfS7h2xnVWn2OKurDNh5e58zBkeyzPHEXUyKoUcYxLDMX9sD3go5cjYcQ5bThXD2M5z26WkU2xxv/OfHIT6eWLKAJ4JkKizCvTxwPShMTh3tQqbjhXh5OVKTL6nK7qH2T9tsbOQ/BZ3VpEOq/cX4NXUpIbrHxJR59U9zA/PjumOflEB+HzvBaw7XIiKWqOrYzmVpJtOCIG/bDqFSf0iMSQ2yNVxiMhJVAo5Hry7C34/KgHXqg14ctUhbDh62W12Xkq6uD/dnY8zxVVY9DAPbydyRxH+3nh6ZDyeTI7BnzJPYuaqQyjU1ro6lsNJtrh3nrmKtzafwbtp/RHWwiG2RNT5yWUyTB0YhW3zR0AuAx58Zxfe/+Ec9NevmtQZSbK4c0qqMO+ro/ifh3piTK9wV8chog4gKlCNVbMGY8m0fvjqYCHGLv0J3x0vgtXa+YZPJFfcB/O1eGL5fozv0wVzhvNgGyK6SSaT4ZG+EdixYCQeGxSNRetP4pFlu7Ht9JVOVeCSmQ4ohMBXBwvxl02nMHdEPJ5/oAcvAEtEzfJSKfDsmO5IT47Bx7vy8NzaY4jw98J/D+uG3wyIkvxJ6OxKv3btWmzYsAFKpRKvv/46YmKce0mwg/la/H3LGWQV6bD0sf6Y2C/Sqa9PRNIU6OOBhQ8n4ncj4/HVoYv48MdcvP59Nh7o3QUT+0ZgWEIIfCRY4jYTV1RU4Ntvv8XatWuRlZWFf/zjH8jIyHBoKCEELmprse10Cf59shgnL1cibXA0/nf6AO6IJKI281er8PTIeMwZHod9uWVYf/QSFqw7DoPJisGxgRgYE4R7ogNwd6QGoX6eHf6veZvFffz4cQwdOhQKhQJ9+vTBhQsXHBamrNqA5785jmOFFaisMyGxix/G94lARto9uCtY7bDXJSL3oJDLcF/3ENzXPQRmixVHCyvwc841HL1YjlV78lGlN8PPU4mYEHXDRZ7rL8HmAY23Cp5KOTxuXN5NKYdFCBhMFuhNVhjMFlgFMKlfpMO34m2uXafTQaPRNHzvyAnuXioFhncPwZP3xiCpq3+LV9IgIrpTSoUcg7sFYXC3+oP3rFaBYp0euVerUaCtRWmVAWU1BlworcHhC1ro9CaYLAJGs7XhMnJyGa5f1FkBT6Ucfl5K3OeE4Reba9doNMjJyWn4Xi63byJKz549bz8VEZFE/Ppwn7HvOP41bRZ3v3798OGHH8JiseDMmTN27Zg8e/Zsu4QjIqKmbBZ3QEAAJk+ejOnTpzfMKiEiIteRCXc5KwsRUSchuSMniYjcHYubiEhiWNxERBLD4iYikhiHFHdBQQFmzJiBtLQ0rF27tsnjmzdvxrRp0zB9+nTMnz8fRqPrLzu0du1apKWlYcaMGSgoKGj0mK3342qtZX/xxReRlpaGadOmITMz00UJW9ZadgCwWCx46KGH8Omnn7ogXetay15TU4NFixZh5syZSE9Pd1HClrWWfdeuXXj00UeRlpaGV1991UUJm2c0GpGWloZBgwZhy5YtTR7vyJ9VW9nb9FkVDjBv3jxx8uRJYTKZxNSpU0V5eXmjxy9evChMJpMQQojFixeLDRs2OCKG3crLy8XUqVOF2WwWJ06cEPPmzWv0uK3340q2sufn5wshhDAYDGLcuHHCaDS6IGXzbGUXQoivv/5azJkzR6xYscIFCVtmK/tbb70lDh065KJ0rbOV/fHHHxeXL18WQgjx29/+VmRnZ7siZrMsFosoKSkRGRkZYvPmzU0e78ifVVvZ2/JZdcgW94ULF5CUlASlUokhQ4bgxIkTjR6Pjo6GUlk/hVyhUEChUDgiht1snY/F1vtxJVvZu3XrBgANP297j3x1BlvZDQYDfvjhB4wbN841AVthK/svv/yC77//Hunp6VizZo1rQrbAVvYePXpAp9PBYrHAYDAgICDANUGbIZfLERYW1qubeb8AAALGSURBVOLjHfmzait7Wz6rDvkUi1umhvv7+6OysrLZ5XJzc7Fnzx48+OCDjohhN1vnY7H3/biCveeS+eSTT/DII4+4/JfkrWxlX716NR5//PEOeaY2W9lPnz6NMWPGYNWqVdi2bRvOnz/v7IgtspV93LhxeOqpp/DQQw8hLi4OXbp0cXbE29aRP6v2suezetvFffXqVTz22GNN/u3YsaPRbwqdTgd/f/8mzy8pKcHChQuxdOlSeHh43G6MdqHRaFBVVdXw/a9/09nzflzFVnYA2LRpE7Kzs/H73//emdFsai17VVUVDh48iFGjRrkimk22fu6BgYFISUmBUqlEcnJyo/P9uJqt7K+99hrWr1+PrVu3ory8HEeOHHF2xNvWkT+r9rD3s3rbxR0WFoZvvvmmyb8xY8YgJiYGp0+fhsViwaFDh9C3b99Gz9XpdHjuuefwyiuv4K677rrdCO2mX79+OHjwICwWC06fPt3kfCy23o8r2cq+a9cuZGZm4u233+5QwyRA69nz8vJQXl6O2bNnY9WqVVi3bh327NnjwrSN2fq5Dxw4EFlZWQCAU6dOOf3iI62xlV2pVMLPzw9yuRwajQY6nc5FSduuI39WbWnLZ9Uhh7wXFBTgpZdegtlsxuTJk5GWlgYAWLRoEd58800sWbIEGzZsQGxsLADgN7/5DR599NH2jtEmX331FTZu3NhwPpaCggJUV1dj/PjxLb6fjqK17MOGDUNYWBh8fX0BAEuXLkVoaKiLE9/UWvYb1q9f31DiHUlr2QsLC/Hyyy9Dr9djyJAheP75510dt5HWsv/73//GqlWroFKpEBkZib///e8N464dwbx585CVlQW1Wo1hw4YhJSVFMp/V1rK35bPKc5UQEUlMx/rbmYiIbGJxExFJDIubiEhiWNxERBLD4iYikhgWNxGRxLC4iYgkhsVNRCQx/w+GB88nVmjXYwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "groups = np.where(brca['ER Status']=='Positive', 1, 0)\n",
    "pvals_t = np.zeros(exprs.shape[1])\n",
    "for i in range(exprs.shape[1]):\n",
    "    stat, pvals_t[i] = sc.stats.ttest_ind(exprs.iloc[groups==1, i],\n",
    "                              exprs.iloc[groups==0, i],\n",
    "                              nan_policy = 'omit')\n",
    "sns.distplot(pvals_t);\n",
    "plt.title('Results of t-test');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAEHCAYAAABhm8cjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXhTdaI+8Dd7mrbp3tIWKGUvFFqKA53iiILKIlAGBKu0I1dgmFHRUa9XGETnp9crIuNSr4zOgOCACPqzgCsqgqLsyFLWQvcCpS3d0yXr9/5RqEVoE2iT9DTv53l42jQnOW/ycN5z8s1ZZEIIASIikhS5uwMQEdGNY3kTEUkQy5uISIJY3kREEsTyJiKSIJY3EZEEsbzpGoMHD0ZycjImTpyIp556CmazucOeOy0tDTk5OaipqcHHH3/c7uc7e/YspkyZgqlTp8JgMDT/ff78+dizZw8AoLGxEYMHD77qdlJSEoQQmD17NgAgIyMDy5cvb3ee9srIyEBFRcUNP66j3k+SDpY3XcPf3x9btmzB559/jvLycnz55ZcdPo+OKpsdO3Zg+vTp2Lx5M3x8fJr/HhcXhyNHjgAAjh8/jn79+l11e9CgQZDJZFizZk27M3SkTZs2obKy8oYfx/L2PCxvapVcLkdsbCxKSkoAALm5uZg9ezamTZuGefPmobKyElarFU8//TTuueceTJ48GZ999hkAYMyYMTAajQCuv1Wbnp6OM2fOIDk5GStXrsS+ffswefJkJCcnIyUl5ZosDQ0NePLJJzF58mSkpKQgLy8Pu3fvxvvvv4/33nsPjz322FXTDxs2DEePHgUAZGZmIjU1FZmZmc234+PjAQCjRo26Zl45OTlITU3FlClTMHPmTJjNZlRUVGDevHmYPHky5syZg0uXLqGwsBCTJk1CfX09DAYDJkyYgOLiYhQUFOCBBx7A5MmT8fjjj6O+vr75PUlPT8eUKVMwa9asqz4pAMB3332H48eP49FHH8UDDzwAANi+fTtmzJiB5ORkvPjiiwCA0tJSpKSkIDk5GZMnT0ZOTs417yd5AEH0K0lJSUIIIRobG8WsWbPEyZMnhRBCzJ49W1y4cEEIIcSmTZvE0qVLxbFjx8SsWbOaH1tTUyOEEOKOO+4QjY2NQgghPvnkE/Hqq68KIYRITU0V2dnZoqioSMyYMaP5cfPnzxd79uy56jla+uc//yleeuklIYQQ33//vZg9e7YQQoj09HSxfv36a6avra0Vo0aNEkII8dhjj4mLFy+KlJQUIYQQCxYsEDt37rzqtbbMOG3atOYs1dXVwmazieeff16sWbNGCCHEunXrxOLFi4UQQqxatUq8+OKL4tlnnxXr1q0TQggxd+5c8e233wohhHjllVfEihUrmt+TTZs2CSGEeOGFF8TGjRuvyX3l/RFCiPLycjF79mxhNBqFEEIsXLhQbN++XaxatUq8+eabQgghTCaTaGhouOb9pK6PW950jaqqKiQnJyMxMRF6vR4xMTEwGAw4fPgw/vSnPyE5ORmrVq1CcXExevTogQsXLuC///u/sWfPHvj6+t7UPIcNG4bly5fjgw8+gMlkuub+w4cPY9KkSQCA0aNHIzs7u83n8/HxQUBAAIqKilBSUoKwsDBERkaiqKgIx44dQ1xc3HUfZzAYUFdXh8TERACAXq+HTCa7av7Jycn4+eefAQAPPvggjhw5guzs7Oat5VOnTuHOO+8EAEyZMqV5WgAYO3YsACAmJgbnz59v8zUcOXIEWVlZzVvehw4dQmFhIWJjY7Flyxa8/fbbKCwshFarbfN5qGtSujsAdT5XxrwrKiqQlpaGbdu2ITExEWFhYdiyZcs103/66afYuXMn3n33XRw8eBALFiyAXC6HzWYDgOuW8a/Nnz8ft912G3bs2IF7770XGRkZCAgIaL5f3MQpeOLj47Ft2zaEhoYCaBoH37ZtG7y8vKDX61t9nEwmu+ZvQojr/t1gMKC2thZarRZmsxlqtfq6012hVqsBNA1JWa3WNvMLITB27Njm4ZKWPvjgA2zfvh2PPPIInn/+efTo0aPN56Kuh1ve1KrAwEA89dRTWLlyJXx8fKDX6/HTTz8BaCrk3Nzc5j0jJk6ciPnz5+PUqVMAgIiICJw6dQo2mw0//PDDNc+t0+lQV1fXfLuoqAgxMTF4+OGHERERgYsXL141fUJCQvMXpzt37kTfvn3t5o+Pj8fatWsxdOhQAE3lvXbt2la3uoGmLXadTof9+/cDaPoiUAiBhIQEfPHFFwCAzz77DMOHDwcAvPzyy5gzZw7uuusurFixAgAwcOBAbN++HQDw+eefN0/riJbvS3x8PPbs2dP8nUNFRQVKS0tx/vx5hISE4IEHHsCECROQlZV1zftJXR+3vKlNd9xxB958800cPXoUr776Kp5//nksW7YMNpsNDz/8MKKjo7Fo0SIIIaBQKPDcc88BAP785z9j4cKFCAsLQ3R09DXPGxgYiL59+zZ/SXnhwgXs27cPcrkcCQkJGDhw4FXTp6amYvHixZg8eTK8vb3x8ssv280eHx+P8+fPN5f3wIEDUVZW1mZ5A8CyZcuwZMkSGAwGeHl5Yd26dViwYAGeeeYZfPTRRwgJCcGyZcuwa9cuFBcXY+bMmTCZTJg5cybGjx+PZ599FosWLcLrr7+O3r17O5T1iuTkZDz99NMICgrC+vXr8dxzz+HPf/5z81b90qVLcfz4caxcuRIqlQr+/v547bXXrnk/586d6/A8SZpk4mY+jxIRkVtx2ISISIJY3kREEsTyJiKSIJY3EZEEsbyJiCTIKbsKDhgwwBlPS0TU5WVlZTk0ndP283Y0ABERNbmRDV8OmxARSRDLm4hIgljeREQSxPImIpIgljcRkQSxvImIJIjlTUQkQSxvIiIJcuggnczMTLzxxhswmUwYPXo05s2b5+xcnUJlvQk19WaHptXrVAjQqZ2ciIioid3yNplMeOutt/D222/Dy8vLFZk6jZp6M3bllDs07ag+QSxvInIZu8Mmhw8fhpeXFx577DHMmTMHZ86ccUUuIiJqg90t77KyMmRlZWHTpk0oLi7GkiVLsH79eldkIyKiVtjd8tbr9UhISIBOp0OfPn1QW1vrilxERNQGu+UdFxeHvLw8WK1WlJaWQqvVuiIXERG1we6wiZ+fH2bMmIG0tDRYrVY888wzrshFRERtcGhXwenTp2P69OnOzkJERA7iQTpERBLE8iYikiCWNxGRBLG8iYgkiOVNRCRBLG8iIglieRMRSRDLm4hIgljeREQSxPImIpIgljcRkQSxvImIJIjlTUQkQSxvIiIJYnkTEUkQy5uISIJY3kREEsTyJiKSIJY3EZEEsbyJiCSI5U1EJEEsbyIiCWJ5ExFJEMubiEiCWN5ERBLE8iYikiClIxPFx8djyJAhAIA//OEPuOuuu5waioiI2uZQeXfv3h1r1651dhYiInKQQ8MmxcXFSE1NxVNPPYWKigpnZyIiIjscKu9t27Zh3bp1GDt2LJYuXersTEREZIdD5R0QEAAAmDBhAk6fPu3UQEREZJ/d8q6vr4fVagUAHDx4EJGRkU4PRUREbbP7hWVubi6WLFkCnU4HhUKBF154wRW5iIioDXbLOzY2Fps2bXJFFiIichAP0iEikiCWNxGRBLG8iYgkiOVNRCRBLG8iIglieRMRSRDLm4hIgljeREQSxPImIpIgljcRkQSxvImIJIjlTUQkQSxvIiIJYnkTEUkQy5uISIJY3kREEsTyJiKSIJY3EZEEsbyJiCSI5U1EJEEsbyIiCWJ5ExFJEMubiEiCWN5ERBLE8iYikiCWNxGRBDlc3gcPHsSAAQNQUVHhzDxEROQAh8v7/fffR2xsrDOzEBGRgxwq7+3bt2P48OHQ6XTOzkNERA6wW942mw0ffvgh7r//flfkISIiB9gt788++wxjxoyBRqNxRR4iInKA3fI+c+YMvv76a8yZMwdZWVl44oknXJGLiIjaoLQ3wdNPP938e1paGl5//XWnBiIiIvvslndLa9eudVYOIiK6ATxIh4hIgljeREQSxPImIpIgljcRkQSxvImIJIjlTUQkQSxvIiIJYnkTEUkQy5uISIJY3kREEsTyJiKSIJY3EZEEsbyJiCSI5U1EJEEsbyIiCWJ5ExFJEMubiEiCWN5ERBLE8iYikiCWNxGRBLG8iYgkiOVNRCRBLG8iIglieRMRSRDLm4hIgljeREQSpLQ3QUlJCR599FFoNBpYLBb87W9/w8CBA12RjYiIWmG3vIODg7Fx40bI5XLs2bMHK1euxPLly12RjYiIWmG3vBUKRfPvdXV1GDRokFMDERGRfXbLGwCys7Px7LPPori4GG+99ZazMxERkR0OfWHZt29fbNiwAe+88w5efPFFZ2ciIiI77Ja3yWRq/l2v10Or1To1EBER2Wd32CQzMxNvvPEGZDIZAGDhwoVOD0VERG2zW9633HIL1q1b54osRETkIB6kQ0QkQSxvIiIJYnkTEUkQy5uISIJY3kREEsTyJiKSIJb3DRJCwGK1uTsGEXk4h85tQk1qGs34cF8hCivqEeSjQY8AL9wzNBw6Nd9GInItbnk7qLC8Dm/vyIZSIcMfb+uN0f2DUWYwYvWufDSare6OR0QehuXtgNpGM97bnY+hkX6YnRSNqCBvDI8KxH8kRUNA4P3d+ag3Wdwdk4g8CMvbAdtOlSLMV4OJQ8KhkMua/+6lVuChpGg0mK343x3ZbkxIRJ6G5W1HSU0jfi6owMQh4c0n52pJp1Hi3uHdsfX4RRwtqnJDQiLyRCxvO7Yev4hB4XpEBXm3Ok33AB3Gx3bD85+egM0mXJiOiDwVy7sNR89VIbvUgHGDu9mddt6tvZFTakDG4fMuSEZEno7l3YYvMosxOFKPIB+N3WkDvNV4/M5+WLb1NIwW7n1CRM7F8m5FvcmCnWfLMKxHgMOPSU2Mgk0Amw5x65uInIvl3YqvT1yEl0qBvqE+Dj9Gq1LgoVt74d2dubBy7JuInIjl3YqMQ+dxZ0zYVbsGOiI1MQqXao345sRFJyUjImJ5X9fF6kb8lH3JoS8qf02vVWFWYhT+8UMOhODWNxE5B8v7OjYfOY8BYb43NGTS0kOjeuF0cS325JZ3cDIioiYs7+v48lgxkuMjb/rxoXotpsRHYM2u/I4LRUTUAsv7V8pqjcg8V407Y0Lb9Tyzk3ph26kSFFXUd1AyIqJfsLx/5YczZYj097rpIZMrYiP9MKxnANbtLeigZEREv2B5/8qOrFLcPiDkuucxuVGzk3phw4EiNJh40A4RdSyWdwsWqw07z5ThjgHtGzK5YnxsN2hVcmw5woN2iKhjsbxbOFRYBaPZhqS+QR3yfCqFHKkjo7Bmdz53GySiDmW3vHNycnD//fdj1qxZSEtLQ1FRkStyucWOrFKM7B3YoZc1u39kT+SW1WFfXkWHPScRkd3yDggIwLvvvosPPvgA8+bNwzvvvOOKXG6x43Qpbu+gIZMrgn00mBQXzt0GiahD2S3vwMBA6PV6AIBSqYRCoXB6KHcoqWnE6Yu1uH1ASIc/9+ykXvjm5EWcr2ro8OcmIs/k8Jh3Q0MD0tPT8eCDDzozj9vszS1HN70WvYNbv+jCzRra3R9xPfyxdg93GySijuFQeVssFjzxxBOYO3cu+vTp4+xMbrEvrwIjewd2yC6C19O022AhrzRPRB3CbnkLIbB48WLcdtttuPPOO12RyS325pZjZHTH7GVyPRNiw6FSyPHpkQtOmwcReQ675f3jjz9i69at+Oqrr5CWloaXXnrJFblcqrS2EblldRjZO9Bp81Arm3YbXM3dBomoA9jdJ+62227D0aNHXZHFbfbnVSDYR+OU8e6W7h/ZA/+74ywO5FdiRLTzVhRE1PXxIB0A+3KdO959RaivFpOGRmDN7jynzoeIur6OOxpFwvbllSMtMapdz2Gy2lBwqc7udOMGheGR9YdxoaoBEf5e7ZonEXkujy/vijoTzpQYkNi7fV9W1hktOFBc69C0A7r5YM3ufPx1Yky75klEnsvjh03255UjyFvd7lPA3ohZI6Owbm8BKupMLpsnEXUtHl/ee3MrMCLa+ePdLSX1CULPQB1W7+LYNxHdHJZ3bjlGunjPD7lMhkfH9MWaXfmobjC7dN5E1DV4dHlX1ZuQVVKLke0c774ZE2LDEarX4N+7810+byKSPo8u7/15FfDzUmFAmK/L562QN219/+vHXFTVc+ybiG6MR5f3vrwK/KZXIORy1413tzQlLhLdA3RI/y7bLfMnIuny8PJ2/Xh3Swq5DIvvicHavfnId2AfcSKiKzy2vGsazTh5oabd+3e316i+wfhdvxC8svW0W3MQkbR4bHkfzK+At0aJmHC9u6Ng0YSB+OZkCXbnXHJ3FCKSCI8t7325FRjRKxAKN413t9QvzBdzbo3GooxjaDDxfN9EZJ/Hlvfe3HKnngL2Rj15V3/IZTL8/Zssd0chIgnwyPI2GC04fqHGqRdfuFFalQKvTB+K1bvzcaiw0t1xiKiT88jyPphfAS+VAoMj3D/e3dKI6ECkJUbhLxuOoLqeR14SUes8srz35VVgeFQAlIrO9/IXTRyIAG81Ht94GDYbr7hDRNfX+drLBfZ1svHuljRKBf4xKwHHzlXjje/OujsOEXVSHnc+73qTBZnnqvHspEFuy+DIhRuevScGT///THQP8MLMW3q4KBkRSYXHlfehgiqolXIMifRzWwZHL9ywcMJA/DXjGHRqBSYNjXBBMiKSCo8r73155RgeFQBVJxzv/rW7YsLgo1HiiY1HoJDJMGFIuLsjEVEn4XHlvTe3HKP7h7g7hsNSRvSExSaw4MPDWFjVgDm3Rrv0whFE1Dl1/s3PDtRotuJoUbVbzt/dHqmJUXg3bTj+/s0ZLNlyHEYLj8Ik8nQeVd6HCishkwFDu7tvvPtmjY0Jw8d/+i2+zyrD1Ld3I7vUsYsdE1HX5FHlvTu7HCOiA6FRKtwd5abERvrhy8d/h/5hPpj01k9454ccmCw2d8ciIjfwqPL+KfsSRvUNdneMdtFrVXjjvngsnxGH937Kw8T0H/Hj2TJ3xyIiF7Nb3iaTCSkpKbjllluwdetWV2Ryiup6MzLPVeFWiZc3AMhkMkwaGoHvnhqN2/uH4KE1B3D/P/fi54IKd0cjIhexu7eJUqlEeno6Nm7c6Io8TrMntxx+XioM6gTn73aUIwfzpCVG4a6YMKw/UIgZ7+zBsJ4BeGhUNMYNDuuUh/8TUcewW95yuRyhoaGuyOJUu7IvIalvsNuuV3kzHD2YBwCevLM//vPuAVi7twALMzLx0hdK/CGpF1J+0wP+OrWTkxKRq3nMptlP2Zfwuy4wZNKWHoE6/HViDPYuGos/3d4HHx0owoj/+Q6PrD+EHadLYbHyy02irsIjDtI5V1mPvEt1kv+y0lHeGiX+8NteSB0Zhb155cg4dB6PrD8EnVqJ5PgITI2PRGykngf7EEmYR5T37uxyRAXp0CNQ5+4oTtPa+HiknxcW3NEXc0dF48fsMmw9UYLVu/IQ4e+FCbHdMD42HMN6+EtqOImIHCzvBQsW4OTJk9DpdDhy5AgWLlzo7Fwd6scusIugPY6Mj3trVJie0B3PjB+A/XkV2HmmDO/tykegToXf9QvB7f1DEBvp13xdT71OhQCOlxN1Sg6V91tvveXsHE5jttrwfVYpXp8Z7+4onYZaIUegtwZTh3XH3YO74XRxLY5fqMaWoxegVcrRP8wXA7r5InVkT5Y3USfV5YdN9udVwGy1dfkt75ulUyuREBWAhKgANJqtOFtqQNbFGnyWWYyPDxZheK9AjBkYijEDQ9Ev1Ifj5ESdRJcv729PluB3/ULgpZbmIfGupFUpMCTSD0Mi/WATAn5eSvycX4lNh85j6Ven0U2vwbCeARjWwx/Devoj1Ffb/FgOsRC5VpcubyEEvj1ZgsfH9nN3FMmRy2ToEaBDbaMV/bvpUdtoxtlSA3LL6rDi+xxUNZgR5K1G7xAf9Anxxn239EBAFMubyFW6dHmfvliLC9UNuGOg9A8ycjdfrQoJPQOQ0DMAQghU1puRU2ZAbpkBX2QWY8OBIvQI9MLQ7v6I6+6HuO7+iI30g7emS/8XI3KbLr1kfXuyBAk9AxDiq3F3lC5FJpMh0FuNQO9A/KZXIIQQ6BmoQ5nBiMxz1fjq+EUs/+YMTBYbIv290DvEG31CfNAn1AfRQd7o5qdBmF4LX63K3S+FSLK6dHlvO1WCCbG8dJizyWQy9AzU4Xf9QzAtoTuApr18sksNyCkzIKe0DtllBuzfV4jCinoYjBYAgLdagTA/LYK9NfDVKqH3UjX91Db9VCpkkEMGnVoBL7UC3moldGpF8221Qt78BSrH3MnTdNnyPl/VgMxz1XhtZpy7o3gklUKOmHA9Yq5zIrBzlfXILTXgksGEMoMR1Q1m1BktMBgtKK1pRG5ZHQxGC2obzaisN8NotsJoscFiE1c9j1wGaJQKaFRyhPho0M1PiyBvNQK9NQjyUV/+dKBGsI8Gob4ahPhqoFXxi2vqGrpseW8+fB6DI/ToG+rr7ij0K1arwLmqRgBNY+mtDZ8MCvfFyRYHHlltAkZLU5EbLbbmUjdabAjXawCZDBV1JlwyGHGmpBbldSZU1BlRbjA1F79eq0SoXttc5qG+GoT6an/5Xa9BiI8Wei8ld4ukTq1LlrcQAp/8fA6zEqPcHcVjOHL62ivqzTd3DU6FXAadWonrjY78plcA1K2cAtcmBGoazJfL3ITKehPKao2oqDMhu9SAfXkVqKgzodxgQsPlbGqlHME+anTTXyn2FoWv/+V2oLcaSoUclfUm1NSb7b6G9g7vmK1NKyshRPPKRXb5vVEr5F3qNAeOvqeAZw6bdcnyPlxUhcKKeiTHR7g7ise4kdPXDgrv+E9DNzL/3/QKxMniWvS6znFbRosVhkYLahotiPDTQgAoMxhRWmPE/vwKlNUaUVZrRHmdCUDT0E2gtwb+uqZPD2qFHCqFDCqFHCqlHEqZDE0d2/SzZ6AXvDUqGC1WmC5/ajBd/nflU0Wj2YpGsw31JgsaTL98ujBZrLCKazO3pFXJ4aVSNP27/N2Al0oBnVoJb40CSrkcSrkMXle+O1BduV9x+W/K5seG+WkR4ecFrUreYZ9CbqSQ681WHC6scmjaUX2CWN5dQcahc7h9QAiCfbiXCd0YjVIBjY8CQT4ajOoThKhg7+tOZ7baUG4wobS2EaU1Rpy+WIPMc9UwWW0wW2wwWQUaTNam4RohIAAIAVisNui9VFAr5dAo5dCq5PBrcVujVECravppaDQjr7z+l5WBXAalQg6lQgYZmspUQAACiOvuhyBfDRpMVjSYrWg0N/1sMDWtBOpNVtSZLCipbkTepbqmlYG1aejJZLVdtSK53ncLXioFNCoFtEr5VT81Sjm0Sjl8vFTw81JdteLQNv8uh1apgFatQE29GVkltdB7qeCjUULexkrBGSv5rqTLlbfRYsVnR4uxdNoQd0ehLkylkKObnxbd/JqOMu0X6oNAb/sbC20N7/xavdkKHwe3PHuH+LS6ommp4FIdduWUtzmN1SZgstjQM9ALJy7UNBe9+cpPq4DZamvxT0CrlMNmE7hkMF5nBdL0SaLBbEW90YLGyxfNVshkCLw8NBXup0VUkDd6BHjxClAO6nLl/d2pUgDAmBgemEPt44xxfGcNLzma1ZGcisvDKk3j+1q70wNo81NKSwWX6vBj9iXUNJhRVW/GJYMRJTWNOFNSi+2nm5bdPiE+GBLph54BXg7N21N1ufJ+76c8TE/oDo2Su4RR+7h7HP9GOJrVWTlvZOUhl8ngr1PDX6dGrxaFb7baUFhRj1PFNfj2VAk+PXoBQyL9MLJ3ILoHdN1z8d+sLlXe+/MqcPRcFdLvH+buKEQepSNWHiqFvOlI3BAfTBwSDqvVho0Hz+GdH3IQHeyNMQPDEO3A1r2n6FLl/Y/vszE1PhIR/vy4RSRlcpkMsT38oVIqUFFnwvdZpVj1Uy76hvpg0pAIBPOUF13nAsQnL9TghzNlmD+6j7ujEFEHCvRWY1pCdzx51wAo5XK8uf0svj5xESaLZ19Qu8tsef/jhxzcPagb+ob6uDsKETlBoLcaqYlROFNSi88zL+BIURUmxHbDkEg/d0dziy6x5X0gvwJfHivGo2P6ujsKETlZ/zBfPDa2HxJ7ByHj0Hms2Z2P4qoGd8dyOcmXt8liw18zjiEtMQqxHroGJvI0Srkco/uH4C939oNCLsPsNQew8sdcWKyeM5Qi+WGTf+7MQW2jBf85boC7oxCRi/nr1EhLjILJYsPb32djy5ELWDp9CAZHdP0NOUlveZ8tqUX69mz8v+TB8OEVW4g8kkwmw5iBodj25GjEhPsi+X934eWvTqHBdHMnQJMKyZZ3aU0jZq8+gClxERg3uJu74xCRm/nr1Fh2bxz+/dAIbD1+EePf3IkfzpS5O5bTSLK864wWPPT+AUQHe+N/fs9zmBDRL5L6BuPrv9yGe4aEY977BzF79X5klzp2pKyUSK68y2qN+I/VB2CxCqxITYBaKbmXQEROplUp8F/jB2Lbk6OhUytw9+s78cTGI8gpM7g7WoeRVPMdzK/APek/Qi4H1s4ZCT0vYEtEbegZpMOKWcOR8fAoVDeYcddrP+CP/z6In85eghB2To7eyTn0Ld+GDRuwefNmKJVKvPTSS4iKcu0VagrK65D+XTY2HzmPubdG4+lxA3jaSCJyWHwPf7w3+zc4fr4aa3bn46H3DyDcT4tJQ8MxcUg4BoXrJXfZO7vlXVVVhU8++QQbNmzAyZMn8fe//x3p6elOD1ZuMGL76VJ8faIEO7JKkdQnCB/N/y2GRwU4fd5E1DXFRvph+Yw4/HViDL7IvIDPM4ux4vscBPtokNQnCLf0CkRsRNOFszv7xartlvfRo0cxcuRIKBQKDBkyBPn5+U4LU1lnwn99kolj56pxsaYREX5a3D24Gz7+02+R0JOlTUQdI9BbjbTf9kLab3vhksGIPTnl2J1zCR/sLcDZUgOsNoFgHzXC/bwQ7qdFhL8Xgn3U8Lp8mTjd5SsFAU2nsnKG5r8AAAVzSURBVDVdvlAFAEyJi4C3C3ZdtjuHmpoa6PX65tvOHCdSKeVI6BmA5PgIxITr0TvYW3IfZYhIWoJ9NJgcF4HJcU3XvG00W5FTZkBxVSOKqxtw/vLP7FLDL1cHunyFIKDpYtUqhRxqhRzeGgVu7RvcOcpbr9fjzJkzzbflcsfGmgcM4BGPROR57nzdNfOxW95xcXFYsWIFrFYrTp8+7dCXlVlZWR0SjoiIrs9uefv7+2Pq1KmYNWtW894mRETkXjIh9Z0diYg8EHeWJiKSIJY3EZEEsbyJiCSI5U1EJEFOK++CggKkpqYiJSUFGzZsuOb+r776CjNmzMCsWbPwxBNPwGQyOSuKQzZs2ICUlBSkpqaioKDgqvvsvRZ3ayv7M888g5SUFMyYMQObNm1yU8LWtZUdAKxWK8aPH49Vq1a5IV3b2speV1eHRYsW4cEHH0RaWpqbEraurew7d+7EtGnTkJKSghdeeMFNCVtnMpmQkpKCW265BVu3br3m/s68vNrLfkPLq3CSBQsWiGPHjgmz2SzuvfdeUVlZedX9hYWFwmw2CyGEePXVV8XmzZudFcWuyspKce+99wqLxSIyMzPFggULrrrf3mtxJ3vZ8/LyhBBCGI1GMW7cOGEymdyQ8vrsZRdCiI0bN4p58+aJlStXuiFh6+xlX7p0qThw4ICb0rXNXvb77rtPnD9/XgghxB//+Edx6tQpd8RsldVqFSUlJSI9PV189dVX19zfmZdXe9lvZHl12pZ3fn4+YmNjoVQqMWLECGRmZl51f48ePaBUNu1mrlAooFC47yQw9s7fYu+1uJO97L169QKA5vfa0SNkXcFedqPRiO3bt2PcuHHuCdgGe9kPHTqEL7/8EmlpaVi3bp17QrbCXvb+/fujpqYGVqsVRqMR/v7+7gnaCrlcjtDQ0Fbv78zLq73sN7K8Om1JFi12H/fz80N1dfV1p8vJycGuXbtw9913OyuKXfbO3+Loa3EHR889869//Qv33HOPW1eSv2Yv+9q1a3Hfffd1yvPb2Mt+4sQJjB07FqtXr8Y333yD7OxsV0dslb3s48aNw9y5czF+/Hj07t0b3bpJ6zKDnXl5dZQjy2u7yru0tBQzZ8685t9333131RqjpqYGfn7XXs25pKQECxcuxGuvvQa1Wt2eKO2i1+tRW/vLZZJ+vbZz5LW4i73sAPDpp5/i1KlTeOSRR1wZza62stfW1mL//v2444473BHNLnvve0BAAJKSkqBUKpGYmHjV+YHczV72F198ERkZGfj6669RWVmJn3/+2dUR26UzL6+OcHR5bVd5h4aG4qOPPrrm39ixYxEVFYUTJ07AarXiwIEDGDp06FWPrampweOPP47nnnsOPXv2bE+MdouLi8P+/fthtVpx4sSJa87fYu+1uJO97Dt37sSmTZuwbNmyTjVkArSdPTc3F5WVlZgzZw5Wr16Njz/+GLt27XJj2qvZe9+HDx+OkydPAgCOHz/u8guYtMVedqVSCV9fX8jlcuj1etTU1Lgp6c3pzMurPTeyvDrt8PiCggIsXrwYFosFU6dORUpKCgBg0aJFePnll7F8+XJs3rwZ0dHRAIDf//73mDZtmjOiOOTDDz/Eli1bms/fUlBQAIPBgIkTJ7b6WjqLtrKPGjUKoaGh8PHxAQC89tprCAkJcXPiX7SV/YqMjIzmIu9M2speVFSEJUuWoLGxESNGjMCTTz7p7rhXaSv7F198gdWrV0OlUiEiIgKvvPJK8xhsZ7FgwQKcPHkSOp0Oo0aNQlJSkmSW17ay38jyynObEBFJUOf6HE1ERA5heRMRSRDLm4hIgljeREQSxPImIpIgljcRkQSxvImIJIjlTUQkQf8Ho1tsFpLjleYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pvals_w = np.zeros(exprs.shape[1])\n",
    "for i in range(exprs.shape[1]):\n",
    "    stats, pvals_w[i] = sc.stats.mannwhitneyu(exprs.iloc[groups==1,i], \n",
    "                                            exprs.iloc[groups==0, i],\n",
    "                                             alternative='two-sided')\n",
    "sns.distplot(pvals_w);\n",
    "plt.title('Results of Wilcoxon test');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can directly compare the graphs, which appear quite similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD5CAYAAADItClGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXwTdd4H8M9MJkfTJml6lxYKFmjEXgtYlcsLpeCq6KoUl8MDH1ZZVMAVWGR37YqCWNzFZ11QXFdxpahcsgiouI/FY6UKUmg55GgBW1p65mqTzPH8kXbaQNukpUkPvu/XK6+5J9+gyacz85vfMJIkSSCEEEIAsN1dACGEkJ6DQoEQQoiMQoEQQoiMQoEQQoiMQoEQQoiMQoEQQoiM6+4COiopKam7SyCEkF7n2LFjPq3X60IB8P3DEUII6dgf03T6iBBCiKxXHin0BTV2J8x2l9f19FoljFpVACoihBAKhW5jtrvw9ckqr+uNTgynUCCEBAydPiKEECLz6Ujhm2++wWuvvYaysjLwPC/P/+qrr/xWGCGEkMDzKRT+9Kc/4dVXX8WwYcPAMIy/ayKEENJNfAqFiIgIXHPNNf6uhRBCSDfzKRQSExPx6KOP4qabboJK1XzRc8qUKX4rjBBCSOD5FArR0dGIjo5GXV1dh3aenp6OlJQUAMCMGTNw2223yctKSkqwZMkS8DyPyZMnIysrq0P7JoQQ0vV8CoXf/va3AIDq6moAQFhYmE87j4+Px/r161tdlpOTg0WLFsFkMmHq1KnIzMxEaGioT/slhBDiHz6FQkFBARYuXIigoCBIkgSn04mXXnoJqamp7W5XVlaGadOmITo6GkuWLPEIk+LiYiQnJwMAMjIyUFBQgHHjxl3GRyGEEHK5fAqF559/Hn/9618xdOhQAMBPP/2EhQsXYvPmze1u9/nnn8NoNOKTTz7B8uXL8fLLL8vLWj4a2mAwdPjUFCGEkK7n081rTqdTDgQAGDJkCFwu7100GI1GAMDEiRNx9OhRzzdmm9/abDbDYDD4VDAhhBD/8elIISMjA08++SQmTZoEhmHwySefICMjo91t7HY71Go1FAoFvv/+e8TFxXksT0hIQGFhIUwmE/Lz8zFr1qzOfwpCCCFdwqdQeO6557Br1y7s378fkiQhMzMTmZmZ7W5z6tQpLF26FFqtFgqFAtnZ2cjLy4PVasWkSZOwYMECufXRPffcQxeZCSGkB2Cklif3L8LzPDiOg9PpbHV5y3sWAiUpKalPPE+hpNLmc4d4CRHBAaiIENJXdeR3s90jhTlz5mDt2rXIzMz06N5CkiQwDIM9e/ZcXqWEEEJ6lHZDYe3atQCA7du3IzjY869Vm83mv6oIIYR0C59aH02bNs2neYQQQnq3do8Uzp49i+LiYthsNo9usq1WKxwOh9+LI4QQEljthsKxY8ewZ88e1NXVYceOHfJ8rVaL7OxsvxdHCCEksNoNhfHjx2P8+PE4fPiw3CUFIYSQvsun+xQGDhyIdevW4eTJkx7NU3NycvxWGCGEkMDz6ULzM888A47jsH//fvzyl7+EIAiIioryd22EEEICzKdQqKiowEMPPQSVSoWbb74Zq1atwqFDh/xdGyGEkADz6fQRx7lXi4yMxL59+xAVFYXy8nK/FkYIISTwfAqF2bNnw2KxYOHChXjhhRdgs9mwaNEif9dGCCEkwHwKBZ1OB51Oh6SkJPlJavv27fNrYYQQQgLPp2sKL730kk/zCCGE9G7tHil8++23+Oabb3DhwgWsWrVKnm+1Wj06yCOEENI3tBsKRqMRV111FZRKJQYNGiTP12q1mDt3rt+LI4QQEljthoLJZILJZMI999wTqHoIIYR0I58uNI8ZM6bV+S07ySOEENL7+RQKLX/8nU4n/vOf/+DEiRPtbnPy5Ek899xzYFkWLMvixRdfRP/+/eXlixYtwvHjxxEcHIxBgwZRB3uEENID+BQKLalUKkyYMAHr1q3DnDlz2lzPaDRi7dq10Ov1yMvLw5o1a7Bs2TKPdZ5//nmkpKR0vGpCCCF+4VMobNy4UR4XRRFHjhxBUFBQu9uEhYU1vwnHQaFQXLJOdnY2NBoNZs+e3eYpqr7OJYgoq61Hlc0JcwMPnYZDRLAKsaFBUCp8ajFMCCFdxqdQuHDhgjzOsiyGDx/u8x3N9fX1WL169SVHCQsXLoTRaERVVRUeeughpKWlQafTdaD03q3K6sBnReXYd7oKDS4RoVol9EFKmOtdqLE7oVSwSI03IEyrREJEsPcdEkJIF/ApFH772992auc8z2PevHmYNWsWEhMTPZYZjUYAQHh4OJKTk3H69GmkpqZ26n16E0mSsOXAz1iy5TCMWiUmpsQiNc4ArsVRAS+K+Knciu9LajDn/f0YPywaCzNNuCoypBsrJ4RcCXwKhcLCQrz99tsoKyuDIAjy/Nzc3Da3kSQJS5Yswbhx4zB+/PhLllssFuh0OjgcDhQVFSEuLq4T5fcuNgeP3285hE8Ly/HbmxMRqlW1ehMgx7K4OlaPq2P1iA/V4N3/nsHtr+Zh9o1X4clbh0DNXXoqjhBCuoJPoTB//nw8++yzGDp0KFjWt/Pce/fuxa5du1BaWoqdO3fCZDJh7NixsFqtmDRpEubPnw+LxQKe5zF9+nSEh4df1gfp6cwNLjz8dj4sDS5snzsGSpbB1yervG6XEB6MdTNH4qufKrFwUwH2HKlAzgNpuKafIQBVE0KuNIwkSZK3lR588EG8//77gajHq6SkJBw7dqy7y+iQapsTM/7xHViGwTsPZ8AYrEJJpc2nUBidGC5fU7A0uPDCv49g84FzmHvLEDx+UyJdjCaEeNWR302fjhSeeeYZPPnkk8jIyIBSqZTnT5kypXMVXkHsTh4z/7EPGk6Bfzx8LfQapfeN2qDTKLHivlRkJsdg4aYCfFZUjpX3p8IUo+/CigkhVzKf/sx8/fXXIYoiqqurceHCBflF2ieIEp7K/RENLuGyA6Glm01R+GzejRgSFYI7X/sKf/38J7gEsUv2TQi5svl0pFBeXo7t27f7u5Y+Z8WuozhwpgZbnhjdZYHQxKBVYtWUdPwyLRa/33wYuwrPY+V9qUiOo2sNhJDO8+lI4bbbbsMnn3wCm80Gp9Mpv0jb3v/uDP75TTHemDES/cO0fnufW0zR+HT+OKTFGzD5b19j5e6jsDt5v70fIaRv8+lIYevWrQCAV155BQzDQJIkMAyDPXv2+LW43mrvTxfwh22H8eqUdAwfYPT7++k1Siz/VSruSI3F0q2H8eH35/C7CUm4d3g8FCw994IQ4juvoSAIAhYuXIgJEyYEop5e76dyC554bz+eHj8Ed6b1u+z9OQURJZU2n9ZNjjPg03k3Yv1/S/DCjiP4+5cnMeemwbgrvR+1UiKE+MRrKCgUCrz77rsUCj6otDrw8D/zcfs1MZhz8+Au2afNwSO/zOLTuqMTw2HUqvDomEG4b0Q81n9bjBd2FOGVT49hasYATLm2P6L1mi6pixDSN/l0+mjkyJF49dVXMWHCBI+O8Fo+je1K1+AS8Ni73yMuNAgv3ZvSLY8rvfio4s7UfrjVFIXPj1Rg248/46+fH8cvBhhxiykKmSkxuCqCus0ghHjyKRT279/vMQQAhmHw7rvv+qeqXkYUJSz48CBq7S5sfnwUVFz3nKpp66giPESNh0cPws+19Th0rg5v5J3Cyt3HkBJvwOjBEUjvH4q0+FDEGOgogpArnU+hsH79en/X0ast++QIvj1ZhU2Pj4IxWNXd5bSKYRjEG7WIN2qRmRyDWIMGJy5Y8c3JKmzMP4tqmxPRejVS40NxdaweA8O1SAgPxsBwLcKCW++jiRDS9/gUCufOncPLL7+MyspKvP/++zhx4gS+/fZbTJ8+3d/19Xhv5p3C+9+dwfuPXYdBvaSLa4ZhkBgZgluujsb/jEuEJEk4V1OPgnN1OHiuFgXnarH9YCnOVtvBixJ0ag4xBg0iQtSI0KkREaJCRIgakSFqROhUMGpV0GmU0Gs46DRKaJQshQghvZRPobB48WI89dRTeOGFFwAAiYmJePrpp6/4UNiYfwYv7z6KN6aPxC8C0PS0K7XWqim5nx7J/Zq7zOBFEZU2J85W2XHB4kCN3Ykauwtnq+0oOFuLarsLNTYnzA0uiC160FIqGOg0Sug0nPulbhp3D/UaDlo1h2A1hxC1AloVh5CLpoPVHIJVCo8uxQkh/udTKNTX12PkyJHyNMMwrT5J7Uryr+9K8KePC/HqlHTcbIrq7nI6zNdWTcNidXAKEgxaFQxaFQa2so4kSXAKIpKidXDyImwOHlYHD5uTh7WBh80hwOrkYXPw+LnGjuNOAfUuAXYH7x463a96p4CLe2dUc6wcGE1B4Q4PDtoW4+7lCgSrmse1Kg4aJQuNUgE1x0LNKaBRuodKBUNHM4S0wqdQiIyMxNGjR+Uv0Ycffoj4+Hi/FtZTSZKEt746jRW7jmJ11i8wMSW2u0vqdgzDQM0poFUpUFxll+drlBw0Sg4RrTxQb1isDkUXhZIkSXAJEhy8ACcvwsGLGBShhYuXGsODd4dHiyCptdvlaXfQCLC7BNQ7edQ7BQht9AHMMoCaU0CtZKFpHKq55gBRKppeDJQKFlzTOMtCyTHgWBYqjgXHMh7rtbpN43xOwUClaNyGYz331bjcc3v3+7F0AyIJIJ9CITs7Gy+99BIqKiowduxYjBw5EtnZ2f6urcdx8AL+sLUQ2wtK8fdfj8D4YdHdXVKfwjAMVBzj0XorMTIERWUWqJUKqJUKGNu5bNNa0AiiBF4Q4Woc8oIElyhiSFQIJAlyADkFEQ5X45AX3euKEgRRgksQ4RIk8KIEu5MH3yDJy12CBEF0LwcACQAviHAKje/buK2raX1ehEsU4eIlCN57rQcAKFjmktBpK5S4i9dTsHIQsQwDlnX/O7MM3NMMA0YeR+N0y+UALpruyiMsH3ruB+B+T41SgSClAkEqtnHIIUjp/mPEEKREqFaJEDVHR4CXyWsofPbZZzhz5gzuvvturFq1KhA19UhnquyY/8GPOG9uwOYnRlF31b2EgmWgYBVQXzR/QJhWDhCWZaFhWbTWZ2FrQdOWawcaofLxGojdJeCHkhqIojscBLH5JUru6zmiCCTFhIABwDeGkksQ5aDiRalxvjt4REhwuprmey538iIkSJAkyNd/BFGCJEkQ4f5xFqVLh03rAc3zBQlo7WeXYYCLf+Pb+n1mAPlUYbs/4Y07FSXAwYtocAlw8AIaXO7xBt792ZpwLINQrRKhWhWMjcNovRr9QoPQzxDkHoZqEGsIoi5g2tBuKCxduhTFxcVIT0/H3/72Nxw4cABPP/10oGrrEQRRwj+/KcYru49h9OBwrJ0+AuEhF//EENKxu8+Hxercf3krmHa/hHGhQZeGEsNAySlwcYZ1JMB8Xdcf++zq9xclCfVOAYOjgmF3CjDXu2Bu4GGud6Gu3oVKixNFpWaUmx24YGmAU5CgUrCINwZhQJjW/QrX4qqIYAwI03p0CaPXKmHU9sxm5v7SbigcOHAAH3/8MViWRUNDA6ZOnXrFhAIviNheUIrXvjiBWrsLy3+VgrvS+tGhKSE9DMswCFZzMGpVKKuzAGhq/aZE3EWNAiVJgtXBI0StQH5xLSqtDnx3uhrbC0pRY3eBZYBInRrReg1i9BrcnBSJsUMjERcadMV899sNBaVSKT+TWaPp+N2uubm52Lp1KziOw7Jly5CQkCAvKykpwZIlS8DzPCZPnoysrKwO77+rSZKEY+UWfPxjKbb9WAqrg8ejYwZh5qiBMAR17fMQCCGBxzDuwBgWqwPDeJ7qc7gElFscKK9rwHlzA36qsOK701VYvOUwdGoOQ2N0SIrRwRSjQ1K0DqYYPQzavve70G4oHD9+HGPGjJGna2trPaa/+uqrNretra3Fpk2bkJubi6KiIuTk5GD16tXy8pycHCxatAgmkwlTp05FZmYmQkNDL+ez+Kzpr4UqqxOnK204UWHFwXO12He6GhUWB64bFIYnbk7EXWn9oOvih+MQQnomtVIhn05qMuqqMGjVHI6ct+DYeTOOnrdgY/5Z/FRhhZMXEaPXYEC4FvGhTdcr3NcswoPVCNUqYQxWIVil6FVHGe2GQmFhYad3fPDgQVx33XVQKBRISUlBcXGxx/Li4mIkJycDADIyMlBQUIBx48Z1+v28+fv/ncS2H39Gtc2JGrtTbi1iCFJicFQIrumnxx/uHIaMQWGI0lEfQIQQ95FFlF6DKL0GNw6NlOfzgojiKjuOl1twttqO0tp6HD1vxp6jFSitrUddvUteV6lgYAhSetwn03Ko4li5FRjTdNmdcV+AZxjG3RhBlPDLtFj8MvXyu+P3xqcmqZ1hNpuh1ze30Lm46VnLaYPBgLq6Op/3nZSUdFm1sYDcGqUBwOHG18bL2ishhLhd3BTF0vi6HP8BsOAy9+ELv4WCXq/H8ePH5emmaxOtTZvNZphMJp/2e+zYsa4pkBBCyCX81rFMWloa9u3bB0EQUFhY6HGRGQASEhJQWFgIQRCQn5+P1NRUf5VCCCHER4zk6y2FnbBhwwZs27ZNbn1UUlICq9WKSZMm9cjWR4QQcqXzaygQQgjpXahfYkIIITIKBUIIITIKBUIIITIKBUIIITIKBUIIITIKBUIIITIKBUIIITIKBUIIITK/9X3kL5fbGR4hhFyJfO03rteFAkCd4hFCSEd05I9pOn1ECCFE1iuPFPqCGrsTZrvL63pX4oPDCSHdh0Khm5jtLnx9ssrreqMTwykUCCEBQ6ePCCGEyCgUCCGEyCgUCCGEyCgUCCGEyPx6oTk9PR0pKSkAgBkzZuC2226Tl9HjOAkhpOfxayjEx8dj/fr1rS7LycnBokWLYDKZMHXqVGRmZiI0NNSf5RBCCPHCr6ePysrKMG3aNCxYsADV1dUey4qLi5GcnAyO45CRkYGCggJ/lkIIIcQHfg2Fzz//HO+99x5uvfVWLF++3GOZJEnyuMFgQF1dnT9LIYQQ4gO/hoLRaAQATJw4EUePHvV8Y7b5rc1mMwwGgz9LIYQQ4gO/hYLdbocgCACA77//HnFxcR7LExISUFhYCEEQkJ+fj9TUVH+VQgghxEd+u9B86tQpLF26FFqtFgqFAtnZ2cjLy4PVasWkSZOwYMECufXRPffcQxeZCSGkB2Cklif3e4GkpKQ+0XV2SaXN576PEiKCA1ARIaSv6sjvJt28RgghREahQAghREahQAghREahQAghROY1FP7973/7NI8QQkjv5zUU3nrrLZ/mEUII6f3avE9h9+7d2L17N0pLS7FgwQJ5vtVqRUhISECKI4QQElhthkJycjKMRiMqKiowZcoUeb5Wq0VSUlJAiiOEEBJYbYZCXFwc4uLisGbNGmi1WrAsi1OnTuH06dMYOnRoIGskhBASIF6vKcyYMQMulwulpaWYNWsWPv30U/zud78LRG2EEEICzGsoSJIEtVqN3bt3Y8aMGVixYgXOnDkTiNoIIYQEmNdQYBgGW7ZswUcffYRbbrkFAOByufxeGCGEkMDzGgorVqxAYWEhHn/8cQwYMADnzp3DXXfdFYjaCCGEBJjXUBgyZAgeeOABaDQaAEBoaCjuu+8+vxdGCCEk8LyGwhtvvIEVK1Zg5cqVANxPSZs7d67fCyOEEBJ4XkNhx44dePPNNxEUFAQA6NevH8xms98LI4QQEnheQ0GpVIJlWTAMAwCwWCzyOCGEkL7F6+M4p0yZgnnz5qG2thZvvPEGduzYgUceeSQQtRFCCAkwr6Fw//33Iz09Hd999x0kScIrr7yCIUOGeN3xyZMn8dxzz4FlWbAsixdffBH9+/eXly9atAjHjx9HcHAwBg0ahOzs7Mv7JIQQQi6b11CYP38+Vq1a5REETfPaYzQasXbtWuj1euTl5WHNmjVYtmyZxzrPP/88UlJSOlk6IYSQrtZmKIiiCJ7ncerUKbhcLkiSBMDdS+qRI0e87jgsLKz5TTgOCoXiknWys7Oh0Wgwe/ZsjBkzpjP192qSJOFEhRVHzltQbXOg1u5CiIZDRLAaCeFaXNPP0N0lEkKuMG2Gwvr16/HOO++goqICmZmZcigEBwd79JrqTX19PVavXn3JUcLChQthNBpRVVWFhx56CGlpadDpdJ38GL2LJEnYXXge6746jWqbE1fH6hFv1CK5nxIWB49KiwM7D5/HxwdLsf9MDRZPMiHWENTdZRNCrgBthsLMmTMxc+ZM5ObmIisrq1M753ke8+bNw6xZs5CYmOixzGg0AgDCw8ORnJyM06dPIzU1tVPv05vU2p1YtOkQvjx+AWOGRCBjYBiC1Zf+ZxBECcfLLTh0rhY3rfw/PDpmEObcPLjVdQkhpKt4bZLaMhCWLl3q844lScKSJUswbtw4jB8//pLlFosFAOBwOFBUVIS4uDif991bHT1vxqS/7kVZXT3WzRiBm5Oi2vyRV7AMro7V4y9T0vG3B4djV+F5ZP41D/89VRXgqgkhV5IO/dl56NAhn9fdu3cvdu3ahdLSUuzcuRMmkwljx46F1WrFpEmTMH/+fFgsFvA8j+nTpyM8PLzDxfcmh87VYfo/vsOEYTH48+RklNXWo6S63ut2DMNg/LBojBkSgVc/P45fr/sO069PwMJME4JUl16nIYSQy+E1FP7xj3/I9yU0PXGt5by2jBs3DgcPHmxz+ZtvvtmROnu1/WdqMPOtffjViHj88c5hnbr5T6NUYPHEq3H7sBg88+FBfHn8Al65PxUjEsK8b0wIIT7yevpo+/bt8viKFSsAANu2bfNfRX1McaUNj/wzH1kZ/TsdCC2NSDDikyfH4qakSDyw9r944d9FqHcKXVQtIeRK1+aRwocffoiPPvoIxcXFHtcVbDbbJReNSevq7C488s98jE6MwOKJV3dZ9yBBKgX+eOc1mJgci2c/OojPjpRjxa9Scf1VffsUHCHE/9oMhczMTNxwww147bXX8OSTT8rztVqt3HKItM0liHj8Xz9AF6REzgNpYNmu7y8qY1AYdj41Tr7W8OvrBmBhpolaKBFCOo2Rmm5A6CWSkpJw7Nix7i6jXZIkYfHmQ9j7UyW2zBmFKJ3mknVKKm34+qT3lkSjE8OREBHsdb0DZ2rw7EcFsDsFLJxowp2psdRxISEEQMd+N+lPSj9Yt/c0th8sxUePtx4IHeEURJRU2ryuF6ZV4b1Z1+GjH87h95sP4e2vT+N3tyfhhsRwCgdCiM8oFLrYp4XnsWLXUbwxYwSujtVf9v5sDh75ZRaf1h2dGI45Nw/GAyP74697juOht/ORHKfH4zcNxi2mKCj8cAqLENK3UCh0ocM/1+Gp3B/x+0lX4xZTdLfVEalT44XJKfjtzUOwNu8kns49AEOQElkZAzA5PQ4DwrXdVhshpGejUOgi5+saMOud7/GrEXF4ePTAbqmhtVNND90wEA+MiMfnRyrw74JSrPrsOJKidRg/LAoTrolBcj+DXy6CE0J6JwqFLmB18Hj0nXwMiQ7Bn+68ptvO4bd3qik8RI2ZowahyurAoZ/r8OWxC/j7/52EPkiJ1PhQpMUbkBYfitT+hsu+DkII6b0oFC6TSxDx+Hs/QBAl/O3Xw8EpvN4P2K3CQ9S4KSkKoxPDERqswr7T1Sg4V4uD5+rw7rclqKt3ISJEjUERWiSEB2NQRDASwrXoFxqEyBA1InVqaJTUvQYhfRWFwmWQJAkLNxXg1AUbNj8xCnqNsrtL8plTEFFrc2JoVAiGRoXgvuHxkCQJpXUNOHXBip9r6/FzbT2+OFqBc9V2VFodEBobL4eoOYQFqxAerEJYsAqhWiV0ag6Reg1i9BoYgpQwaJUwBCkRGqSEPkhJQUJIL0Gh0EmiKGHptsP44mgFPvrNDYjW965TLu2damJZFv3DgtE/LBjDYnUoKrNAlCTYnQKsDTwsDS5YHLw8fvKCE/VOAZyCQYNLQF09D3O9C05BlPep5ljoNBxC1ByC1e5hiJpDiMY9rVO3WKZpXh6s5qDTeG5DragI8R8KhU4QRQlLth7CrsPn8a9Z12FwVN9/OBDLMPKPcoyh9QC8dqARqsbTZ5IkwcGLsLQIkXqnAJuTh80hoIF3B4zdKaDO7kRpTT3sTgF2Jy8PbU4BdocA4aL7K4OUCjk4gtUKBCkV0CjdwyBVi+nGcfc0C03jfKWChZpjoVSwUCoYKDkWKoV7WsW55zVNK1tM0/0e5EpAodBBDS4BizYVIO+nSrz/2PVdci9CX+HLPRUaJQeNkpOPQLwZmRAKSIC9MVDcgdE8Xu8U0OAS4OBFOAUR9Y2BUm1zwuES0MCLHkMHL8IlShBECbzg3sYluKe9USqYxiBpDBAF0xgiLLjGafc407i8eVwOoBbjXIv9cG2so1Sw4Njm4PIYb1qXZaHkPMc51r09BRnpKAqFDqgwN+Cx9T+gzu7EB7NvwOCokO4uqc+zO4U2woOBVsVBq2r+X9jXoAE8j2oA95PueFEEL0hwNQYFL4gQ4f5DwCU0L+NFCU6h5brueYIkwcU3h0zTNrzontfgEmFt4N3rtthH0/s2rcc3BlbTeu7aGvct+h5igDvImgJCDhuOcYdHKwHWdGqu5cGZhOYJj/kXrSNJaF5Tap7XOAlJkhqHkENZ8thP8ztxrDsoNUoFtCqFXGPT51BxLNScAjoN1+KlREjj6UZ9kBKROjV0ao6CsYMoFHz0eVE5fr/lEIZG6/DuwxkwaHvPRWVyKV/vFB8Wq0Nxlb2VJQy4xr/wg1qs62so+bruxeHVRJQkj0Bxie4fWIfLHVJyKIkXrdc4T5QkOPnm8GtaT5AkNP2GMgAYloEkSmDQ4oe19VH3/S5S80wGAMOgeVumcR4AlyjhbE29vP3Fv9tiY3DEGjSQJM+gdvIi7E4eDS730OoQYHO4jxytDh52J4+mzFRzLMKDVQgPUSFSp0FcqAZxoUHoH6ZFrF6DsGBVu6Gh1yph1Kra+0/U51AoeFFubsCyHUew6/B5PHnrYPzmxsQe3+yU9B0d6eak7QBrxrIsVKzvoeSPoOvIup3ZpyS5j8osDlfjNS33da3aehf2na5Bta0cNXYneFGCRskiRnhE140AABmrSURBVK9BjEGDaL0GsXr3UN3YWm50YjiFQlfKzc3F1q1bwXEcli1bhoSEBHlZSUkJlixZAp7nMXnyZI9nNvQE52rseCPvFHLzzyIt3oAdT47BkOi+f0GZkN6OYRh3IwOVAm21ATHFhOC/p6pRYXHgfF0Dzpsb8N2palywOCBIEsKCVYjRa3DsvBk3JIbDFKPHgDDtFXH3v99Coba2Fps2bUJubi6KioqQk5OD1atXy8tzcnKwaNEimEwmTJ06FZmZmQgNDfVXOV5JkoSz1fXYe+ICPv6xFPuKqzEywYi3Zo7EmMERdF6SkD6EZRiEalUI1aowtMUfe7wootLiRFldPcrNDThSZsbOw+dRYXFAq1IgKUYHU4wephgdBkYEY2C4FnGhQX3q7IHfQuHgwYO47rrroFAokJKSguLiYo/lxcXFSE5OBgBkZGSgoKAA48aN81c5EEUJ5gYXqm1O1NidqLa5UGV1oLjKjhMVVhz6uRblZgcGhmsxKSUW2XcnIymGjgwIuZJwLIsYg0Zudt30PJMqqwNHz1twpMyMI2UWbNp/DqcrbbA08OBYBvHGIMQZgxCl0yBSp0aUzn33vyFI2Xh/jtLdjFrFQa10tyJTsD2zdZjfQsFsNkOvb26uefGzfFpOGwwG1NXV+asUAMCfdxTh7a+LAbhbNhgb78gdEKbFkOgQ3JXeDxkDw9psg08IuXKFh6gxerAaowdHyPMkSUKt3YXiKhuKq2worW3ABYsDP9fU48CZGlRYHDDXu69r8K20FmMYNLYCa2qS3HQZXoIoNbfWEkX38NlME6Zfn3DJfrqa30JBr9fj+PHj8jTLeh5etZw2m80wmUw+7zspKalTNalbjJsbX6cBfNmpvRFCiHeKxldb+MZXg5f9vLABeKHLqmqb30IhLS0Nr7/+OgRBwNGjRz0uMgNAQkICCgsLYTKZkJ+fj1mzZvm0357+KE5CCOnN/PqM5g0bNmDbtm1y66OSkhJYrVZMmjSpx7c+IoSQK5FfQ4EQQkjv0nfaURFCCLlsFAqEEEJkFAqEEEJkFAqEEEJkFAqEEEJkFAqEEEJkFAqEEEJkFAqEEEJkFAqEEEJkve7Ja53tDI8QQq5kvvYb1+tCAaBO8QghpCM68sc0nT4ihBAi65VHCn1Bjd0Js93ldT29VnnFPTicENJ9KBS6idnuwtcnq7yuNzoxnEKBEBIwdPqIEEKIjEKBEEKIjEKBEEKIjEKBEEKIjEKBEEKIzK+tj9LT05GSkgIAmDFjBm677TZ5WUlJCZYsWQKe5zF58mRkZWX5sxRCCCE+8GsoxMfHY/369a0uy8nJwaJFi2AymTB16lRkZmYiNDTUn+UQQgjxwq+nj8rKyjBt2jQsWLAA1dXVHsuKi4uRnJwMjuOQkZGBgoICf5ZCCCHEBz4dKZSXl2PDhg0oLS2FIAjy/JycnHa3+/zzz2E0GvHJJ59g+fLlePnll+VlkiTJ4waDAXV1dR2tnRBCSBfzKRQef/xx3Hnnnbj++uvBsr4fXBiNRgDAxIkTsWbNGo9lLfdjNpthMpl83i8hhBD/8PmawsMPP9yhHdvtdqjVaigUCnz//feIi4vzWJ6QkIDCwkKYTCbk5+dj1qxZHdo/IYSQrudTKIwaNQp///vfccstt0Clau6HZ9CgQW1uc+rUKSxduhRarRYKhQLZ2dnIy8uD1WrFpEmTsGDBArn10T333EMXmQkhpAdgpJYn99swffr0SzdkGLz77rt+Kao9SUlJfeJ5CiWVNp87xEuICA5ARYSQvqojv5s+HSm01ayUEEJI3+JTKFRXV+Mvf/kLfvjhBwDAiBEj8PTTTyMsLMyvxRFCCAksn5oSPfvsszCZTNi4cSM2btyIq6++Gr/73e/8XRshhJAA8ykUKioq8OCDDyIkJAQhISGYOnUqLly44O/aCCGEBJhPoRATE4P169ejuroaNTU1eO+99xAdHe3v2gghhASYT9cUli9fjtdeew2PPPIIAGD48OFYsWKFXwsjhBASeD6FQllZGf74xz96zNu7dy/Gjh3rl6IIIYR0D59OH/3xj3/EypUr4XQ6UVtbi2effRbvvPOOv2sjhBASYD6FwsaNG6HX63H33XcjKysLo0aNwrp16/xdGyGEkADzKRSqq6tx8OBBJCYmIigoCMXFxXC5XP6ujRBCSID5FAozZ87Evffei//93//FBx98AIVCgfvvv9/ftRFCCAkwny405+bmQq/XAwCUSiXmzp2LiRMn+rUwQgghgedTKNTU1ODpp5/GiRMnwDAMBg8ejD/84Q/+ro0QQkiA+XT6aPHixZg5cyby8vLw5ZdfYubMmVi8eLG/ayOEEBJgPoWC1WrFjTfeKE+PGzcOVqvVb0URQgjpHj6dPhoyZAiWL1+OO+64AwzDYMeOHRgyZIi/ayOEEBJgPoXCsmXL8N5772HNmjWQJAnDhw/HU0895e/aCCGEBJhPoWCxWDBjxgz5OcpOpxN1dXXQaDRtbnPy5Ek899xzYFkWLMvixRdfRP/+/eXlixYtwvHjxxEcHIxBgwYhOzv7Mj8KIYSQy+VTKPzmN7/Bhg0b5GlJkvD444/jo48+anMbo9GItWvXQq/XIy8vD2vWrMGyZcs81nn++eeRkpLSydL7BkmSUGl1oqTKhgqLA0oFC42SRX+jFgPCtd1dHiHkCuNTKPA8D5VKJU+r1Wo4nc52t2n5VDaO46BQKC5ZJzs7GxqNBrNnz8aYMWN8rblPkCQJhaV1+KyoHBUWByJC1IgxaMALIuxOAZ8WlkOtZDFhWDSeyTQhLjSou0smhFwBfAqF6OhobN26FZMnTwYAbN261efnKdTX12P16tWXHCUsXLgQRqMRVVVVeOihh5CWlgadTtfB8nuns9V2zNlwAMfLLRg9OAKPXhUOnUbpsU6DS8BPFVYUltbhppX/Qda1AzDvtqEIC1a1sVdCCLl8PjVJfeGFF/DFF19g3LhxuPHGG/Gf//znkh/51vA8j3nz5mHWrFlITEz0WGY0GgEA4eHhSE5OxunTpztRfu/z9YlK3Pm/X8GoVeKZ25Nw+7CYSwIBADRKBVLiDFg9JR3vPJKBgnO1GL/qS2w5cA6SJHVD5YSQK4FPRwpRUVFYvXp1h3YsSRKWLFmCcePGYfz48Zcst1gs0Ol0cDgcKCoqQlxcXIf23xttzD+DJVsO48lbh+DO1Fh8e6ra6zYMw2BUYgQ2PzEa735bjOe2HMa2H0vx8q9SEaVv+0I/IYR0RruhMH/+fDAM0+bynJycNpft3bsXu3btQmlpKXbu3AmTyYSxY8fCarVi0qRJmD9/PiwWC3iex/Tp0xEeHt75T9ELfPj9WSzZchirpqTjrrR+KKm0dWh7Bcvg4dGDcPs1MXjmg4O4/S95eGFyMn6Z2s9PFRNCrkSM1M65iH379rW7cUZGRpcX5E1SUhKOHTsW8Pe9HNt+/BkLPjiInAfScHe6+4iopNKGr09Wed12dGI4EiKCPeaJooS3vynGil1HMTE5Btl3JcOgvfQUFCGEAB373Wz3SGH48OHgOM5rSyPStm9PVuGZDw/ixXtT5EC4XCzL4NExgzBuSATmf3AQE/6Sh5X3p2LskMgu2T8h5MrVbijMmTMHa9euRWZmpnwaqenAgmEY7Nmzx/8V9mIlVTY8/q8f8PiNiXhgZH/vG3TQkGgdNj8xCq/t+QkPv52PX183AIsmXo0g1aXNfwkhxBfthsLRo0dxxx134LrrrkN6ejrS09MxZMgQsKxPjZauaOYGFx75Zz5GJYbj6fFDO70fpyB6vf7wq+HxuDpWj+W7juI/xy7g+buuwc2mqE6/JyHkytVuKHz55ZcoLy/HwYMH8eOPP2L79u0oLi5GYmIi0tPTMW/evEDV2avwgojfvn8AWhWHnPvTwbJtX6z3xubgkV9m8WndN6aNwL8PlWH2+h9wU1IkFk004arIkE6/NyHkyuP1T/7o6GjcfvvtmDNnDubOnYsHH3wQ586dw44dOwJRX6/0wo4jOHbejDdnjAzoqRyNUoEFtydh19NjwYsSbns1D4s3F+BcjT1gNRBCerd2jxQ2bdqEgwcP4vTp09BoNEhOTkZqaio+/PBDj24sSLP3/luC3Pwz+GD2DYgxdM99BFdFhuAfD12L/OJqrNx9DDeu/D9MSonFI6MHIr1/aLvNjAkhV7Z2Q2H16tUICwvD3XffjfT0dAwbNsyjDyTi6ZsTlXh+eyH+MuUXSI0P7e5ycO3AMHww+wYcOFODdXtP47413yIxMhi/Gh6PSSmx6B9GHe4RQjy1e58CAI9rCkeOHAHP8xg6dChSU1Nx9913B6pOWU+9T+HUBSvuef0bPDpmEJ681fsDiHy9T2FYrA5FPl5TuHagESpF22cEq6wOfHakHLsLy3Gq0gZTjA43JUXh2oFGjEgwIlRLgU9IX9SR302vodDEbrfj0KFD+OGHH7Bp0yaUl5fj8OHDl1VoZ/TEUKi2OXHf37/BNXEGrM5K9+n0jD9CoSPrDgzXoqjMjG9OVuH74mqYG3gMjQ7BiIQwJMfpMTRahyFRIRQUhPQBXXbzWtM1hR9//BE1NTVISUlBWloaXnzxRaSmpnZJsb2d3cnjkX/mIzxEhZX3pfaa8/VxoUEYNTgCs8ZeBVGU8FOFFfnF1fi+uBrrvy3BqQs2OAURkTo1hkSFYECYFnGhQYgPC0K80T0erddAcRktqwghPU+7oVBUVISRI0fiscce83hqGnFzCSKe+Nd+2J08Ppw9Chpl77lp7OL7HzQci7GDIzB2cAQAgBdFlNY2oNzSgPN1DThXU4/vTldj84F6/FxTD6cggmMZxIZqEKPXIEqnQaROjUidGlE6NaL0GkSGqBGlVyNMq7qsZrmEkMBpNxSWLl0aqDp6HScv4skNB3D8vAWbnhjV6/oe8vX+h2sHGnFtgud1ClGSUGNzoszcgHJzAyqtTlRbnSitrcehc7WosjlRZXPC0sADcHfmF6ZVIixYhbBgd1DEGDQI06pgDFbJw/Bg91Cv4XrNERchfY1PXWcTTw0uAU/8az+Ol1uwcfYNiDX03aeieQsPjZJDvJFDvFF7yTUNlyDC6uBhaeBhaXDJQ0GUcLTMgrp6F+rsTvew3gWh8eqWgmVg0HAwBCkRFqJGWLAKIWrO/dK4h7rGYbCag07NQavmoFQwUHMslAoWqhZDlcL9oqMVQryjUOigapsTT/zrB5SbHfhg9g3oR4/JbJNSwcKoVcF40cXq1i6IS5KEBpcIm5OH3cHD5hRgd/KICFFDqWDlcLlgcbjHHTysDS7YHAKsDh42Jw9vTSYULAOlgoGCYaBgGXCse6hgWXCK5mmucZ6CZaBRstAoFeBYBpyChVLBgGNZcCzTHDpci/DhWKgvmr54XM2xUCkUbW9LAUa6EYVCBxz+uQ6z1/+AWIMGH8y+AZE6dXeX1GcwDIMglcJ9B3hI87+rt2a2TSRJAligwSGCF0U4BRG8IMEliHA1Dp2CCEGUYHMKOHbeAlGUIEoSBAkQGsdFUYIgD4F+jTcgCqIEXnTv0z0uod7Fo67evW9elODkBTib3pMXG9+zqYbGab55/fZwLCMf7TQFkFLByKfVmiKj6SybojHQmMYloiRBahxC8pwWRAmC4J6W5PkAJAmie3V5viS530tqfK/mwAQ4lgXbOFSwzWGraBGurEf4Ns5jGHCKxmGL/bXcb8v9tAzxVvfHMnDwIlwuESzr/rcIVnEeR5c6NQd14zU/hYKBIHhvdKnXKi/5g+ZKQKHgA5cgYt3e0/jL58fxwMj+WPrLYVBx1ClgIHSk7ydfm+QOi9V5ParozD47cj8JxzLNgSUHiGeAuXgR1sYA40UJgih61C21GBkUoZVbgkkSwDIMGAaNL3dUsI0J4hJFFFfa3cuA5qBpZdoUrZP/X3eHpjsgm4KTbwxTCYCLF5tDR3QHTNO6zfMkiACExoBuWu5ySRAlwWM9tjEMeLF5+5YvXhQbA11Eg1OAuYGH2BiADpeABpcIocU/WJBSAUOQEv1C3a3mQrUqRIa4G0eEh6jAXdTR5+jEcAoF4kmSJHx7qgrZ24tQbXPi1SnpmJQS291lkV7O5uB9DpDkfnr5x7w9HQmlX8SGIkTtW8OIpJjW98swDBQKBgpFx9/f13V9PUoEALtLwIEztR7zJEmCS5DQ4BJgdwowN7hgrndBxbE4UWHFyQtWfHeqCuYGHiwDGLUqROndreliDBrEh2oQZwwC52MNfYVfQyE3Nxdbt24Fx3FYtmwZEhIS5GUlJSVYsmQJeJ7H5MmTkZWV5c9SOsTBC/jiSAXW5p1CYWkdHswYgAUTkqDX9K4WRoT0Zh09SrwYwzBQce7TcPogpdwX2cWh1OASUGl1oMLiQIW5AT/X2rH/TA027DsDFcdiaHQIkqL1MMXokBSjgylGh0idus+2kPNbKNTW1mLTpk3Izc1FUVERcnJysHr1anl5Tk4OFi1aBJPJhKlTpyIzMxOhod3TX5AkSThbXY/vS6rx1U+V+KyoHACQldEfa6aN6LaO7Qgh/qdRKhBv1CLe6NkXWGqc3n39qdyCo+ct2Hm4DH/d8xOsDh5hwSokRTeHxJBoHeJCgxCpU/f6Gzr9FgoHDx7EddddB4VCgZSUFBQXF3ssLy4uRnJyMgD3s54LCgowbtw4f5WD05U2HCkzo8buRK3dhVq7E1VWJ05X2XDqgg119S7EG4OQMSgMr05Jx9ihEVBzvedmNEJI19JplEiOD8V1V4XL8yRJwrmaehw7b8GxcguOlJnx1leVOF1pAy9KULAMonVqxIYGITJEDUOQEqFaJfRBShiClNCqFI2tzBSXtDpTsAzc7QKkxvdqrqVfaBDCggNzfcNvoWA2m6HX6+Xpi7tYajltMBhQV1fnr1IAACt3H8XBs3UIbWxRENp4M9UvU/vhqohgDOunR7SejggIIW1jGAb9w7ToH6bF+GHR8nxBlFBpdaC0th5ldQ0oq2vABYsDdfUunKupR21pHerqXbA7BTh50f0SRHncW2u0O9P64bWpv/D3xwPgx1DQ6/U4fvy4PH3xIzxbTpvNZphMJp/3nZSU1Om6Kju9JSGEdD1F46s9n24Bkv4UgGLgx1BIS0vD66+/DkEQcPToUY+LzACQkJCAwsJCmEwm5OfnY9asWT7tt6f1kEoIIX2Jz11nd8aGDRuwbds2ufVRSUkJrFYrJk2a1KNbHxFCyJXKr6FACCGkd7my7soghBDSLgoFQgghMgoFQgghMgoFQgghsl4XCiUlJZg2bRqysrKQm5t7yfKdO3fi/vvvx69//WvMmzcPTqezG6pslpubi6ysLEybNg0lJSUey7x9lu7WXu0LFy5EVlYW7r//fmzZsqWbKmxbe7UDgCAIyMzMxFtvvdUN1bWvvdptNhsWL16MmTNnYvr06d1UYdvaqz0vLw/33nsvsrKykJ2d3U0Vts7pdCIrKwsjR47Erl27Llnek7+r3mrv8HdV6mXmzp0rHTp0SHK5XNJ9990n1dTUeCw/c+aM5HK5JEmSpJUrV0pbt27tjjIlSZKkmpoa6b777pN4npcKCgqkuXPneiz39lm6k7faT58+LUmSJDkcDmnChAmS0+nshipb5612SZKkjRs3So899pi0bt26bqiwbd5qX758uZSfn99N1bXPW+1TpkyRfv75Z0mSJOl//ud/pCNHjnRHma0SBEEqLy+XVq9eLe3cufOS5T35u+qt9o5+V3vdkUJTn0kcx8l9JrXUv39/cJz7njyFQgGFovv6L/K1/6e2Pkt38lb7wIEDAUD+t774jvXu5K12h8OBL774AhMmTOieAtvhrfb9+/fjk08+wfTp0/Hee+91T5Ft8Fb70KFDYTabIQgCHA5Ht3WA2RqWZREVFdXm8p78XfVWe0e/qz3nm+wjycc+k06ePImvv/4at99+e6BKu0RP6/+pI7zV3uTNN9/EHXfc0a3hezFvta9fvx5TpkzpkV0fe6u9sLAQt956K95++218+umnOHHiRKBLbJO32idMmIBZs2YhMzMTV111FWJiYgJdYqf15O+qr3z9rvbIUKioqMADDzxwyWvPnj2X9JlkMBgu2b68vByLFi3CqlWroFJ135OT9Ho9LJbmftu99f/U2mfpLt5qB4CPP/4YR44cwZw5cwJZmlft1W6xWLBv3z7cfPPN3VGaV97+3Y1GI0aNGgWO43D99dd79C/W3bzV/uc//xmbN2/G7t27UVNTgx9++CHQJXZaT/6u+qIj39UeGQpRUVH44IMPLnndeuutcp9JgiAgPz8fqampHtuazWY89dRT+MMf/oABAwZ00ydwS0tLw759+yAIAgoLC9vs/6mtz9KdvNWel5eHLVu24OWXX+5Rp46A9ms/deoUampq8Oijj+Ltt9/Ghx9+iK+//robq/Xk7d99xIgRKCoqAgAcPnz4kuXdyVvtHMdBp9OBZVno9XqYzeZuqrTjevJ31ZuOfld7XTcXbfWZtHjxYrz00kt45ZVXsHXrVgwaNAgAcM899+Dee+/ttnp7c/9P7dU+evRoREVFISQkBACwatUqREZGdnPFzdqrvcnmzZvlgOhJ2qv97NmzWLp0KRoaGpCRkYH58+d3d7ke2qt9x44dePvtt6FUKtGvXz+sWLFCPs/dE8ydOxdFRUXQarUYPXo0Ro0a1Wu+q+3V3tHvaq8LBUIIIf7Ts477CSGEdCsKBUIIITIKBUIIITIKBUIIITIKBUIIITIKBUIIITIKBUIIITIKBUIIIbL/BxqwzlBevs/uAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(3,1, sharex = True)\n",
    "\n",
    "sns.distplot(pvals, ax = ax[0]); ax[0].set_ylabel('Permutation');\n",
    "sns.distplot(pvals_t, ax = ax[1]); ax[1].set_ylabel('t-test');\n",
    "sns.distplot(pvals_w, ax = ax[2]); ax[2].set_ylabel('Wilcoxon');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also compare how many proteins will be chosen if we employ a p-value cutoff of 0.0001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "permutation    896\n",
       "ttest          499\n",
       "wilcoxon       396\n",
       "dtype: int64"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pvalues = pd.DataFrame({'permutation' : pvals, 'ttest' : pvals_t,\n",
    "                           'wilcoxon' : pvals_w})\n",
    "pvalues.apply(lambda x: np.sum(x < 0.0001))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The **lambda function** employed above is an anonymous (un-named) function that\n",
    "can be used on-the-fly. In the above statement, this function takes one (vector) argument *x* and computes the number of *x* values less than 0.0001. This function is then applied to each column of the `pvalues` dataset using the `apply` function.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting a confidence interval using the bootstrap\n",
    "\n",
    "We can use simulations to obtain a model-free confidence interval for particular parameters of interest based on our observed data. The technique we will demonstrate is called the bootstrap. The idea is that if we sample with replacement from our observed data to get another data set of the same size as the observed data, and compute our statistic of interest, and then repeat this process many times, then the distribution of our statistic that we will obtain this way will be very similar to the true sampling distribution of the statistic if we could \"play God\". This has strong theoretical foundations from work done by several researchers in the 80s and 90s. \n",
    "\n",
    "1. Choose the number of simulations `nsim`\n",
    "1. for each iteration (1,...,nsim)\n",
    "    - Simulate a dataset with replacement from the original data. \n",
    "    - compute and store the statistic\n",
    "1. Compute the 2.5th and 97.5th percential of the distribution of the statistic. This is your confidence interval.\n",
    "\n",
    "Let's see this in action. Suppose we tossed a coin 100 times. We're going to find a confidence interval for the proportion of heads from this coin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "name": "04-python-stat-33"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0,\n",
       "       1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0,\n",
       "       1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1,\n",
       "       1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1,\n",
       "       1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rng = np.random.RandomState(304)\n",
    "x = rng.binomial(1, 0.7, 100)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives the sequence of heads (1) and tails (0), assuming the true probability of heads is 0.7. \n",
    "\n",
    "We now create 100000 bootstrap samples from here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "name": "04-python-stat-34"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD5CAYAAAA+0W6bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXhb5aEm8Ff7YkuyvK+x4yXOHmdlCTjsBMIUCqFNoSHtQChdgIFOB4alTyeUDgylTcO0wC293DYBAlwolC0JS0tICMQmq+14S7zvtiRL1mJt5/4RsmEntmVZR0d6f8/jx+HYst7okd8cvvOd75MJgiCAiIgkRy52ACIiCg0LnIhIoljgREQSxQInIpIoFjgRkUSxwImIJEoZyScrLS2N5NMREcWMurq6EcciWuBnC0FERGd3tpNfDqEQEUkUC5yISKJY4EREEsUCJyKSKBY4EZFEscCJiCSKBU5EJFEscCIiiYr4jTxEUmd1eWF3+UYcN+pVMOvVIiSieMUCJ5ogu8uH3UcHRhxfXpTCAqeI4hAKEZFE8QycaIqNNuTC4RYKBxY40RQbbciFwy0UDixwonMY7ezZ5QuIlIboTCxwonMY7ex5dpZBpDREZxrzIqbX68WaNWuwZMkSbNu2DQCwefNmrF69GmvWrMFjjz025SGJiGikMQtcqVRi06ZNWLdu3clj5eXleP3117F161ZYLBZUVlZOaUgiIhppzCEUuVyO9PT0M47l5+ef/LNCoYBCoQh/MiIiOqdJzQOvqKiAxWLBwoULw5WHiIjGKeSLmA0NDfjtb3+LZ599Npx5iIhonEIq8M7OTjz44IPYuHEjkpOTw52JiIjGYVwFfvfdd6OmpgZ6vR4HDhxAT08PbDYbHnroIQDA+vXrUV5ePqVBiYjoTOMq8GeeeWaqcxAR0QTxRh4iEXgDQbT0O0cc5xopNBEscCIROIf9qOhyjDjONVJoIricLBGRRLHAiYgkigVORCRRLHAiIoligRMRSRRnoRDh7DvNc/MGimYscCKcfad5bt5A0YxDKEREEsUCJyKSKBY4EZFEscCJiCSKBU5EJFEscCIiiWKBExFJFAuciEiiWOBERBLFAicikigWOBGRRLHAiYgkigVORCRRXI2QKIpwt3qaCBY4URThbvU0ERxCISKSqDEL3Ov1Ys2aNViyZAm2bdsGAHC73bj33ntxyy234Je//CWCweCUByUiojONWeBKpRKbNm3CunXrTh574403sGDBArz88stQKpXYuXPnlIYkIqKRxixwuVyO9PT0M45VVlZixYoVAIAVK1agsrJyatIREdFZhTQGbrfbYTQaAQAmkwmDg4NhDUVERGMLaRaK0WiEw+FAWloa7HY7TCZTuHMRTdpoO81zOh7FkpAKfOnSpfj0009RWFiInTt34qKLLgp3LqJJG22neU7Ho1gyrgK/++67UVNTA71ejwMHDuDee+/Fgw8+iFtuuQXFxcUoLy+f6pxERPQN4yrwZ555ZsSxP/zhD2EPQ0RE48cbeYiIJIoFTkQkUSxwokkQBAHPfNKA2m672FEoDrHAiSZhYMiLrkEPDrTZxI5CcYgFTjQJDX1DUMhkqOt2wOvnmkAUWSxwoklo7HFgSYEZAoB9rVax41CcYYEThSgQFHCs34lZWUaUZhiws6Ff7EgUZ7ihA1GI2iwu+IMCClIS4PEFsK2qG4GgAIVcJnY0ihM8AycKUUPvEPJT9FAr5SjNMMDp9aOy2SJ2LIojLHCiEDX2OlCSbgAAaFQKLMk3Y1t1t8ipKJ6wwInG4A8G8XFtD4Z9gZPH3N4A2q1ulKQnnjx2SWk63jnYBV/g1GyUDw53cYohTRkWONEYqjvs+PhIL75sOjU8cqjDhkStEpkm7cljl8xIw7A/gI+P9AIAHB4f/tcbh/DwW4dhcXojnptiHwucaAxfHBtAcoIau4/2wx8IwhcI4l91fSgvSYNcduqCpValwA1lOdha0QoA2Lq3DWa9Ggtyk7C1ohV+7h1LYcYCJzqHY/1OtFpcuO38fAQF4ECbDR/W9CAoCFg2PXnE969ZlodP6/vQMuDEv+9uwh0XT8eDK2diaNiP7VUcH6fwYoETncMHh7tQmmlAulGL5UUp2NnQh//c14FLZqRBpRj56zMn24S52Sb8aPNX8PgCuHlxHow6FW5alIs9xwbg9gZGeRai0LDAic7C7Q3g04Z+nF+YAgA4b3oK7B4/ZACWFow8+z5hzbI81HY7sO7CAujUCgDA9NQEJGqUYV30yuryoqXfOeLD6uJ4e7zgjTxEZ7G/zYqUBDWKv55polMr8K0F2ZiTZYBylLPvE64vy8G+Fhtuu6Dg5DG5TIZZWUZUd9qxcJo5LPlG2zIO4LZx8YRn4ERnUdftwAWFKWdcqFw0zTxmASdqlHj6OwuQnHBmic7JNqGhl4teUfiwwIlG4QsE0dTvxMJpSWH7mdNTE6CUy9HY6wjbz6T4xgInGkVTvxNyuQwzMw1h+5mKr39edacdgaCA9w934a+fN8Ph8YXtOSi+sMCJRtHQ40BhasKoM00mY3a2EbXdDvzm/SM42G6DxxfAH//ZiJYBZ1ifh+IDC5xoFPW9Q5iREb6z7xNK0g3wB4MYcHrxk0uKccfFhViQl4QXPmuCnWfiNEEscKJvsLm86HMMn7HOSbiolXLctaII//fGeTDpVFDIZVg5JxMJGgWa+nkWThPDAicC0G514a0DHehzDKOhZwjJCWqkJGqm5LmyTDroVIqT/y2TyVCQmoBmFjhNUMjzwDds2ICqqioEg0Hcc889KC8vD2cuoojxBYJ47L0j6Br0oLLZgkSNErOzjRHNkJ+SgIomriVOExNSgTc3N6OxsRGvvfYaBgYGcOedd7LASbL+8FEDBoaGcfdlxege9OCfdb0oywvPzTbjVZCix7sHOzHk8Uf0eUnaQirw1NRU6PV6+P1+OBwOmM2RfbMThUtFswXPfXoUT62eD7cviMK0RBSmhX/seywZRi00Kjlqu+3Qa1QRf36SppAKPCEhAZmZmVi5ciVcLhc2bdoU7lxE42Z1eWF3jZzB4fKNvXDU//+kEd9dmodF08yj3pYeKXKZDPnJCajpcmDJOdZZITpdSAW+e/duDA4OYseOHejv78ddd92FN998M9zZiMblbGuCzM469zRAfyCIymYLbr9o8VRFm5CC1ATUdNlZ4DRuIc1CCQaDMJlMkMvlSExMhMvlCncuoilX02WHxx/EovzoGAIsSNGjsXfojC3ZiM4lpAJfvnw5nE4nbrnlFqxduxY/+clPwp2LaMp9ecyCudlGJGqiY1HOnCQd5DIZ2qw8IaLxCemdq1Ao8NRTT4U7C1FEfdlkGXVXHbEoFXKUZhqwvaob18zNQkFqAoaG/ei0ubE0Sv4vgaJLdJx6EEVYMCigotmCNUvzxI5yhnsuK8bzO4/hhV3HYNCqMOg+fnE2N0mHkjAurEWxgQVOcam+1wG7x3fOnXXEkGbQ4MZFubikNB1N/U7kmXX4qLYXNV3h28mHYgcLnOLSl8csKM0wwKSPzjnXyQnqkxtCTDPrwroVG8UOroVCcWlvk+XkXpfRLtesR0PvEHfyoRFY4BR3XF5/1F3APJfsJB2CQYFn4TQCC5zihiAI+KyhD1f+bieMWiWWF6eKHWlc1Eo5pqcm4ECbTewoFGU4Bk5xwTXsx98PdKChdwg/u7QYd5YXQnvakq7RblaWEQfabLjtArGTUDRhgVPMO9Y3hNcq22DUqfDiuiVYXpImdqQJm5VlxBv72sWOQVGGBU4x7/Wv2jEn24Rr52Uh16wXO05IZmYacLTPCbvHB6M2OmfOUORxDJxi2qDbh0G3DxeVpEIhl4kdJ2QFKQnQqxU41DYodhSKIixwimntVhcSNEok6aR91qqQyzAvx4SD7byQSaewwCmmtVvdyE3SQSaT7tn3CWV5Sahs5rZrdAoLnGJau9WF3GSd2DHCYkVpGj4/OgCXl9uu0XEscIpZQUFAu9WNPIleuPymZQXJ0KoU2FnfL3YUihIscIpZXTYPhv1B5CTFxhm4UiHH5bPSsaOmW+woFCVY4BSz6nsdSE5QIyFKNmwIh6tmZ+CT2l74uWsPgfPAKYY19A4h1xwbZ98nlM9Ig9sbQEWzFdkm7ajf4w0E0dLvHHHcqFfBrFdPdUSKIBY4xayGniGUpCeKHSOs9GolLi5JxY6abvzggoJRv8c57EdFl2PE8eVFKSzwGMMhFIpJ/mAQTf1Oyd55eS5Xzc7EjuoeNPc7EQgKYschEbHAKSZ1Wt0ICgKyY+QC5umunJ0BvVqBdf9Rgf/zTjUO8eaeuMUCp5hU2WLF4nwz1MrYe4ubE9T48P4VePdny7Fomhn7Wq1iRyKRcAycopLV5YXd5RtxfDwX4jy+AA622/DgyplTFS8qGLQqzMs14W97muEPBKFUxN4/VnRuLHCKSnaXD7uPDow4Pp4LcftarUjUKLFwWhJqu4emKmJUmJashyAArVYXClNj64ItjY3/ZFNMEQTh6+3SUiCPgfVPxqJSyFGQmoCjvbH9DxWNjgVOMaVpwAmL04vF+Waxo0RMcVoiGlngcSnkIZRDhw5h48aN8Hq9WLFiBdavXx/OXEQTcrDNhqrOQRztG8L8HBMSY+juy7EUpSdie3U33N4AdGrpbBNHkxfSu9zr9eKZZ57BH//4R+h0sTdNi6Sle9CDVyvbcN70ZKxelIeSjPgaC84yaaFVKdDUP4TZ2Sax41AEhTSEsn//fuh0Otxzzz24/fbbUV9fH+5cRON2qMOGJJ0K15flYHa2Eao4m40hl8lQlJ6Ixj4Oo8SbkM7A+/r6UFdXh7///e/o6urCo48+ipdffjnc2YjG5XDHIApSE8SOIaritETsbuQys/EmpFMVo9GIRYsWQa/Xo6ioCA7HyHUXiCLlUPsg8lNi75b5ichP0aNvaBhub0DsKBRBIRX4ggUL0NTUhEAggN7eXmi1o6+KRjTVrE4vmgdcKEiJ7zPwNIMGaqUc7TaX2FEogkIaQjGZTLj55puxdu1aBAIBPPDAA+HORTQulS1WGLRKpBk0YkcRlVwmQ06SDu1Wt9hRKIJCnmt100034aabbgpnFqIJq2y2YF6OKS5u2hlLnlmHdgvPwONJfF2up5izt9mC+TmcOgcAuWY92qxuCAKXmI0XLHCSLLc3gMPtg5iXywIHgFyzDkPDfvQPecWOQhHCAifJOtBmg0IuQ2mGQewoUcGkU8GgVaKhl7PC4gULnCSrotmCsrykuLtx52xkMhlyzXrU9/CGnnjBdz5JVkWzBcumJ4sdI6rkmXVo4MJWcYMFTpLkDwSxr8WKpQUs8NPlmvU42juEIC9kxgUWOEnSkS4H3L4AFk5LEjtKVMlJ0sHjD6LH7hE7CkUAC5wkqaLZgtnZRhi0KrGjRBWdWoH5uSZ8UtsrdhSKABY4SVJFs4XDJ2fxo4sLUdftQE2nXewoNMVY4CQ5giCgopnj32eTY9bhktJ0/ONgBzw+Lm4Vy1jgJDnNAy70Dw1jSUH8bJs2UeUzUqFVKfDxkR6xo9AUip99pyhmVDRbUJCiR7qBq2CejVIux9VzMvFqRRsun5UBrWr0rdasLi/sLt+I40a9Cma9eqpj0iSxwElyKpo4/j0epZkGJGgU+KrFiuXFqaN+j93lw+6jAyOOLy9KYYFLAIdQSFIae4ewrbobF5WMXkh0ilwmw/mFKfji2ADnhccoFjhJRq/dg5+/fhDXl2XjWwuyxY4jCYvzzbB7fGjk3ZkxiUMoJKqzjcG6vjF7wuHx4S+7m7C8KAUbvjUXMq7/PS56tRILcpOw5+gAfnhhgdhxKMxY4CSqs43Bzs46c4XBPccGYNAo8YurSyGXs7wn4vzCFPzxn40YcA4jP843f441HEKhqOcLBLG3yYLlxalQyvmWnagskxYGrRIH2mxiR6Ew428DRb1D7TbIZTLM4847IZHJZChMS8S+VhZ4rGGBU1QTBAGfHx3AsunJUHLd75AVpiZgPws85vA3gqJa84ALvfZhnMd1vyelMC0RHTY3uga5a30sYYFTVKtotmBODlcdnKzkBDUyjRrsGeWCMUkXC5yiVlAQUN/jwJxsjn2HQ1meGZ+zwGMKC5yiVpfNA7c3gOK0RLGjxIRF05J4Bh5jWOAUtRp6HchL1kOnHn0hJpqYhdOS0GFzo83iEjsKhcmkCryyshKlpaWwWCzhykN0Un3PEErSefYdLukGLQpS9NjZ0Cd2FAqTSRX4X//6V8ydOzdcWYhOcnn9aLU4UZJhGPubady+vTAXf9nVBH8gKHYUCoOQC/yTTz7B4sWLodfrw5mHCABwuMMOjVKBXLNO7Cgx5b9fVACL04u3DnSKHYXCIKQCDwaDeOWVV/C9730v3HmIAAD7W60oTk+EnItWhZVBq8JdK4qw8aN6eP08C5e6kAr8nXfewWWXXQaNRhPuPEQQBAH722wc/54it12QD48viNcq28SOQpMUUoHX19dj+/btuP3221FXV4f77rsv3Lkojh1st2HQ7UNpJse/p4JercTPLi3Cpo8b4Bz2ix2HJiGk5WR/8YtfnPzz2rVr8fvf/z5sgSi+ub0BvHe4G7csm8a7L6fQrefn46UvW/Hi581YOI2bQ0vVpOeBb968GcnJXKeCwmN7TTeMWiVWzcsSO0pMUynk2HD9XLy5vwPdgx6x41CIeCMPRY1WiwuVzRbcUJYDBTdtmHIXFKXgstI0vH2wAwL3zJQkFjhFhUBQwNsHOrC0IBl5yZyaGik/vqQIXTYPGrhnpiSxwCkq7DnaD4fHj6tmZ4odJa6kJGgwIyMRR7rsYkehELDASXQ2lxcfHenFqnlZXPdEBKWZRtR1OziMIkHc1JgiZrQd6F2+AD6o6sa0ZD3m53LZWDHMyEjEm/va0WMfRqZJCwDwBoJo6XeO+F6jXgWzXh3piHQWLHCKmNF2oM8za1HdOYj1FxdCxrsuRWHQqpBr1qGu236ywJ3DflR0OUZ87/KiFBZ4FOEQColqb7MVBq2KFy5FVpppQG33yMKm6MYCJ1F93tiPudlGrnkispmZRrRaXHDxzkxJYYGTaNzeAA62D2JuDse+xZZl0sKgVaK+l2fhUsICJ9Ec6bLDpOPwSTSQyWQozTSgqoPTCaWEBU6iqeocxAWFKRw+iRLnTU/BkS47+oeGxY5C48QCJ1G4vQE09A7hwuIUsaPQ17KTdChOT8Rn3HJNMljgJIqdDX0w69WYySVjo0r5jDTsa7XB4vSKHYXGgQVOEdfvGMauhn78t/lZHD6JMoWpCcgyafGPg9xyTQpY4BRRgiDgnUOdKM00cMPiKCSTybBiRhq2VXejx85lZqMdC5wiqrrTjqZ+J9f7jmKzsoxYXpSKF3Y1scSjHAucIuaLYwN4rbINV83OgDmBt2NHK7lMhp9eWoSZGQa8sKsJ/Q7OSolWLHCKiH8c7MTDb1XhqtkZuKgkTew4NAa5TIZvL8pBTpIW/6rnrJRoxQKnKdc96MH9rx7AfVeUsLwlRC6T4fzpKajuHITXHxQ7Do2CBU5T7pW9rZiZZeC4twSVZBigkMtQ2807NKMRC5ymlD8QxNaKVnz/vHwuFytBCrkM83NNONBmEzsKjYIFTlPqoyO9cA0H8K2ybLGjUIjK8syo73FgiCsVRh1u6EAhG22HHeDMXVte+rIFNy7KgV6tBMDZDFKUZ9bBrFfjcMcgrp6dIXYcOg0LnEI22g47wKldW5r7ndjV2I9Hr5stQjoKF5lMhrK8JBxotYodhb6BBU5T5sXdTVhWkIwZvONS8srykvBxbS+aBrhPZjQJucCPHj2KRx55BHK5HHK5HL/5zW+Ql5cXzmwkYX2OYWytaMO/3bZE7CgUBimJGuSZddhW1Y35uUlnfI37ZIon5IuYZrMZzz//PF566SWsX78ezz33XDhzkcT9++4mlGQkorwkVewoFCZl08z4tL4PgiCIHYW+FnKBJycnw2g0AgCUSiUUCkXYQpG0OTw+bN7Tgp9eUsypgzFkXo4JvY5htFvdYkehr016GqHb7camTZuwbt26cOShGPD3/R3IMGpw9ZxMsaNQGCVqlFiYl4T9nBMeNSZV4H6/H/fddx/uuOMOFBUVhSsTSVhFswV/3dOC+68shVzOs+9Yc8mMNBxqtyEQ5DBKNAi5wAVBwMMPP4zy8nJcccUV4cxEEhQUBGyr6sY7Bzvx8LWzsGo+b5uPRUunmyEIwO7GfrGjECZR4J999hm2bduGDz74AGvXrsXjjz8ezlwkMXubLNjbPIDbL5qOy2amix2HpohGqcB3luRiR0036nscYseJeyFPIywvL8fBgwfDmYUkatDtw/bqblw3Pxv5KQlix6EpVpppxJWzM7G1ohU/WVEsdpy4xrVQaFIEQcDbBzqQZ9Zj0bSksR9AMaG8JBUl6Qa8vLeVS82KiAVOk3K4YxBH+4Zww8IcThmMIzKZDN9emINhfwB/2dUkdpy4xQKnkA37A9hW1Y3LZmYgmVukxR2tSoGbF+fh9X3t+PwoL2qKgQVO42J1edHS7zzj49XKdggALixKETseiaQgNQHfW5qH//naQTi53GzEcTErGpdvrjzo9gbwakUbrp6TCZWC5wHx7AcXFmBXYz9e+KwJ915RInacuMLfPArJv+p7kZKgxkJeuIx7KoUcv7h6Jv5t51H0cQf7iGKB04QdbLfh88YBrLswH3JeuCQA187LRHGGAZs+bjh5bLRht5Z+J6wur4hJYwuHUGhCdjX2Y3tVN64vy8aiaWbUdPFmDjo+K+Wha2bi1he+xLoLC1Ccnjjmhh80eTwDp3EJCgI+ONyFD2u6cev507CkIFnsSBRlzitMwcq5mfjR5kpYnDzLjgQWOI3J6w/iN+8fwVetVtx+USFmZhrFjkRR6qnVC5CcoMYP/6MCLi9npUw1Fjidkz8QxI+3fIXDHYP4UXkRpiXrxY5EUUynVuCF25Zi2BfAw29VYdgXEDtSTGOB0zk9/v4RHO4YxDNrFiLNoBE7DkmASa/C325fhkG3Dy/sasIQ54dPGRZ4nBrPDIGXvmzBy1+24oV1S5Bu1IqYlqQm3aDFpjULoVbK8dynRzHo9okdKSaxwOPUiRkC3/ywu3wQBAF/29OMX/2jGk9/Z8GITWyJxiNRo8QPLixASoIaW75ogS/ARa/CjQVOZ3B4fPjxln34f9vq8PvvluG6+dliRyIJUynkWLN0Gob9Aby5r50bIocZC5wAAP5gELsb+3HrX75Eh82N9+65iOVNYaFTK7D2/ALU9TjwcW0vSzyMeCMPocfuwZYvWuANBHFXeSF+tKIISq5vQmGUZtDg1vPyseWLFjy1ow73XzkDSvmp95hRr+LNPSFggce52m47Xq1ow5xsE761IBuXlqaxvGlKFKUl4s7yQrz8ZSvu2rIPN5TlwPz1MsS8OzM0/E2NUy6vHx9UdWHLFy24bGY6blqUA7WSbweaWlkmHZ68aR4EAfjdR/X44HAXPJwrHjKegccBq8sLu+v4NC6Hx4edDf34y+4myCDDD5dPR1FaosgJKZ6kJmrww+UFaOgdwvuHu1DdZUeeWYf8VO6nOlEs8Dhgd/nwty9a8Gl9H471DSFBo8QNZdkoyTCcMQ5JFCkymQwzMgyYnpqA9w934acv78PDq2Zj7fn5kMu5wuV4scBjXE2nHb/6RzW+arFiSYEZV84qQo5Zh7nZRq4kSKJTKeS4viwHV8/JwNM76vBBVReevGk+8lN4Nj4eLPAY1WZx4Xcf1uPtAx24tDQd/+OKEqQk8lZ4ik6Xz8zAdfOz8chbVbjq9zuxan4WblyYi5lZBjg9I2/F56yV41jgMcLq9OLj2h7sbhzAgVYrmgZcWJJvxvPfX4zcZD32t9rEjkh0TulGLZ5fuxifNfTjP79qxx1/q0CiRolZmUaUTUtClkl38ns5a+U4FrhEBYIC6rod+OhIDz6s6UFV5yCMWhXyzDrMy03C9WU5SDdq0Tfk5SJUJBkymQzlM9JQPiMNDo8PL3/Rite+asOuT/qRZdJiUb4ZMzIMCPJmIACTKPCtW7firbfeglKpxOOPP478/Pxw5iIAwaCAY/1DaOwZQo/dg06bB+02F1otbjT2DsHjD2DRNDNWzc/CUzfPh0Yhx55jFrFjE4WFQavCyrmZMOhUsLm82N9mw56jA3j3UBee+/QoZqQnoiTDgBkZiZiRbsCcXBNS42yYMKQCt9lseOONN7B161bU1NTg6aefxqZNm8KdLaZ5fAHYXD7Y3F70OYbRaXOjw+ZBp8196mPQA68/CLns+Js5OUGNlAQ1Li5Oxf++ZibmZJtg0qtO/syWfqeIfyOiqZOkV+PS0nRcWpqOoWE/VHJgzzELjnTZ8XFtLyxOLzRKOYrTE5GTpEOuWY9csw45Zh1yknTIMmmRnKCGLMb2cA2pwA8ePIjzzjsPCoUC8+bNQ3Nzc5hjnckfCKJvaBiCAAgABEHAif+DOn5MOPNrXx/HGcdP+77T/vzNnxEUBAQFAf6AgIAgIBAU4A8KCH7jc+C0D3/w6+8NBOEPCvAGghjy+DE07MeQxw/HsB92tw+Dbh9sLh+sLi+G/adWZkvUKJFp0iItUY10gxZzs424bGY6MgwaGHUqtFrcUJw2tWp5UQrnzFLcStQoMTvLAI3qVH25vQEYtAp0D3rQYx9G84ATXxwbQI/dA4vTCwGARilHlkmLTJMWJp0KCRolDBolErVKJGpUX39WQKNUQCmXQaWQQ6mQQSmXQ6WQQak4/lmlkEMhl0GG40M+xz8DMshw4t8HmezMrxm1x58v3EL6iXa7HUbjqW21pnpxmmf/dRRPf1g/pc9xOqVcBoVcdvLz8Q855DJABpx2TAa5TAal4vhnuez4MbVCBr1GiUSNEjqVAhkGDYrTEmDQqpBp0iInSYckvRpmvQpGnQpalQIt/c4RG8A6hgPIS9ZDIfdE7O9OJEU6tQIzMgzwB4FUgxZzckwnv7aswAyNSoFOmxvddg+6Bj1weHzoH/Kie9ADZ18ALq8fLm8ATq8fPv/xEzF/4MTJ2qn/DrXpFuQl4e2fLg/PX/Y0IRW40WhEff2pQpVP4Oyf6jsAAASaSURBVGaQ0tLSUJ4SYoxs+b/+IKL4JgcwmTkvtQBKp2CUOaQCX7BgAf70pz8hEAigtrZ23Bcw6+rqQnk6IiIaRUgFnpSUhBtuuAG33nrryVkoREQUWTKBq6sTEUkSVzIiIpIoFjgRkUSxwImIJIoFTkQkUZJczOpc67A8+OCDqK+vR0JCAqZPn44NGzYAAMrKyjBv3jwAwG233YYrr7wy4tmcTid+/etfo7OzE8FgEJs3bx7zMWJni4bX7b777kN/fz8A4PDhw9i6dStmzpwZ0fV4QskXDa/dzp07sXHjRqjVasyePRu//OUvx3yM2Nmi4XXbsWMH/vznP0OpVOKuu+7CihUrxnyMKASJsVqtwurVqwW/3y8cOnRIuPvuu8/4+gMPPCAcOnRoxONWrVolerYnnnhCqKiomNBjxMwmCNHxup0wMDAgXHvttRN6jFj5BCE6Xrvvfve7QkdHhyAIgnDnnXcKR44ciZr33GjZBEH8183v9wurVq0S3G634HK5hBtvvFHw+/0Rfc+Nl+SGUMazDsuGDRuwdu1a7Nq16+Sxrq4ufP/738fPf/5zWCxTs2LfWNn27duH999/H2vXrsWWLVvG/fcRKxsQHa/bCdu2bcPKlSsn9Bix8gHR8drNmDEDdrsdgUAAw8PDSEpKipr33GjZAPFfN6vVirS0NGi1Wuh0OhgMBrS0tER8DajxkFyBj7UOywMPPIDXX38dGzduxJNPPgmH4/i2YR999BG2bNmCyy+/HE888YQo2aqrq3H55ZfjxRdfxI4dO9DY2BixdWVCyQZEx+t2wnvvvYfrrrtuQo8RKx8QHa/d1VdfjTvuuAMrV65EYWEhMjMzo+Y9N1o2QPzXLTk5Gb29vbBYLBgYGMCRI0dgt9sjvgbUeEiuwI1G48lSBkauw2I2mwEAKSkpmDt3Lpqams44fs0116C2tla0bBdeeCGUSiXOP/981NfXj/kYMbOdOA6I+7oBx8/KhoeHMX369HE/Rsx8QHS8do899hjefPNNbN++HVarFV999VXUvOdGywaI/7rJ5XI88sgjuPfee/GrX/0Ks2bNQlpaWkTfc+MlfoIJWrBgAfbu3YtAIIDq6uoRFxFOvMDDw8OoqalBTk4OXC4XAoEAAKCyshI5OTmiZFu8eDFqamoAAFVVVcjPzx/zMWJmi5bXDQDeffddrFq1akKPETNftLx2SqUSBoMBcrkcRqMRdrs9at5zo2WLltftggsuwObNm7FhwwZoNBrk5ORE9D03XpK8lf6VV17B22+/ffJKcEtLC4aGhnDttddi/fr1cDgc8Pv9WLNmDVavXo2qqio8+uij0Ov1UCgU2LBhAwoKCiKera2tDY8++ig8Hg+WLVuG+++/f9THTNUbY6LZouV1A4Abb7wRzz77LDIyMs76mKn8hZpovmh57d577z28+OKLUKlUyM7OxpNPPgmlUhkV77nRstXW1kbF6/bEE0+guroaGo0GDz30EAoLC0d9jNglLskCJyIiCQ6hEBHRcSxwIiKJYoETEUkUC5yISKJY4EREEsUCJyKSKBY4EZFEscCJiCTqvwC5YPHnFnjrjgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nsim = 100000\n",
    "\n",
    "boots = np.random.choice(x, (len(x), nsim), replace = True) # sample from the data\n",
    "boot_estimates = boots.mean(axis = 0) # compute mean of each sample, i.e proportion of heads\n",
    "\n",
    "sns.distplot(boot_estimates);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "name": "04-python-stat-35"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.66, 0.83])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.quantile(boot_estimates, (0.025, 0.975)) # Find 2.5 and 97.5-th percentiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our 95% bootstrap confidence interval is (0.66, 0.83). Our true value of 0.7 certainly falls in it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression analysis\n",
    "\n",
    "### Ordinary least squares (linear) regression\n",
    "\n",
    "The regression modeling frameworks in Python are mainly in `statsmodels`, though some of it can be found in `scikit-learn` which we will see tomorrow. We will use the diamonds dataset for demonstration purposes. We will attempt to model the diamond price against several of the other diamond characteristics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "name": "04-python-stat-36"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import statsmodels.api as sm\n",
    "import statsmodels.formula.api as smf # Use the formula interface to statsmodels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "name": "04-python-stat-361"
   },
   "outputs": [],
   "source": [
    "diamonds = sm.datasets.get_rdataset('diamonds','ggplot2').data\n",
    "mod1 = smf.ols('price ~ np.log(carat) + clarity + depth + cut * color', data = diamonds).fit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "name": "04-python-stat-37"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>price</td>      <th>  R-squared:         </th>  <td>   0.786</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th>  <td>   0.786</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th>  <td>   4598.</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Wed, 03 Jun 2020</td> <th>  Prob (F-statistic):</th>   <td>  0.00</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>05:58:07</td>     <th>  Log-Likelihood:    </th> <td>-4.8222e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td> 53940</td>      <th>  AIC:               </th>  <td>9.645e+05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td> 53896</td>      <th>  BIC:               </th>  <td>9.649e+05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>    43</td>      <th>                     </th>      <td> </td>     \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>      <td> </td>     \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "               <td></td>                  <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>                   <td> 2745.0643</td> <td>  415.804</td> <td>    6.602</td> <td> 0.000</td> <td> 1930.085</td> <td> 3560.043</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>clarity[T.IF]</th>               <td> 4916.7221</td> <td>   83.694</td> <td>   58.746</td> <td> 0.000</td> <td> 4752.681</td> <td> 5080.763</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>clarity[T.SI1]</th>              <td> 2686.1493</td> <td>   71.397</td> <td>   37.623</td> <td> 0.000</td> <td> 2546.210</td> <td> 2826.088</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>clarity[T.SI2]</th>              <td> 2060.8180</td> <td>   71.809</td> <td>   28.699</td> <td> 0.000</td> <td> 1920.072</td> <td> 2201.564</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>clarity[T.VS1]</th>              <td> 3710.1759</td> <td>   72.891</td> <td>   50.900</td> <td> 0.000</td> <td> 3567.309</td> <td> 3853.043</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>clarity[T.VS2]</th>              <td> 3438.3999</td> <td>   71.792</td> <td>   47.894</td> <td> 0.000</td> <td> 3297.687</td> <td> 3579.112</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>clarity[T.VVS1]</th>             <td> 4540.1420</td> <td>   77.314</td> <td>   58.724</td> <td> 0.000</td> <td> 4388.606</td> <td> 4691.678</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>clarity[T.VVS2]</th>             <td> 4343.0545</td> <td>   75.136</td> <td>   57.803</td> <td> 0.000</td> <td> 4195.788</td> <td> 4490.321</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Good]</th>                 <td>  708.5981</td> <td>  161.869</td> <td>    4.378</td> <td> 0.000</td> <td>  391.334</td> <td> 1025.862</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Ideal]</th>                <td> 1198.2067</td> <td>  149.690</td> <td>    8.005</td> <td> 0.000</td> <td>  904.812</td> <td> 1491.601</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Premium]</th>              <td> 1147.1417</td> <td>  152.896</td> <td>    7.503</td> <td> 0.000</td> <td>  847.464</td> <td> 1446.820</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Very Good]</th>            <td> 1011.3463</td> <td>  152.977</td> <td>    6.611</td> <td> 0.000</td> <td>  711.510</td> <td> 1311.183</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>color[T.E]</th>                  <td>  -59.4094</td> <td>  190.227</td> <td>   -0.312</td> <td> 0.755</td> <td> -432.256</td> <td>  313.437</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>color[T.F]</th>                  <td>  -86.0097</td> <td>  178.663</td> <td>   -0.481</td> <td> 0.630</td> <td> -436.191</td> <td>  264.172</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>color[T.G]</th>                  <td> -370.6455</td> <td>  178.642</td> <td>   -2.075</td> <td> 0.038</td> <td> -720.784</td> <td>  -20.507</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>color[T.H]</th>                  <td> -591.0922</td> <td>  179.786</td> <td>   -3.288</td> <td> 0.001</td> <td> -943.474</td> <td> -238.710</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>color[T.I]</th>                  <td>-1030.7417</td> <td>  201.485</td> <td>   -5.116</td> <td> 0.000</td> <td>-1425.655</td> <td> -635.829</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>color[T.J]</th>                  <td>-1210.6501</td> <td>  223.111</td> <td>   -5.426</td> <td> 0.000</td> <td>-1647.949</td> <td> -773.351</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Good]:color[T.E]</th>      <td>  -30.3553</td> <td>  212.126</td> <td>   -0.143</td> <td> 0.886</td> <td> -446.123</td> <td>  385.413</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Ideal]:color[T.E]</th>     <td> -211.3711</td> <td>  195.630</td> <td>   -1.080</td> <td> 0.280</td> <td> -594.807</td> <td>  172.065</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Premium]:color[T.E]</th>   <td>  -91.3261</td> <td>  199.440</td> <td>   -0.458</td> <td> 0.647</td> <td> -482.230</td> <td>  299.578</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Very Good]:color[T.E]</th> <td>  -45.2968</td> <td>  199.656</td> <td>   -0.227</td> <td> 0.821</td> <td> -436.625</td> <td>  346.031</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Good]:color[T.F]</th>      <td> -365.4060</td> <td>  202.035</td> <td>   -1.809</td> <td> 0.071</td> <td> -761.397</td> <td>   30.585</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Ideal]:color[T.F]</th>     <td> -198.0428</td> <td>  184.498</td> <td>   -1.073</td> <td> 0.283</td> <td> -559.661</td> <td>  163.575</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Premium]:color[T.F]</th>   <td> -322.8527</td> <td>  188.465</td> <td>   -1.713</td> <td> 0.087</td> <td> -692.246</td> <td>   46.540</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Very Good]:color[T.F]</th> <td> -186.0519</td> <td>  189.090</td> <td>   -0.984</td> <td> 0.325</td> <td> -556.670</td> <td>  184.566</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Good]:color[T.G]</th>      <td>  -93.0430</td> <td>  202.404</td> <td>   -0.460</td> <td> 0.646</td> <td> -489.757</td> <td>  303.671</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Ideal]:color[T.G]</th>     <td>  -65.8579</td> <td>  183.980</td> <td>   -0.358</td> <td> 0.720</td> <td> -426.461</td> <td>  294.745</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Premium]:color[T.G]</th>   <td>   35.4302</td> <td>  187.596</td> <td>    0.189</td> <td> 0.850</td> <td> -332.260</td> <td>  403.121</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Very Good]:color[T.G]</th> <td>  -81.2595</td> <td>  188.786</td> <td>   -0.430</td> <td> 0.667</td> <td> -451.282</td> <td>  288.764</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Good]:color[T.H]</th>      <td>  137.0235</td> <td>  205.696</td> <td>    0.666</td> <td> 0.505</td> <td> -266.142</td> <td>  540.189</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Ideal]:color[T.H]</th>     <td>  -83.4763</td> <td>  186.060</td> <td>   -0.449</td> <td> 0.654</td> <td> -448.155</td> <td>  281.202</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Premium]:color[T.H]</th>   <td>  -44.4372</td> <td>  189.378</td> <td>   -0.235</td> <td> 0.814</td> <td> -415.620</td> <td>  326.745</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Very Good]:color[T.H]</th> <td>  -43.2485</td> <td>  190.851</td> <td>   -0.227</td> <td> 0.821</td> <td> -417.318</td> <td>  330.821</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Good]:color[T.I]</th>      <td>  331.4048</td> <td>  228.614</td> <td>    1.450</td> <td> 0.147</td> <td> -116.681</td> <td>  779.490</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Ideal]:color[T.I]</th>     <td>  106.2368</td> <td>  208.391</td> <td>    0.510</td> <td> 0.610</td> <td> -302.210</td> <td>  514.684</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Premium]:color[T.I]</th>   <td>  357.1453</td> <td>  212.341</td> <td>    1.682</td> <td> 0.093</td> <td>  -59.045</td> <td>  773.335</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Very Good]:color[T.I]</th> <td>  149.1555</td> <td>  213.697</td> <td>    0.698</td> <td> 0.485</td> <td> -269.693</td> <td>  568.004</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Good]:color[T.J]</th>      <td> -406.8484</td> <td>  256.938</td> <td>   -1.583</td> <td> 0.113</td> <td> -910.448</td> <td>   96.752</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Ideal]:color[T.J]</th>     <td> -330.0602</td> <td>  234.063</td> <td>   -1.410</td> <td> 0.159</td> <td> -788.826</td> <td>  128.706</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Premium]:color[T.J]</th>   <td> -156.8065</td> <td>  236.860</td> <td>   -0.662</td> <td> 0.508</td> <td> -621.055</td> <td>  307.442</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cut[T.Very Good]:color[T.J]</th> <td> -381.5722</td> <td>  238.799</td> <td>   -1.598</td> <td> 0.110</td> <td> -849.620</td> <td>   86.475</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>np.log(carat)</th>               <td> 6630.7799</td> <td>   15.605</td> <td>  424.923</td> <td> 0.000</td> <td> 6600.195</td> <td> 6661.365</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>depth</th>                       <td>   -0.7353</td> <td>    5.961</td> <td>   -0.123</td> <td> 0.902</td> <td>  -12.418</td> <td>   10.948</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>13993.592</td> <th>  Durbin-Watson:     </th> <td>   0.134</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th>  <td> 0.000</td>   <th>  Jarque-Bera (JB):  </th> <td>34739.732</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>           <td> 1.432</td>   <th>  Prob(JB):          </th> <td>    0.00</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>       <td> 5.693</td>   <th>  Cond. No.          </th> <td>7.08e+03</td> \n",
       "</tr>\n",
       "</table><br/><br/>Warnings:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.<br/>[2] The condition number is large, 7.08e+03. This might indicate that there are<br/>strong multicollinearity or other numerical problems."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                  price   R-squared:                       0.786\n",
       "Model:                            OLS   Adj. R-squared:                  0.786\n",
       "Method:                 Least Squares   F-statistic:                     4598.\n",
       "Date:                Wed, 03 Jun 2020   Prob (F-statistic):               0.00\n",
       "Time:                        05:58:07   Log-Likelihood:            -4.8222e+05\n",
       "No. Observations:               53940   AIC:                         9.645e+05\n",
       "Df Residuals:                   53896   BIC:                         9.649e+05\n",
       "Df Model:                          43                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "===============================================================================================\n",
       "                                  coef    std err          t      P>|t|      [0.025      0.975]\n",
       "-----------------------------------------------------------------------------------------------\n",
       "Intercept                    2745.0643    415.804      6.602      0.000    1930.085    3560.043\n",
       "clarity[T.IF]                4916.7221     83.694     58.746      0.000    4752.681    5080.763\n",
       "clarity[T.SI1]               2686.1493     71.397     37.623      0.000    2546.210    2826.088\n",
       "clarity[T.SI2]               2060.8180     71.809     28.699      0.000    1920.072    2201.564\n",
       "clarity[T.VS1]               3710.1759     72.891     50.900      0.000    3567.309    3853.043\n",
       "clarity[T.VS2]               3438.3999     71.792     47.894      0.000    3297.687    3579.112\n",
       "clarity[T.VVS1]              4540.1420     77.314     58.724      0.000    4388.606    4691.678\n",
       "clarity[T.VVS2]              4343.0545     75.136     57.803      0.000    4195.788    4490.321\n",
       "cut[T.Good]                   708.5981    161.869      4.378      0.000     391.334    1025.862\n",
       "cut[T.Ideal]                 1198.2067    149.690      8.005      0.000     904.812    1491.601\n",
       "cut[T.Premium]               1147.1417    152.896      7.503      0.000     847.464    1446.820\n",
       "cut[T.Very Good]             1011.3463    152.977      6.611      0.000     711.510    1311.183\n",
       "color[T.E]                    -59.4094    190.227     -0.312      0.755    -432.256     313.437\n",
       "color[T.F]                    -86.0097    178.663     -0.481      0.630    -436.191     264.172\n",
       "color[T.G]                   -370.6455    178.642     -2.075      0.038    -720.784     -20.507\n",
       "color[T.H]                   -591.0922    179.786     -3.288      0.001    -943.474    -238.710\n",
       "color[T.I]                  -1030.7417    201.485     -5.116      0.000   -1425.655    -635.829\n",
       "color[T.J]                  -1210.6501    223.111     -5.426      0.000   -1647.949    -773.351\n",
       "cut[T.Good]:color[T.E]        -30.3553    212.126     -0.143      0.886    -446.123     385.413\n",
       "cut[T.Ideal]:color[T.E]      -211.3711    195.630     -1.080      0.280    -594.807     172.065\n",
       "cut[T.Premium]:color[T.E]     -91.3261    199.440     -0.458      0.647    -482.230     299.578\n",
       "cut[T.Very Good]:color[T.E]   -45.2968    199.656     -0.227      0.821    -436.625     346.031\n",
       "cut[T.Good]:color[T.F]       -365.4060    202.035     -1.809      0.071    -761.397      30.585\n",
       "cut[T.Ideal]:color[T.F]      -198.0428    184.498     -1.073      0.283    -559.661     163.575\n",
       "cut[T.Premium]:color[T.F]    -322.8527    188.465     -1.713      0.087    -692.246      46.540\n",
       "cut[T.Very Good]:color[T.F]  -186.0519    189.090     -0.984      0.325    -556.670     184.566\n",
       "cut[T.Good]:color[T.G]        -93.0430    202.404     -0.460      0.646    -489.757     303.671\n",
       "cut[T.Ideal]:color[T.G]       -65.8579    183.980     -0.358      0.720    -426.461     294.745\n",
       "cut[T.Premium]:color[T.G]      35.4302    187.596      0.189      0.850    -332.260     403.121\n",
       "cut[T.Very Good]:color[T.G]   -81.2595    188.786     -0.430      0.667    -451.282     288.764\n",
       "cut[T.Good]:color[T.H]        137.0235    205.696      0.666      0.505    -266.142     540.189\n",
       "cut[T.Ideal]:color[T.H]       -83.4763    186.060     -0.449      0.654    -448.155     281.202\n",
       "cut[T.Premium]:color[T.H]     -44.4372    189.378     -0.235      0.814    -415.620     326.745\n",
       "cut[T.Very Good]:color[T.H]   -43.2485    190.851     -0.227      0.821    -417.318     330.821\n",
       "cut[T.Good]:color[T.I]        331.4048    228.614      1.450      0.147    -116.681     779.490\n",
       "cut[T.Ideal]:color[T.I]       106.2368    208.391      0.510      0.610    -302.210     514.684\n",
       "cut[T.Premium]:color[T.I]     357.1453    212.341      1.682      0.093     -59.045     773.335\n",
       "cut[T.Very Good]:color[T.I]   149.1555    213.697      0.698      0.485    -269.693     568.004\n",
       "cut[T.Good]:color[T.J]       -406.8484    256.938     -1.583      0.113    -910.448      96.752\n",
       "cut[T.Ideal]:color[T.J]      -330.0602    234.063     -1.410      0.159    -788.826     128.706\n",
       "cut[T.Premium]:color[T.J]    -156.8065    236.860     -0.662      0.508    -621.055     307.442\n",
       "cut[T.Very Good]:color[T.J]  -381.5722    238.799     -1.598      0.110    -849.620      86.475\n",
       "np.log(carat)                6630.7799     15.605    424.923      0.000    6600.195    6661.365\n",
       "depth                          -0.7353      5.961     -0.123      0.902     -12.418      10.948\n",
       "==============================================================================\n",
       "Omnibus:                    13993.592   Durbin-Watson:                   0.134\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):            34739.732\n",
       "Skew:                           1.432   Prob(JB):                         0.00\n",
       "Kurtosis:                       5.693   Cond. No.                     7.08e+03\n",
       "==============================================================================\n",
       "\n",
       "Warnings:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "[2] The condition number is large, 7.08e+03. This might indicate that there are\n",
       "strong multicollinearity or other numerical problems.\n",
       "\"\"\""
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mod1.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the basic syntax for modeling in statsmodels using the *formula* interface. This formula interface mimics the way regression formula are written in R. We will use this formula interface here since it allows for a more concise expression of the regression formula, and handles several things, as we will see. \n",
    "\n",
    "> `statsmodels` provides a traditional input syntax as well, where you \n",
    "specify the dependent or _endogenous_ variable _y_ as a vector array, and the \n",
    "independent or _exogenous_ variables _X_ as a numerical matrix. The typical syntax would be `mod2 = sm.OLS(y, X).fit()`. The formula interface,\n",
    "which uses the Python package __patsy__, takes care of the conversions, as\n",
    "well as modifying the design matrix to accommodate interactions and \n",
    "transformations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go through and parse it. \n",
    "\n",
    "One thing you notice is that we've written a formula inside the model\n",
    "\n",
    "```\n",
    "mod1 = smf.glm('price ~ np.log(carat) + clarity + depth + cut * color', \n",
    "    data = diamonds).fit()\n",
    "```\n",
    "\n",
    "This formula will read as \n",
    "\"price depends on log(carat), clarity, depth, cut and color, and the interaction of cut and color\". Underneath a lot is going on.\n",
    "\n",
    "1. color, clarity, and cut are all categorical variables. They actually need to be expanded into dummy variables, so we will have one column for each category level, which is 1 when the diamond is of that category and 0 otherwise. We typically use the **treatment** contrast formulation, which deems one category (usually the first) to be the reference category, and so creates one less dummy variable than the number of category levels, corresponding to the reference level. \n",
    "1. An intercept term is added\n",
    "1. The variable `carat` is transformed using `np.log`, i.e. the natural logarithm available in the `numpy` package. Generally, any valid Python function can be used here, even ones you create. \n",
    "1. Interactions are computed. The syntax `cut * color` is a shortcut for `cut + color + cut:color`, where the `:` denotes interaction. \n",
    "1. The dummy variables are concatenated to the continuous variables\n",
    "1. The model is run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the full design matrix we can drop down and use **patsy** functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "import patsy\n",
    "f = mod1.model.formula\n",
    "y,X = patsy.dmatrices(f, data = diamonds, return_type = 'dataframe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_X_ is the full design matrix with all the transformations and dummy variables and interactions computed, as specified by the formula."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we wanted the Ideal cut of diamond to be the reference level for the `cut` variable. We could specify this within the formula quite simply as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod2 = smf.ols('price ~ np.log(carat) + clarity + depth + C(cut, Treatment(\"Ideal\")) * color', data = diamonds).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This syntax says that we consider `cut` to be a categorical variable, \n",
    "from which we will create dummy variables using _treatment_ contrasts, \n",
    "using Ideal as the reference level. \n",
    "\n",
    "### Logistic regression\n",
    "\n",
    "Logistic regression is the usual regression method used when you have\n",
    "binary outcomes, e.g., Yes/No, Negative/Positive, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic regression does exist as an individual method in **scikit-learn**, whic we will see in the Machine Learning module. However, it resides in its more traditional form within the _generalized linear model_ framework in __statsmodels__\n",
    "\n",
    "We will use a dataset based on deaths from the Titanic disaster in 1912. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1313 entries, 0 to 1312\n",
      "Data columns (total 6 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   Name      1313 non-null   object \n",
      " 1   PClass    1313 non-null   object \n",
      " 2   Age       756 non-null    float64\n",
      " 3   Sex       1313 non-null   object \n",
      " 4   Survived  1313 non-null   int64  \n",
      " 5   SexCode   1313 non-null   int64  \n",
      "dtypes: float64(1), int64(2), object(3)\n",
      "memory usage: 61.7+ KB\n"
     ]
    }
   ],
   "source": [
    "titanic = sm.datasets.get_rdataset('Titanic','Stat2Data').data\n",
    "titanic.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will model `Survived` on the age, sex and passenger class of passengers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Generalized Linear Model Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>Survived</td>     <th>  No. Observations:  </th>  <td>   756</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                  <td>GLM</td>       <th>  Df Residuals:      </th>  <td>   751</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model Family:</th>        <td>Binomial</td>     <th>  Df Model:          </th>  <td>     4</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Link Function:</th>         <td>logit</td>      <th>  Scale:             </th> <td>  1.0000</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                <td>IRLS</td>       <th>  Log-Likelihood:    </th> <td> -347.57</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Wed, 03 Jun 2020</td> <th>  Deviance:          </th> <td>  695.14</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>05:58:09</td>     <th>  Pearson chi2:      </th>  <td>  813.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Iterations:</th>          <td>5</td>        <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td></td>           <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>     <td>    1.8664</td> <td>    0.217</td> <td>    8.587</td> <td> 0.000</td> <td>    1.440</td> <td>    2.292</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Sex[T.male]</th>   <td>   -2.6314</td> <td>    0.202</td> <td>  -13.058</td> <td> 0.000</td> <td>   -3.026</td> <td>   -2.236</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>PClass[T.1st]</th> <td>    1.8933</td> <td>    0.208</td> <td>    9.119</td> <td> 0.000</td> <td>    1.486</td> <td>    2.300</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>PClass[T.2nd]</th> <td>    0.6013</td> <td>    0.148</td> <td>    4.052</td> <td> 0.000</td> <td>    0.310</td> <td>    0.892</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>PClass[T.3rd]</th> <td>   -0.6282</td> <td>    0.132</td> <td>   -4.754</td> <td> 0.000</td> <td>   -0.887</td> <td>   -0.369</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Age</th>           <td>   -0.0392</td> <td>    0.008</td> <td>   -5.144</td> <td> 0.000</td> <td>   -0.054</td> <td>   -0.024</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                 Generalized Linear Model Regression Results                  \n",
       "==============================================================================\n",
       "Dep. Variable:               Survived   No. Observations:                  756\n",
       "Model:                            GLM   Df Residuals:                      751\n",
       "Model Family:                Binomial   Df Model:                            4\n",
       "Link Function:                  logit   Scale:                          1.0000\n",
       "Method:                          IRLS   Log-Likelihood:                -347.57\n",
       "Date:                Wed, 03 Jun 2020   Deviance:                       695.14\n",
       "Time:                        05:58:09   Pearson chi2:                     813.\n",
       "No. Iterations:                     5                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "=================================================================================\n",
       "                    coef    std err          z      P>|z|      [0.025      0.975]\n",
       "---------------------------------------------------------------------------------\n",
       "Intercept         1.8664      0.217      8.587      0.000       1.440       2.292\n",
       "Sex[T.male]      -2.6314      0.202    -13.058      0.000      -3.026      -2.236\n",
       "PClass[T.1st]     1.8933      0.208      9.119      0.000       1.486       2.300\n",
       "PClass[T.2nd]     0.6013      0.148      4.052      0.000       0.310       0.892\n",
       "PClass[T.3rd]    -0.6282      0.132     -4.754      0.000      -0.887      -0.369\n",
       "Age              -0.0392      0.008     -5.144      0.000      -0.054      -0.024\n",
       "=================================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mod_logistic = smf.glm('Survived ~ Age + Sex + PClass', data=titanic,\n",
    "  family = sm.families.Binomial()).fit()\n",
    "mod_logistic.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `family = sm.families.Binomial()` tells us that we're fitting a logistic\n",
    "regression, since we are stating that the outcomes are from a Binomial distribution. (See the [API documentation](https://www.statsmodels.org/stable/glm.html#families) for a list of available distributions for GLMs). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coefficients in a logistic regression are the _log-odds ratios_. To get the odds ratios, we would need to exponentiate them. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sex[T.male]      0.071981\n",
       "PClass[T.1st]    6.640989\n",
       "PClass[T.2nd]    1.824486\n",
       "PClass[T.3rd]    0.533574\n",
       "Age              0.961581\n",
       "dtype: float64"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(mod_logistic.params.drop('Intercept'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The intercept term in a logistic regression is **not** a log-odds ratio, so we omit it by using the `drop` function. \n",
    "\n",
    "### Survival analysis\n",
    "\n",
    "Survival analysis or reliability analysis deals typically with data on \n",
    "time to an event, where this time can be _censored_ at the end of observation. Examples include time to death for cancer patients, time to failure of a car transmission, etc. Censoring would mean that the subject is still alive/working when we last observed. \n",
    "\n",
    "A common regression method for survival data is Cox proportional hazards regression. As an example, we will use a data set from a VA lung cancer study. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>        <td>PH Reg</td>  <td>Sample size:</td> <td>137</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent variable:</td>  <td>time</td>   <td>Num. events:</td> <td>128</td>\n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Ties:</td>        <td>Breslow</td>       <td></td>        <td></td>  \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "            <td></td>            <th>log HR</th>  <th>log HR SE</th>   <th>HR</th>      <th>t</th>     <th>P>|t|</th> <th>[0.025</th> <th>0.975]</th>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(trt)[T.2]</th>           <td>0.1734</td>   <td>0.2016</td>   <td>1.1893</td> <td>0.8600</td>  <td>0.3898</td> <td>0.8011</td> <td>1.7655</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>celltype[T.large]</th>     <td>-0.8817</td>  <td>0.2962</td>   <td>0.4141</td> <td>-2.9761</td> <td>0.0029</td> <td>0.2317</td> <td>0.7400</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>celltype[T.smallcell]</th> <td>-0.0956</td>  <td>0.2649</td>   <td>0.9088</td> <td>-0.3609</td> <td>0.7182</td> <td>0.5407</td> <td>1.5275</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>celltype[T.squamous]</th>  <td>-1.1738</td>  <td>0.2997</td>   <td>0.3092</td> <td>-3.9173</td> <td>0.0001</td> <td>0.1718</td> <td>0.5563</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(prior)[T.10]</th>        <td>0.0378</td>   <td>0.2064</td>   <td>1.0385</td> <td>0.1833</td>  <td>0.8546</td> <td>0.6930</td> <td>1.5563</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age</th>                   <td>0.0042</td>   <td>0.0096</td>   <td>1.0042</td> <td>0.4401</td>  <td>0.6598</td> <td>0.9855</td> <td>1.0233</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                              Results: PHReg\n",
       "===========================================================================\n",
       "Model:                         PH Reg            Sample size:           137\n",
       "Dependent variable:            time              Num. events:           128\n",
       "Ties:                          Breslow                                     \n",
       "---------------------------------------------------------------------------\n",
       "                       log HR log HR SE   HR      t    P>|t|  [0.025 0.975]\n",
       "---------------------------------------------------------------------------\n",
       "C(trt)[T.2]            0.1734    0.2016 1.1893  0.8600 0.3898 0.8011 1.7655\n",
       "celltype[T.large]     -0.8817    0.2962 0.4141 -2.9761 0.0029 0.2317 0.7400\n",
       "celltype[T.smallcell] -0.0956    0.2649 0.9088 -0.3609 0.7182 0.5407 1.5275\n",
       "celltype[T.squamous]  -1.1738    0.2997 0.3092 -3.9173 0.0001 0.1718 0.5563\n",
       "C(prior)[T.10]         0.0378    0.2064 1.0385  0.1833 0.8546 0.6930 1.5563\n",
       "age                    0.0042    0.0096 1.0042  0.4401 0.6598 0.9855 1.0233\n",
       "===========================================================================\n",
       "Confidence intervals are for the hazard ratios\n",
       "\"\"\""
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "veteran = sm.datasets.get_rdataset('veteran', 'survival').data\n",
    "\n",
    "mod_cph = smf.phreg('time ~ C(trt) + celltype + age + C(prior)',\n",
    "  data = veteran, status = veteran.status).fit()\n",
    "mod_cph.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For survival regression, we need to input the status of the subject \n",
    "at time of last follow-up, coded as 1 for failure/death, 0 for censored."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** Why did I use `C(trt)` instead of `trt` in the formula?\n",
    "\n",
    "We can do a few more basic things for this data. First, let's draw the \n",
    "survival curve, which plots the proportion of subjects still alive against time, using the Kaplan-Meier method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEICAYAAAC55kg0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAcuklEQVR4nO3de3BU9d3H8U82iYEgS+ARtPi0oYq5VCFWELyCENE0FEXUkoVQC2HGMhppYNAolSoVdWwDTVBa69goXlhroyhFtEygpPoMA4INldwcMJkWR4hCSCImkeU8f9Cs2SSb3VzO3s779Y/dPTlnv7/q7Gd/53c5UYZhGAIAWJIt2AUAAIKHEAAACyMEAMDCCAEAsDBCAAAsjBAAAAszLQTa2tqUlZWliRMn6t133+1yvK6uTtnZ2crKypLT6TSrDABAD2JMu3BMjIqKivTaa691e7ygoED5+flKSUmRw+FQRkaGEhISfF43OTl5oEsFgIhXXV3d7fumhYDNZtOoUaO8Hq+trdVll10mSZo0aZIOHDigKVOm+HVtb43pycLiPWpubtbrudN7fW44q6ysVGpqarDLCCjaHPms1l6pf23u6cezaSHgS8eFysOGDdPJkyf9PreysrLXn7ezur7P54azlpYW2mwBVmuz1dormdfmoIWAzfbtcERjY6NSUlL8PrdvaXi4H+eGL34xWYPV2my19krmtTlos4MSExN18OBBuVwu7d27V+PHjw9WKQBgWab2BHJzc1VRUaH4+Hj985//1DXXXKPm5mZlZmZq+fLlWrlypU6fPq3bbrvNr0FhAMDAMjUE1q9f7/VYYmKiXn75ZTM/vlsLi/eoeOGkgH8uAIQiyywWm5Y8UtK3A8QAAAuFAL/+AaAry4RARwuL9wS7BAAICZYKgSsvHCyJW0IA0M5SIbD6xu8EuwQACCmWCgEAgCdCAAAsjBAAAAsjBADAwggBALAwQgAALIwQAAALIwQAwMIsGwJsHQEAFgwBdhMFgG9ZLgTYTRQAvmW5EAAAfIsQAAALIwQAwMIIAQCwMEIAACzM0iHAWgEAVmfJEGCtAACcZckQYK0AAJxlyRAAAJxFCACAhRECAGBhlg8BZggBsDLLhgAzhADAwiHADCEAsHAIAAAIAQCwNEIAACyMEBAzhABYl6VDgBlCAKzO1BBwOp3KyspSdna26urqPI6VlZVpzpw5ysrK0urVq80swytmCAGwOtNCoKGhQSUlJXrllVf0wAMPqKCgwOP4hg0b9PTTT8vpdOrIkSOqqqoyqxQAgBemhUB5ebkmT56s6OhojRs3TrW1tR7Hk5KS1NjYKJfLpdbWViUkJJhVCgDAixizLtzY2Ci73e5+bRiGx/Gbb75Zixcv1uDBg3X99dfrggsu8PvalZWVfaqppaXF67l9vWao66nNkYo2Rz6rtVcyr82mhYDdbldNTY37tc3m2en49a9/rTfeeEPnnXeeli9frn379mnChAl+XTs1NbVPNVVWVnZz7uF+XTPUdd/myEabI5/V2iuZ12bTbgelpaVpz549crlcOnjwoBITEz2Ox8TEaOjQobLZbLLb7WpsbDSrFACAF6b1BBISEjR79mzNnz9fMTExWrNmjcrKytTc3KzMzEwtWbJECxYsUGxsrEaPHq3rr7/erFIAAF6YFgKS5HA45HA43K879gZmzpypmTNnmvnxAAAfLL1YrKN122t8/xEARBhC4L8KSz8JdgkAEHCWD4H2rSMAwIosHwJsHQHAyiwfAgBgZYQAAFiYqVNEQ9G67TVeB4HH5G/t8t7S9EuUNyPJ7LIAICgsFwJ5M5K6fKm3f/nXPsm6BQDWwu0gALAwQqADHjMJwGoIAfGYSQDWRQjIc60AvQEAVkII/NeY/4mXRG8AgLUQAv/19xXTgl0CAAQcIQAAFkYIAICFEQIAYGGEAABYGCEAABZGCHSDtQIArIIQ6ICVwwCshhDogKeMAbAaQsALbgkBsAK/QmDXrl166aWXJEn19fU6dOiQqUUFE7eEAFiJzxB47LHH9N577+nVV1+VJEVHRys/P9/0woKl4y2hMflb6REAiGg+Q2Dfvn16/PHHFRcXJ0kaMWKEWltbTS8smNp7AxI9AgCRzWcIxMbG6ptvvlFUVJQk6ejRo4qJieynUhYvnMSjJgFYgs9v8yVLlujuu+/WsWPHtGrVKu3evVurVq0KRG0AAJP5DIFp06YpLS1N5eXlMgxDv/jFLzRixIhA1AYAMJnPEMjJyVFGRoZuvPFGDR8+PBA1AQACxOeYwP3336/PP/9cP/vZz7Ro0SI5nU59+eWXgagtZDBDCECk8hkCycnJys3N1VtvvaVf/vKX2rt3r6ZMmRKI2oKONQMAIp1fi8WqqqpUWFiopUuXqr6+XitXrjS7rpDAA+gBRDqfYwI/+tGPNHr0aN1000168cUXLTcoPC15pHZW19MbABCRfIbApk2blJCQEIhaQlLxwkkak7812GUAgCm8hsDatWu1bNkyrV692r1QrKOCggKfF3c6ndq8ebNiYmK0Zs0aJSYmuo999dVXeuyxx/TZZ5/pzJkz7r2JAACB4zUEbrrpJklSVlZWny7c0NCgkpISOZ1OVVRUqKCgQEVFRe7jTz/9tG6//XZNnDixT9cHAPSf1xBISkpSW1ubLr/88j5duLy8XJMnT1Z0dLTGjRun2tpaj+P79+9Xa2urCgsLdfPNNys7O7tPnwMA6DuvIZCRkeFxG8gwDEVFRbn/WVpa2uOFGxsbZbfbPc7v6ODBg7rvvvv00EMPadGiRbrqqqs0duxYv4qurKz06+86a2lp6fO5knTn+h1afeN3+nx+MPS3zeGINkc+q7VXMq/NXkNgx44d/bqw3W5XTU2N+7XN5jkbdfjw4brmmmsUFRWlq666SjU1NX6HQGpqap9qqqys7NO505KbtLO6XnuPfN3nzw6WvrY5nNHmyGe19krmtdmv7UAPHDigQ4cOqa2tzf3e3LlzezwnLS1NGzZskMvlUlVVlcegsCRNmDBBFRUVuvTSS/Xxxx9r6tSpfSg/MJghBCBS+QyBp556SnV1dSovL9esWbNUWlqqyy+/3GcIJCQkaPbs2Zo/f757dlBZWZmam5uVmZmp5cuX6+GHH1ZLS4smTZqkSy+9dMAaBQDwj88Q+Mc//qG3335bt956qx544AEtWbJEeXl5fl3c4XDI4XC4X3fsDXz3u9/VCy+80PuKAQADxue2EYMGDVJUVJTi4uJ0/PhxDRkyRP/5z38CURsAwGQ+ewJTpkxRY2OjFi1apDlz5shms+mWW24JRG0AAJP5DIHc3FxJZ/cQSk9PV2trq4YOHWp6YQAA8/XqYcHnnHOOzjnnHLNqAQAEmF9bSeNbbCkNIJIQAn7iATMAIpHP20GGYWj37t06cuSIXC6X+31f6wQiTccFYwuL93g8cAYAwpXPELj33nvlcrn0gx/8oMvWD1bDA2YARBqfIVBXV6e//vWvgagl5LF9BIBI4/On/TXXXKP9+/cHopawwgAxgEjgsyfwzjvvaOPGjRo2bJhiY2Pd77///vumFhaquCUEIJL4DAGrftl70/GWUPs/pyWPZKAYQFjya3bQ1q1btW/fPknSxIkTlZmZ2e1zh62ivTfQjl4BgHDlMwRWr16thoYGzZo1S4Zh6J133tHevXv1yCOPBKC80NTxVz8DxQDCmc8Q2Ldvn95++2336/T0dDaQ68aY/K2aljxS4/83QXkzkoJdDgD4xefsoNjYWFVVVblfV1dXewwQW137SmLp7G2hwtJPtG57TQ9nAEDo8NkT+NWvfqUVK1a4HxRvs9m0Zs0a0wsLF+23hjreFios/YTeAICw4DMExo8fry1btqipqUmGYchutweiLgBAAHgNgZdfflnZ2dkqKCjodibQsmXLTC0s3HSeMQQA4cBrCIwePVqSdNFFF3U5ZuXpod6wpQSAcOQ1BKZPny5JOnHihBYtWuRx7E9/+pO5VYWBddtrVFj6idfjnQNhafoljBMACDk+Zwdt2bKly3tvvfWWKcWEk7wZSVqafonff8+sIQChyGtP4PXXX9df/vIX1dbWKisry/3+V199pYsvvjggxYW6vBlJXX7ds5UEgHDiNQQyMjJ09dVXa/369brvvvvc78fHx2v48OEBKS6cMUgMIBx4DYGhQ4cqPj5e1dXVuvDCCwNZU8RYWLyHFcQAQlqPYwLR0dFKTU3VoUOHAlVPRGlfQQwAocrnYrHq6mrdeuutuuiiixQfHy/DMBQVFSWn0xmI+sLO0vRLVFj6CesGAIQFnyGwfv36QNQRMdpv/eTNSHIPEq/bXsMtIQAhyecU0QsvvFDHjh3Ttm3btG3bNh09epQxAh86f+FzSwhAqPIZAk8//bSeeuopxcXF6ZxzzlFBQYGeeeaZQNQWUVgjACAU+bwd9N577+nNN99UTMzZP503b55uu+023XPPPaYXF058rSAuLP3E4zgriAGEAp8hYLPZdPz4cY0aNUrS2W0kbDafHQjL6W7hmOS5fUTtkzMDWRIA+OQzBJYtWyaHw6GxY8dKkg4dOqRVq1aZXhgAwHw+Q2Dq1Kl655139Omnn0qSvv/97ysuLs70wsIRm8oBCDc+Q+Crr77Sxo0btX//fkVFRemHP/yhfvrTn2rIkCGBqC+s9LSXkMTtIAChx6/bQZdddplWrFgh6exAcV5env74xz/6vLjT6dTmzZsVExOjNWvWKDEx0eO4y+XSzJkzdeeddyonJ6ePTQgtPfUGunveAL0BAMHkMwQ+++wzPfvss+7XSUlJmjNnjs8LNzQ0qKSkRE6nUxUVFSooKFBRUZHH35SUlOh73/teH8oOXZ17A51Dgd4AgFDic5rPDTfcoD//+c86ceKEGhoa9Nprryk9PV1tbW1qa2vzel55ebkmT56s6OhojRs3TrW1tR7HW1tbtWPHDt188839bkQo6/wrf2HxniBVAgBd+ewJbN169hbGH/7wB4/3S0pKFBUVpdLS0m7Pa2xs9HgovWEYHsdfeuklzZ07VydOnOh10ZWVlb0+R5JaWlr6fO5A2VldH9AaQqHNgUabI5/V2iuZ12afIbBjx44+Xdhut6um5ttVsh3XFjQ1NWnPnj1avHix3njjjV5fOzU1tU81VVZW9vnc/liaHu1xS+ip3U0Be+BMsNocTLQ58lmtvZJ5bfYZAi0tLXrxxRe1b98+SdKECRN01113adCgQT2el5aWpg0bNsjlcqmqqspjUPjw4cM6ceKEcnJydOzYMX3zzTdKSUnRtdde28/mhKa8GUkeO4uyuyiAUOEzBB588EGNHj1ay5cvl2EY2rp1qx588EGtW7eux/MSEhI0e/ZszZ8/3z07qKysTM3NzcrMzNTrr78uSXrjjTd04sSJiA2Adu2zgLqbIQQAweIzBA4fPuzxhZ+SkqJbb73Vr4s7HA45HA73685TRCX5NdMoEjANFEAo8jk76Nxzz9XOnTvdr//+97/r3HPPNbUoAEBg+AyBJ554Qps2bdL111+vqVOnatOmTXriiScCUVtEY6oogFDQ4+0gl8ul3//+936tDoZ/GBwGEEp8Pmi+vr5eLS0tgaon4nWeGsrDZgAEk8+B4fZZPtddd53i4+Pd7y9btszUwqxgYfEe7ayuZ9AYQND4DIFrr7024qdvBhq3hACEih5DYPv27Tp+/LjGjh2rqVOnBqqmiFe8cBLrBQCEBK9jAg8//LA2btyohoYGPfPMM/rd734XyLoi3rTkkcEuAQC8h8BHH32kF198UcuXL9fGjRu1a9euQNYV8ToOEDNdFECweL0dFBsb6970zdc+QfCtp4fN7Kyu59GTAILCawjU1NTouuuuc79uaGjweP3++++bW1mE6e7Rk5Ln08amJY8M2O6iACD1EAIHDx4MZB2QmC0EIOB8bhsB8/HISQDBQgiEmDH5WxkoBhAwhECI6DhllNtCAAKFEAiypemXSDo7ZZTbQgACjRAIMqaBAggmQiBEMS4AIBAIgRDTPjaws7qeIABgOkIgxBQvnOQRBABgJkIgBHVcNcyUUQBmIgRCFFNGAQQCIRCimDIKIBAIAQCwMEIgTDAuAMAMhECIY6YQADMRAiGOmUIAzEQIhAFmCgEwCyEQBvyZKbRue02AqgEQSQiBMNTdLSFvzy8GgJ4QAmGEQWIAA83rM4YReooXTnI/mL7zA+oBoC/oCYSZ7r7w6RkA6Ct6AmGm45RRybNHsLB4T5fjANATQiDMrNte43UQeGd1vUcoSNLooTH6v5WpgSgNQBgyNQScTqc2b96smJgYrVmzRomJie5jDzzwgOrq6uRyuTRv3jzddtttZpYSMfJmJHX7SMrOX/7tPms6bXZJAMKYaSHQ0NCgkpISOZ1OVVRUqKCgQEVFRe7jS5Ys0ZgxY9TW1qZbbrlFP/7xjxUbG2tWOZbQeS2Bt2AAgHamDQyXl5dr8uTJio6O1rhx41RbW+txfMyYMZKkmJizOWSzMUYdLCw0A6zLtJ5AY2Oj7Ha7+7VhGN3+3XPPPaeZM2cqOjra72tXVlb2qaaWlpY+nxsOvLWtc4/gygsHa/WN33G/Liw9rIz/dZlaWyBF+r/n7litzVZrr2Rem00LAbvdrpqab39hdvdL/+2331ZlZaXWrl3bq2unpvZtoLOysrLP54a+w13aNi25qdvpo3uPfN3pb7ueG84i+99z96zWZqu1VzKvzaaFQFpamjZs2CCXy6WqqiqPQWFJKisr05tvvqlnn32WW0EDYGn6JV3eK144qct/OJ0Xm7HQDLA200IgISFBs2fP1vz5892zg8rKytTc3KzMzEw9+OCDGjVqlHJyciRJa9eu1ciRfCH1VXczhrozLXmkR++AhWaAtZk6RdThcMjhcLhfd+wNfPDBB2Z+NLzo/HwCANbGYjEL6WmhWXeBsDT9Er97GADCEyFgId0tNOv85T8teSRbTwAWwoisxXUeGGaMALAWQsDi2n/1d1xtPCZ/K88zBiyCEIAbvQLAehgTgFt3M4c6ridgrACIPIQAul1o1t16AgaRgchDCKDbaaAdv9wXFu/p9tYQt4uA8EcIwKfufu13vF1EjwAIXwwMo086DiLTIwDCFyGAPileOMljWinTSYHwRAigX9p7BPQGgPDEmAD6pXjhpC7TSbvDuAEQmugJoN/8eSYBPQUgNNETQL/5+oVPTwEIXfQEYDp6CkDooicA0/W1p0DvADAfPQEEnbeeAr0DwHz0BBB0vlYkd4deAjAw6AkgJPkaR6CXAAwMegIIST39ymffImDg0BNA2GHfImDgEAIIO+xbBAwcQgBhi32LgP4jBBC2Oj/4BkDvEQIIa/QGgP5hdhDCmvddTA9LYj0B4As9AYS9ntYU0EMAekZPAGGv8y/9yspKpaam+rV7qT/oTSCS0RNAxPJn91J/0JtAJKMngIg1EL/eB6o34Q96HAgGegJADwaqN+EPehwIBnoCQA8C9cu8tz0Oeg0YKPQEgBDQ2x4HvQYMFHoCQAjoza96772GwwNYUTiwRnvN7vWZ2hNwOp3KyspSdna26urqPI7V1dUpOztbWVlZcjqdZpYBRJRAjlMg+Mzu9ZnWE2hoaFBJSYmcTqcqKipUUFCgoqIi9/GCggLl5+crJSVFDodDGRkZSkhIMKscIGJ096uwfW2EVVilvR17fVdeOFivm9Bm03oC5eXlmjx5sqKjozVu3DjV1tZ6HK+trdVll12mmJgYTZo0SQcOHDCrFAAISx17fXuPfG3KZ5gWAo2NjbLb7e7XhmF4HO/4etiwYTp58qRZpQBAWOr87AwzmHY7yG63q6amxv3aZvPMm46vGxsblZKS4ve1k5OT+18gAISJuP/+M/nNZQN+bdNCIC0tTRs2bJDL5VJVVZUSExM9jicmJurgwYNKSUnR3r17tXjxYr+uW11dbUa5AGBJUUbn+zQDaNOmTXrrrbcUExOjNWvWqK6uTs3NzcrMzFRdXZ1Wrlyp06dPa/bs2crKyjKrDACAF6aGAAAgtLFiGAAsjBAAAAsjBADAwggBALAwy4RAT/sYhbNDhw7J4XBo/vz5WrBggf7973/r66+/1tKlSzVv3jytWrVKZ86ckXR2FXdWVpbmzp2r0tLSIFfefx9++KGSk5N1/PhxS7T5wIEDWrRokbKzs/Xcc89Zos2rV6/WT37yE91xxx0qKyuLyDa3tbUpKytLEydO1LvvvitJvW5nYWGhHA6HFi9erOPHj/euAMMCTpw4Ydxxxx3G6dOnjQMHDhi5ubnBLmnAfPnll8bJkycNwzCMXbt2GQ899JDx0ksvGc8//7xhGIbx6KOPGjt37jQMwzDmzp1rfP7550Zzc7Mxe/Zs45tvvglW2QPi3nvvNebMmWN8+eWXEd/m1tZWY/HixcapU6fc70V6mz/99FNjwYIFhmEYxhdffGHMmTMnItvscrmMo0ePGkVFRca2bdsMw+jdv9vq6mrj7rvvNgzDMLZt22Y8+eSTvfp8S/QEfO1jFM5GjBjh3p4jJiZG0dHR+vDDDzV16lRJ0tSpU/Xhhx+qtbVVZ86c0fnnn68hQ4YoMTExrHtEO3bs0IQJExQfHy9JEd/mjz76SIMHD9Z9992nnJwc1dTURHybzzvvPMXHx+v06dNqamrS8OHDI7LNNptNo0aN8nivN+3s+Lc33HCD9u/f37vPH5hmhDZf+xhFgq+//lpFRUW66667PNrbvi9TQ0ODhg4d6v57u90etvs1nTlzRps2bZLD4XC/F+ltrq+vV3V1tQoLC/XQQw/pkUceifg2DxkyRBdccIEyMjI0b948/fznP4/4NrfrTTs7/u2gQYN06tSpXn2WJULAbrerqanJ/brzPkbh7vTp08rLy9PixYt18cUXe7S3sbFRw4YNU0JCgpqbm93nNDU1adiwYcEquV+2bNmi6dOnKy4uzv1epLfZbrfriiuuUHx8vC6++GI1NTVFfJs/+OADnTx5Un/729+0efNmPf744xHf5na9aWfHv21tbXX3jv0VWd+GXqSlpWnPnj1yuVw6ePBgl32MwplhGFq5cqWmTJmiG2+8UZJ05ZVXateuXZKksrIyTZw4UXFxcbLZbDp27JhOnTql2trasP3/oaamRu+9955ycnJUXV2tvLy8iG9zWlqaPv30U7lcLh07dkyDBg2K+DafOXNGw4YNk81m07nnnqtTp05FfJvb9aadHf92165duuKKK3r1WZbZNqLzPkbh/h9Ju7KyMuXm5mr8+PGSpJSUFC1btkz5+fmqr6/X2LFj9cgjj8hms6m8vFxPPPGEzpw5o7vvvlvp6elBrr7/FixYoMLCQg0ePDji21xSUqKSkhK5XC6tWLFCl156aUS32eVyKT8/X0eOHFFra6vuuusuzZgxIyLbnJubq4qKCsXHx+vaa6/V0qVLe9XOwsJC7d69W/Hx8frNb36jESNG+P3ZlgkBAEBXlrgdBADoHiEAABZGCACAhRECAGBhhAAAWJhpzxgGwt2cOXPkcrn0xRdfKCYmRgkJCTp06JB++9vfKiMjI9jlAQOCKaKAD+vXr9d5553nsU0FECm4HQT0wvr167Vp0yZJ0vTp01VYWKhZs2YpJydH+/btk8Ph0E033aR//etfkqTm5mYtX75ct99+u+688059/PHHwSwf6IIQAPrhoosu0pYtWzRo0CAVFxfrlVde0f33368XXnhBkrRhwwZlZmaqpKREa9eu1aOPPhrcgoFOGBMA+mHatGmSpKSkJA0fPlw2m03Jycl6/vnnJUm7d+/W+++/r6KiIkny2MgQCAWEANAPsbGxks7uTNv+v6OiouRyuSSd3eDvueee0/nnnx+0GoGecDsIMNHVV1+tV1991f26qqoqiNUAXRECgInuueceHT16VLNmzVJmZqa2bNkS7JIAD0wRBQALoycAABZGCACAhRECAGBhhAAAWBghAAAWRggAgIURAgBgYYQAAFjY/wO+PpOW43THOQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sf = sm.duration.SurvfuncRight(veteran.time, veteran.status)\n",
    "sf.plot();\n",
    "plt.grid(True);\n",
    "plt.xlabel('Time');\n",
    "plt.ylabel('Proportion alive');\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we now want to see if there is any difference between treatment groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEICAYAAAC55kg0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de1hUdf4H8DcDKKKO4Gq12v7AG5fE0CQpNQ1YL+FaaqkgsOalNTNvmDZaGaWolWig2ZqPa0nqeMHrmrmJJmu7eUHDRAQfFZ4VTElEQARlOL8/iImBmTkDzJnbeb+epwfOnDNnPl/smc/5fr/n+zlOgiAIICIiWVJYOwAiIrIeJgEiIhljEiAikjEmASIiGWMSICKSMSYBIiIZkywJPHjwABEREQgKCsK3337bYH9eXh6io6MREREBtVotVRhERGSEi2QndnFBUlIStm/frnd/QkICVCoV/Pz8EBkZieHDh8PDw0P0vL6+vuYOlYjI4WVnZ+t9XbIkoFAo8Mgjjxjcn5ubi4CAAABAv379cP78eQwaNMikcxtqjDEZHw3B7bIHSO76CTZN6tfo99urrKws+Pv7WzsMi2KbHZ/c2gs0r83GLp4lSwJi6i5UbteuHe7evWvye7Oyshr9eYH3TwHOwOTswia9315VVFTIqr0A2ywHcmsvIF2brZYEFIrfpyNKSkrg5+dn8nubewUgpysIXjHJg9zaLLf2AtK12WpJwMvLC5mZmfDz88Pp06cxdepUa4VCRACqq6tRUFCAhw8fWjsUUYIg4Nq1a9YOw6JMbbOrqys6deqkc6FtjKRJYObMmbh48SLc3d3x008/oX///igrK0N4eDjmzZuHd955B1VVVRg9erRJk8JEJJ2CggIolUoolUprhyLq/v37aNWqlbXDsChT21xSUoKCggI8/vjjJp1X0iSwZs0ag/u8vLzw9ddfS/nxRNQIDx8+tIsEQMYplUrcvn3b5ONluVhs0qZT1g6BiMgmyDIJHMsutHYIRLL34MEDxMTEICYmBkFBQYiIiEBMTAx++OGHRp9ry5YtEkRYY/fu3bh//36D17///nuEh4cjKChIss+2BFkmASKyvhYtWiA5ORnJycnw9/fHunXrkJycjAEDBkCj0TTqXNu2bZMoSmDPnj16k0Dv3r2xe/duPPbYY5J9tiVY7e4gIqK68vPz8cYbb6BTp07w9PTE2LFjsXz5clRXV6NLly744IMPcOXKFcTHx6OiogJKpRJJSUlITU3F9evXERMTg9DQUJSVlSE3NxelpaW4ffs2Xn/9dajVahQWFmLNmjXw8vLCf//7X6xduxaCIKB///548803sWbNGty8eRO//vorfvnlFyQmJuLmzZvIysrCG2+8gaeeegoLFizQxusoN7PILgnkuk3AUU1vACOsHQqRTZq06ZTZhkxDfDs2aoV+fn4+vvzyS7i5uSEmJgarVq1Cx44dsWLFCpw4cQJPP/00Nm7ciMrKSqxbtw5HjhzBiBEjtL0IoOaGlEcffRQJCQn47LPP8O2332Ljxo3Yu3cv9u3bh5kzZ2LlypVITk6Gu7s7Zs2ahatXrwKo+WJfunSp9thZs2bB398fiYmJaN++vVn+JrZGPkmgx1Dg8r8AAKHOP1k5GCLSx9fXF25ubgCAy5cvIzY2FgBw79499OjRAwUFBfjoo49QWlqKoqIieHp66j1Pr169AAB//OMf0bJlS+3vP//8M+7cuYPr169j2rRpAGpuqbxx4wYAoGfPnjrHyoF8kkDUzpoVd9ufsXYkRDbNmrW1nJ2dtb/7+Pjg008/1V6BP3z4EB999BHGjRuH/v37Y82aNdryM05OTjrnqbtd93dBEODp6Qlvb29s2LABbm5uqK6uhiAIOHv2bINjgZrFV42do7An8kkCRGRXFi1ahHnz5qGqqgoKhQLvvvsuQkJCEB8fjx07dkCpVOIPf/gDAOD555/H3/72N4SFhYme18nJCXPmzMFrr70GoOZL/pNPPjF4fFhYGObPn4/g4GBMnz5d+3pGRgZWr16N/Px8vPrqq5g6dSoGDhzYzFZbnpNQt5KbHfD19W1SFVEAuj2BONML1tkz1liRB3O0+dq1a+jSpYuZIpIWVwwbV//f0tj3Jm8RJSKSMdkmAa4aJiKScRLgqmEiIhknASJqntXf5Vg7BDIDJgEiapLE1MvWDoHMgEmAiEjGmASIyCpYRfR3hYWFSEhIMHrM9u3bm/05+sg2CeS6TQC2jLV2GESyxSqiv+vYsSPmzZtn9JgdO3Y0+3P0kd+K4To1hLQ/icjqHK2K6MmTJ7Fx40a4urri+vXrmDVrFsLCwnDlyhW8//77EAQBXbp0wZIlS5Cfn48lS5Zg/fr1iImJQZ8+fbS1i7744gts3boV+fn5iImJwbhx41BeXo5du3bBzc0Nw4YNQ3R0dJP/7vJLAlE74a06WNMTICJRq7/LMTgJ7K062OC12WE9MHeIT5M+y9GqiN66dQu7du1CeXk5JkyYgLCwMKxcuRIqlQoBAQGIi4tDamoq/Pz8dN4XFBSE2NhYLFy4EOnp6Zg4cSK2b9+ubWNMTAzWr1+P9u3bo7q6ukl/61rySwJE1Chzh/jo/VL3Vh1E7grzlmR3tCqiAQEBcHFxgVKphLu7O8rLy3H9+nUEBAQAAPr06YNr1641SAK1+zt16oS7dxuWuFGpVFi5ciWqqqowbty4Zs1LMAkQkc1wtCqimZmZ0Gg0KC8vR3l5Odzd3dG5c2dcuHABAQEBOHfunGjROX1t7NatG5YtW4abN29i9uzZUKvVTYoPYBIgIhvlCFVEO3TogBkzZuDGjRuYM2cOAOCtt97C4sWLAQBeXl4IDQ1FQUGBaNx+fn6YMWMGRo8ejSNHjiA/Px+VlZWYMKF5Q9vyqyLq7687J+Dg1URZUVMerFFFVIrhIFPZQxXRkydP4vDhw9ov/OZiFVEiIjI7DgcRUZPMDuth7RBsWnBwMIKDg60dhij2BIioSZp6GyjZFlkmgRDfjr9vcNUwEcmYLJPApkn9cFTTu2aDq4aJSMZkmQQAYPLDBeIHEZFk7L2AXHJyMl555RVERERgyZIlkn2+1GSbBHRwSIjI4uy9gNygQYOwc+dOqNVqFBUV4cyZM5LFICXZ3h0U4tsRR6/2RqjzTxwSIrIB9lZAzsvLS/u7s7OzzmpneyLbJLBpUj94qxYg15mF5Ih0bBlrvgujHkOBqJ0mH26PBeROnz6NoqIi9OnTp/l/LyuQbRIgIttjbwXkLl++jJUrV+Lzzz9vRquti0mAiHQ14srd3OypgFxBQQFUKpVOjPaISYCIbJKtF5D75JNPUFxcjEWLFgEAXnvtNQwaNKg5TbYKSQvIqdVq7N27Fy4uLoiPj9eZSElLS8Onn36KFi1a4IknnjC5yJI5CsjVkkMhORZTkwdrFJCzJnsoIGdudldArri4GCkpKdiyZQvefvvtBg9RXrduHdauXQu1Wo38/HxcunRJqlCIiMgAyZJARkYGgoOD4ezsjF69eiE3N1dnv4+PD0pKSqDRaFBZWSn6vE4iIjI/yeYESkpKoFQqtdv1R52GDRuGqVOnolWrVnjuuefw2GOPmXzurKysJsVUUVFh8L1NPaetM9ZmR8U2N40gCHoXRdkie4rVXBrT5vv375v8/4NkSUCpVCInJ0e7rVDodjqWLFmC3bt3o0OHDpg3bx7S09PRt29fk87d1LHP+uOmIb6lQF7zzmnrOD4uD+aaE2jUOPux5UDIwmZ9ZlNxTsC4Vq1amTy/I9lwUGBgIE6dOgWNRoPMzEydSWEAcHFxQdu2baFQKKBUKlFSUiJVKAZtmtTP4p9J5DCOr7B2BGQGkvUEPDw8MGrUKERFRWnvDkpLS0NZWRnCw8Mxffp0xMTEwNXVFZ06dcJzzz0nVShERGSApOsEIiMjERkZqd2u2xsYMWIERoywzvNJ9bmaGI6us7+xdhhEsvHgwQNMmTIFQM1wVvfu3eHq6orXX38dAwYMaNS5tmzZgqioKCnCxO7du/HCCy80GIpJTk7Gvn374OLigp49e+K9995r8mcUFhZi8+bNmDdvnsFjtm/fjldffbXJn2GI7KuI1j5XoOudxpevJaKmYxXR33Xs2NFoAgCAHTt2NPn8xsh+xfDkhywiR2QLHK2K6MmTJ7Fx40a4urri+vXrmDVrFsLCwnDlyhW8//77EAQBXbp0wZIlS5Cfn48lS5Zg/fr1iImJQZ8+fbS1i7744gts3boV+fn5iImJwbhx41BeXo5du3bBzc0Nw4YNQ3R0dJP/7rJPAkQk4thyw5PAce0avjZY1eS7hhytiuitW7ewa9culJeXY8KECQgLC8PKlSuhUqkQEBCAuLg4pKamws/PT+d9QUFBiI2NxcKFC5Geno6JEydi+/bt2jbGxMRg/fr1aN++Paqrq5v0t67FJEBExoUs1P+lHtfO7OVWHK2KaEBAAFxcXKBUKuHu7o7y8nJcv34dAQEBAIA+ffrg2rVrDZJA7f5OnTrh7t2Gf2OVSoWVK1eiqqoK48aNQ1BQkGishjAJ1DFp0yneNkpkRY5WRTQzMxMajQbl5eUoLy+Hu7s7OnfujAsXLiAgIADnzp3DwIEDjf5N9LWxW7duWLZsGW7evInZs2dDrVYbPYcxsksCq7/LQWLqZb37jmUXwlt1UOe12WE9MHeIjyVCI6I6HKGKaIcOHTBjxgzcuHEDc+bMAQC89dZb2oKZXl5eCA0NRUFBgWjcfn5+mDFjBkaPHo0jR44gPz8flZWVmDCheXOaklYRlYI5q4hq/Tau6V2xFbkrbOe2VXPg6ll5sEoVUQmGg0xlDyuGT548icOHD5tcIVmM3VURtVeTNp2ydghERBYju+EgMceyC60dApF9GKyydgQ2LTg4GMHBwdYOQxR7AkTUNFYqHkfmxSRARABqJkitUciRzKukpASurq4mH8/hICICUHNPekFBAW7fvm3tUETZw8SwuZna5tqinKZiEiAiADXP/Hj88cetHYZJsrKy7OZ5yOYiVZs5HEREJGNMAnrwNlEikgsmgTpy3SbgH64f8zZRIpINJgEA6DFU+2uo809WDISIyLI4MQwAUTtrftYpi1tbQyjEtyOLyhGRw2JPQASHhojIkTEJ6JG7YoTDFZIjItKHScAE3qqDvGOIiBwSk4ARIb4dtb9zWIiIHBGTgBGbJvXjsBAROTSTksDx48e1DzguLCzElStXJA3KVnFIiIgcjWgSWLp0KQ4fPoytW7cCqHkGqErl4HXEt4zV2awdFuKQEBE5GtEkkJ6ejmXLlqFly5YAgPbt26OyslLywKyidtHY5X/pvMx1AkTkqESTgKurKx4+fKh90v3Nmzfh4uKga8xqF40REcmEaBKYPn06pk2bhlu3bmHx4sWIiYlBbGysJWKzrnpDQkREjkj0kj4kJASBgYHIyMiAIAiYM2cO2rdvb4nYrKPH0JrhoHpDQkREjki0JzBlyhSkpqaid+/eCA0NdewEAHBIiIhkRTQJLFiwAL/88gteffVVTJ48GWq12i4eP2cWce04LEREDk00Cfj6+mLmzJnYt28f3n33XZw+fRqDBg2yRGzWU6e0NIeFiMiRmXSbz6VLl3D48GEcOXIEnp6eeOedd6SOy7r0lJYmInJEoknghRdeQKdOnTB06FB89dVXjj8nIGLSplNcN0BEDkM0CWzbtg0eHh6WiMWmhfh2xLHsQq4aJiKHYjAJrFq1CrGxsfjwww+1C8XqSkhIED25Wq3G3r174eLigvj4eHh5eWn33bt3D0uXLkVBQQGqq6u1tYls1aZJ/bRPGyMichQGk8DQoTWToxEREU06cXFxMVJSUqBWq3Hx4kUkJCQgKSlJu3/t2rV4+eWXERQU1KTzWxOHhIjIURhMAj4+Pnjw4AF69+7dpBNnZGQgODgYzs7O6NWrF3Jzc3X2nz17FpWVlUhMTMSwYcMQHR3dpM+xJA4JEZGjMZgEhg8frjMMJAgCnJyctD9TU1ONnrikpARKpVLn/XVlZmZi1qxZWLRoESZPnoxnnnkG3bt3NynorKwsk46rr6KiolHv9a/3eQueaatNAE2NwdIa22ZHwDY7Prm1F5CuzQaTwNGjR5t1YqVSiZycHO22QqG7JMHT0xP9+/eHk5MTnnnmGeTk5JicBPz9/cUP0iMrK6tJ79V9z1UAwMc/ltrFkFBT22zP2GbHJ7f2AtK12aSHypw/fx579uzB9u3btf+JCQwMxKlTp6DRaJCZmakzKQwAffv2xcWLFwEAFy5caLDfptRZNcxnCxCRIxG9RfTjjz9GXl4eMjIyMHLkSG0dofHjxxt9n4eHB0aNGoWoqCjt3UFpaWkoKytDeHg45s2bh/feew8VFRXo168fevbsabZGmY2eYnK8S4iIHIloEvj3v/+N/fv346WXXsLbb7+N6dOnY+7cuSadPDIyEpGRkdrtulf7f/rTn/Dll182PmJLitppdNVwbTII8e1oF0NDRET1iQ4Hubm5wcnJCS1btkRRURFat26N69evWyI226JnSKgWh4aIyF6J9gQGDRqEkpISTJ48GWPGjIFCocCLL75oidhsg4EhoVocGiIieyaaBGbOnAmgpoZQWFgYKisr0bZtW8kDsxkiQ0JERPasUQ8LbtGiBVq0aCFVLHaN8wNEZI9MukWUDOP8ABHZs0b1BKghzg8QkT0TTQKCIODHH39Efn4+NBqN9nWxdQJyxqEhIrIXokngzTffhEajwRNPPNGg9APpqi0wV+tYdiFWf1dTOmPuEB9rhUVEZJBoEsjLy8M///lPS8Ri9/QNDSWmXgbAJEBEtkn00r5///44e/asJWIhIiILE+0JfPPNN9i8eTPatWsHV1dX7esnTpyQNDCbtGXs7w+hF1F/aIiIyBaJJgFZftnXp2fVsBgWmiMie2DS3UEHDx5Eeno6ACAoKAjh4eF6nzvssLhqmIgclGgS+PDDD1FcXIyRI0dCEAR88803OH36NOLi4iwQnv1Y/V2OdhJYH329gtlhPThhTERWJZoE0tPTsX//fu12WFiYvArImWjuEB+9X+i1X/65K0ZYOiQiIlGidwe5urri0qVL2u3s7GydCWIyzaRNp6wdAhFRA6I9gffffx/z58/XPiheoVAgPj5e8sAcDe8UIiJbJJoEnnzySRw4cAClpaUQBAFKpdIScRERkQUYTAJff/01oqOjkZCQoPdOoNjYWEkDIyIi6RmcE+jUqRMAoGvXrujSpYvOf127drVYgDYnrp3OoybFzA7rof2d8wJEZGsM9gRCQ0MBAHfu3MHkyZN19v3jH/+QNipbVLtgDGjUorG5Q3xw/noxjmUXcl6AiGyO6N1BBw4caPDavn37JAnGpkXtBOLuNumtLCdNRLbKYE9g586d2LVrF3JzcxEREaF9/d69e+jWrZtFgrNpdVcQ9xhqck2hSZtOMSkQkc0wmASGDx+OZ599FmvWrMGsWbO0r7u7u8PT09MiwdmkusNCtUwYHqotKMchISKyJQaTQNu2beHu7o7s7Gx07tzZkjHZtvpX/CbWFGJBOSKyRUbnBJydneHv748rV65YKh771ci7hoiIbIHoYrHs7Gy89NJL6Nq1K9zd3SEIApycnKBWqy0Rn+1rwl1D3qqDfP4wEdkE0SSwZs0aS8Rhv2qHh2qHhY4tB0IW6j207oNmODdARLZA9BbRzp0749atWzh06BAOHTqEmzdvco7AmOMrDO7aNKkfq4kSkU0RTQJr167Fxx9/jJYtW6JFixZISEjAZ599ZonYiIhIYqLDQYcPH8aePXvg4lJz6IQJEzB69GjMmDFD8uDslpEhISIiWyLaE1AoFCgqKtJu37lzBwqF6NvkzciQEBGRLRHtCcTGxiIyMhLdu3cHAFy5cgWLFy+WPDA54OphIrI20SQwePBgfPPNN7h27RoAoEuXLmjZsqXkgdm8Y8uNX/HXX0Q2WKUdIuLqYSKyFaJJ4N69e9i8eTPOnj0LJycn9OnTB3/961/RunVrS8Rnu0IWNhz3r/vFb6TYXN3Vw6u/y+HD5onIakQH92NjY1FVVYX58+fjrbfeQlVVFebOnWvSydVqNSIiIhAdHY28vLwG+zUaDYYPH46NGzc2PnIHkZh62dohEJGMifYECgoKsH79eu22j48PxowZI3ri4uJipKSkQK1W4+LFi0hISEBSUpLOMSkpKfi///u/JoRtI4wNCemrKVRnSIiIyBaIJoHnn38eO3bswJAhQ+Dk5ITDhw8jLCwMDx48AAC0aNFC7/syMjIQHBwMZ2dn9OrVC7m5uTr7KysrcfToUQwbNgzFxcXNb4k1GBsSEnn2QN3Vw0RE1iKaBA4erBm7/vvf/67zekpKCpycnJCamqr3fSUlJToPpRcEQWd/cnIyxo8fjzt37jQ66KysrEa/BwAqKiqa/F5T+f/2s2z9C/jfoFUGj1vwTFttEpAyJku02dawzY5Pbu0FpGuzaBI4evRok06sVCqRk5Oj3a67tqC0tBSnTp3C1KlTsXv37kaf29/fX/wgPbKyspr8XpO17woUXUWbG/8x4bOuAgA+/rFUsltFLdJmG8M2Oz65tReQrs2iSaCiogJfffUV0tPTAQB9+/bFxIkT4ebmZvR9gYGBWLduHTQaDS5dugQvLy/tvqtXr+LOnTuYMmUKbt26hYcPH8LPzw8DBgxoZnNswKxzeucDVn+XY3AS+Fh2YYNnDcwO68G7hohIcqJJYOHChejUqRPmzZsHQRBw8OBBLFy4EKtXrzb6Pg8PD4waNQpRUVFwcXFBfHw80tLSUFZWhvDwcOzcWVN9c/fu3bhz545jJAAj5g7x0fulXvvlz8JyRGQNokng6tWrOl/4fn5+eOmll0w6eWRkJCIjI7XbdXsDtUy508huxbVr1POHiYgsTXSdQJs2bXDs2DHt9vfff482bdpIGpTd6zH0998b8aCZSZtOSRQQEZF+oj2B5cuXY+nSpVi8eDEUCgX8/PywfPlyS8Rmv+o/aMZEvGWUiCzNaBLQaDT4/PPP8cUXX1gqHse0ZazRIaHcFSP4EHoisgrRB80XFhaioqLCUvE4ltphISNDQrPDeuhsc1iIiCxJdDio9i6fgQMHwt3dXft6bGyspIE5hKidos8err1jiM8fJiJrEE0CAwYMcPjbNy3i+AqjdYNqF4txWIiILMloEvjuu+9QVFSE7t27Y/DgwZaKiYiILMTgnMB7772HzZs3o7i4GJ999hk+/fRTS8Yle5wXICJLMNgTOHfuHPbv3w+FQoGKigpERkZizpw5loxNlvjUMSKyJIM9AVdXV23RN7E6QWSiuHY1/20Za/AQPnOYiCzJYE8gJycHAwcO1G4XFxfrbJ84cULayOyV2LOHgZpbRmvvGjLyoBk+iJ6IpGYwCWRmZloyDseh70EzwG+9gLsmPXSGQ0JEZCmitYPI8nj1T0SWwiRARCRjTAI2jmUkiEhKTAI2KsS3o/Z3zg0QkVSYBCxlsKpRh2+a1I9PGyMiyTEJWIqRukFERNbCJEBEJGNMAkREMiZaSpokou/Rk3woPRFZGHsCllb3IfT1GXkCGW8TJSIpsCdgaYau9A08lJ4lJIhISuwJ2Jp6VUZZQoKIpMQkYCvqDhMZGBbi6mEiMjcmAVsRtdNgZVGuHiYiqTAJ2AGuHiYiqXBi2FbpmSjOrX3A2xbeSkpE5sGegK0xdgtpLSO3khIRNQZ7ArbGyBW+t+ogct0mWDAYInJ0TAL2iiuOicgMOBxkZ45qehveyWEiImok9gTsSIhvR0zOXgA8bLiPw0RE1BRMAnZE3+phb9XB5p/42HI+74BIppgEHE29uYI//bE/4H/I+HuOr2ASIJIpSecE1Go1IiIiEB0djby8PJ19b7/9NiIiIjB27Fjs2bNHyjBkwdBcQZsb/7FwJERkTyTrCRQXFyMlJQVqtRoXL15EQkICkpKStPunT58Ob29vPHjwAC+++CL+8pe/wNXVVapwHMbq73KQmHq5weuTHzacK+A8ARGJkSwJZGRkIDg4GM7OzujVqxdyc3N19nt7e9cE4FITgkLBG5VMMXeID+YO8WnwurfqYMPSEnGWiYmI7JdkSaCkpARKpVK7LQiC3uM2bNiAESNGwNnZ2eRzZ2VlNSmmioqKJr/XHtRvm3/tLwaeVWDsvfbM0f+d9ZFbm+XWXkC6NkuWBJRKJXJycrTb+q709+/fj6ysLKxatapR5/b39xc/SI+srKwmv9f2XW3QtqOa3gh1/smkd/tvf6bhi4NVdjlh7Nj/zvrJrc1yay8gXZslSwKBgYFYt24dNBoNLl26BC8vL539aWlp2LNnD9avX8+hIIkkd/0Ekw2UntYOHdX2EgyUsSYixyZZEvDw8MCoUaMQFRUFFxcXxMfHIy0tDWVlZQgPD8fChQvxyCOPYMqUKQCAVatWoWPHjiJnpcbYNKlfg6sHg+sKapMBS08QyYqk6wQiIyMRGRmp3a7bG/jhhx+k/GjZmR3Wo1HH1yaDfZ79EHi/ztPKWHqCSFY4DuMg9N0xpE/dp5QBwEt35tT8wuEgIlniimGZqVt6wiwlJ4jIrjEJUM1dQHWZcEupDs4jENktDgcRvA8/CW/VQeNlqo3hPAKR3WJPQMZCfDviWJ1bSPWWnhB7wH1jew1EZFOYBGRMX2nqWpwvIJIHJgEyj+b2CDivQGQVnBMgo7xVB+GtOohJm07pP6DHUPN8EOcViKyCPQHSq/58wTED5SfMcvXOeQUiq2ESIL24noBIHpgEyHbU7xFwnoBIckwCZLL6PYIQ345G7zAyWY+h+ucEOE9AJDkmARJVf36glsF5gsbSd7XPeQIii2ASIFH6rvY5T0DkGJgEyLYZ6hFwvoDILLhOgJpFdB1BU4mtP+B8AZFZsCdATWLyOoKmMnaVz/kCIrNhEqAm4ToCIsfAJEBmYywZmO120rr4XGSiZuOcADVb/UdW6mPW4aL68wWcHyBqMvYEqNnErvDNPlxU96qf8wNEzcKeABGRjLEnQBYjxQRyrttvv9TpEfib+0M450AOjD0BkpwpcwZN1eTnIjcG5xzIgbEnQJIz+11Bv/FWHcTkhwuQu8dS+PkAAAfhSURBVET3OchZWVnw9zdTf4BzDuTg2BMgIpIx9gTI7nmrDkqzDqEuW+sRcJ6CzIQ9AbJbdecazF62opa5nqFsbpynIDNhT4DsVu2Vv6RlK2zxatvWeiVk19gTICKSMfYEyGHo9giuApCoZpGtkHJthI2TVXslnv9hT4DsnrF1CJLNFViTrc5TkDQknv9hT4DsXv0r/dp1Ag5b4lrPVaFZ10bYAdm01wLzP+wJEBHJGHsC5PDspUfg0PMX1Dxx7fCnP/YH/A+Z/dSS9gTUajUiIiIQHR2NvLw8nX15eXmIjo5GREQE1Gq1lGGQTElZs0gKDjl/Qc1TZ/6nzY3/SPIRkvUEiouLkZKSArVajYsXLyIhIQFJSUna/QkJCVCpVPDz80NkZCSGDx8ODw8PqcIhGbKnq2p76a2QhdXO/0g4NyBZTyAjIwPBwcFwdnZGr169kJubq7M/NzcXAQEBcHFxQb9+/XD+/HmpQiEiIgMkSwIlJSVQKpXabUEQdPbX3W7Xrh3u3r0rVShERGSAZMNBSqUSOTk52m2FQjff1N0uKSmBn5+fyef29fVtfoBENqTlbz9998RaNQ6yVY/V/Nhm/u8+yZJAYGAg1q1bB41Gg0uXLsHLy0tnv5eXFzIzM+Hn54fTp09j6tSpJp03OztbinCJiGTJSag/TmNG27Ztw759++Di4oL4+Hjk5eWhrKwM4eHhyMvLwzvvvIOqqiqMGjUKERERUoVBREQGSJoEiIjItnHFMBGRjDEJEBHJGJMAEZGMMQkQEcmYbJKAsTpG9uzKlSuIjIxEVFQUYmJi8L///Q/379/H7NmzMWHCBCxevBjV1dUAalZxR0REYPz48UhNTbVy5M135swZ+Pr6oqioSBZtPn/+PCZPnozo6Ghs2LBBFm3+8MMPMW7cOLzyyitIS0tzyDY/ePAAERERCAoKwrfffgsAjW5nYmIiIiMjMXXqVBQVFTUuAEEG7ty5I7zyyitCVVWVcP78eWHmzJnWDslsbt++Ldy9e1cQBEE4fvy4sGjRIiE5OVnYuHGjIAiC8MEHHwjHjh0TBEEQxo8fL/zyyy9CWVmZMGrUKOHhw4fWCtss3nzzTWHMmDHC7du3Hb7NlZWVwtSpU4Xy8nLta47e5mvXrgkxMTGCIAjCr7/+KowZM8Yh26zRaISbN28KSUlJwqFDhwRBaNy/bXZ2tjBt2jRBEATh0KFDwooVKxr1+bLoCYjVMbJn7du315bncHFxgbOzM86cOYPBgwcDAAYPHowzZ86gsrIS1dXVePTRR9G6dWt4eXnZdY/o6NGj6Nu3L9zd3QHA4dt87tw5tGrVCrNmzcKUKVOQk5Pj8G3u0KED3N3dUVVVhdLSUnh6ejpkmxUKBR555BGd1xrTzrrHPv/88zh79mzjPt88zbBtYnWMHMH9+/eRlJSEiRMn6rS3ti5TcXEx2rZtqz1eqVTabb2m6upqbNu2DZGRkdrXHL3NhYWFyM7ORmJiIhYtWoS4uDiHb3Pr1q3x2GOPYfjw4ZgwYQJef/11h29zrca0s+6xbm5uKC8vb9RnySIJKJVKlJaWarfr1zGyd1VVVZg7dy6mTp2Kbt266bS3pKQE7dq1g4eHB8rKyrTvKS0tRbt20j+6TgoHDhxAaGgoWrZsqX3N0dusVCrx1FNPwd3dHd26dUNpaanDt/mHH37A3bt38a9//Qt79+7FsmXLHL7NtRrTzrrHVlZWanvHpnKsb0MDAgMDcerUKWg0GmRmZjaoY2TPBEHAO++8g0GDBuHPf/4zAODpp5/G8ePHAQBpaWkICgpCy5YtoVAocOvWLZSXlyM3N9du/w45OTk4fPgwpkyZguzsbMydO9fh2xwYGIhr165Bo9Hg1q1bcHNzc/g2V1dXo127dlAoFGjTpg3Ky8sdvs21GtPOusceP34cTz31VKM+SzZlI+rXMbL3/0lqpaWlYebMmXjyyScBAH5+foiNjYVKpUJhYSG6d++OuLg4KBQKZGRkYPny5aiursa0adMQFhZm5eibLyYmBomJiWjVqpXDtzklJQUpKSnQaDSYP38+evbs6dBt1mg0UKlUyM/PR2VlJSZOnIghQ4Y4ZJtnzpyJixcvwt3dHQMGDMDs2bMb1c7ExET8+OOPcHd3xyeffIL27dub/NmySQJERNSQLIaDiIhIPyYBIiIZYxIgIpIxJgEiIhljEiAikjHJnjFMZO/GjBkDjUaDX3/9FS4uLvDw8MCVK1ewcuVKDB8+3NrhEZkFbxElErFmzRp06NBBp0wFkaPgcBBRI6xZswbbtm0DAISGhiIxMREjR47ElClTkJ6ejsjISAwdOhQ///wzAKCsrAzz5s3Dyy+/jLFjx+LChQvWDJ+oASYBombo2rUrDhw4ADc3N2zatAlbtmzBggUL8OWXXwIA1q1bh/DwcKSkpGDVqlX44IMPrBswUT2cEyBqhpCQEACAj48PPD09oVAo4Ovri40bNwIAfvzxR5w4cQJJSUkAoFPIkMgWMAkQNYOrqyuAmsq0tb87OTlBo9EAqCnwt2HDBjz66KNWi5HIGA4HEUno2WefxdatW7Xbly5dsmI0RA0xCRBJaMaMGbh58yZGjhyJ8PBwHDhwwNohEengLaJERDLGngARkYwxCRARyRiTABGRjDEJEBHJGJMAEZGMMQkQEckYkwARkYwxCRARydj/A1yo0ElrG6lzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sf1 = sm.duration.SurvfuncRight(veteran.time[veteran.trt==1], veteran.status[veteran.trt==1], title='Treatment 1')\n",
    "sf2 = sm.duration.SurvfuncRight(veteran.time[veteran.trt==2], veteran.status[veteran.trt==2], title='Treatment 2')\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "\n",
    "plt.grid(True)\n",
    "sf1.plot(ax); # Draw on previously defined axis\n",
    "sf2.plot(ax);\n",
    "\n",
    "plt.xlabel('Time');\n",
    "plt.ylabel('Proportion alive');\n",
    "plt.legend(loc='upper right');\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also perform a statistical test (the _log-rank test_) to see\n",
    "if there is a statistical difference between these two curves.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.928"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chisq, pval = sm.duration.survdiff(veteran.time, veteran.status, veteran.trt)\n",
    "np.round(pval,3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,Rmd"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "nteract": {
   "version": "0.23.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}