A-B Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding business problem & data\n",
    "\n",
    "Detect and resolve problems in the data (Missing Value, Outliers, Unexpected Value)\n",
    "\n",
    "Look summary stats and plots\n",
    "\n",
    "Apply hypothesis testing and check assumptions\n",
    "\n",
    "Check Normality & Homogeneity\n",
    "\n",
    "Apply tests (Shapiro, Levene Test, T-Test, Welch Test, Mann Whitney U Test)\n",
    "\n",
    "Evaluate the results\n",
    "\n",
    "Make inferences\n",
    "\n",
    "Recommend business decision to your customer/director/ceo etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Statistical significance mainly depends on the sample size, the quality of the data and the power of the statistical procedures.\n",
    " \n",
    "\n",
    "1. Determine the effect, sample size instead of P-Hacking \n",
    "==\n",
    "(p-hacking: gradually adding samples until rejecting Ho--bad practice)\n",
    "2. Use power analysis to determine required sample size. \n",
    "===\n",
    "power=0.8= want 80% chance of rejecting Ho correctly\n",
    "3. State your Hypothesis\n",
    "==\n",
    "4. Analyze\n",
    "=="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " The product manager (PM) told you that the current conversion rate is about 14% on average throughout the year. They want an increase of 3%, meaning that the new design will be considered a success if it raises the conversion rate to 17%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Required sample Size 5335\n"
     ]
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import scipy.stats as stats\n",
    "import statsmodels.stats.api as sms\n",
    "\n",
    "#effect size=(mean difference)/(standard deviation)\n",
    "#calculate effect size using Cohen's D\n",
    "def cohend(d1, d2):\n",
    "\t# calculate the size of samples\n",
    "\tn1, n2 = len(d1), len(d2)\n",
    "\t# calculate the variance of the samples\n",
    "\ts1, s2 = var(d1, ddof=1), var(d2, ddof=1)\n",
    "\t# calculate the pooled standard deviation\n",
    "\ts = sqrt(((n1 - 1) * s1 + (n2 - 1) * s2) / (n1 + n2 - 2))\n",
    "\t# calculate the means of the samples\n",
    "\tu1, u2 = mean(d1), mean(d2)\n",
    "\t# calculate the effect size\n",
    "\treturn (u1 - u2) / s\n",
    "\n",
    "#since we dont know actual mean we will calculate proportion effect size\n",
    "effect_size = sms.proportion_effectsize(0.14, 0.17)    # Calculating effect size based on our expected rates\n",
    "required_n = sms.NormalIndPower().solve_power(\n",
    "    effect_size, \n",
    "    power=0.99, \n",
    "    alpha=0.05, \n",
    "    ratio=1\n",
    "    )      \n",
    "required_n=int(required_n)\n",
    "print(\"Required sample Size\", int(required_n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need atleast  2279 samples\n",
    "=="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user_id          int64\n",
      "timestamp       object\n",
      "group           object\n",
      "landing_page    object\n",
      "converted        int64\n",
      "dtype: object\n",
      "1472390\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "data=pd.read_csv(\"https://raw.githubusercontent.com/salammemphis/ML/main/dataset/ab_data.csv\")\n",
    "print(data.dtypes)\n",
    "print(data.size)\n",
    "data.head()\n",
    "#load data\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>group</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>control</td>\n",
       "      <td>147202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>treatment</td>\n",
       "      <td>147276</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       group   count\n",
       "0    control  147202\n",
       "1  treatment  147276"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=data.groupby(\"group\")[\"converted\"].agg([\"count\"]).reset_index()\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1433450\n"
     ]
    }
   ],
   "source": [
    "#make sure no bias and all participant has equal influence \n",
    "#if one user participated multiple times just delete since we have enough data\n",
    "df=data.groupby([\"user_id\"])[\"converted\"].agg([\"count\"]).reset_index()\n",
    "df=df[df[\"count\"]>1]\n",
    "df.size\n",
    "data=data[~data[\"user_id\"].isin(df[\"user_id\"])]\n",
    "print(data.size)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=[]\n",
    "y=[]\n",
    "for i in range(500):\n",
    "    x.append(data[data[\"group\"]=='control'].sample(n=33450)[\"converted\"].mean())\n",
    "    y.append(i)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAD3NJREFUeJzt3X+MZWddx/H3x5ZGBEJ36TBZW8apUov8YQuMtQgh0rVQqGHXpGlAhEndZE1UAtEoK/5l4h9b/xAw/kg2LTIm2FILuCtFYF1ANEBlCwXaLrDbuoWt+wto5YeJuPD1jzkrt+sM98zMvTPTZ96vZHLPee45937vk7OffeaZc85NVSFJeuL7kbUuQJI0Gga6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqRHnr+abXXTRRTU9Pb2abylJT3j33HPP16pqYth2qxro09PTHDx4cDXfUpKe8JI83Gc7p1wkqREGuiQ1YmigJ7k8yb0DP99M8qYkm5PsT3K4e9y0GgVLkhY2NNCr6ktVdWVVXQm8APgv4H3ALuBAVV0GHOjWJUlrZKlTLluBB6vqYWAbMNe1zwHbR1mYJGlplhrorwZu65Ynq+p4t3wCmBxZVZKkJesd6EkuAF4F/N25z9X81x4t+NVHSXYmOZjk4OnTp5ddqCTph1vKCP0VwGeq6mS3fjLJFoDu8dRCO1XVnqqaqaqZiYmh58VLkpZpKYH+Gn4w3QKwD5jtlmeBvaMqSpK0dL2uFE3yFOBa4DcGmncDdyTZATwM3Dj68qT+pnfdtex9j+6+foSVSGujV6BX1XeAZ5zT9nXmz3qRJK0DXikqSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1IhegZ7kwiR3JvlikkNJXphkc5L9SQ53j5vGXawkaXF9R+hvBz5YVc8BrgAOAbuAA1V1GXCgW5ckrZGhgZ7k6cBLgFsBquq7VfUYsA2Y6zabA7aPq0hJ0nB9RuiXAqeBv07y2SS3JHkKMFlVx7ttTgCT4ypSkjTc+T23eT7whqq6O8nbOWd6paoqSS20c5KdwE6AqampFZYrjcf0rruWve/R3dePsBJp+fqM0I8Bx6rq7m79TuYD/mSSLQDd46mFdq6qPVU1U1UzExMTo6hZkrSAoYFeVSeArya5vGvaCjwA7ANmu7ZZYO9YKpQk9dJnygXgDcC7klwAPATcxPx/Bnck2QE8DNw4nhIlSX30CvSquheYWeCpraMtR5K0XF4pKkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktSI8/tslOQo8C3ge8CZqppJshl4NzANHAVurKpHx1OmJGmYpYzQX1pVV1bVTLe+CzhQVZcBB7p1SdIaWcmUyzZgrlueA7avvBxJ0nL1DfQCPpzkniQ7u7bJqjreLZ8AJkdenSSpt15z6MCLq+qRJM8E9if54uCTVVVJaqEdu/8AdgJMTU2tqFhJjze9665l73t09/UjrETrQa8RelU90j2eAt4HXAWcTLIFoHs8tci+e6pqpqpmJiYmRlO1JOn/GRroSZ6S5Glnl4GXAfcB+4DZbrNZYO+4ipQkDddnymUSeF+Ss9v/bVV9MMmngTuS7AAeBm4cX5nS+rWSaQ9w6kOjMzTQq+oh4IoF2r8ObB1HUZKkpfNKUUlqRN+zXCSNyUqnbKSzHKFLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSI/zGIq0rfnuPtHyO0CWpEb0DPcl5ST6b5P3d+qVJ7k5yJMm7k1wwvjIlScMsZYT+RuDQwPrNwFur6tnAo8COURYmSVqaXoGe5BLgeuCWbj3ANcCd3SZzwPZxFChJ6qfvCP1twO8D3+/WnwE8VlVnuvVjwMUjrk2StARDAz3JLwOnquqe5bxBkp1JDiY5ePr06eW8hCSphz4j9BcBr0pyFLid+amWtwMXJjl72uMlwCML7VxVe6pqpqpmJiYmRlCyJGkhQwO9qv6gqi6pqmng1cBHquq1wEeBG7rNZoG9Y6tSkjTUSs5DfzPwO0mOMD+nfutoSpIkLceSrhStqo8BH+uWHwKuGn1JkqTl8EpRSWqE93LRyHk/FmltOEKXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRviNRdIGtZJvljq6+/oRVqJRcYQuSY0YGuhJfjTJvyX5XJL7k/xR135pkruTHEny7iQXjL9cSdJi+ozQ/xu4pqquAK4ErktyNXAz8NaqejbwKLBjfGVKkoYZGug179vd6pO6nwKuAe7s2ueA7WOpUJLUS6859CTnJbkXOAXsBx4EHquqM90mx4CLx1OiJKmPXoFeVd+rqiuBS4CrgOf0fYMkO5McTHLw9OnTyyxTkjTMks5yqarHgI8CLwQuTHL2tMdLgEcW2WdPVc1U1czExMSKipUkLa7PWS4TSS7slp8MXAscYj7Yb+g2mwX2jqtISdJwfS4s2gLMJTmP+f8A7qiq9yd5ALg9yR8DnwVuHWOdktaRlVyUBF6YNC5DA72qPg88b4H2h5ifT5ckrQNeKSpJjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktSIPrfP1Qaz0lujSlobjtAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWrE0EBP8qwkH03yQJL7k7yxa9+cZH+Sw93jpvGXK0laTJ8R+hngd6vqucDVwG8leS6wCzhQVZcBB7p1SdIaGRroVXW8qj7TLX8LOARcDGwD5rrN5oDt4ypSkjTckubQk0wDzwPuBiar6nj31AlgcqSVSZKWpPe9XJI8FXgP8Kaq+maS/3uuqipJLbLfTmAnwNTU1MqqVW/ej0XaeHqN0JM8ifkwf1dVvbdrPplkS/f8FuDUQvtW1Z6qmqmqmYmJiVHULElaQJ+zXALcChyqqj8deGofMNstzwJ7R1+eJKmvPlMuLwJeB3whyb1d21uA3cAdSXYADwM3jqdESVIfQwO9qv4VyCJPbx1tOZKk5fJKUUlqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGDA30JO9IcirJfQNtm5PsT3K4e9w03jIlScP0GaG/E7junLZdwIGqugw40K1LktbQ0ECvqo8D3zineRsw1y3PAdtHXJckaYmWO4c+WVXHu+UTwOSI6pEkLdP5K32BqqoktdjzSXYCOwGmpqZW+naSGjC9665l73t09/UjrKQtyx2hn0yyBaB7PLXYhlW1p6pmqmpmYmJimW8nSRpmuYG+D5jtlmeBvaMpR5K0XH1OW7wN+CRweZJjSXYAu4FrkxwGfqlblyStoaFz6FX1mkWe2jriWiRJK+CVopLUiBWf5SJJq8kzZBbnCF2SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCC8skrRhtH5RkiN0SWqEgS5JjXDKZcxW8iuepPXjiTBd4whdkhphoEtSIwx0SWqEgS5JjTDQJakRnuUyhGepSHqicIQuSY1YUaAnuS7Jl5IcSbJrVEVJkpZu2VMuSc4D/gK4FjgGfDrJvqp6YFTFDXLqQ5J+uJWM0K8CjlTVQ1X1XeB2YNtoypIkLdVKAv1i4KsD68e6NknSGhj7WS5JdgI7u9VvJ/nSMl/qIuBro6mqafZTf/ZVP/ZTP4v2U25e8Wv/RJ+NVhLojwDPGli/pGt7nKraA+xZwfsAkORgVc2s9HVaZz/1Z1/1Yz/1sx76aSVTLp8GLktyaZILgFcD+0ZTliRpqZY9Qq+qM0l+G/gQcB7wjqq6f2SVSZKWZEVz6FX1AeADI6plmBVP22wQ9lN/9lU/9lM/a95Pqaq1rkGSNAJe+i9JjVjTQE/yjiSnktw30LY5yf4kh7vHTV17kvxZd5uBzyd5/sA+s932h5PMrsVnGacl9tNru/75QpJPJLliYJ+mb9WwlH4aeP7nkpxJcsNAW9PHEyy9r5L8YpJ7k9yf5J8H2j2mfvBv7+lJ/iHJ57p+umlgn9U5pqpqzX6AlwDPB+4baPsTYFe3vAu4uVt+JfCPQICrgbu79s3AQ93jpm5501p+rjXup184+/mBVwz003nAg8BPAhcAnwOeu9afba36aaBPPsL834Fu2CjH0zKOqQuBB4Cpbv2ZHlML9tNbBpYngG90/bJqx9SajtCr6uPMf+hB24C5bnkO2D7Q/jc171PAhUm2AC8H9lfVN6rqUWA/cN34q189S+mnqvpE1w8An2L++gDYALdqWOLxBPAG4D3AqYG25o8nWHJf/Srw3qr6Srfv2f7ymHp8PxXwtCQBntrtd4ZVPKbW4xz6ZFUd75ZPAJPd8mK3GtiotyBYrJ8G7WD+txqwn2Cgn5JcDPwK8FfnbL9R+wkWP6Z+GtiU5GNJ7kny+q59o/bVYv3058DPAP8BfAF4Y1V9n1Xsp3X9BRdVVUk8DWeIhfopyUuZD/QXr01V6885/fQ24M1V9f35AZUGndNX5wMvALYCTwY+meRTa1bcOnJOP70cuBe4BvgpYH+Sf1nNetbjCP1kN5VC93j217vFbjXQ6xYEDVqsn0jys8AtwLaq+nrXbD89vp9mgNuTHAVuAP4yyXY2bj/B4n11DPhQVX2nqr4GfBy4go3bV4v1003MT01VVR0B/h14DqvYT+sx0PcBZ/8KPAvsHWh/fXe2y9XAf3a/9nwIeFmSTd1fm1/WtbVuwX5KMgW8F3hdVX15YPuNequGBfupqi6tqumqmgbuBH6zqv6ejXs8weL/9vYCL05yfpIfA34eOITHFDy+n77C/G8xJJkELmf+D6Crd0yt8V+QbwOOA//D/ChgB/AM4ABwGPgnYHO3bZj/Qo0HmZ+fmhl4nV8HjnQ/N63lZ1oH/XQL8Cjzv/rdCxwceJ1XAl/u+vAP1/pzrWU/nbPfO+nOctkIx9Ny+gr4PebPdLkPeJPH1IL/9n4c+HCXT/cBv7bax5RXikpSI9bjlIskaRkMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGvG/YRL82oB57DEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "dt=np.array(x)*1000/np.min(x)\n",
    "plt.hist(dt,bins=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "control_group=data[data[\"group\"]=='control'].sample(n=required_n,random_state=42)\n",
    "treatment_group=data[data[\"group\"]=='treatment'].sample(n=required_n,random_state=42)\n",
    "df=pd.concat([control_group,treatment_group], axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>group</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>count</th>\n",
       "      <th>sum</th>\n",
       "      <th>std_err</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>control</td>\n",
       "      <td>0.115651</td>\n",
       "      <td>0.319836</td>\n",
       "      <td>5335</td>\n",
       "      <td>617</td>\n",
       "      <td>0.004379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>treatment</td>\n",
       "      <td>0.128210</td>\n",
       "      <td>0.334355</td>\n",
       "      <td>5335</td>\n",
       "      <td>684</td>\n",
       "      <td>0.004578</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       group      mean       std  count  sum   std_err\n",
       "0    control  0.115651  0.319836   5335  617  0.004379\n",
       "1  treatment  0.128210  0.334355   5335  684  0.004578"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfg=df.groupby(\"group\")[\"converted\"].agg(['mean',\"std\",\"count\",\"sum\"]).reset_index()\n",
    "dfg[\"std_err\"]=dfg[\"std\"]/(dfg[\"count\"]**(1/2))\n",
    "dfg\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems treatment group made things little better. The conversion rate went little up than what we expected. The average conversion rate of control group was lower than what we thought.\n",
    "\n",
    "Is the difference Statistically significant. Since, we have very large sample lets assume it follow central limit theorem and poportion based use z-test\n",
    "==="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z statistic: -1.98\n",
      "p-value: 0.047\n",
      "ci 95% for control group: [0.107, 0.124]\n",
      "ci 95% for treatment group: [0.119, 0.137]\n"
     ]
    }
   ],
   "source": [
    "from statsmodels.stats.proportion import proportions_ztest, proportion_confint\n",
    "\n",
    "successes = [dfg[\"sum\"][0], dfg[\"sum\"][1]]\n",
    "nobs = [dfg[\"count\"][0], dfg[\"count\"][1]]\n",
    "z_stat, pval = proportions_ztest(successes, nobs=nobs)\n",
    "(lower_con, lower_treat), (upper_con, upper_treat) = proportion_confint(successes, nobs=nobs, alpha=0.05)\n",
    "\n",
    "print(f'z statistic: {z_stat:.2f}')\n",
    "print(f'p-value: {pval:.3f}')\n",
    "print(f'ci 95% for control group: [{lower_con:.3f}, {upper_con:.3f}]')\n",
    "print(f'ci 95% for treatment group: [{lower_treat:.3f}, {upper_treat:.3f}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since, P-value <alpha, There is significant difference in conversion rate among two groups. Reject Null Hypothesis\n",
    "=="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Independant  sample T-TEST\n",
    "===========\n",
    "Assumption 1: Are the two samples independent? \n",
    "\n",
    "Assumption 2: Are the data from each of the 2 groups following a normal distribution?\n",
    "\n",
    "Assumption 3: Do the two samples have the same variances (Homogeneity of Variance)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>height</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>height</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        height\n",
       "height     1.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "data = 'https://gist.githubusercontent.com/baskaufs/1a7a995c1b25d6e88b45/raw/4bb17ccc5c1e62c27627833a4f25380f27d30b35/t-test.csv'\n",
    "df = pd.read_csv(data)\n",
    "df.head()\n",
    "df_men=df[df[\"grouping\"]=='men']\n",
    "df_women=df[df[\"grouping\"]=='women']\n",
    "df.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"8\" halign=\"left\">height</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>grouping</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>men</th>\n",
       "      <td>7.0</td>\n",
       "      <td>179.871429</td>\n",
       "      <td>6.216836</td>\n",
       "      <td>169.0</td>\n",
       "      <td>176.80</td>\n",
       "      <td>181.5</td>\n",
       "      <td>183.85</td>\n",
       "      <td>187.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>women</th>\n",
       "      <td>7.0</td>\n",
       "      <td>171.057143</td>\n",
       "      <td>5.697619</td>\n",
       "      <td>165.2</td>\n",
       "      <td>166.65</td>\n",
       "      <td>170.3</td>\n",
       "      <td>173.75</td>\n",
       "      <td>181.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         height                                                           \n",
       "          count        mean       std    min     25%    50%     75%    max\n",
       "grouping                                                                  \n",
       "men         7.0  179.871429  6.216836  169.0  176.80  181.5  183.85  187.3\n",
       "women       7.0  171.057143  5.697619  165.2  166.65  170.3  173.75  181.1"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"grouping\").describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_men[\"height\"].corr(df_women[\"height\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assumption 2: Are the data from each of the 2 groups following a normal distribution?\n",
    "===\n",
    "Ho=Sample is normally dirtributed\n",
    "\n",
    "perform Shapiro-Wilks test, Histogram, Q-Q plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ShapiroResult(statistic=0.9550848603248596, pvalue=0.7756242156028748)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import scipy.stats as stats\n",
    "#Shapiro-Wilks test\n",
    "stats.shapiro(df_men[\"height\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ShapiroResult(statistic=0.9197608828544617, pvalue=0.467536598443985)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.shapiro(df_women[\"height\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P-value is greater than alpha hence accept null hypotheis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assumption 3: Do the two samples have the same variances (Homogeneity of Variance)?\n",
    "==\n",
    "if  data does not fulfill the assumption of equal variances, we can use Welch’s t-test instead of Student’s t-test\n",
    "Ho=Both sample has same variance\n",
    "\n",
    "Perform levene TEST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LeveneResult(statistic=0.026695150465104206, pvalue=0.8729335280501348)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.levene(df_men[\"height\"], df_women[\"height\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "T-stat: 2.765444762721087 P-value: 0.017105701528166175 Degree of Freedom: 12.0\n"
     ]
    }
   ],
   "source": [
    "from statsmodels.stats.weightstats import ttest_ind\n",
    "result=ttest_ind(df_men[\"height\"], df_women[\"height\"])\n",
    "print( \"T-stat:\",result[0],\"P-value:\",result[1],\"Degree of Freedom:\",result[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Paired T-Test\n",
    "==========\n",
    "scipy.stats.ttest_rel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chi-square TEST\n",
    "===\n",
    "The table has interest of male and females in three subject math, science, art\n",
    "\n",
    "Ho: Interests do not depend on gender Or interest is independent of gender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import chisquare\n",
    "from scipy.stats import chi2_contingency\n",
    "\n",
    "table = [\t[10, 20, 30],\n",
    "\t\t\t[6,  9,  17]]\n",
    "stat, p, dof, expected=chi2_contingency(table)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}