04 WebApp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "709c1d15-e298-4af3-9049-39f5005fa26a",
   "metadata": {},
   "source": [
    "# WebApp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64ce49c9-1f18-4ec5-892e-73ee8d8ce946",
   "metadata": {},
   "source": [
    "In this notebook, I'll be preparing the models required to run the web application built in flask.\n",
    "\n",
    "The purpose of the webapp is to produce a prediction and classify the movement that is being made, based on phone sensor data. We'll be loading in a csv of new data and seeing how it performs against our ML model.\n",
    "\n",
    "---\n",
    "\n",
    "### Data\n",
    "\n",
    "In the dataset we've been working on, the data was collected from a Samsung Galaxy S II on the waist of participants, collecting data at a sampling rate of 50Hz. I'll be replicating this experiment, collecting my own data, and running it through the ML model that's running on the web app.\n",
    "\n",
    "To replicate this experiment properly, there's a few things that we'll have to do.\n",
    "\n",
    "Since the experiment's clean, feature engineered data was so complicated with 561 features, I'll be using the raw data to train the model.\n",
    "For a time series model, we'll need to use the previous n records to predict the future, so we'll need to do that before training the model.\n",
    "\n",
    "Also, the app I'll be using collects data samples at a rate of 100Hz, I'll need to downsample my data with pandas `resample` method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fe682d2-3c3c-4966-9524-7f1f69b370f7",
   "metadata": {},
   "source": [
    "We can start off by importing the required libraries for this analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e993d57f-1340-4700-be19-7beafb401440",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import joblib\n",
    "import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d6e2d42-bda1-4144-9380-6bfcac4cc4a8",
   "metadata": {},
   "source": [
    "Now, let's load in the raw data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "096ef5be-c806-4054-86bb-83478169dc5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate empty dataframes\n",
    "acc_features = ['acc_x','acc_y','acc_z','activity']\n",
    "gyro_features = ['gyro_x','gyro_y','gyro_z','activity']\n",
    "accelerometer = pd.DataFrame(columns = acc_features)\n",
    "gyroscope = pd.DataFrame(columns = gyro_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84f200e2-f227-4248-984b-1fd3eaa45661",
   "metadata": {},
   "source": [
    "We can use the `glob` function to get the file names in the raw data folder, in order to read the files and add them to the empty dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b71c279e-ae18-411d-8c20-51afac35f904",
   "metadata": {},
   "outputs": [],
   "source": [
    "files = []\n",
    "for file in glob.glob('data/RawData/*.txt'):\n",
    "    files.append(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8c02610e-d36a-4151-b219-64e5870807b3",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data/RawData/acc_exp01_user01.txt',\n",
       " 'data/RawData/acc_exp02_user01.txt',\n",
       " 'data/RawData/acc_exp03_user02.txt',\n",
       " 'data/RawData/acc_exp04_user02.txt',\n",
       " 'data/RawData/acc_exp05_user03.txt',\n",
       " 'data/RawData/acc_exp06_user03.txt',\n",
       " 'data/RawData/acc_exp07_user04.txt',\n",
       " 'data/RawData/acc_exp08_user04.txt',\n",
       " 'data/RawData/acc_exp09_user05.txt',\n",
       " 'data/RawData/acc_exp10_user05.txt',\n",
       " 'data/RawData/acc_exp11_user06.txt',\n",
       " 'data/RawData/acc_exp12_user06.txt',\n",
       " 'data/RawData/acc_exp13_user07.txt',\n",
       " 'data/RawData/acc_exp14_user07.txt',\n",
       " 'data/RawData/acc_exp15_user08.txt',\n",
       " 'data/RawData/acc_exp16_user08.txt',\n",
       " 'data/RawData/acc_exp17_user09.txt',\n",
       " 'data/RawData/acc_exp18_user09.txt',\n",
       " 'data/RawData/acc_exp19_user10.txt',\n",
       " 'data/RawData/acc_exp20_user10.txt',\n",
       " 'data/RawData/acc_exp21_user10.txt',\n",
       " 'data/RawData/acc_exp22_user11.txt',\n",
       " 'data/RawData/acc_exp23_user11.txt',\n",
       " 'data/RawData/acc_exp24_user12.txt',\n",
       " 'data/RawData/acc_exp25_user12.txt',\n",
       " 'data/RawData/acc_exp26_user13.txt',\n",
       " 'data/RawData/acc_exp27_user13.txt',\n",
       " 'data/RawData/acc_exp28_user14.txt',\n",
       " 'data/RawData/acc_exp29_user14.txt',\n",
       " 'data/RawData/acc_exp30_user15.txt',\n",
       " 'data/RawData/acc_exp31_user15.txt',\n",
       " 'data/RawData/acc_exp32_user16.txt',\n",
       " 'data/RawData/acc_exp33_user16.txt',\n",
       " 'data/RawData/acc_exp34_user17.txt',\n",
       " 'data/RawData/acc_exp35_user17.txt',\n",
       " 'data/RawData/acc_exp36_user18.txt',\n",
       " 'data/RawData/acc_exp37_user18.txt',\n",
       " 'data/RawData/acc_exp38_user19.txt',\n",
       " 'data/RawData/acc_exp39_user19.txt',\n",
       " 'data/RawData/acc_exp40_user20.txt',\n",
       " 'data/RawData/acc_exp41_user20.txt',\n",
       " 'data/RawData/acc_exp42_user21.txt',\n",
       " 'data/RawData/acc_exp43_user21.txt',\n",
       " 'data/RawData/acc_exp44_user22.txt',\n",
       " 'data/RawData/acc_exp45_user22.txt',\n",
       " 'data/RawData/acc_exp46_user23.txt',\n",
       " 'data/RawData/acc_exp47_user23.txt',\n",
       " 'data/RawData/acc_exp48_user24.txt',\n",
       " 'data/RawData/acc_exp49_user24.txt',\n",
       " 'data/RawData/acc_exp50_user25.txt',\n",
       " 'data/RawData/acc_exp51_user25.txt',\n",
       " 'data/RawData/acc_exp52_user26.txt',\n",
       " 'data/RawData/acc_exp53_user26.txt',\n",
       " 'data/RawData/acc_exp54_user27.txt',\n",
       " 'data/RawData/acc_exp55_user27.txt',\n",
       " 'data/RawData/acc_exp56_user28.txt',\n",
       " 'data/RawData/acc_exp57_user28.txt',\n",
       " 'data/RawData/acc_exp58_user29.txt',\n",
       " 'data/RawData/acc_exp59_user29.txt',\n",
       " 'data/RawData/acc_exp60_user30.txt',\n",
       " 'data/RawData/acc_exp61_user30.txt',\n",
       " 'data/RawData/gyro_exp01_user01.txt',\n",
       " 'data/RawData/gyro_exp02_user01.txt',\n",
       " 'data/RawData/gyro_exp03_user02.txt',\n",
       " 'data/RawData/gyro_exp04_user02.txt',\n",
       " 'data/RawData/gyro_exp05_user03.txt',\n",
       " 'data/RawData/gyro_exp06_user03.txt',\n",
       " 'data/RawData/gyro_exp07_user04.txt',\n",
       " 'data/RawData/gyro_exp08_user04.txt',\n",
       " 'data/RawData/gyro_exp09_user05.txt',\n",
       " 'data/RawData/gyro_exp10_user05.txt',\n",
       " 'data/RawData/gyro_exp11_user06.txt',\n",
       " 'data/RawData/gyro_exp12_user06.txt',\n",
       " 'data/RawData/gyro_exp13_user07.txt',\n",
       " 'data/RawData/gyro_exp14_user07.txt',\n",
       " 'data/RawData/gyro_exp15_user08.txt',\n",
       " 'data/RawData/gyro_exp16_user08.txt',\n",
       " 'data/RawData/gyro_exp17_user09.txt',\n",
       " 'data/RawData/gyro_exp18_user09.txt',\n",
       " 'data/RawData/gyro_exp19_user10.txt',\n",
       " 'data/RawData/gyro_exp20_user10.txt',\n",
       " 'data/RawData/gyro_exp21_user10.txt',\n",
       " 'data/RawData/gyro_exp22_user11.txt',\n",
       " 'data/RawData/gyro_exp23_user11.txt',\n",
       " 'data/RawData/gyro_exp24_user12.txt',\n",
       " 'data/RawData/gyro_exp25_user12.txt',\n",
       " 'data/RawData/gyro_exp26_user13.txt',\n",
       " 'data/RawData/gyro_exp27_user13.txt',\n",
       " 'data/RawData/gyro_exp28_user14.txt',\n",
       " 'data/RawData/gyro_exp29_user14.txt',\n",
       " 'data/RawData/gyro_exp30_user15.txt',\n",
       " 'data/RawData/gyro_exp31_user15.txt',\n",
       " 'data/RawData/gyro_exp32_user16.txt',\n",
       " 'data/RawData/gyro_exp33_user16.txt',\n",
       " 'data/RawData/gyro_exp34_user17.txt',\n",
       " 'data/RawData/gyro_exp35_user17.txt',\n",
       " 'data/RawData/gyro_exp36_user18.txt',\n",
       " 'data/RawData/gyro_exp37_user18.txt',\n",
       " 'data/RawData/gyro_exp38_user19.txt',\n",
       " 'data/RawData/gyro_exp39_user19.txt',\n",
       " 'data/RawData/gyro_exp40_user20.txt',\n",
       " 'data/RawData/gyro_exp41_user20.txt',\n",
       " 'data/RawData/gyro_exp42_user21.txt',\n",
       " 'data/RawData/gyro_exp43_user21.txt',\n",
       " 'data/RawData/gyro_exp44_user22.txt',\n",
       " 'data/RawData/gyro_exp45_user22.txt',\n",
       " 'data/RawData/gyro_exp46_user23.txt',\n",
       " 'data/RawData/gyro_exp47_user23.txt',\n",
       " 'data/RawData/gyro_exp48_user24.txt',\n",
       " 'data/RawData/gyro_exp49_user24.txt',\n",
       " 'data/RawData/gyro_exp50_user25.txt',\n",
       " 'data/RawData/gyro_exp51_user25.txt',\n",
       " 'data/RawData/gyro_exp52_user26.txt',\n",
       " 'data/RawData/gyro_exp53_user26.txt',\n",
       " 'data/RawData/gyro_exp54_user27.txt',\n",
       " 'data/RawData/gyro_exp55_user27.txt',\n",
       " 'data/RawData/gyro_exp56_user28.txt',\n",
       " 'data/RawData/gyro_exp57_user28.txt',\n",
       " 'data/RawData/gyro_exp58_user29.txt',\n",
       " 'data/RawData/gyro_exp59_user29.txt',\n",
       " 'data/RawData/gyro_exp60_user30.txt',\n",
       " 'data/RawData/gyro_exp61_user30.txt']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files.sort()\n",
    "files.pop() # getting rid of the activity labels file\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dc9b8a6-c831-45d0-9231-0273bf22e1ae",
   "metadata": {},
   "source": [
    "Now, we can create new dataframes with each of the files and add the associated activity labels to each of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2f8ab685-2a0d-4c59-8ce1-5119ce4df075",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>experiment</th>\n",
       "      <th>user</th>\n",
       "      <th>activity</th>\n",
       "      <th>row_start</th>\n",
       "      <th>row_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>250</td>\n",
       "      <td>1232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>1233</td>\n",
       "      <td>1392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1393</td>\n",
       "      <td>2194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>2195</td>\n",
       "      <td>2359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2360</td>\n",
       "      <td>3374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1209</th>\n",
       "      <td>61</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>13842</td>\n",
       "      <td>14574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1210</th>\n",
       "      <td>61</td>\n",
       "      <td>30</td>\n",
       "      <td>3</td>\n",
       "      <td>14751</td>\n",
       "      <td>15427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1211</th>\n",
       "      <td>61</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>15588</td>\n",
       "      <td>16319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1212</th>\n",
       "      <td>61</td>\n",
       "      <td>30</td>\n",
       "      <td>3</td>\n",
       "      <td>16546</td>\n",
       "      <td>17250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1213</th>\n",
       "      <td>61</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>17394</td>\n",
       "      <td>18097</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1214 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      experiment  user  activity  row_start  row_end\n",
       "0              1     1         5        250     1232\n",
       "1              1     1         7       1233     1392\n",
       "2              1     1         4       1393     2194\n",
       "3              1     1         8       2195     2359\n",
       "4              1     1         5       2360     3374\n",
       "...          ...   ...       ...        ...      ...\n",
       "1209          61    30         2      13842    14574\n",
       "1210          61    30         3      14751    15427\n",
       "1211          61    30         2      15588    16319\n",
       "1212          61    30         3      16546    17250\n",
       "1213          61    30         2      17394    18097\n",
       "\n",
       "[1214 rows x 5 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "activities_columns = ['experiment','user','activity','row_start','row_end']\n",
    "activities = pd.read_csv('data/RawData/labels.txt', sep=' ', names = activities_columns)\n",
    "activities"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "341f9fd6-6f1f-404b-989f-0831d8fb02fe",
   "metadata": {},
   "source": [
    "We are going to have to loop through each file when we create the dataframe and add the associated activities to the dataframe by checking if the experiment and users match, then we'll choose the rows and column from the `activities` dataframe and label the rows that way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "882e6fab-62ba-4fd7-8333-64cfb701fb37",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc | experiment:01  user: 01\n",
      "acc | experiment:02  user: 01\n",
      "acc | experiment:03  user: 02\n",
      "acc | experiment:04  user: 02\n",
      "acc | experiment:05  user: 03\n",
      "acc | experiment:06  user: 03\n",
      "acc | experiment:07  user: 04\n",
      "acc | experiment:08  user: 04\n",
      "acc | experiment:09  user: 05\n",
      "acc | experiment:10  user: 05\n",
      "acc | experiment:11  user: 06\n",
      "acc | experiment:12  user: 06\n",
      "acc | experiment:13  user: 07\n",
      "acc | experiment:14  user: 07\n",
      "acc | experiment:15  user: 08\n",
      "acc | experiment:16  user: 08\n",
      "acc | experiment:17  user: 09\n",
      "acc | experiment:18  user: 09\n",
      "acc | experiment:19  user: 10\n",
      "acc | experiment:20  user: 10\n",
      "acc | experiment:21  user: 10\n",
      "acc | experiment:22  user: 11\n",
      "acc | experiment:23  user: 11\n",
      "acc | experiment:24  user: 12\n",
      "acc | experiment:25  user: 12\n",
      "acc | experiment:26  user: 13\n",
      "acc | experiment:27  user: 13\n",
      "acc | experiment:28  user: 14\n",
      "acc | experiment:29  user: 14\n",
      "acc | experiment:30  user: 15\n",
      "acc | experiment:31  user: 15\n",
      "acc | experiment:32  user: 16\n",
      "acc | experiment:33  user: 16\n",
      "acc | experiment:34  user: 17\n",
      "acc | experiment:35  user: 17\n",
      "acc | experiment:36  user: 18\n",
      "acc | experiment:37  user: 18\n",
      "acc | experiment:38  user: 19\n",
      "acc | experiment:39  user: 19\n",
      "acc | experiment:40  user: 20\n",
      "acc | experiment:41  user: 20\n",
      "acc | experiment:42  user: 21\n",
      "acc | experiment:43  user: 21\n",
      "acc | experiment:44  user: 22\n",
      "acc | experiment:45  user: 22\n",
      "acc | experiment:46  user: 23\n",
      "acc | experiment:47  user: 23\n",
      "acc | experiment:48  user: 24\n",
      "acc | experiment:49  user: 24\n",
      "acc | experiment:50  user: 25\n",
      "acc | experiment:51  user: 25\n",
      "acc | experiment:52  user: 26\n",
      "acc | experiment:53  user: 26\n",
      "acc | experiment:54  user: 27\n",
      "acc | experiment:55  user: 27\n",
      "acc | experiment:56  user: 28\n",
      "acc | experiment:57  user: 28\n",
      "acc | experiment:58  user: 29\n",
      "acc | experiment:59  user: 29\n",
      "acc | experiment:60  user: 30\n",
      "acc | experiment:61  user: 30\n",
      "gyro | experiment:01  user: 01\n",
      "gyro | experiment:02  user: 01\n",
      "gyro | experiment:03  user: 02\n",
      "gyro | experiment:04  user: 02\n",
      "gyro | experiment:05  user: 03\n",
      "gyro | experiment:06  user: 03\n",
      "gyro | experiment:07  user: 04\n",
      "gyro | experiment:08  user: 04\n",
      "gyro | experiment:09  user: 05\n",
      "gyro | experiment:10  user: 05\n",
      "gyro | experiment:11  user: 06\n",
      "gyro | experiment:12  user: 06\n",
      "gyro | experiment:13  user: 07\n",
      "gyro | experiment:14  user: 07\n",
      "gyro | experiment:15  user: 08\n",
      "gyro | experiment:16  user: 08\n",
      "gyro | experiment:17  user: 09\n",
      "gyro | experiment:18  user: 09\n",
      "gyro | experiment:19  user: 10\n",
      "gyro | experiment:20  user: 10\n",
      "gyro | experiment:21  user: 10\n",
      "gyro | experiment:22  user: 11\n",
      "gyro | experiment:23  user: 11\n",
      "gyro | experiment:24  user: 12\n",
      "gyro | experiment:25  user: 12\n",
      "gyro | experiment:26  user: 13\n",
      "gyro | experiment:27  user: 13\n",
      "gyro | experiment:28  user: 14\n",
      "gyro | experiment:29  user: 14\n",
      "gyro | experiment:30  user: 15\n",
      "gyro | experiment:31  user: 15\n",
      "gyro | experiment:32  user: 16\n",
      "gyro | experiment:33  user: 16\n",
      "gyro | experiment:34  user: 17\n",
      "gyro | experiment:35  user: 17\n",
      "gyro | experiment:36  user: 18\n",
      "gyro | experiment:37  user: 18\n",
      "gyro | experiment:38  user: 19\n",
      "gyro | experiment:39  user: 19\n",
      "gyro | experiment:40  user: 20\n",
      "gyro | experiment:41  user: 20\n",
      "gyro | experiment:42  user: 21\n",
      "gyro | experiment:43  user: 21\n",
      "gyro | experiment:44  user: 22\n",
      "gyro | experiment:45  user: 22\n",
      "gyro | experiment:46  user: 23\n",
      "gyro | experiment:47  user: 23\n",
      "gyro | experiment:48  user: 24\n",
      "gyro | experiment:49  user: 24\n",
      "gyro | experiment:50  user: 25\n",
      "gyro | experiment:51  user: 25\n",
      "gyro | experiment:52  user: 26\n",
      "gyro | experiment:53  user: 26\n",
      "gyro | experiment:54  user: 27\n",
      "gyro | experiment:55  user: 27\n",
      "gyro | experiment:56  user: 28\n",
      "gyro | experiment:57  user: 28\n",
      "gyro | experiment:58  user: 29\n",
      "gyro | experiment:59  user: 29\n",
      "gyro | experiment:60  user: 30\n",
      "gyro | experiment:61  user: 30\n"
     ]
    }
   ],
   "source": [
    "# This for loop shows how to extract the user number and experiment number from the file name\n",
    "for i in files:\n",
    "    if 'acc' in i:\n",
    "        print(f'acc | experiment:{i[20:22]}  user: {i[-6:-4]}')\n",
    "    if 'gyro' in i:\n",
    "        print(f'gyro | experiment:{i[21:23]}  user: {i[-6:-4]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ebfb036-313b-4f31-bc61-eb122054340b",
   "metadata": {},
   "source": [
    "Now, we'll create individual dataframes for each of the files and add the activity labels.\n",
    "\n",
    "It sounds simple, but there's a level of complexity to it. Let me explain.\n",
    "\n",
    "First, we'll start off by instantiating our `main` sensor dataframes. These will be used to hold all the file data combined together. To make sure the files for both the gyroscope and accelerometer data are compatible to be concatenated together, and are loaded in the same sequential order, we'll instantiate some empty lists.\n",
    "\n",
    "The code below will loop through all the file names obtained above in the `files` variable, and if the name contains `acc` or `gyro`, we'll add the file data to their respective dataframes. During this process, we'll also get some information about the file, such as the experiment number, user id, number of rows and columns for each file. The purpose of this is to match each experiment, user and number of rows for both the accelerometer and gyroscope data, so that when we combine both dataframes everything will be in sync, as long as there are no row discrepancies.\n",
    "\n",
    "**Note:** file data is stored in temp_df\n",
    "\n",
    "When looping through the files, we'll check the experiment number and user id with the activity label dataframe in order to obtain the rows associated with each label. There will be more than one activity label for each experiment, so we get all the indicies of these labels and loop through these to find the correct data to label our file data with. \n",
    "\n",
    "Then, after all the data is labeled correctly, we'll add the file data into the main dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5331b157-335b-4bd3-8f80-f7ca843d6ceb",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc | experiment:01  user: 01\n",
      "acc | experiment:02  user: 01\n",
      "acc | experiment:03  user: 02\n",
      "acc | experiment:04  user: 02\n",
      "acc | experiment:05  user: 03\n",
      "acc | experiment:06  user: 03\n",
      "acc | experiment:07  user: 04\n",
      "acc | experiment:08  user: 04\n",
      "acc | experiment:09  user: 05\n",
      "acc | experiment:10  user: 05\n",
      "acc | experiment:11  user: 06\n",
      "acc | experiment:12  user: 06\n",
      "acc | experiment:13  user: 07\n",
      "acc | experiment:14  user: 07\n",
      "acc | experiment:15  user: 08\n",
      "acc | experiment:16  user: 08\n",
      "acc | experiment:17  user: 09\n",
      "acc | experiment:18  user: 09\n",
      "acc | experiment:19  user: 10\n",
      "acc | experiment:20  user: 10\n",
      "acc | experiment:21  user: 10\n",
      "acc | experiment:22  user: 11\n",
      "acc | experiment:23  user: 11\n",
      "acc | experiment:24  user: 12\n",
      "acc | experiment:25  user: 12\n",
      "acc | experiment:26  user: 13\n",
      "acc | experiment:27  user: 13\n",
      "acc | experiment:28  user: 14\n",
      "acc | experiment:29  user: 14\n",
      "acc | experiment:30  user: 15\n",
      "acc | experiment:31  user: 15\n",
      "acc | experiment:32  user: 16\n",
      "acc | experiment:33  user: 16\n",
      "acc | experiment:34  user: 17\n",
      "acc | experiment:35  user: 17\n",
      "acc | experiment:36  user: 18\n",
      "acc | experiment:37  user: 18\n",
      "acc | experiment:38  user: 19\n",
      "acc | experiment:39  user: 19\n",
      "acc | experiment:40  user: 20\n",
      "acc | experiment:41  user: 20\n",
      "acc | experiment:42  user: 21\n",
      "acc | experiment:43  user: 21\n",
      "acc | experiment:44  user: 22\n",
      "acc | experiment:45  user: 22\n",
      "acc | experiment:46  user: 23\n",
      "acc | experiment:47  user: 23\n",
      "acc | experiment:48  user: 24\n",
      "acc | experiment:49  user: 24\n",
      "acc | experiment:50  user: 25\n",
      "acc | experiment:51  user: 25\n",
      "acc | experiment:52  user: 26\n",
      "acc | experiment:53  user: 26\n",
      "acc | experiment:54  user: 27\n",
      "acc | experiment:55  user: 27\n",
      "acc | experiment:56  user: 28\n",
      "acc | experiment:57  user: 28\n",
      "acc | experiment:58  user: 29\n",
      "acc | experiment:59  user: 29\n",
      "acc | experiment:60  user: 30\n",
      "acc | experiment:61  user: 30\n",
      "gyro | experiment:01  user: 01\n",
      "gyro | experiment:02  user: 01\n",
      "gyro | experiment:03  user: 02\n",
      "gyro | experiment:04  user: 02\n",
      "gyro | experiment:05  user: 03\n",
      "gyro | experiment:06  user: 03\n",
      "gyro | experiment:07  user: 04\n",
      "gyro | experiment:08  user: 04\n",
      "gyro | experiment:09  user: 05\n",
      "gyro | experiment:10  user: 05\n",
      "gyro | experiment:11  user: 06\n",
      "gyro | experiment:12  user: 06\n",
      "gyro | experiment:13  user: 07\n",
      "gyro | experiment:14  user: 07\n",
      "gyro | experiment:15  user: 08\n",
      "gyro | experiment:16  user: 08\n",
      "gyro | experiment:17  user: 09\n",
      "gyro | experiment:18  user: 09\n",
      "gyro | experiment:19  user: 10\n",
      "gyro | experiment:20  user: 10\n",
      "gyro | experiment:21  user: 10\n",
      "gyro | experiment:22  user: 11\n",
      "gyro | experiment:23  user: 11\n",
      "gyro | experiment:24  user: 12\n",
      "gyro | experiment:25  user: 12\n",
      "gyro | experiment:26  user: 13\n",
      "gyro | experiment:27  user: 13\n",
      "gyro | experiment:28  user: 14\n",
      "gyro | experiment:29  user: 14\n",
      "gyro | experiment:30  user: 15\n",
      "gyro | experiment:31  user: 15\n",
      "gyro | experiment:32  user: 16\n",
      "gyro | experiment:33  user: 16\n",
      "gyro | experiment:34  user: 17\n",
      "gyro | experiment:35  user: 17\n",
      "gyro | experiment:36  user: 18\n",
      "gyro | experiment:37  user: 18\n",
      "gyro | experiment:38  user: 19\n",
      "gyro | experiment:39  user: 19\n",
      "gyro | experiment:40  user: 20\n",
      "gyro | experiment:41  user: 20\n",
      "gyro | experiment:42  user: 21\n",
      "gyro | experiment:43  user: 21\n",
      "gyro | experiment:44  user: 22\n",
      "gyro | experiment:45  user: 22\n",
      "gyro | experiment:46  user: 23\n",
      "gyro | experiment:47  user: 23\n",
      "gyro | experiment:48  user: 24\n",
      "gyro | experiment:49  user: 24\n",
      "gyro | experiment:50  user: 25\n",
      "gyro | experiment:51  user: 25\n",
      "gyro | experiment:52  user: 26\n",
      "gyro | experiment:53  user: 26\n",
      "gyro | experiment:54  user: 27\n",
      "gyro | experiment:55  user: 27\n",
      "gyro | experiment:56  user: 28\n",
      "gyro | experiment:57  user: 28\n",
      "gyro | experiment:58  user: 29\n",
      "gyro | experiment:59  user: 29\n",
      "gyro | experiment:60  user: 30\n",
      "gyro | experiment:61  user: 30\n",
      "complete\n"
     ]
    }
   ],
   "source": [
    "# creating column names\n",
    "acc_features = ['acc_x','acc_y','acc_z','acc_activity','acc_user','acc_experiment']\n",
    "gyro_features = ['gyro_x','gyro_y','gyro_z','gyro_activity','gyro_user','gyro_experiment']\n",
    "\n",
    "# instantiate empty dataframe\n",
    "accelerometer = pd.DataFrame(columns = acc_features)\n",
    "gyroscope = pd.DataFrame(columns = gyro_features)\n",
    "\n",
    "# instantiate empty lists for file comparison\n",
    "acc_file_params = []\n",
    "gyro_file_params = []\n",
    "\n",
    "# looping through each file in folder\n",
    "for i in files:\n",
    "    \n",
    "    # if accelerometer experiment\n",
    "    if 'acc' in i:\n",
    "        \n",
    "        # extract user and experiment numbers\n",
    "        experiment = int(i[20:22])\n",
    "        user = int(i[-6:-4])\n",
    "        \n",
    "        # read in the file\n",
    "        temp_df = pd.read_csv(i, sep = ' ', names = acc_features)\n",
    "        \n",
    "        #gather some parameters for file comparison\n",
    "        temp_array = []\n",
    "        temp_array.append(experiment)\n",
    "        temp_array.append(user)\n",
    "        temp_array.append(temp_df.shape[0])\n",
    "        temp_array.append(temp_df.shape[1])\n",
    "        \n",
    "        #add parameters per file to main params list\n",
    "        acc_file_params.append(temp_array)\n",
    "        \n",
    "        # find the rows for the activity label\n",
    "        # by checking the user and experiment number\n",
    "        exp_check = (activities['experiment'] == experiment)\n",
    "        user_check = (activities['user'] == user)\n",
    "        print(f'acc | experiment:{i[20:22]}  user: {i[-6:-4]}')\n",
    "        \n",
    "        #grab the indices of the activity label DF\n",
    "        temp_index = activities[(exp_check) & (user_check)].index\n",
    "        \n",
    "        # loop through the indicies of the activity label DF\n",
    "        # to get the row numbers for each activity, and add the labels to the temp_df\n",
    "        for j in temp_index:\n",
    "            start = activities.loc[j]['row_start']\n",
    "            end = activities.loc[j]['row_end']\n",
    "            activity = activities.loc[j]['activity']\n",
    "            \n",
    "            temp_df['acc_activity'].loc[start:end] = activity\n",
    "            temp_df['acc_user'].loc[start:end] = user\n",
    "            temp_df['acc_experiment'].loc[start:end] = experiment\n",
    "            \n",
    "            # concatenate the temp_df to our main accelerometer df\n",
    "            accelerometer = pd.concat([accelerometer, temp_df], axis = 0)\n",
    "        \n",
    "\n",
    "    # if gyroscope experiment\n",
    "    elif 'gyro' in i:\n",
    "        \n",
    "        # extract user and experiment numbers\n",
    "        experiment = int(i[21:23])\n",
    "        user = int(i[-6:-4])\n",
    "        \n",
    "        # read in the file\n",
    "        temp_df = pd.read_csv(i, sep = ' ', names = gyro_features)\n",
    "        \n",
    "        #gather some parameters for file comparison\n",
    "        temp_array = []\n",
    "        temp_array.append(experiment)\n",
    "        temp_array.append(user)\n",
    "        temp_array.append(temp_df.shape[0])\n",
    "        temp_array.append(temp_df.shape[1])\n",
    "        \n",
    "        #add parameters per file to main params list\n",
    "        gyro_file_params.append(temp_array)\n",
    "        \n",
    "        # find the rows for the activity label\n",
    "        # by checking the user and experiment number\n",
    "        exp_check = (activities['experiment'] == experiment)\n",
    "        user_check = (activities['user'] == user)\n",
    "        print(f'gyro | experiment:{i[21:23]}  user: {i[-6:-4]}')\n",
    "        \n",
    "        #grab the indices of the activity label DF\n",
    "        temp_index = activities[(exp_check) & (user_check)].index\n",
    "        \n",
    "        # loop through the indicies of the activity label DF\n",
    "        # to get the row numbers for each activity, and add the labels to the temp_df\n",
    "        for j in temp_index:\n",
    "            start = activities.loc[j]['row_start']\n",
    "            end = activities.loc[j]['row_end']\n",
    "            activity = activities.loc[j]['activity']\n",
    "\n",
    "            temp_df['gyro_activity'].loc[start:end] = activity\n",
    "            temp_df['gyro_user'].loc[start:end] = user\n",
    "            temp_df['gyro_experiment'].loc[start:end] = experiment\n",
    "\n",
    "            # concatenate the temp_df to our main gyroscope df\n",
    "            gyroscope = pd.concat([gyroscope, temp_df], axis = 0)\n",
    "print('complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae2ea69-d406-44d5-a45c-6710927319fb",
   "metadata": {},
   "source": [
    "Wow, that was a lot of code, and a lot to take in. Let's double check our work!\n",
    "\n",
    "Remember our comparison lists? Let's take a look to see that all the files had the same number of rows and columns for their respective users and experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6695ebbe-cd79-4143-a65a-32d0e66ab304",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc_file_params == gyro_file_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8d729f18-ef30-41de-99f9-c6db18a2aa82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1, 1, 20598, 6],\n",
       " [2, 1, 19286, 6],\n",
       " [3, 2, 18026, 6],\n",
       " [4, 2, 16565, 6],\n",
       " [5, 3, 20994, 6],\n",
       " [6, 3, 17493, 6],\n",
       " [7, 4, 17668, 6],\n",
       " [8, 4, 15888, 6],\n",
       " [9, 5, 16864, 6],\n",
       " [10, 5, 15038, 6]]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc_file_params[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4392bf95-2bc4-4865-8260-2466fe2689fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1, 1, 20598, 6],\n",
       " [2, 1, 19286, 6],\n",
       " [3, 2, 18026, 6],\n",
       " [4, 2, 16565, 6],\n",
       " [5, 3, 20994, 6],\n",
       " [6, 3, 17493, 6],\n",
       " [7, 4, 17668, 6],\n",
       " [8, 4, 15888, 6],\n",
       " [9, 5, 16864, 6],\n",
       " [10, 5, 15038, 6]]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gyro_file_params[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "355c8107-34ee-462e-baf7-dbe00e738498",
   "metadata": {},
   "source": [
    "Everything looks like it's lining up properly, let's now take a look at the final dataframe before we concatenate everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7d8429e8-8cb2-457b-a96b-37765dd26a95",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>acc_x</th>\n",
       "      <th>acc_y</th>\n",
       "      <th>acc_z</th>\n",
       "      <th>acc_activity</th>\n",
       "      <th>acc_user</th>\n",
       "      <th>acc_experiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8461</th>\n",
       "      <td>0.034722</td>\n",
       "      <td>0.783333</td>\n",
       "      <td>0.629167</td>\n",
       "      <td>6.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>61.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2190</th>\n",
       "      <td>0.993056</td>\n",
       "      <td>0.120833</td>\n",
       "      <td>0.226389</td>\n",
       "      <td>4.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>59.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16287</th>\n",
       "      <td>-0.450000</td>\n",
       "      <td>0.759722</td>\n",
       "      <td>0.506944</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6391</th>\n",
       "      <td>1.015278</td>\n",
       "      <td>-0.105556</td>\n",
       "      <td>-0.094444</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8643</th>\n",
       "      <td>0.920833</td>\n",
       "      <td>-0.418056</td>\n",
       "      <td>-0.104167</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          acc_x     acc_y     acc_z  acc_activity  acc_user  acc_experiment\n",
       "8461   0.034722  0.783333  0.629167           6.0      30.0            61.0\n",
       "2190   0.993056  0.120833  0.226389           4.0      29.0            59.0\n",
       "16287 -0.450000  0.759722  0.506944           NaN       NaN             NaN\n",
       "6391   1.015278 -0.105556 -0.094444           NaN       NaN             NaN\n",
       "8643   0.920833 -0.418056 -0.104167           1.0       1.0             1.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accelerometer.sample(5, random_state = 42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "73e29e13-ff55-4f87-9020-5393caf1b481",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gyro_x</th>\n",
       "      <th>gyro_y</th>\n",
       "      <th>gyro_z</th>\n",
       "      <th>gyro_activity</th>\n",
       "      <th>gyro_user</th>\n",
       "      <th>gyro_experiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8461</th>\n",
       "      <td>0.003360</td>\n",
       "      <td>0.002749</td>\n",
       "      <td>-0.001222</td>\n",
       "      <td>6.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>61.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2190</th>\n",
       "      <td>0.002749</td>\n",
       "      <td>0.008858</td>\n",
       "      <td>-0.005498</td>\n",
       "      <td>4.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>59.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16287</th>\n",
       "      <td>-0.071166</td>\n",
       "      <td>-0.029016</td>\n",
       "      <td>0.069944</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6391</th>\n",
       "      <td>-0.001527</td>\n",
       "      <td>0.008858</td>\n",
       "      <td>-0.003054</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8643</th>\n",
       "      <td>0.743423</td>\n",
       "      <td>1.408655</td>\n",
       "      <td>-0.309709</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         gyro_x    gyro_y    gyro_z  gyro_activity  gyro_user  gyro_experiment\n",
       "8461   0.003360  0.002749 -0.001222            6.0       30.0             61.0\n",
       "2190   0.002749  0.008858 -0.005498            4.0       29.0             59.0\n",
       "16287 -0.071166 -0.029016  0.069944            NaN        NaN              NaN\n",
       "6391  -0.001527  0.008858 -0.003054            NaN        NaN              NaN\n",
       "8643   0.743423  1.408655 -0.309709            1.0        1.0              1.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gyroscope.sample(5, random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f152e0bc-63c9-4fb3-bb88-0da72003dedd",
   "metadata": {},
   "source": [
    "There seems to be some missing values to be taken care of, but overall, the columns look like they've been filled properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "af424794-c8b7-4fce-9c9a-5e06577b4300",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accelerometer dataframe has 22496259 rows and 6 columns\n",
      "The gyroscope dataframe has 22496259 rows and 6 columns\n"
     ]
    }
   ],
   "source": [
    "print(f'The accelerometer dataframe has {accelerometer.shape[0]} rows and {accelerometer.shape[1]} columns')\n",
    "print(f'The gyroscope dataframe has {gyroscope.shape[0]} rows and {gyroscope.shape[1]} columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3598a9ec-6bce-449a-965b-d07bde8d1eaf",
   "metadata": {},
   "source": [
    "Both the dataframes have the same number of rows, that leads us to believe that everything is good to combine together now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ecfa27f6-ebf9-4669-98b6-43457fc0a7b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.concat((gyroscope, accelerometer), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b54fec5-0869-48a4-a175-08c3682ffc95",
   "metadata": {},
   "source": [
    "Perfect, let's see if everything concatenated correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "14518550-8e9a-4213-866a-f3ad53561c59",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22496259, 12)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8e6c5810-2ea1-49a5-9b43-f19a08bc6388",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 22496259 entries, 0 to 19081\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Dtype  \n",
      "---  ------           -----  \n",
      " 0   gyro_x           float64\n",
      " 1   gyro_y           float64\n",
      " 2   gyro_z           float64\n",
      " 3   gyro_activity    float64\n",
      " 4   gyro_user        float64\n",
      " 5   gyro_experiment  float64\n",
      " 6   acc_x            float64\n",
      " 7   acc_y            float64\n",
      " 8   acc_z            float64\n",
      " 9   acc_activity     float64\n",
      " 10  acc_user         float64\n",
      " 11  acc_experiment   float64\n",
      "dtypes: float64(12)\n",
      "memory usage: 2.2 GB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1532e1d2-d4f3-411e-8785-aec5a8ce91e9",
   "metadata": {},
   "source": [
    "Nice! There's actually a few more things we can do for our data to slim it down a little.\n",
    "\n",
    "The user and experiment columns aren't necessary for our analysis, these columns were only added to check the compatibility of the individual sensor dataframes.\n",
    "\n",
    "The acc_activity and gyro_activity can also be combined into one column, so we're not repeating information. Then, we should have a total of 7 columns.\n",
    "\n",
    "We can also remove any rows with missing activity labels, these will not add any insights to our analysis. Duplicate values should be fine because movement patterns or static movements will most likely have some duplicate records over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "57e29b42-ce27-44ee-8fcd-5e2a30ab81ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 22496259 entries, 0 to 19081\n",
      "Data columns (total 8 columns):\n",
      " #   Column         Dtype  \n",
      "---  ------         -----  \n",
      " 0   gyro_x         float64\n",
      " 1   gyro_y         float64\n",
      " 2   gyro_z         float64\n",
      " 3   gyro_activity  float64\n",
      " 4   acc_x          float64\n",
      " 5   acc_y          float64\n",
      " 6   acc_z          float64\n",
      " 7   acc_activity   float64\n",
      "dtypes: float64(8)\n",
      "memory usage: 1.5 GB\n"
     ]
    }
   ],
   "source": [
    "#removing user and experiment columns\n",
    "data = data.drop(columns=['gyro_experiment','gyro_user','acc_experiment','acc_user'])\n",
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7085d646-7f1e-47f3-aa14-24971115acce",
   "metadata": {},
   "source": [
    "Let's get rid of the missing values now before we continue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "479761c2-7989-49b4-9dda-3d0a7f8561bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gyro_x                  0\n",
       "gyro_y                  0\n",
       "gyro_z                  0\n",
       "gyro_activity    13755959\n",
       "acc_x                   0\n",
       "acc_y                   0\n",
       "acc_z                   0\n",
       "acc_activity     13755959\n",
       "dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a38e712-0314-4c41-8135-af9de4532bd2",
   "metadata": {},
   "source": [
    "We have 13,755,959 missing values in both activity columns, if we subtract that from our 22,496,259 total rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "498d1089-54b6-4ae2-b5bd-509bd159c8ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "we'll have 8740300 rows left\n"
     ]
    }
   ],
   "source": [
    "print(f\"we'll have {22496259-13755959} rows left\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "36acae7a-5b8f-4603-aae6-a64ae2d62232",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dropping na rows\n",
    "data = data.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "aa91b475-2009-46d0-8ee8-6afd4012e133",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 8740300 entries, 250 to 18097\n",
      "Data columns (total 8 columns):\n",
      " #   Column         Dtype  \n",
      "---  ------         -----  \n",
      " 0   gyro_x         float64\n",
      " 1   gyro_y         float64\n",
      " 2   gyro_z         float64\n",
      " 3   gyro_activity  float64\n",
      " 4   acc_x          float64\n",
      " 5   acc_y          float64\n",
      " 6   acc_z          float64\n",
      " 7   acc_activity   float64\n",
      "dtypes: float64(8)\n",
      "memory usage: 600.1 MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce1e2629-8a1e-4afd-abf3-ff4cfb9fd4dd",
   "metadata": {},
   "source": [
    "Everything's looking good so far, we can see that we have the expected number of rows left."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f77e30cc-bdd5-4358-a977-68cedad2ea90",
   "metadata": {},
   "source": [
    "For combining the activity column, we can check to see if all the rows have the same activity label, if so, we can just use one row to fill in our new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ccb522f2-cf0a-4287-9dff-0870d84ed963",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gyro_x</th>\n",
       "      <th>gyro_y</th>\n",
       "      <th>gyro_z</th>\n",
       "      <th>gyro_activity</th>\n",
       "      <th>acc_x</th>\n",
       "      <th>acc_y</th>\n",
       "      <th>acc_z</th>\n",
       "      <th>acc_activity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [gyro_x, gyro_y, gyro_z, gyro_activity, acc_x, acc_y, acc_z, acc_activity]\n",
       "Index: []"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking if the columns contain same values\n",
    "data[data['gyro_activity'] != data['acc_activity']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25047f64-1bd6-4783-a786-fe4233459f84",
   "metadata": {},
   "source": [
    "The cell above is checking to see if there are any rows in the activity columns that are not aligned, and as we can see, there are no rows that have discrepant values. We can now safely use a single one of these columns as our `activity` column, and drop the individual sensor's activity columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "266fafda-1bef-4a62-b061-1c54e23338fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['activity'] = data['gyro_activity']\n",
    "data = data.drop(columns = ['gyro_activity','acc_activity'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6a35d0a4-705f-4008-8b8d-9be70634b4c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 8740300 entries, 250 to 18097\n",
      "Data columns (total 7 columns):\n",
      " #   Column    Dtype  \n",
      "---  ------    -----  \n",
      " 0   gyro_x    float64\n",
      " 1   gyro_y    float64\n",
      " 2   gyro_z    float64\n",
      " 3   acc_x     float64\n",
      " 4   acc_y     float64\n",
      " 5   acc_z     float64\n",
      " 6   activity  float64\n",
      "dtypes: float64(7)\n",
      "memory usage: 533.5 MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3ba30cd1-3a1c-49b0-96c8-8ca989cea8ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gyro_x</th>\n",
       "      <th>gyro_y</th>\n",
       "      <th>gyro_z</th>\n",
       "      <th>acc_x</th>\n",
       "      <th>acc_y</th>\n",
       "      <th>acc_z</th>\n",
       "      <th>activity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>250</th>\n",
       "      <td>-0.002749</td>\n",
       "      <td>-0.004276</td>\n",
       "      <td>0.002749</td>\n",
       "      <td>1.020833</td>\n",
       "      <td>-0.125000</td>\n",
       "      <td>0.105556</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251</th>\n",
       "      <td>-0.000305</td>\n",
       "      <td>-0.002138</td>\n",
       "      <td>0.006109</td>\n",
       "      <td>1.025000</td>\n",
       "      <td>-0.125000</td>\n",
       "      <td>0.101389</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252</th>\n",
       "      <td>0.012217</td>\n",
       "      <td>0.000916</td>\n",
       "      <td>-0.007330</td>\n",
       "      <td>1.020833</td>\n",
       "      <td>-0.125000</td>\n",
       "      <td>0.104167</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>253</th>\n",
       "      <td>0.011301</td>\n",
       "      <td>-0.001833</td>\n",
       "      <td>-0.006414</td>\n",
       "      <td>1.016667</td>\n",
       "      <td>-0.125000</td>\n",
       "      <td>0.108333</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>0.010996</td>\n",
       "      <td>-0.001527</td>\n",
       "      <td>-0.004887</td>\n",
       "      <td>1.018056</td>\n",
       "      <td>-0.127778</td>\n",
       "      <td>0.108333</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       gyro_x    gyro_y    gyro_z     acc_x     acc_y     acc_z  activity\n",
       "250 -0.002749 -0.004276  0.002749  1.020833 -0.125000  0.105556       5.0\n",
       "251 -0.000305 -0.002138  0.006109  1.025000 -0.125000  0.101389       5.0\n",
       "252  0.012217  0.000916 -0.007330  1.020833 -0.125000  0.104167       5.0\n",
       "253  0.011301 -0.001833 -0.006414  1.016667 -0.125000  0.108333       5.0\n",
       "254  0.010996 -0.001527 -0.004887  1.018056 -0.127778  0.108333       5.0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce495024-2c92-4edb-85ca-441ee3ab9bce",
   "metadata": {},
   "source": [
    "Let's also reset the indicies of the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4206ad54-2b30-4d12-9818-c8960131e994",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d97e8e1f-84c9-4a4d-ae43-176577c46931",
   "metadata": {},
   "source": [
    "Now that we've got our raw data all organized, we can feature engineer some columns for our analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28f76b7f-1d8b-4430-a35d-a265de7a40f1",
   "metadata": {},
   "source": [
    "We will create a csv file from the data that we have to continue in google colab in notebook 05 WebApp.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3d7309e-f354-46a1-99bb-2d39075b2ff2",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.to_csv('data.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cf5a62a-e0c2-40e5-82b9-23d3de888970",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "capstone",
   "language": "python",
   "name": "capstone"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}