Data Cleaning - IPL Case Study.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing the libraries\n",
    "import pandas as pd               # for data manipulation\n",
    "import numpy as np                # for mathematical operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reading the raw dataset\n",
    "df = pd.read_csv(\"Raw_Data.csv\", index_col=0)\n",
    "\n",
    "# displaying the dataset\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# displaying the info of the dataframe\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are missing values present in many of the columns. The datatypes of a few of the numerical columns like \"Number of balls bowled\", \"runs given\", etc. are also incorrect. \n",
    "\n",
    "Before that, few columns which do not add any information to the dataset or are redundant are removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# removing the columns\n",
    "df.drop(['Variable Tag', 'Name', 'Unnamed: 25', 'matches played', 'innings played.1'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unique values in all the columns are displayed to get a detailed look of the values stored in them individually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# iterating over the column names\n",
    "for col in df.columns:\n",
    "    # printing the column name\n",
    "    print(col)\n",
    "    # printing the unique values in that specified column\n",
    "    print(df[col].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few observations from the above output are:\n",
    "1. The highest score has an asterisk(*) at the end of some of the scores. This means that the batsman stayed not out at the end of that inning. Since a column 'no' is present which stores the number of matches in which the batsman stayed not out, this column can be cleaned by removing the asterisk and converting to a numeric datatype.\n",
    "2. The symbol '-' is used along with 'nan' to represent null values in many of the features like 'Number of balls bowled', 'runs given' etc. The '-' needs to be replaced with Python understandable null values. \n",
    "3. The columns 'Best inning bowling' and 'best match bowling' store impossible and incorrect dates and hence, need to be removed. \n",
    "4. The column '10 wickets haul' stores 0, '-' and nan, or no useful information, and can be removed.\n",
    "\n",
    "All these issues are corrected in the next few steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replacing all the '-' with np.nan\n",
    "df.replace('-', np.nan, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no. of missing values in each of the columns\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# filling all the missing values with 0\n",
    "df.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# removing all the unnecessary columns\n",
    "df.drop(['10 wicket hauls', 'best match bowling', 'Best innings bowling'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unique values of the 'highest score' column\n",
    "df['highest score'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# removing the asterisk and converting the datatype\n",
    "df['highest score'] = [int(score.strip('*')) for score in df['highest score']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unique values of the 'highest score' column\n",
    "df['highest score'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# numerical variables which have incorrect datatype\n",
    "var = ['Number of balls bowled', 'runs given', 'wkts taken', 'Bowling econ', 'sr', '4w', '5w']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting the datatype of the above variables\n",
    "df[var] = df[var].astype('float')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving the cleaned dataset\n",
    "df.to_csv(\"Clean_Data.csv\", index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}