Pandas_File_Format_Comparison.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMTRlFRDnZQlCEmBOz0f59v",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/satyakisen/pandas-ff-comparison/blob/main/Pandas_File_Format_Comparison.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Pandas File Format Comparision\n",
        "## Overview\n",
        "In this notebook let us compare the following pandas file formats.\n",
        "1. csv - common text file that is comma seperated.\n",
        "2. hdf5 - an open source file format that supports large, complex, heterogeneous data\n",
        "3. parquet - an open source, column-oriented data file format designed for efficient data storage and retrieval.\n",
        "4. feather - a portable file format for storing Arrow tables or data frames (from languages like Python or R) that utilizes the Arrow IPC format internally\n",
        "\n",
        "Comparision parameters to consider are:\n",
        "1. Time to write.\n",
        "2. Time to read.\n",
        "3. File size on disk.\n",
        "4. Memory Usage.\n"
      ],
      "metadata": {
        "id": "97FNsEkHjZaj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Testing with Numerical Data\n",
        "Let us begin by creating a dummy dataset containing only random float."
      ],
      "metadata": {
        "id": "7vs4sgxDYiBh"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "Cz0sTNA1jHuL"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "def make_data(row_n, col_n):\n",
        "  arr = np.random.randn(row_n, col_n)\n",
        "  df = pd.DataFrame(arr, columns=['col_{0}'.format(i) for i in range(col_n)])\n",
        "  return df\n",
        "\n",
        "df = make_data(100000, 10)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let us check the dummy dataset we made."
      ],
      "metadata": {
        "id": "Hw9cgrxjquTr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.head(5)"
      ],
      "metadata": {
        "id": "ovnaAuM2smjW",
        "outputId": "74232917-8337-45bf-fa63-35633eecc70f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      col_0     col_1     col_2     col_3     col_4     col_5     col_6  \\\n",
              "0  1.641336 -1.621931  0.796686 -1.679665  0.435253 -0.197769 -2.721646   \n",
              "1  0.153340 -0.069513 -0.748609  1.237218  0.644955  0.323406  0.669177   \n",
              "2  0.649113  1.348819  1.938892  0.614601  0.485035  0.352973 -0.306332   \n",
              "3  0.404498 -1.440968 -0.385486  0.441557 -0.036578 -0.293221  1.587384   \n",
              "4  0.767174  1.118811  0.021381 -0.216749  0.331357  0.790685 -1.101796   \n",
              "\n",
              "      col_7     col_8     col_9  \n",
              "0  0.343617  0.702246  0.433517  \n",
              "1 -0.611342 -0.503564 -0.778894  \n",
              "2 -0.079343 -1.674584 -1.566218  \n",
              "3  0.709433 -0.395009 -1.840616  \n",
              "4 -0.090205  0.595651 -0.359788  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-16c14e9d-3353-4ac3-943d-e6941f8adaed\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>col_0</th>\n",
              "      <th>col_1</th>\n",
              "      <th>col_2</th>\n",
              "      <th>col_3</th>\n",
              "      <th>col_4</th>\n",
              "      <th>col_5</th>\n",
              "      <th>col_6</th>\n",
              "      <th>col_7</th>\n",
              "      <th>col_8</th>\n",
              "      <th>col_9</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.641336</td>\n",
              "      <td>-1.621931</td>\n",
              "      <td>0.796686</td>\n",
              "      <td>-1.679665</td>\n",
              "      <td>0.435253</td>\n",
              "      <td>-0.197769</td>\n",
              "      <td>-2.721646</td>\n",
              "      <td>0.343617</td>\n",
              "      <td>0.702246</td>\n",
              "      <td>0.433517</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.153340</td>\n",
              "      <td>-0.069513</td>\n",
              "      <td>-0.748609</td>\n",
              "      <td>1.237218</td>\n",
              "      <td>0.644955</td>\n",
              "      <td>0.323406</td>\n",
              "      <td>0.669177</td>\n",
              "      <td>-0.611342</td>\n",
              "      <td>-0.503564</td>\n",
              "      <td>-0.778894</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.649113</td>\n",
              "      <td>1.348819</td>\n",
              "      <td>1.938892</td>\n",
              "      <td>0.614601</td>\n",
              "      <td>0.485035</td>\n",
              "      <td>0.352973</td>\n",
              "      <td>-0.306332</td>\n",
              "      <td>-0.079343</td>\n",
              "      <td>-1.674584</td>\n",
              "      <td>-1.566218</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.404498</td>\n",
              "      <td>-1.440968</td>\n",
              "      <td>-0.385486</td>\n",
              "      <td>0.441557</td>\n",
              "      <td>-0.036578</td>\n",
              "      <td>-0.293221</td>\n",
              "      <td>1.587384</td>\n",
              "      <td>0.709433</td>\n",
              "      <td>-0.395009</td>\n",
              "      <td>-1.840616</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.767174</td>\n",
              "      <td>1.118811</td>\n",
              "      <td>0.021381</td>\n",
              "      <td>-0.216749</td>\n",
              "      <td>0.331357</td>\n",
              "      <td>0.790685</td>\n",
              "      <td>-1.101796</td>\n",
              "      <td>-0.090205</td>\n",
              "      <td>0.595651</td>\n",
              "      <td>-0.359788</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-16c14e9d-3353-4ac3-943d-e6941f8adaed')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-16c14e9d-3353-4ac3-943d-e6941f8adaed button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-16c14e9d-3353-4ac3-943d-e6941f8adaed');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Time to write\n",
        "\n",
        "Let us now check the writing time of the above dataframe we created. We will first create a decorator for calculating time and then write a function to save the dataframe into different file formats."
      ],
      "metadata": {
        "id": "rPkE4jkEst7x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_csv('test.csv', index=False)"
      ],
      "metadata": {
        "id": "1Iz7XFyoudXQ",
        "outputId": "6e481b23-aac2-4982-b449-82c749d5376d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 1.59 s, sys: 69.5 ms, total: 1.66 s\n",
            "Wall time: 1.74 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_hdf('test.h5', key='root')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nRxSLmokctIG",
        "outputId": "652e5247-c170-4c18-e42d-4d6d5e7193db"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 81.1 ms, sys: 19.9 ms, total: 101 ms\n",
            "Wall time: 267 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_parquet('test.parquet')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DFRHG-Cac4Fp",
        "outputId": "1a89f420-68e3-4638-b2c9-61793dd99c8a"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 208 ms, sys: 38.6 ms, total: 246 ms\n",
            "Wall time: 510 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_feather('test.feather')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XnU1Sp8ndHNI",
        "outputId": "2b4aa21e-89ff-46ab-f703-27ab03208c1d"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 30.9 ms, sys: 21.1 ms, total: 51.9 ms\n",
            "Wall time: 77.2 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "From above we can see that **feather** & **hdf** are the file formats which works pretty well for writing to disk. Now let us consider the second parameter.\n",
        "\n",
        "### File size on Disk\n",
        "Let us now check the file size on disk."
      ],
      "metadata": {
        "id": "uUtar4J_dP3v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "du -sh test.*"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3CBJ1VPHdzaz",
        "outputId": "8ed5f920-982c-45fe-cb93-dfb8ee20e0f8"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "19M\ttest.csv\n",
            "7.7M\ttest.feather\n",
            "8.5M\ttest.h5\n",
            "9.7M\ttest.parquet\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "From the above result we can see that **feather** and **hdf** outperforms others. But we will check the performance again with some big data afterward.\n",
        "\n",
        "### Time to read\n",
        "Let us check how much time does it take for a read operation."
      ],
      "metadata": {
        "id": "IH90ZALweQEA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time \n",
        "df_csv=pd.read_csv('test.csv')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jKZ2CLMAetEP",
        "outputId": "ae03f1b3-66e1-4460-86ac-fba87647d3cb"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 217 ms, sys: 36.9 ms, total: 254 ms\n",
            "Wall time: 254 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_hdf=pd.read_hdf('test.h5')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KnHr0FaHfJbD",
        "outputId": "37d5f5d1-5799-4bb1-85f5-d0cd0bedcbd0"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 17.1 ms, sys: 9.94 ms, total: 27 ms\n",
            "Wall time: 27.4 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_parquet=pd.read_parquet('test.parquet')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IoIcukvlfUdh",
        "outputId": "bb925653-d96e-4cb6-8bd7-bb0de41c58af"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 31.3 ms, sys: 46.3 ms, total: 77.6 ms\n",
            "Wall time: 99.4 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_feather=pd.read_feather('test.feather')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L3ownw65fZsC",
        "outputId": "5e804fb3-ee0a-4332-e0fe-51fd595f4f2c"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 16.5 ms, sys: 7.1 ms, total: 23.6 ms\n",
            "Wall time: 18 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Memory Usage"
      ],
      "metadata": {
        "id": "D9T7IF5gZv28"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_csv.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2UPHXAEPZ_--",
        "outputId": "b1facd56-c27e-4790-8aab-b7d0cc146a21"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 100000 entries, 0 to 99999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Non-Null Count   Dtype  \n",
            "---  ------  --------------   -----  \n",
            " 0   col_0   100000 non-null  float64\n",
            " 1   col_1   100000 non-null  float64\n",
            " 2   col_2   100000 non-null  float64\n",
            " 3   col_3   100000 non-null  float64\n",
            " 4   col_4   100000 non-null  float64\n",
            " 5   col_5   100000 non-null  float64\n",
            " 6   col_6   100000 non-null  float64\n",
            " 7   col_7   100000 non-null  float64\n",
            " 8   col_8   100000 non-null  float64\n",
            " 9   col_9   100000 non-null  float64\n",
            "dtypes: float64(10)\n",
            "memory usage: 7.6 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_hdf.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OeHoPg6JaJen",
        "outputId": "befe8ded-7a4f-4ac2-b767-cc0cc2043720"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 100000 entries, 0 to 99999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Non-Null Count   Dtype  \n",
            "---  ------  --------------   -----  \n",
            " 0   col_0   100000 non-null  float64\n",
            " 1   col_1   100000 non-null  float64\n",
            " 2   col_2   100000 non-null  float64\n",
            " 3   col_3   100000 non-null  float64\n",
            " 4   col_4   100000 non-null  float64\n",
            " 5   col_5   100000 non-null  float64\n",
            " 6   col_6   100000 non-null  float64\n",
            " 7   col_7   100000 non-null  float64\n",
            " 8   col_8   100000 non-null  float64\n",
            " 9   col_9   100000 non-null  float64\n",
            "dtypes: float64(10)\n",
            "memory usage: 8.4 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_parquet.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AMMSuClnatj6",
        "outputId": "8be37b5f-da49-4cf1-a42e-e618ca5b0993"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 100000 entries, 0 to 99999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Non-Null Count   Dtype  \n",
            "---  ------  --------------   -----  \n",
            " 0   col_0   100000 non-null  float64\n",
            " 1   col_1   100000 non-null  float64\n",
            " 2   col_2   100000 non-null  float64\n",
            " 3   col_3   100000 non-null  float64\n",
            " 4   col_4   100000 non-null  float64\n",
            " 5   col_5   100000 non-null  float64\n",
            " 6   col_6   100000 non-null  float64\n",
            " 7   col_7   100000 non-null  float64\n",
            " 8   col_8   100000 non-null  float64\n",
            " 9   col_9   100000 non-null  float64\n",
            "dtypes: float64(10)\n",
            "memory usage: 7.6 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_feather.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OxDZeXeia0V3",
        "outputId": "25a8334b-8d41-489c-a043-4138c19e0435"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 100000 entries, 0 to 99999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Non-Null Count   Dtype  \n",
            "---  ------  --------------   -----  \n",
            " 0   col_0   100000 non-null  float64\n",
            " 1   col_1   100000 non-null  float64\n",
            " 2   col_2   100000 non-null  float64\n",
            " 3   col_3   100000 non-null  float64\n",
            " 4   col_4   100000 non-null  float64\n",
            " 5   col_5   100000 non-null  float64\n",
            " 6   col_6   100000 non-null  float64\n",
            " 7   col_7   100000 non-null  float64\n",
            " 8   col_8   100000 non-null  float64\n",
            " 9   col_9   100000 non-null  float64\n",
            "dtypes: float64(10)\n",
            "memory usage: 7.6 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Testing with categorical data.\n",
        "Now let us create data with both the categorical and numerical values and check the performances of different file formats."
      ],
      "metadata": {
        "id": "GQ7qXsuifhn_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install lorem"
      ],
      "metadata": {
        "id": "VkD23bbVg1rw",
        "outputId": "6af8b09f-4b62-4fa8-cde2-3c00c84e57ce",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting lorem\n",
            "  Downloading lorem-0.1.1-py3-none-any.whl (5.0 kB)\n",
            "Installing collected packages: lorem\n",
            "Successfully installed lorem-0.1.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Prepare Data"
      ],
      "metadata": {
        "id": "gzCGLidZjyFQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from lorem import sentence\n",
        "\n",
        "words = np.array(sentence().strip().lower().replace(\".\", \" \").split())\n",
        "\n",
        "np.random.seed(0)  \n",
        "n = 5000000\n",
        "df = pd.DataFrame(np.c_[np.random.randn(n, 5),\n",
        "                  np.random.randint(0,10,(n, 2)),\n",
        "                  np.random.randint(0,1,(n, 2)),\n",
        "np.array([np.random.choice(words) for i in range(n)])] , \n",
        "columns=list('ABCDEFGHIJ'))\n",
        "\n",
        "df=df.astype(dtype={'A': float, 'B': float, 'C': float, 'D': float, 'E': float, 'F': int, 'G': int, 'H': int, 'I': int, 'J': str}, copy=True)\n",
        "df.loc[::10, 'A'] = np.NaN\n",
        "len(df)"
      ],
      "metadata": {
        "id": "NrBYk7PdgYxK",
        "outputId": "10fff12b-3305-43cd-8395-387ba1b9a126",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5000000"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Time to write"
      ],
      "metadata": {
        "id": "HgM5-O4bj1mi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_csv('test_big.csv', index=False)"
      ],
      "metadata": {
        "id": "P3-sl79VhEd1",
        "outputId": "519551ad-5daf-45b0-9282-4a589f70f999",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 44.9 s, sys: 941 ms, total: 45.9 s\n",
            "Wall time: 46.1 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_hdf('test_big.h5', key='root', index=False)"
      ],
      "metadata": {
        "id": "GRrmh2jkjUw1",
        "outputId": "ccfb34ff-dceb-49cf-f786-1e6cdad9a5f6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 2.25 s, sys: 1.25 s, total: 3.5 s\n",
            "Wall time: 3.87 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_parquet('test_big.parquet', index=False)"
      ],
      "metadata": {
        "id": "Z-vCOXP4jgXE",
        "outputId": "2159cfd4-ec76-4b3a-fa4e-be2173c55f55",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 1.43 s, sys: 364 ms, total: 1.79 s\n",
            "Wall time: 1.77 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_feather('test_big.feather')"
      ],
      "metadata": {
        "id": "_fta9qUMjpHK",
        "outputId": "ea0a18f5-a65b-4ef4-d373-c3862a759687",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 965 ms, sys: 345 ms, total: 1.31 s\n",
            "Wall time: 1.15 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### File size on disk"
      ],
      "metadata": {
        "id": "q_6fAC0Bj7JN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "du -sh test_big.*"
      ],
      "metadata": {
        "id": "mW4OhVMsjvuj",
        "outputId": "fa5360e2-2ce9-4d42-aa42-9ef3349954d1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "530M\ttest_big.csv\n",
            "245M\ttest_big.feather\n",
            "425M\ttest_big.h5\n",
            "195M\ttest_big.parquet\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Time to read"
      ],
      "metadata": {
        "id": "Z9__uJEukjAx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time \n",
        "df_big_csv=pd.read_csv('test_big.csv')"
      ],
      "metadata": {
        "id": "zEXxFX8jkCX1",
        "outputId": "e037f5b9-5da6-42d9-89cb-9da8b3931805",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 6.07 s, sys: 10.2 s, total: 16.3 s\n",
            "Wall time: 16.3 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_big_hdf=pd.read_hdf('test_big.h5', key='root')"
      ],
      "metadata": {
        "id": "iijyDHY-ktn9",
        "outputId": "a55a92b0-3a38-43c0-ebae-dc772d062316",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 874 ms, sys: 620 ms, total: 1.49 s\n",
            "Wall time: 1.49 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_big_parquet=pd.read_parquet('test_big.parquet')"
      ],
      "metadata": {
        "id": "-1ts1znJk_XP",
        "outputId": "a0607ee7-e672-4935-e8fc-bf8478409049",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 542 ms, sys: 597 ms, total: 1.14 s\n",
            "Wall time: 841 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_big_feather=pd.read_feather('test_big.feather')"
      ],
      "metadata": {
        "id": "KqrcH_melJmw",
        "outputId": "115ef343-66b3-4705-8fdc-e76b6a441653",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 432 ms, sys: 534 ms, total: 966 ms\n",
            "Wall time: 648 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_parquet.dtypes"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q2ZaZUrvg15A",
        "outputId": "69d2fdcb-0270-4c39-f5c3-513c6d337c5b"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "A    float64\n",
              "B    float64\n",
              "C    float64\n",
              "D    float64\n",
              "E    float64\n",
              "F      int64\n",
              "G      int64\n",
              "H      int64\n",
              "I      int64\n",
              "J     object\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Memory Usage"
      ],
      "metadata": {
        "id": "vTpugGkxbnXb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_csv.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D6Acigzfbp9W",
        "outputId": "0bd98cfa-764a-4284-e4b0-0cc8d72cda8b"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5000000 entries, 0 to 4999999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Dtype  \n",
            "---  ------  -----  \n",
            " 0   A       float64\n",
            " 1   B       float64\n",
            " 2   C       float64\n",
            " 3   D       float64\n",
            " 4   E       float64\n",
            " 5   F       int64  \n",
            " 6   G       int64  \n",
            " 7   H       int64  \n",
            " 8   I       int64  \n",
            " 9   J       object \n",
            "dtypes: float64(5), int64(4), object(1)\n",
            "memory usage: 642.5 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_hdf.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AbB5GcoCcU2b",
        "outputId": "793cec08-1d1e-4bf4-f1e4-1ce511a87c5b"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 5000000 entries, 0 to 4999999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Dtype  \n",
            "---  ------  -----  \n",
            " 0   A       float64\n",
            " 1   B       float64\n",
            " 2   C       float64\n",
            " 3   D       float64\n",
            " 4   E       float64\n",
            " 5   F       int64  \n",
            " 6   G       int64  \n",
            " 7   H       int64  \n",
            " 8   I       int64  \n",
            " 9   J       object \n",
            "dtypes: float64(5), int64(4), object(1)\n",
            "memory usage: 680.7 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_parquet.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f1DWT9IXchbT",
        "outputId": "cec18d75-02a0-4be6-c0d1-346701ace27f"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5000000 entries, 0 to 4999999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Dtype  \n",
            "---  ------  -----  \n",
            " 0   A       float64\n",
            " 1   B       float64\n",
            " 2   C       float64\n",
            " 3   D       float64\n",
            " 4   E       float64\n",
            " 5   F       int64  \n",
            " 6   G       int64  \n",
            " 7   H       int64  \n",
            " 8   I       int64  \n",
            " 9   J       object \n",
            "dtypes: float64(5), int64(4), object(1)\n",
            "memory usage: 642.5 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_feather.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a4dcJf-_cosf",
        "outputId": "39f12cbc-2819-4444-90a4-9f72d328cf29"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5000000 entries, 0 to 4999999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Dtype  \n",
            "---  ------  -----  \n",
            " 0   A       float64\n",
            " 1   B       float64\n",
            " 2   C       float64\n",
            " 3   D       float64\n",
            " 4   E       float64\n",
            " 5   F       int64  \n",
            " 6   G       int64  \n",
            " 7   H       int64  \n",
            " 8   I       int64  \n",
            " 9   J       object \n",
            "dtypes: float64(5), int64(4), object(1)\n",
            "memory usage: 642.5 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see in the above cells that when we use raw categorical fields the size of the file gets large in case of feather and hdf. Now let us use the pd.Categorical function and modify the data and check what happens."
      ],
      "metadata": {
        "id": "ZkkWZG_NlcXt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df['J'] = pd.Categorical(df['J'])"
      ],
      "metadata": {
        "id": "H6mKI4K6lPY7"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.dtypes"
      ],
      "metadata": {
        "id": "a9R2_J4F8xdY",
        "outputId": "ad471965-dfd8-42a5-8edc-22524a6ca8d9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "A     float64\n",
              "B     float64\n",
              "C     float64\n",
              "D     float64\n",
              "E     float64\n",
              "F       int64\n",
              "G       int64\n",
              "H       int64\n",
              "I       int64\n",
              "J    category\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Time to write"
      ],
      "metadata": {
        "id": "lrW1nWFDl8Aq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_csv('test_big_pd_cat.csv')"
      ],
      "metadata": {
        "id": "NmTTty61qTn6",
        "outputId": "e4a68789-b220-432e-ec5a-b1f60af070fa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 47.3 s, sys: 910 ms, total: 48.3 s\n",
            "Wall time: 48.4 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_hdf('test_big_pd_cat.h5', key='root', format='table')"
      ],
      "metadata": {
        "id": "vyJ_aN6pl15B",
        "outputId": "9435e281-f8c5-4ad7-a2d1-d132506eddae",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 3.49 s, sys: 610 ms, total: 4.1 s\n",
            "Wall time: 4.38 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_parquet('test_big_pd_cat.parquet')"
      ],
      "metadata": {
        "id": "Vma2CNVKqwQo",
        "outputId": "ac9f8ed1-9f28-45b8-f8de-6e77b3607414",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 1e+03 ms, sys: 238 ms, total: 1.24 s\n",
            "Wall time: 1.25 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%time df.to_feather('test_big_pd_cat.feather')"
      ],
      "metadata": {
        "id": "RRojwlU2rI4Q",
        "outputId": "d105202e-0dec-4669-f68a-879fcf8042ff",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 699 ms, sys: 250 ms, total: 950 ms\n",
            "Wall time: 831 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### File size on disk"
      ],
      "metadata": {
        "id": "ILTWFCEorXw9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "du -sh test_big_pd_cat.*"
      ],
      "metadata": {
        "id": "Ee8iYOJHrOiC",
        "outputId": "ba67955a-8d88-4843-95e3-b104b231bd36",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "567M\ttest_big_pd_cat.csv\n",
            "219M\ttest_big_pd_cat.feather\n",
            "389M\ttest_big_pd_cat.h5\n",
            "195M\ttest_big_pd_cat.parquet\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Time to read"
      ],
      "metadata": {
        "id": "DxjMRro_reoc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_big_pd_cat_csv = pd.read_csv('test_big_pd_cat.csv')"
      ],
      "metadata": {
        "id": "XaKCawgqreSj",
        "outputId": "4b072a14-12d6-4b53-9b98-982dd65039a4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 7.51 s, sys: 1.2 s, total: 8.71 s\n",
            "Wall time: 8.78 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_big_pd_cat_hdf = pd.read_hdf('test_big_pd_cat.h5')"
      ],
      "metadata": {
        "id": "KdhKoHgErUwq",
        "outputId": "564f0aa9-fbad-49a1-8f0d-1f7c7c3a5eb1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 824 ms, sys: 309 ms, total: 1.13 s\n",
            "Wall time: 1.14 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_big_pd_cat_parquet = pd.read_parquet('test_big_pd_cat.parquet')"
      ],
      "metadata": {
        "id": "ODTDAtIwrreR",
        "outputId": "e00f43a6-ed18-4b7b-de22-de779d738482",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 405 ms, sys: 581 ms, total: 987 ms\n",
            "Wall time: 704 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "df_big_pd_cat_feather = pd.read_feather('test_big_pd_cat.feather')"
      ],
      "metadata": {
        "id": "rYicnhboryuR",
        "outputId": "d3b86b33-66c9-4283-c1d0-b39054bac826",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 323 ms, sys: 431 ms, total: 753 ms\n",
            "Wall time: 515 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Memory Usage"
      ],
      "metadata": {
        "id": "0HOatRcjw8e7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_pd_cat_csv.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "id": "j1pyKruxw-wQ",
        "outputId": "9c73584a-8ca7-4d8f-be13-94755a32ef2e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5000000 entries, 0 to 4999999\n",
            "Data columns (total 11 columns):\n",
            " #   Column      Dtype  \n",
            "---  ------      -----  \n",
            " 0   Unnamed: 0  int64  \n",
            " 1   A           float64\n",
            " 2   B           float64\n",
            " 3   C           float64\n",
            " 4   D           float64\n",
            " 5   E           float64\n",
            " 6   F           int64  \n",
            " 7   G           int64  \n",
            " 8   H           int64  \n",
            " 9   I           int64  \n",
            " 10  J           object \n",
            "dtypes: float64(5), int64(5), object(1)\n",
            "memory usage: 680.7 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_pd_cat_hdf.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "id": "3cz9G4jFxNSe",
        "outputId": "8f82bf6b-4691-4be5-834e-2546e0fd6c25",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 5000000 entries, 0 to 4999999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Dtype   \n",
            "---  ------  -----   \n",
            " 0   A       float64 \n",
            " 1   B       float64 \n",
            " 2   C       float64 \n",
            " 3   D       float64 \n",
            " 4   E       float64 \n",
            " 5   F       int64   \n",
            " 6   G       int64   \n",
            " 7   H       int64   \n",
            " 8   I       int64   \n",
            " 9   J       category\n",
            "dtypes: category(1), float64(5), int64(4)\n",
            "memory usage: 386.2 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_pd_cat_parquet.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "id": "yYgiseNJxTs2",
        "outputId": "62fddc85-735f-48fe-d67c-a002c326fc0c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5000000 entries, 0 to 4999999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Dtype   \n",
            "---  ------  -----   \n",
            " 0   A       float64 \n",
            " 1   B       float64 \n",
            " 2   C       float64 \n",
            " 3   D       float64 \n",
            " 4   E       float64 \n",
            " 5   F       int64   \n",
            " 6   G       int64   \n",
            " 7   H       int64   \n",
            " 8   I       int64   \n",
            " 9   J       category\n",
            "dtypes: category(1), float64(5), int64(4)\n",
            "memory usage: 348.1 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_big_pd_cat_feather.info(memory_usage='deep', verbose=True)"
      ],
      "metadata": {
        "id": "6uqgysyMxhhp",
        "outputId": "bcf75219-aa3e-4022-cc14-2cdd801bee90",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5000000 entries, 0 to 4999999\n",
            "Data columns (total 10 columns):\n",
            " #   Column  Dtype   \n",
            "---  ------  -----   \n",
            " 0   A       float64 \n",
            " 1   B       float64 \n",
            " 2   C       float64 \n",
            " 3   D       float64 \n",
            " 4   E       float64 \n",
            " 5   F       int64   \n",
            " 6   G       int64   \n",
            " 7   H       int64   \n",
            " 8   I       int64   \n",
            " 9   J       category\n",
            "dtypes: category(1), float64(5), int64(4)\n",
            "memory usage: 348.1 MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Feather or Parquet\n",
        "1. Parquet format is designed for long-term storage, where Arrow is more intended for short term or ephemeral storage because files volume are larger.\n",
        "\n",
        "2. Parquet is usually more expensive to write than Feather as it features more layers of encoding and compression.\n",
        "\n",
        "3. Feather is unmodified raw columnar Arrow memory. We will probably add simple compression to Feather in the future.\n",
        "\n",
        "4. Due to dictionary encoding, RLE encoding, and data page compression, Parquet files will often be much smaller than Feather files\n",
        "\n",
        "5. Parquet is a standard storage format for analytics that’s supported by Spark. So if you are doing analytics, Parquet is a good option as a reference storage format for query by multiple systems\n",
        "\n",
        "[Source StackOverflow](https://stackoverflow.com/questions/48083405/what-are-the-differences-between-feather-and-parquet)"
      ],
      "metadata": {
        "id": "I0N-Z2ZSr-aJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Bibliograpgy\n",
        "This notebook is inspired from the following link.<br>\n",
        "\n",
        "[File Format - Python tools for big data](https://pnavaro.github.io/big-data/14-FileFormats.html)"
      ],
      "metadata": {
        "id": "4AIPs8_GsWUQ"
      }
    }
  ]
}