02_multiclass_classification_random_forest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiclass Classification w/ Random-Forest and Transformer-based Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Multiclass Classification` is a fundamental problem type in supervised learning where the goal is to classify instances into one or more classes. This notebook illustrates how to train a `random forest` model with hyperparameter tuning for multiclass classification. It then assesses the perfomance of combining said random forest with:\n",
    "- [Cohere-embeddings](https://docs.cohere.com/reference/embed), and\n",
    "- [bag-of-words vectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Random forests](https://web.archive.org/web/20160417030218/http://ect.bell-labs.com/who/tkh/publications/papers/odt.pdf) is an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees. Random forests correct for decision trees' habit of overfitting to their training set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Dataset Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install --upgrade cohere datasets python-dotenv scikit-learn tqdm --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import tqdm as notebook_tqdm\n",
    "\n",
    "from datasets import load_dataset\n",
    "from dotenv import find_dotenv, load_dotenv\n",
    "\n",
    "SAMPLE_SIZE = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [AG_News](https://huggingface.co/datasets/ag_news) dataset is a collection of more than 1 million news articles. News articles have been gathered from more than 2000 news sources by ComeToMyHead in more than 1 year of activity. ComeToMyHead is an academic news search engine which has been running since July, 2004. The dataset is provided by the academic comunity for research purposes in data mining (clustering, classification), information retrieval (ranking, search), data compression, data streaming, and any other non-commercial activity. For more information, please refer to http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text', 'label'],\n",
       "    num_rows: 120000\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = load_dataset(\"ag_news\", split=\"train\")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's work with a smaller subset (by default 1000 samples). We shuffle the dataset to maintain the balance among classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text', 'label'],\n",
       "    num_rows: 1000\n",
       "})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_sm = ds.shuffle(seed=42).select(range(SAMPLE_SIZE))\n",
    "ds_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEpCAYAAAB1Fp6nAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0W0lEQVR4nO3deVhU9f4H8PewzbAjCAwIIoqKuN5cEFMEJcXMNL3u13BJU8FUrIyuikvFzbL8ZaZmBVZ6KzUtzQ1RXBI3yr1ccFdQQRBxYf38/vDhXEdQEVk979fznOfhfM/3nPM5Z4Z5z1lmRiMiAiIiUiWjyi6AiIgqD0OAiEjFGAJERCrGECAiUjGGABGRijEEiIhUjCFARKRiDAEiIhVjCBARqRhDgJ5ZZ8+ehUajQUxMTIWtMz4+HhqNBvHx8RW2zvIQExMDjUaD/fv3V8j6NBoNpk+fXiHrIkMMAZU6ePAgNBoNjh8/DgD49NNPUadOnWL77ty5E926dUOtWrWg0+lQu3Zt9OjRA8uWLavAitWrTp060Gg0xQ7BwcGVXd4jHThwAP/617/g7u4OrVYLe3t7BAUFITo6Gvn5+ZVdHgEwqewCqHLs2bMH9vb2aNCgAQAgISEBbdu2LdJv+fLl6N+/P1q0aIHx48ejRo0aOHPmDLZv347Fixdj0KBBFV16lebv7487d+7AzMysTJfbokULTJo0qUi7q6trma6nLH311VcYPXo0nJ2dMWTIENSvXx83b95EXFwcRowYgeTkZLz77ruVXabqMQRUau/evWjTpg00Gg2AeyEQHh5epN/06dPh4+OD3bt3F3lhu3r1aoXUWp0YGRlBp9OV+XJr1aqFf/3rX2W+3PKye/dujB49Gn5+fli3bh2sra2VaRMmTMD+/ftx5MiRSqyQCvF0kIqkp6cjNTUVqamp2LNnD5o0aYLU1FQcPXoUFy9eRP369ZGamoqsrCxlnqSkJLRu3brYd7ZOTk4G4x9//DHatWsHBwcHmJubo2XLllixYkWR+TQaDcLCwrB8+XL4+PjA3Nwcfn5+OHz4MABg0aJF8PLygk6nQ0BAAM6ePWswf0BAAJo0aYLExES0a9cO5ubm8PT0xMKFC0u0H/7++2/885//hL29PXQ6HVq1aoVff/3VoE9ubi5mzJiB+vXrQ6fTwcHBAe3bt0dsbOwjl13cNYHCeo8dO4bAwEBYWFigVq1amD17donqLalDhw5h6NChqFu3LnQ6HfR6PYYPH460tLQifS9duoQRI0bA1dUVWq0Wnp6eGDNmDHJycgz6ZWdnIzw8HI6OjrC0tMQrr7yCa9euPbaWGTNmQKPRYOnSpQYBUKhVq1YYOnToQ+c/d+4cxo4di4YNG8Lc3BwODg7o27dvkedCSR6nlJQUDBs2DG5ubtBqtXBxcUHPnj2LLEuteCSgIv/4xz9w7tw5ZfzIkSP4+OOPlfEePXoAAEJCQpSLqR4eHoiLi8PFixfh5ub2yOX/3//9H15++WUMHjwYOTk5+OGHH9C3b1+sXbsW3bt3N+i7Y8cO/PrrrwgNDQUAREVF4aWXXsLbb7+NL774AmPHjkV6ejpmz56N4cOHY8uWLQbzp6en48UXX0S/fv0wcOBA/PTTTxgzZgzMzMwwfPjwh9Z49OhRPP/886hVqxbeeecdWFpa4qeffkKvXr2wcuVKvPLKKwDuHQFFRUXhtddeQ5s2bZCZmYn9+/fjjz/+wAsvvPCYPV1Ueno6goOD0bt3b/Tr1w8rVqzA5MmT0bRpU3Tr1u2x8+fm5iI1NbVIu6WlJczNzQEAsbGxOH36NIYNGwa9Xo+jR4/iyy+/xNGjR7F7927lqO/y5cto06YNMjIyMGrUKHh7e+PSpUtYsWIFbt++bRD448aNQ40aNRAZGYmzZ89i7ty5CAsLw48//vjQWm/fvo24uDj4+/ujdu3aT7qrAAD79u3Drl27MGDAALi5ueHs2bNYsGABAgICcOzYMVhYWAAo2ePUp08fHD16FOPGjUOdOnVw9epVxMbG4vz58w+9DqYqQqqxc+dOiY2NlalTp4qJiYmsX79eYmNjpVu3btKqVSuJjY2V2NhYOXr0qDLP119/LQDEzMxMAgMDZerUqbJjxw7Jz88vsvzbt28bjOfk5EiTJk2kU6dOBu0ARKvVypkzZ5S2RYsWCQDR6/WSmZmptEdERAgAg74dO3YUADJnzhylLTs7W1q0aCFOTk6Sk5MjIiJnzpwRABIdHa3069y5szRt2lTu3r2rtBUUFEi7du2kfv36Slvz5s2le/fuj9mjRW3dulUAyNatW4vU++233xrUq9frpU+fPo9dpoeHhwAodoiKilL6Pbj/RUT++9//CgDZvn270vbqq6+KkZGR7Nu3r0j/goICERGJjo4WABIUFKS0iYhMnDhRjI2NJSMj46H1Hjx4UADI+PHjH7tthQBIZGTkI7clISGhyH583OOUnp4uAOSjjz4qcS1qw9NBKvL8888jKCgIWVlZaN26NYKDgxEUFITz58/jpZdeQlBQEIKCguDj46PMM3z4cGzYsAEBAQHYuXMnZs2ahQ4dOqB+/frYtWuXwfIL35EC99753rhxAx06dMAff/xRpJbOnTsbvAvz9fUFcO9d2/2nDwrbT58+bTC/iYkJXn/9dWXczMwMr7/+Oq5evYrExMRit//69evYsmUL+vXrh5s3byqnxtLS0tC1a1ecPHkSly5dAgDY2dnh6NGjOHny5CP3aUlZWVkZnNM3MzNDmzZtimzXw/j6+iI2NrbIMHDgQKXP/fv/7t27SE1NVS72Fz4GBQUFWL16NXr06IFWrVoVWU/h0UKhUaNGGbR16NAB+fn5BkeUD8rMzASAYk8DldT925Kbm4u0tDR4eXnBzs7O4Pn0uMfJ3NwcZmZmiI+PR3p6eqnreZYxBFTixo0byoteXFwcfH19kZqaihMnTuDo0aNo3rw5UlNTcePGjSLzdu3aFRs3bkRGRga2b9+O0NBQnDt3Di+99JLBxeG1a9eibdu20Ol0sLe3h6OjIxYsWFDsMh88TWBrawsAcHd3L7b9wX9gV1dXWFpaGrQV3un0sHO9p06dgohg6tSpcHR0NBgiIyMB/O9i98yZM5GRkYEGDRqgadOmeOutt3Do0KFil1sSbm5uRV5ga9SoUeIXppo1ayohff/g4eGh9Ll+/TrGjx8PZ2dnmJubw9HREZ6engCgPAbXrl1DZmYmmjRpUqL1Pvg41ahRA0DRx+N+NjY2AICbN2+WaB3FuXPnDqZNm6bcWlqzZk04OjoiIyPD4Pn0uMdJq9Xiww8/xPr16+Hs7Ax/f3/Mnj0bKSkppa7tWcMQUImePXsqL3iHDh3C3Llz4ejoiIYNGwIAXnnlFTg6OqJnz54PXYaFhQU6dOiAzz//HFOmTEF6ejrWr18P4N45/pdffhk6nQ5ffPEF1q1bh9jYWAwaNAhSzC+YGhsbF7uOh7UXt4wnVVBQAAB48803i31XHRsbCy8vLwD3bvVMSkrCN998gyZNmuCrr77Cc889h6+++qpU6y7P7SrUr18/LF68GKNHj8bPP/+MTZs2YcOGDQD+t+1PqjR1e3l5wcTERLnQXxrjxo3D+++/j379+uGnn37Cpk2bEBsbCwcHB4NtKcnjNGHCBJw4cQJRUVHQ6XSYOnUqGjVqhD///LPU9T1LeGFYJebMmYP09HQkJCRgxowZWLt2LUxMTDBv3jxcunQJ//nPfwD8753e4xSeSkhOTgYArFy5EjqdDhs3boRWq1X6RUdHl/GW3HP58mXcunXL4GjgxIkTAPDQi31169YFAJiamiIoKOix67C3t8ewYcMwbNgwZGVlwd/fH9OnT8drr7329BtQxtLT0xEXF4cZM2Zg2rRpSvuDp0kcHR1hY2NTrrdnWlhYoFOnTtiyZQsuXLhQ5OiuJFasWIGQkBDMmTNHabt79y4yMjKK9C3J41SvXj1MmjQJkyZNwsmTJ9GiRQvMmTMH33//fam28VnCIwGVaNmyJYKCgpCXl4cmTZoo1wOuXLlicHqhZcuWBvPFxcUVu7x169YBgHIkYWxsDI1GY/Ap0LNnz2L16tXlsj15eXlYtGiRMp6Tk4NFixbB0dGxyDYUcnJyQkBAABYtWqSE1/3uv/Xxwdsqrays4OXlhezs7DLagrJV+I79wXfoc+fONRg3MjJCr169sGbNmmK/EqKsjkwiIyMhIhgyZIjBLceFEhMTsWTJkofOb2xsXKSWefPmFfmU8eMep9u3b+Pu3bsGferVqwdra+sq+1hWNB4JqMzvv/+Odu3aAbj3zurPP/985Kc2e/bsCU9PT/To0QP16tXDrVu3sHnzZqxZswatW7dWbivt3r07PvnkEwQHB2PQoEG4evUq5s+fDy8vr6c6l/4wrq6u+PDDD3H27Fk0aNAAP/74Iw4cOIAvv/wSpqamD51v/vz5aN++PZo2bYqRI0eibt26uHLlChISEnDx4kUcPHgQAODj44OAgAC0bNkS9vb22L9/P1asWIGwsLAy35aSuHTpUrHvWq2srNCrVy/Y2Ngo57tzc3NRq1YtbNq0CWfOnCkyzwcffIBNmzahY8eOGDVqFBo1aoTk5GQsX74cO3fuhJ2d3VPX265dO8yfPx9jx46Ft7e3wSeG4+Pj8euvv+K999576PwvvfQSvvvuO9ja2sLHxwcJCQnYvHkzHBwcDPo97nE6ceIEOnfujH79+sHHxwcmJiZYtWoVrly5ggEDBjz1dj4TKu2+JKpweXl5YmVlJd99952I3LtlFIBcvXr1ofP897//lQEDBki9evXE3NxcdDqd+Pj4yL///W+DWzlF7t1OWr9+fdFqteLt7S3R0dESGRkpDz7NAEhoaKhBW+HtnA/eyld4y+Xy5cuVto4dO0rjxo1l//794ufnJzqdTjw8POTzzz8vdpn33yIqIpKUlCSvvvqq6PV6MTU1lVq1aslLL70kK1asUPq899570qZNG7GzsxNzc3Px9vaW999/X7n99GEedoto48aNi/QNCQkRDw+PRy5P5NG3iN4//8WLF+WVV14ROzs7sbW1lb59+8rly5eL3H4pInLu3Dl59dVXxdHRUbRardStW1dCQ0MlOztbRP53i+iDt5EWt32PkpiYKIMGDRJXV1cxNTWVGjVqSOfOnWXJkiUGtxk/WGN6eroMGzZMatasKVZWVtK1a1f5+++/xcPDQ0JCQpR+j3ucUlNTJTQ0VLy9vcXS0lJsbW3F19dXfvrppxLVrwYakTK8MkVUAQICApCamsqvHSAqA7wmQESkYgwBIiIVYwgQEakYrwkQEakYjwSIiFSMIUBEpGL8sBjufa/K5cuXYW1tXeRLvoiIqiMRwc2bN+Hq6gojo4e/32cI4N730JTm+02IiKq6CxcuPPIHoRgC+N/3nl+4cEH5GlwiouosMzMT7u7uj/1dB4YA/vdDGjY2NgwBInqmPO4UNy8MExGpGEOAiEjFGAJERCrGECAiUjGGABGRijEEiIhUjCFARKRiDAEiIhXjh8WIqpBFJ3+o7BIqxev1+aPvlYVHAkREKsYQICJSMYYAEZGKMQSIiFSMIUBEpGIMASIiFWMIEBGpGEOAiEjFGAJERCrGECAiUjGGABGRijEEiIhUrFJDICoqCq1bt4a1tTWcnJzQq1cvHD9+3KBPQEAANBqNwTB69GiDPufPn0f37t1hYWEBJycnvPXWW8jLy6vITSEiqpYq9VtEt23bhtDQULRu3Rp5eXl499130aVLFxw7dgyWlpZKv5EjR2LmzJnKuIWFhfJ3fn4+unfvDr1ej127diE5ORmvvvoqTE1N8cEHH1To9hARVTeVGgIbNmwwGI+JiYGTkxMSExPh7++vtFtYWECv1xe7jE2bNuHYsWPYvHkznJ2d0aJFC8yaNQuTJ0/G9OnTYWZmVq7bQERUnVWpawI3btwAANjb2xu0L126FDVr1kSTJk0QERGB27dvK9MSEhLQtGlTODs7K21du3ZFZmYmjh49Wux6srOzkZmZaTAQEalRlflRmYKCAkyYMAHPP/88mjRporQPGjQIHh4ecHV1xaFDhzB58mQcP34cP//8MwAgJSXFIAAAKOMpKSnFrisqKgozZswopy0hIqo+qkwIhIaG4siRI9i5c6dB+6hRo5S/mzZtChcXF3Tu3BlJSUmoV69eqdYVERGB8PBwZTwzMxPu7u6lK5yIqBqrEqeDwsLCsHbtWmzduhVubm6P7Ovr6wsAOHXqFABAr9fjypUrBn0Kxx92HUGr1cLGxsZgICJSo0oNARFBWFgYVq1ahS1btsDT0/Ox8xw4cAAA4OLiAgDw8/PD4cOHcfXqVaVPbGwsbGxs4OPjUy51ExE9Kyr1dFBoaCiWLVuGX375BdbW1so5fFtbW5ibmyMpKQnLli3Diy++CAcHBxw6dAgTJ06Ev78/mjVrBgDo0qULfHx8MGTIEMyePRspKSmYMmUKQkNDodVqK3PziIiqvEoNgQULFgC494Gw+0VHR2Po0KEwMzPD5s2bMXfuXNy6dQvu7u7o06cPpkyZovQ1NjbG2rVrMWbMGPj5+cHS0hIhISEGnyt4Wiv2XiuzZVUn/2zjWNklEFE5q9QQEJFHTnd3d8e2bdseuxwPDw+sW7eurMqiMpKxYV5ll1Ap7ILHVXYJRCVWZe4OIiIqrawVP1Z2CZXC6p/9n3oZVeLuICIiqhwMASIiFWMIEBGpGEOAiEjFGAJERCrGECAiUjGGABGRijEEiIhUjCFARKRiDAEiIhVjCBARqRhDgIhIxRgCREQqxhAgIlIxhgARkYoxBIiIVIwhQESkYgwBIiIVYwgQEakYQ4CISMUYAkREKsYQICJSMYYAEZGKMQSIiFSMIUBEpGIMASIiFWMIEBGpGEOAiEjFKjUEoqKi0Lp1a1hbW8PJyQm9evXC8ePHDfrcvXsXoaGhcHBwgJWVFfr06YMrV64Y9Dl//jy6d+8OCwsLODk54a233kJeXl5FbgoRUbVUqSGwbds2hIaGYvfu3YiNjUVubi66dOmCW7duKX0mTpyINWvWYPny5di2bRsuX76M3r17K9Pz8/PRvXt35OTkYNeuXViyZAliYmIwbdq0ytgkIqJqxaQyV75hwwaD8ZiYGDg5OSExMRH+/v64ceMGvv76ayxbtgydOnUCAERHR6NRo0bYvXs32rZti02bNuHYsWPYvHkznJ2d0aJFC8yaNQuTJ0/G9OnTYWZmVhmbRkRULVSpawI3btwAANjb2wMAEhMTkZubi6CgIKWPt7c3ateujYSEBABAQkICmjZtCmdnZ6VP165dkZmZiaNHj1Zg9URE1U+lHgncr6CgABMmTMDzzz+PJk2aAABSUlJgZmYGOzs7g77Ozs5ISUlR+twfAIXTC6cVJzs7G9nZ2cp4ZmZmWW0GEVG1UmWOBEJDQ3HkyBH88MMP5b6uqKgo2NraKoO7u3u5r5OIqCqqEiEQFhaGtWvXYuvWrXBzc1Pa9Xo9cnJykJGRYdD/ypUr0Ov1Sp8H7xYqHC/s86CIiAjcuHFDGS5cuFCGW0NEVH1UagiICMLCwrBq1Sps2bIFnp6eBtNbtmwJU1NTxMXFKW3Hjx/H+fPn4efnBwDw8/PD4cOHcfXqVaVPbGwsbGxs4OPjU+x6tVotbGxsDAYiIjWq1GsCoaGhWLZsGX755RdYW1sr5/BtbW1hbm4OW1tbjBgxAuHh4bC3t4eNjQ3GjRsHPz8/tG3bFgDQpUsX+Pj4YMiQIZg9ezZSUlIwZcoUhIaGQqvVVubmERFVeZUaAgsWLAAABAQEGLRHR0dj6NChAIBPP/0URkZG6NOnD7Kzs9G1a1d88cUXSl9jY2OsXbsWY8aMgZ+fHywtLRESEoKZM2dW1GYQEVVblRoCIvLYPjqdDvPnz8f8+fMf2sfDwwPr1q0ry9KIiFShSlwYJiKiysEQICJSMYYAEZGKMQSIiFSMIUBEpGIMASIiFWMIEBGpGEOAiEjFGAJERCrGECAiUjGGABGRijEEiIhUjCFARKRiDAEiIhVjCBARqRhDgIhIxRgCREQqxhAgIlIxhgARkYoxBIiIVIwhQESkYgwBIiIVYwgQEalYqUKgU6dOyMjIKNKemZmJTp06PW1NRERUQUoVAvHx8cjJySnSfvfuXezYseOpiyIiooph8iSdDx06pPx97NgxpKSkKOP5+fnYsGEDatWqVXbVERFRuXqiEGjRogU0Gg00Gk2xp33Mzc0xb968MiuOiIjK1xOFwJkzZyAiqFu3Lvbu3QtHR0dlmpmZGZycnGBsbFzmRRIRUfl4ohDw8PAAABQUFJRLMUREVLGeKATud/LkSWzduhVXr14tEgrTpk176sKIiKj8leruoMWLF6NRo0aYNm0aVqxYgVWrVinD6tWrS7yc7du3o0ePHnB1dYVGoyky79ChQ5VrEIVDcHCwQZ/r169j8ODBsLGxgZ2dHUaMGIGsrKzSbBYRkeqU6kjgvffew/vvv4/Jkyc/1cpv3bqF5s2bY/jw4ejdu3exfYKDgxEdHa2Ma7Vag+mDBw9GcnIyYmNjkZubi2HDhmHUqFFYtmzZU9VGRKQGpQqB9PR09O3b96lX3q1bN3Tr1u2RfbRaLfR6fbHT/vrrL2zYsAH79u1Dq1atAADz5s3Diy++iI8//hiurq5PXSMR0bOsVKeD+vbti02bNpV1LcWKj4+Hk5MTGjZsiDFjxiAtLU2ZlpCQADs7OyUAACAoKAhGRkbYs2dPhdRHRFSdlepIwMvLC1OnTsXu3bvRtGlTmJqaGkx/4403yqS44OBg9O7dG56enkhKSsK7776Lbt26ISEhAcbGxkhJSYGTk5PBPCYmJrC3tzf4INuDsrOzkZ2drYxnZmaWSb1ERNVNqULgyy+/hJWVFbZt24Zt27YZTNNoNGUWAgMGDFD+btq0KZo1a4Z69eohPj4enTt3LvVyo6KiMGPGjLIokYioWitVCJw5c6as6yiRunXrombNmjh16hQ6d+4MvV6Pq1evGvTJy8vD9evXH3odAQAiIiIQHh6ujGdmZsLd3b3c6iYiqqqq1VdJX7x4EWlpaXBxcQEA+Pn5ISMjA4mJiUqfLVu2oKCgAL6+vg9djlarhY2NjcFARKRGpToSGD58+COnf/PNNyVaTlZWFk6dOqWMnzlzBgcOHIC9vT3s7e0xY8YM9OnTB3q9HklJSXj77bfh5eWFrl27AgAaNWqE4OBgjBw5EgsXLkRubi7CwsIwYMAA3hlERFQCpb5F9H65ubk4cuQIMjIynuj3BPbv34/AwEBlvPAUTUhICBYsWIBDhw5hyZIlyMjIgKurK7p06YJZs2YZfFZg6dKlCAsLQ+fOnWFkZIQ+ffrgs88+K81mERGpTqlCYNWqVUXaCgoKMGbMGNSrV6/EywkICICIPHT6xo0bH7sMe3t7fjCMiKiUyuyagJGREcLDw/Hpp5+W1SKJiKiclemF4aSkJOTl5ZXlIomIqByV6nTQ/bdXAoCIIDk5Gb/99htCQkLKpDAiIip/pQqBP//802DcyMgIjo6OmDNnzmPvHCIioqqjVCGwdevWsq6DiIgqQal/VAYArl27huPHjwMAGjZsaPBzk0REVPWV6sLwrVu3MHz4cLi4uMDf3x/+/v5wdXXFiBEjcPv27bKukYiIykmpQiA8PBzbtm3DmjVrkJGRgYyMDPzyyy/Ytm0bJk2aVNY1EhFROSnV6aCVK1dixYoVCAgIUNpefPFFmJubo1+/fliwYEFZ1UdEROWoVEcCt2/fhrOzc5F2Jycnng4iIqpGShUCfn5+iIyMxN27d5W2O3fuYMaMGfDz8yuz4oiIqHyV6nTQ3LlzERwcDDc3NzRv3hwAcPDgQWi12gr72UkiInp6pQqBpk2b4uTJk1i6dCn+/vtvAMDAgQMxePBgmJubl2mBRERUfkoVAlFRUXB2dsbIkSMN2r/55htcu3YNkydPLpPiiIiofJXqmsCiRYvg7e1dpL1x48ZYuHDhUxdFREQVo1QhkJKSovzE4/0cHR2RnJz81EUREVHFKFUIuLu74/fffy/S/vvvv/NnHYmIqpFSXRMYOXIkJkyYgNzcXOXnJOPi4vD222/zE8NERNVIqULgrbfeQlpaGsaOHYucnBwAgE6nw+TJkxEREVGmBRIRUfkpVQhoNBp8+OGHmDp1Kv766y+Ym5ujfv36Bj8AT0REVd9TfZW0lZUVWrduXVa1EBFRBSvT3xgmIqLqhSFARKRiDAEiIhVjCBARqRhDgIhIxRgCREQqxhAgIlIxhgARkYoxBIiIVKxSQ2D79u3o0aMHXF1dodFosHr1aoPpIoJp06bBxcUF5ubmCAoKwsmTJw36XL9+HYMHD4aNjQ3s7OwwYsQIZGVlVeBWEBFVX5UaArdu3ULz5s0xf/78YqfPnj0bn332GRYuXIg9e/bA0tISXbt2NfiB+8GDB+Po0aOIjY3F2rVrsX37dowaNaqiNoGIqFp7qu8OelrdunVDt27dip0mIpg7dy6mTJmCnj17AgC+/fZbODs7Y/Xq1RgwYAD++usvbNiwAfv27UOrVq0AAPPmzcOLL76Ijz/+mL9tQET0GFX2msCZM2eQkpKCoKAgpc3W1ha+vr5ISEgAACQkJMDOzk4JAAAICgqCkZER9uzZ89BlZ2dnIzMz02AgIlKjKhsCKSkpAABnZ2eDdmdnZ2VaSkoKnJycDKabmJjA3t5e6VOcqKgo2NraKoO7u3sZV09EVD1U2RAoTxEREbhx44YyXLhwobJLIiKqFFU2BPR6PQDgypUrBu1XrlxRpun1ely9etVgel5eHq5fv670KY5Wq4WNjY3BQESkRlU2BDw9PaHX6xEXF6e0ZWZmYs+ePfDz8wMA+Pn5ISMjA4mJiUqfLVu2oKCgAL6+vhVeMxFRdVOpdwdlZWXh1KlTyviZM2dw4MAB2Nvbo3bt2pgwYQLee+891K9fH56enpg6dSpcXV3Rq1cvAECjRo0QHByMkSNHYuHChcjNzUVYWBgGDBjAO4OIiEqgUkNg//79CAwMVMbDw8MBACEhIYiJicHbb7+NW7duYdSoUcjIyED79u2xYcMG6HQ6ZZ6lS5ciLCwMnTt3hpGREfr06YPPPvuswreFiKg6qtQQCAgIgIg8dLpGo8HMmTMxc+bMh/axt7fHsmXLyqM8IqJnXpW9JkBEROWPIUBEpGIMASIiFWMIEBGpGEOAiEjFGAJERCrGECAiUjGGABGRijEEiIhUjCFARKRiDAEiIhVjCBARqRhDgIhIxRgCREQqxhAgIlIxhgARkYoxBIiIVIwhQESkYgwBIiIVYwgQEakYQ4CISMUYAkREKsYQICJSMYYAEZGKMQSIiFSMIUBEpGIMASIiFWMIEBGpGEOAiEjFqnQITJ8+HRqNxmDw9vZWpt+9exehoaFwcHCAlZUV+vTpgytXrlRixURE1UuVDgEAaNy4MZKTk5Vh586dyrSJEydizZo1WL58ObZt24bLly+jd+/elVgtEVH1YlLZBTyOiYkJ9Hp9kfYbN27g66+/xrJly9CpUycAQHR0NBo1aoTdu3ejbdu2FV0qEVG1U+WPBE6ePAlXV1fUrVsXgwcPxvnz5wEAiYmJyM3NRVBQkNLX29sbtWvXRkJCQmWVS0RUrVTpIwFfX1/ExMSgYcOGSE5OxowZM9ChQwccOXIEKSkpMDMzg52dncE8zs7OSElJeeRys7OzkZ2drYxnZmaWR/lERFVelQ6Bbt26KX83a9YMvr6+8PDwwE8//QRzc/NSLzcqKgozZswoixKJiKq1Kn866H52dnZo0KABTp06Bb1ej5ycHGRkZBj0uXLlSrHXEO4XERGBGzduKMOFCxfKsWoioqqrWoVAVlYWkpKS4OLigpYtW8LU1BRxcXHK9OPHj+P8+fPw8/N75HK0Wi1sbGwMBiIiNarSp4PefPNN9OjRAx4eHrh8+TIiIyNhbGyMgQMHwtbWFiNGjEB4eDjs7e1hY2ODcePGwc/Pj3cGERGVUJUOgYsXL2LgwIFIS0uDo6Mj2rdvj927d8PR0REA8Omnn8LIyAh9+vRBdnY2unbtii+++KKSqyYiqj6qdAj88MMPj5yu0+kwf/58zJ8/v4IqIiJ6tlSrawJERFS2GAJERCrGECAiUjGGABGRijEEiIhUjCFARKRiDAEiIhVjCBARqRhDgIhIxRgCREQqxhAgIlIxhgARkYoxBIiIVIwhQESkYgwBIiIVYwgQEakYQ4CISMUYAkREKsYQICJSMYYAEZGKMQSIiFSMIUBEpGIMASIiFWMIEBGpGEOAiEjFGAJERCrGECAiUjGGABGRijEEiIhUjCFARKRiz0wIzJ8/H3Xq1IFOp4Ovry/27t1b2SUREVV5z0QI/PjjjwgPD0dkZCT++OMPNG/eHF27dsXVq1cruzQioirtmQiBTz75BCNHjsSwYcPg4+ODhQsXwsLCAt98801ll0ZEVKWZVHYBTysnJweJiYmIiIhQ2oyMjBAUFISEhIRi58nOzkZ2drYyfuPGDQBAZmZmsf1vZ90sw4qrj8xM7dPNf+tOGVVSvRg95HlUEneybpdhJdXHw/73Sirrtjr3W8Ej9lvhPhWRRy6j2odAamoq8vPz4ezsbNDu7OyMv//+u9h5oqKiMGPGjCLt7u7u5VIjqc3kyi6g2pmIEZVdQjU1/LE9bt68CVtb24dOr/YhUBoREREIDw9XxgsKCnD9+nU4ODhAo9FUYmWGMjMz4e7ujgsXLsDGxqayy6k2uN+eHPdZ6VTl/SYiuHnzJlxdXR/Zr9qHQM2aNWFsbIwrV64YtF+5cgV6vb7YebRaLbRaw1MddnZ25VXiU7OxsalyT7DqgPvtyXGflU5V3W+POgIoVO0vDJuZmaFly5aIi4tT2goKChAXFwc/P79KrIyIqOqr9kcCABAeHo6QkBC0atUKbdq0wdy5c3Hr1i0MGzassksjIqrSnokQ6N+/P65du4Zp06YhJSUFLVq0wIYNG4pcLK5utFotIiMji5y6okfjfnty3Gel8yzsN4087v4hIiJ6ZlX7awJERFR6DAEiIhVjCBARqRhDoJqbPn06WrRo8cg+Q4cORa9evSqkHqJnWXx8PDQaDTIyMiq7lDLDECgHCxcuhLW1NfLy8pS2rKwsmJqaIiAgwKBv4ZMqKSmpgqusWq5du4YxY8agdu3a0Gq10Ov16Nq1K37//fdyX3edOnUwd+7ccl9PZYiJiam0D0JWxpuPoUOHQqPRKIODgwOCg4Nx6NChMll+u3btkJycXKIPYVUXDIFyEBgYiKysLOzfv19p27FjB/R6Pfbs2YO7d+8q7Vu3bkXt2rVRr169J1qHiBiETHXXp08f/Pnnn1iyZAlOnDiBX3/9FQEBAUhLSyu3debk5JTbssvS0wRk//79ceLEiSLtS5YsgZubm8ELZnFDTExMOWxR+QoODkZycjKSk5MRFxcHExMTvPTSS2WybDMzM+j1+ir19TJPiyFQDho2bAgXFxfEx8crbfHx8ejZsyc8PT2xe/dug/bAwEBkZ2fjjTfegJOTE3Q6Hdq3b499+/YZ9NNoNFi/fj1atmwJrVaLnTt3Fll3fn4+wsPDYWdnBwcHB7z99tuP/RbBypaRkYEdO3bgww8/RGBgIDw8PNCmTRtERETg5ZdfBgBoNBosWLAA3bp1g7m5OerWrYsVK1YYLOfw4cPo1KkTzM3N4eDggFGjRiErK0uZXvjO9P3334erqysaNmyIgIAAnDt3DhMnTlRe+ADg3Llz6NGjB2rUqAFLS0s0btwY69atq7idcp+nCUhzc3M4OTkVaf/ll18wbtw45cUyOTkZkyZNQuPGjQ3a+vfvXx6bVK4Kg1Kv16NFixZ45513cOHCBVy7dq3Y0zkHDhyARqPB2bNnATz6sX9w/sIjrY0bN6JRo0awsrJSQuh+X331FRo1agSdTgdvb2988cUXyrScnByEhYXBxcUFOp0OHh4eiIqKAnDvzd706dOVNwCurq544403ynaHCZWLQYMGSZcuXZTx1q1by/Lly2X06NEybdo0ERG5ffu2aLVaiYmJkTfeeENcXV1l3bp1cvToUQkJCZEaNWpIWlqaiIhs3bpVAEizZs1k06ZNcurUKUlLS5PIyEhp3ry5sp4PP/xQatSoIStXrpRjx47JiBEjxNraWnr27FmRm/9EcnNzxcrKSiZMmCB3794ttg8AcXBwkMWLF8vx48dlypQpYmxsLMeOHRMRkaysLHFxcZHevXvL4cOHJS4uTjw9PSUkJERZRkhIiFhZWcmQIUPkyJEjcuTIEUlLSxM3NzeZOXOmJCcnS3JysoiIdO/eXV544QU5dOiQJCUlyZo1a2Tbtm3lvi8elJ6eLgAkPj7+kX1GjRolTk5OotVqpXHjxrJmzRoREYmOjhZbW1uD/nfu3BFLS0v566+/DNoffC7l5+fLBx98IHXq1BGdTifNmjWT5cuXG8xz5MgR6d69u1hbW4uVlZW0b99eTp06JSL39nfPnj3lo48+Er1eL/b29jJ27FjJycl5ij3yaIXrLHTz5k15/fXXxcvLS/Lz85X/o/T0dKXPn3/+KQDkzJkzIvLox/7B+aOjo8XU1FSCgoJk3759kpiYKI0aNZJBgwYpy//+++/FxcVFVq5cKadPn5aVK1eKvb29xMTEiIjIRx99JO7u7rJ9+3Y5e/as7NixQ5YtWyYiIsuXLxcbGxtZt26dnDt3Tvbs2SNffvllme4zhkA5Wbx4sVhaWkpubq5kZmaKiYmJXL16VZYtWyb+/v4iIhIXFycA5OzZs2JqaipLly5V5s/JyRFXV1eZPXu2iPzvybd69WqD9Tz4j+vi4qLMI3LvBdbNza1Kh4CIyIoVK6RGjRqi0+mkXbt2EhERIQcPHlSmA5DRo0cbzOPr6ytjxowREZEvv/xSatSoIVlZWcr03377TYyMjCQlJUVE7r1AODs7S3Z2tsFyPDw85NNPPzVoa9q0qUyfPr0sN7FUHheQ+fn50rZtW2ncuLFs2rRJedFat26diBQfAmvXrpUGDRoUWdaDz6X33ntPvL29ZcOGDZKUlCTR0dGi1WqVQLp48aLY29tL7969Zd++fXL8+HH55ptv5O+//xaRe/vbxsZGRo8eLX/99ZesWbNGLCwsyvxF7H4hISFibGwslpaWYmlpKQDExcVFEhMTRaToi7hI0RB41GNfXAgAUIJPRGT+/Pni7OysjNerV095US80a9Ys8fPzExGRcePGSadOnaSgoKDI+ubMmSMNGjQo1+BkCJSTkydPCgDZtWuX/Pbbb+Lj4yMiIpcuXRKtVit37tyRqVOnSt26deXgwYNKGNyvV69eMmzYMBH535Pv4sWLBn3u/8fNyMgQAEXesfbq1avKh4DIvXeomzZtkpkzZ4qfn58YGxtLdHS0iNwLgSVLlhj0nzBhggQEBIiIyMSJE5W/Cz24P0JCQiQoKKjIeosLgcWLF4uJiYm0a9dOpk2bZhBIFe1RAblx40YxMjKS48ePFztvcSEwcuRIefPNN4v0vf+5dPfuXbGwsJBdu3YZ9BkxYoQMHDhQREQiIiLE09PzoS9QISEh4uHhIXl5eUpb3759pX///iXa7tIofIxPnjwpJ0+elL1798rQoUPFyclJzp49W6IQeNRjX1wIWFhYGNTw888/i0ajEZF7R6gAxNzcXAkmS0tL0Wq14uTkJCIiiYmJYm9vL/Xr15dx48bJxo0blWWdP39e3N3dxc3NTV577TX5+eefJTc3t0z3Ga8JlBMvLy+4ublh69at2Lp1Kzp27AgAcHV1hbu7O3bt2oWtW7eiU6dOT7RcS0vL8ii3StDpdHjhhRcwdepU7Nq1C0OHDkVkZGSZrqOk+++1117D6dOnMWTIEBw+fBitWrXCvHnzyrSWkurTpw8uX76MX3/9FcHBwYiPj8dzzz2HmJgYHDhwAG5ubmjQoEGJliUiWLNmjXKt5WFOnTqF27dv44UXXoCVlZUyfPvtt8qdbAcOHECHDh1gamr60OU0btwYxsbGyriLi0u5//a3paUlvLy84OXlhdatW+Orr77CrVu3sHjxYhgZ3XvJk/uuk+Xm5hrM/6SP/YPbr9FolOUXXpNavHgxDhw4oAxHjhxRrg0+99xzOHPmDGbNmoU7d+6gX79++Oc//wng3g9dHT9+HF988QXMzc0xduxY+Pv7F6n5aTAEylFgYCDi4+MRHx9vcGuov78/1q9fj7179yIwMBD16tWDmZmZwd0eubm52LdvH3x8fEq8PltbW7i4uGDPnj1KW15eHhITE8tkeyqaj48Pbt26pYzff0G9cLxRo0YAgEaNGuHgwYMG/X///XcYGRmhYcOGj1yPmZkZ8vPzi7S7u7tj9OjR+PnnnzFp0iQsXrz4aTbnqTwsIM3NzZ9oOXv37kVeXh7atWv3yH6FL16//fabwYvXsWPHlAvyJVl3cS+QBQUFT1Tz09JoNDAyMsKdO3fg6OgIAAYXbg8cOFBknrJ67J2dneHq6orTp08rwVQ4eHp6Kv1sbGzQv39/LF68GD/++CNWrlyJ69evA7i3n3v06IHPPvsM8fHxSEhIwOHDh0tVT3GeiW8RraoCAwMRGhqK3Nxc5UgAADp27IiwsDDk5OQgMDAQlpaWGDNmDN566y3Y29ujdu3amD17Nm7fvo0RI57sZ/fGjx+P//znP6hfvz68vb3xySefVPkPtqSlpaFv374YPnw4mjVrBmtra+zfvx+zZ89Gz549lX7Lly9Hq1at0L59eyxduhR79+7F119/DQAYPHgwIiMjERISgunTp+PatWsYN24chgwZ8thvk61Tpw62b9+OAQMGQKvVombNmpgwYQK6deuGBg0aID09HVu3blUCpyrw8fHB6tWr0axZM1y8eBEnTpwo0dHAL7/8gu7duxu8O3/Y8rVaLc6fP2/w3L1fs2bNsGTJEuTm5j7yaKCiZWdnIyUlBQCQnp6Ozz//HFlZWejRowe8vLzg7u6O6dOn4/3338eJEycwZ84cg/nL+rGfMWMG3njjDdja2iI4OBjZ2dnYv38/0tPTER4ejk8++QQuLi74xz/+ASMjIyxfvhx6vR52dnaIiYlBfn4+fH19YWFhge+//x7m5ubw8PB4qn1koExPLpGBM2fOCADx9vY2aD979qwAkIYNGyptd+7ckXHjxknNmjVFq9XK888/L3v37lWmF3cuU6Toxbzc3FwZP3682NjYiJ2dnYSHh8urr75apa8J3L17V9555x157rnnxNbWViwsLKRhw4YyZcoUuX37tojcuyYwf/58eeGFF0Sr1UqdOnXkxx9/NFjOoUOHJDAwUHQ6ndjb28vIkSPl5s2byvQH7xwplJCQIM2aNROtViuF/xJhYWFSr1490Wq14ujoKEOGDJHU1NTy2wkPkZqaKoGBgfLdd9/JwYMH5fTp0/LTTz+Js7OzDB8+XEREAgICpEmTJrJp0yY5ffq0rFu3TtavXy8iRa8JNG7cWFauXFnsuh58Lv373/8WBwcHiYmJkVOnTkliYqJ89tlnyl0tqamp4uDgoFwYPnHihHz77bcGF4Yf3N/jx4+Xjh07ls3OKUZISIgAUAZra2tp3bq1rFixQumzc+dOadq0qeh0OunQoYMsX77c4JrAox774q4JPHjNZdWqVfLgS+vSpUulRYsWYmZmJjVq1BB/f3/5+eefReTeTQ0tWrQQS0tLsbGxkc6dO8sff/yhLMvX11dsbGzE0tJS2rZtK5s3by7TfcYQoGoBgKxataqyy6hwJQnItLQ0GTZsmDg4OIhOp5MmTZrI2rVrRcTwRerUqVOi1WoN7qC634MhUFBQIHPnzpWGDRuKqampODo6SteuXQ1uPDh48KB06dJFLCwsxNraWjp06CBJSUkiUjkhQE+OvydA1YJGo8GqVav4HUhP4ZNPPsHmzZsr7UNvVDXxwjCRSri5uSEiIqKyy6AqhkcCREQqxiMBIiIVYwgQEakYQ4CISMUYAkREKsYQICJSMYYAEZGKMQSIiFSMIUBEpGIMASIiFft/A0FMvwfmYwoAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 400x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "class_names = [\"World\", \"Sports\", \"Business\", \"Sci/Tech\"]\n",
    "label_names = [class_names[label] for label in ds_sm['label']]\n",
    "\n",
    "plt.figure(figsize=(4, 3))\n",
    "sns.countplot(x=label_names, hue=label_names, palette=\"pastel\")\n",
    "plt.title(\"#Samples in Each Class\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Random Forest with Transformed-based Embeddings (Cohere)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cohere\n",
    "\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file\n",
    "cohere_api_key = os.environ['COHERE_API_KEY']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.1 Vectorization with Cohere"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vectorize the input sentences using the [Cohere Embeddings](https://docs.cohere.com/reference/embed) model *embed-english-v3.0*. Learn more about this model at https://txt.cohere.com/introducing-embed-v3/ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "co = cohere.Client(cohere_api_key)\n",
    "\n",
    "# split ds_sm into 2 datasets to avoid hitting the rate limit\n",
    "ds_sm1 = ds_sm.select(range(500))\n",
    "ds_sm2 = ds_sm.select(range(500, 1000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_1 = co.embed( texts=ds_sm1['text'],\n",
    "                input_type=\"classification\",\n",
    "                model=\"embed-english-v3.0\",\n",
    "                truncate=\"END\").embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_2 = co.embed( texts=ds_sm2['text'],\n",
    "                input_type=\"classification\",\n",
    "                model=\"embed-english-v3.0\",\n",
    "                truncate=\"END\").embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1024"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# join X1 and X2\n",
    "X_co = np.concatenate((X_1, X_2), axis=0)\n",
    "# Cohere embeddings dimensions\n",
    "len(X_co[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# labels\n",
    "labels = {0: \"World\", 1: \"Sports\", 2: \"Business\", 3: \"Sci/Tech\"}\n",
    "y = [labels[x] for x in ds_sm['label']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.2 Train Model w/ basic Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike model parameters, which are learned from the data during training, `hyperparameters` are set prior to training and have a significant impact on how the model behaves and performs. For the sake of simplicity we aim to optimize just the following random-forest hyperparemeters:\n",
    "\n",
    "- Number of Trees (`n_estimators`): Generally, more trees increase the model's performance and robustness but also increase computational complexity and training time.\n",
    "- Maximum number of leaf nodes (`max_leaf_nodes`): This parameter can influence the shape and complexity of the trees that are created during training the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_co, y, test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`GridSearchCV` evaluates all the possible combinations of hyperparameter values using `cross-validation`. With our param grid, the grid search will explore 6 combinations (2 from `n_estimator`, and 3 from `max_leaf_nodes`), and will train each model 3 times (since we are defining three-fold `cross-validation`) using all available CPU cores (`n_jobs=-1`). That is, 18 rounds of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters: {'max_leaf_nodes': 8, 'n_estimators': 500}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid = {\n",
    "    'n_estimators': [250, 500],\n",
    "    'max_leaf_nodes': [8, 10, 12],\n",
    "}\n",
    "\n",
    "clf = RandomForestClassifier(random_state=42)\n",
    "\n",
    "grid_search = GridSearchCV(estimator=clf, param_grid=param_grid, cv=3, n_jobs=-1)\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "best_params = grid_search.best_params_\n",
    "print(\"Best parameters:\", best_params)\n",
    "\n",
    "# Train the best model\n",
    "co_clf = grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.3 Assess Best Model Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [scikit classification_report](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) provides several key metrics for evaluating the performance of a classification model:\n",
    "\n",
    "- *Precision*: Precision is the ratio of correctly predicted positive observations to the total predicted positive observations. It's a measure of a classifier's exactness. For example, a precision of 0.84 for Business means that 84% of the instances predicted as Business were actually Business.\n",
    "- *Recall*: Recall (also known as sensitivity) is the ratio of correctly predicted positive observations to all observations in the actual class. It's a measure of a classifier's completeness. For example, a recall of 0.90 for Sports means that 90% of the actual Sport instances were correctly identified by the model.\n",
    "- *F1-Score*: The F1-score is the weighted average of Precision and Recall. It takes both false positives and false negatives into account. An F1-score reaches its best value at 1 (perfect precision and recall).\n",
    "- *Support*: The support is the number of actual occurrences of the class in the specified dataset. For example, there are 20 instances of the class Business in the dataset.\n",
    "\n",
    "- *Accuracy*: The accuracy of the model is the ratio of correctly predicted observations to the total observations. An accuracy of 0.88 means that 88% of all predictions were correct.\n",
    "- *Macro Avg*: Computes the metric independently for each class and then takes the average (hence treating all classes equally). For example, the macro avg precision of 0.88 means the average precision across all classes is 88%.\n",
    "- *Weighted Avg*: Takes the average of the metrics, weighted by the number of instances (support) of each class. This is useful if class imbalance is present in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 88.80%\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Business       0.85      0.82      0.83        55\n",
      "    Sci/Tech       0.89      0.85      0.87        65\n",
      "      Sports       0.90      0.93      0.91        69\n",
      "       World       0.91      0.95      0.93        61\n",
      "\n",
      "    accuracy                           0.89       250\n",
      "   macro avg       0.89      0.89      0.89       250\n",
      "weighted avg       0.89      0.89      0.89       250\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = co_clf.predict(X_test)\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(f\"Accuracy: {accuracy * 100:.2f}%\\n\")\n",
    "\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Random Forest with Bag-of-Words Embeddings (CountVectorizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score, classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 3.1 Vectorization with CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`CountVectorizer` is used to transform a block of text into a vector on the basis of the frequency (count) of each word that occurs in the entire text. In practice the process is as follows:\n",
    "\n",
    "- *Tokenization*: It splits the text into individual words (tokens). By default, it splits text by whitespace and punctuation.\n",
    "- *Lowercasing*: All words are converted to lowercase to ensure that the same word in different cases is counted as the same token.\n",
    "- *Filtering Stop Words*: Optionally, it can remove common words ('stop words') that usually don't carry much meaning.\n",
    "- *Counting*: It counts the occurrences of each unique token in each document.\n",
    "- *Vectorization*: Finally, it converts the counts into a numerical format suitable for machine learning algorithms, creating a document-term matrix. Each document is represented as a row, and each unique token is represented as a column. The value in each cell is the count of the token in that particular document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 7764)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = CountVectorizer()\n",
    "X_cv = vectorizer.fit_transform(ds_sm['text'])\n",
    "X_cv.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# labels\n",
    "labels = {0: \"World\", 1: \"Sports\", 2: \"Business\", 3: \"Sci/Tech\"}\n",
    "y = [labels[x] for x in ds_sm['label']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 3.2 Train Model w/ basic Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike model parameters, which are learned from the data during training, `hyperparameters` are set prior to training and have a significant impact on how the model behaves and performs. For the sake of simplicity we aim to optimize:\n",
    "\n",
    "- Number of Trees (`n_estimators`): Generally, more trees increase the model's performance and robustness but also increase computational complexity and training time.\n",
    "- Maximum number of leaf nodes (`max_leaf_nodes`): This parameter can influence the shape and complexity of the trees that are grown during the training of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_cv, y, test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`GridSearchCV` evaluates all the possible combinations of hyperparameter values using `cross-validation`. With our param grid, the grid search will explore 6 combinations (2 from `n_estimator`, and 3 from `max_leaf_nodes`), and will train each model 3 times (since we are defining three-fold `cross-validation`) using all available CPU cores (`n_jobs=-1`). That is, 18 rounds of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters: {'max_leaf_nodes': 12, 'n_estimators': 250}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid = {\n",
    "    'n_estimators': [250, 500],\n",
    "    'max_leaf_nodes': [8, 10, 12],\n",
    "}\n",
    "\n",
    "clf = RandomForestClassifier(random_state=42)\n",
    "\n",
    "grid_search = GridSearchCV(estimator=clf, param_grid=param_grid, cv=3, n_jobs=-1)\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "best_params = grid_search.best_params_\n",
    "print(\"Best parameters:\", best_params)\n",
    "\n",
    "cv_clf = grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 3.3 Assess Model Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 66.00%\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Business       0.79      0.67      0.73        55\n",
      "    Sci/Tech       0.46      0.85      0.59        65\n",
      "      Sports       0.83      0.51      0.63        69\n",
      "       World       0.93      0.62      0.75        61\n",
      "\n",
      "    accuracy                           0.66       250\n",
      "   macro avg       0.75      0.66      0.67       250\n",
      "weighted avg       0.75      0.66      0.67       250\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = cv_clf.predict(X_test)\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(f\"Accuracy: {accuracy * 100:.2f}%\\n\")\n",
    "\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}