speed_test_fastai_vs_hf_nlp_datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Speed: Fastai vs HuggingFace nlp Datasets\"\n",
    "\n",
    "> Speed-test: Fastai's `TextDataloders` vs HuggingFace's `nlp` Datasets\n",
    "\n",
    "- badges: true\n",
    "- categories: [nlp, fastai, dataloader]\n",
    "- image: images/bokeh_mini.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tl;dr\n",
    "- Fastai's `Textdataloader` is well optimised and appears to come in faster on `nlp` Datasets for a dataset of 1.6M\n",
    "\n",
    "## Speed\n",
    "I started playing around with HuggingFace's [`nlp` Datasets library](https://huggingface.co/nlp) recently and was blown away by the speed at which you can iterate through the data, thanks to some PyArrow wizardry its seriously fast!\n",
    "\n",
    "> twitter: https://twitter.com/Thom_Wolf/status/1272512974935203841\n",
    "\n",
    "So I wondered if there was a significant speed up to be gained by doing as much text processing as I could with this library as opposed to Fastai's default text processing. (Also, after previously discovering Fastai's functionality to do [faster text loading](https://forums.fast.ai/t/nlp-speed-up-if-using-sorteddl/74636) I was in the mood for further speed-ups! 💨)\n",
    "\n",
    "\n",
    "## `nlp` Datasets\n",
    "\n",
    "The [`nlp` Datasets library](https://huggingface.co/nlp) is incredibly memory efficient:\n",
    "\n",
    "> It provides a very efficient way to load and process data from raw files (CSV/JSON/text) or in-memory data (python dict, pandas dataframe) with a special focus on memory efficency and speed. As a matter of example, **loading a 18GB dataset like English Wikipedia allocate 9 MB in RAM** and you can iterate over the dataset **at 1-2 GBit/s** in python.\n",
    "\n",
    "It also hosts 130+ common nlp research datasets AND (thanks to [this pointer from Thomas Wolf](https://discuss.huggingface.co/t/nlp-0-3-0-is-out/50/3) on the new HuggingFace forums) I also learned that you can also easily load your own CSVs (or jsons, pandas dataframes) and bask in all of that speedy goodness, for example like below:\n",
    "\n",
    "```\n",
    "from nlp import load_dataset\n",
    "dataset = load_dataset('csv', data_files='my_file.csv')\n",
    "```\n",
    "\n",
    "By the way, if you're curious to learn more about PyArrow then I highly recommend Dejan Simic's [post about it](https://towardsdatascience.com/apache-arrow-read-dataframe-with-zero-memory-69634092b1a)\n",
    "\n",
    "> twitter: https://twitter.com/simicdds/status/1276521257304023046\n",
    "\n",
    "> Note: If you love the sound of laptop fans spinning like sonic the hedgehog 🦔, redhot battery packs 🔥 and the adrenaline 😰 of living on the edge  of pandas' capabilities as you explore, plot and manipulate your giant text datasets, then the `nlp` Datasets library probably isn't for you. \n",
    "\n",
    "Otherwise, regardless about using it for your final DL pipeline or not, `nlp` Datasets is definitely worth using just for the shear speed at which it can apply functions to GB's of data.\n",
    "\n",
    "So, is it faster? Lets see! \n",
    "\n",
    "## The Setup\n",
    "\n",
    "To find out we'll be comparing Fastai's high-level `TextDataloders` class to a custom dataprocessing pipeline using HuggingFace's `nlp` datasets library.\n",
    "\n",
    "#### Fastai\n",
    "This Fastai class does a bunch of different things, all by calling just 1 line of code, including:\n",
    "- Pre and Post Processing\n",
    "- Tokenization with Spacy's tokenizer, including creating a vocabulary and **parallelising** the tokenization\n",
    "- Speed optimizations including sorting data by text sample length and padding only to the longest item in the sequence, [similar what was described here](https://towardsdatascience.com/divide-hugging-face-transformers-training-time-by-2-or-more-21bf7129db9q-21bf7129db9e)\n",
    "- Creating the train and validation dataloaders and putting them onto the gpu\n",
    "\n",
    "#### HuggingFace `nlp` Datasets\n",
    "The `nlp` Datasets pipeline I wrote tries to replicate all of the core functionality of `TextDataloaders` as best I could. \n",
    "\n",
    "> Note: I couldn't figure out how to parallelise the text processing with `nlp` although this is probably down to my lack of experience with parallelism as opposed to a limitation of the library\n",
    "\n",
    "## Sentiment Dataset\n",
    "For this experiment I used the [Sentiment140](https://huggingface.co/datasets/sentiment140) dataset, a sentiment classifcation dataset of Twitter data. \n",
    "\n",
    "For our experiment we'll use\n",
    "- 10% of sentiment dataset (160,000 tweets, 11.8M space-separated tokens), pulled from nlp library\n",
    "- 80/20 train/val split\n",
    "\n",
    "## Experiment Settings\n",
    "\n",
    "The timings will consist of 2 elements, an \"init\" and \"1 epoch\". The former will covering the entire process from loading the data (already downloaded) to creating the dataloaders. The second element will simply consist of iterating through the entire training dataset once.\n",
    "\n",
    "### init Details\n",
    "\"init\" consists of:\n",
    "\n",
    "0. Reading the data from disk, from a csv for fastai and from a PyArrow file for `nlp`\n",
    "\n",
    "1. Applying [fastai's default text pre-processing functions](http://dev.fast.ai/text.core#Preprocessing-rules). These will:\n",
    "\n",
    "\n",
    "    Fix various messy bits of html sometimes seen in documents\n",
    "    Replace repetitions at the character level, e.g. `cccc` becomes: `TK_REP 4 c`\n",
    "    Replace word repetitions, e.g. `cow cow cow cow` becomes: `TK_WREP 4 cow`\n",
    "    Add spaces around / and #\n",
    "    Remove multiple spaces \n",
    "    Replace tokens in ALL CAPS by their lower version and add TK_UP before.\n",
    "    Replace characters in ALL CAPS by their lower version and add TK_UP before.\n",
    "    Lowercases everything\n",
    "\n",
    "\n",
    "2. Tokenizing based on Spacy's tokenizer (fastai's default)\n",
    "\n",
    "3. Applying a post-processing rule which replaces embedded spaces in a token with unicode line char to allow for split/join\n",
    "\n",
    "4. Performing 1 epoch iterating through the training data, bs = 64\n",
    "\n",
    "\n",
    "## Results\n",
    "\n",
    "#### 10% Data\n",
    "Results are...mixed! While the Fastai convienience function had a faster init (48s vs 71s), the PyArrow-backed `nlp` run through a single epoch was significantly faster (11s vs 14s).\n",
    "\n",
    "| 0.16M ROWS: | Init (s)| 1 epoch (s) | 1 mini-batch [bs=64] (ms) |\n",
    "| :- | :-: | :-: | :-: |\n",
    "| **Fastai** | 124 | 14.3 | 7.4 | \n",
    "| **Fastai w/sorted** | **48.1** | 14.3 | 7.4 |\n",
    "| **nlp** | 71.2 | **11.3** | **5.6** |\n",
    "\n",
    "#### 100% Data\n",
    "Surprisingly for the full dataset of 1.6M tweets, Fastai dramatically outperforms `nlp` Datasets.\n",
    "\n",
    "| 1.6M ROWS: | Init (s) | 1 epoch (s) |\n",
    "| :- | :-: | :-: |\n",
    "| **Fastai w/sorted** | **484** | **142** |\n",
    "| **nlp**| 1290 | 323 |\n",
    "\n",
    "## Thoughts\n",
    "- **Memory** Note that Fastai was much faster on the full dataset, but I also had to delete the pandas dataframe used to calculate the text lengths as it was taking up too much memory and causing my dataloaders to fail, I probably should find a more memory efficient way to do this calculation. On the other hand, `nlp` datasets won't incur this issue.\n",
    "- **nlp pipeline** Given the large difference for the full dataset, I am suspicious about some parts of my implementation, for example sorting the entire dataset takes 416s and splitting it into a train and validation set takes 416s. Do I need to sort the full dataset? Should chopping the dataset in 2 really take so long? Removing these two steps bring the timing down to 458s, slightly faster than Fastai."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU1fn48c+TfQ8JSVgSthA2IYosIuACWoqIorW11K1a/Wlra61drK1+Xdqvttpaa22r/VK/av2q1KpVaq1LKyhtbQW0iiCyhiVhCxASsmdmnt8fZ5KZyUISSDKZ5Hm/Xvc1M3fuvXNmIM9z7znnniOqijHGGBMV7gIYY4zpHSwhGGOMASwhGGOM8bOEYIwxBrCEYIwxxs8SgjHGGMASgokQIhItIpUiMrwrt+3tRKRARHq0b7iIbBSR03vyM03vEBPuApi+SUQqg14mAXWA1//6y6r6dGeOp6peIKWrt+3vROQpYIuq3tW4TlXHha9EJpwsIZhuoapNAVlEtgP/T1X/1tb2IhKjqp6eKFtX6s3l7s1lM72TVRmZsBCRu0XkWRFZKiJHgMtFZKaI/FtEDovIHhF5SERi/dvHiIiKyEj/66f8778qIkdE5F8iMqqz2/rfXyAim0SkXER+KSL/FJGrOlHup0TkrqBtPuVPgo2vi0XkWyLykf8zlopIfBvHjxaRn4vIQRHZCpzT7P1iEZnTrDxP+J8X+L/3l0RkJ/CGiESJyPMistf/u74lIhP8238VWAzc6q9ie7H5Z4hIgv+32yMiJSLygIjEBX9PEfmuiJSKyG4R+WLb/+qmt7OEYMLpM8AzQDrwLOABvgFkAbNxwfDLR9n/UuB2IBPYCfx3Z7cVkRzgD8DN/s8tAk7pZLk74vPAPCAfmApc0cZ21wOfBk7yl+PzHTx+sDOA8cBC/+s/A2OAwcA64P8AVPVhXPl/pKopqvqZVo51BzANOBE4Gffv8v2g9/OARGAo8BXgERFJO4Yym17AEoIJp3+o6suq6lPVGlVdrarvqqpHVbcBS4Azj7L/86q6RlUbgKeBycew7XnAB6q6zP/ez4EDnSl3h74pPKiqe1X1IC5At1XWzwM/V9Vi/7b3dvD4we5U1Wr/b+pT1SdU9Yiq1gJ3AVNFJLmDx7oMuEtVS1V1P/BDQpNZLXC3qjao6p9wbUVjj6HMphewhGDCaVfwCxEZLyKv+Ks3KnDBJ+so++8Nel7N0RuS29p2aHA51I32WNyZcndQR8saUh5gxzF8VtP+/iqon4jINv9vusX/1tF+12BDmpVhB5Ab9PqAvxG/UXv/DqYXs4Rgwql5d8r/wVVpFKhqGq66Qrq5DHtw1R4AiIgQGvBa07zcVbieVI0GH2d5hgW9bt51tt3P0tAhjL8InAuchaviKvCvb/xd2+vSugcY0aw8Je3sYyKUJQTTm6QC5UCVv+HzaO0HXeXPwBQROV9EYnBtGNmdPMYHwEIRyRCRIcCNx1GePwA3iUiuiAwEbmnls77gbzg/BbioneOl4qpxDuISyT3N3t+Ha9doy1LgDhHJEpFsXDvMUx37KibSWEIwvcm3gSuBI7irhY422B4zVd2H62nzAC5ojgb+gwuiHfUEsAFXnfIa8PvjKNIjwJvAR8Bq4Plm79+GazA+jAvOz7RzvMeB3f5lPfBOs/cfBU4SkTIRaf5ZAD8APvSXZy3wLvDjjn4ZE1nEJsgxJkBEonHB83Oq+vdwl8eYnmRXCKbfE5FzRCTdf2/A7bjur6vCXCxjepwlBGPgNGAbrrvpOcCFqtqZKiNj+gSrMjLGGAPYFYIxxhi/iB7cLisrS0eOHBnuYhhjTER57733Dqhqi+7VEZ0QRo4cyZo1a8JdDGOMiSgi0uod8BFZZeS/iWhJeXl5uItijDF9RkQmBP/AYtelp6eHuyjGGNNnRGRCMMYY0/UsIRhjjAEsIRhjjPGLyIRgjcrGGNP1IjIhWKOyMcZ0vYi+D8EYY/o8Vag7DEeK4cguqPQ/TvwSDDjaVBadZwnBGGPCpa1gfyTosbIYGqpC95MoGDLTEoIxxkQEVagrbzvQN65vLdgnD4HUPMguhFELIHWYe506DFLyIGUIRHV9+LaEYIwxndUY7I8W6I/s6lXBviMiMiGIyPnA+QUFBe1ua4wxnebzQuVuqNgOFTv8i/95Y+BvqAzdJzjYD5wII88JDfSpwyB5METHhuMbdUhEz4cwbdo0tcHtjDGd5q13gb0x2JdvhyNBzyuLwecJ3ScpB9JGQOrwlmf1ERDsg4nIe6o6rfn6iLxCMMaYo2qohoqdrZ/hV+xwZ/8EnwwLpOS6gJ872z2mjYC0kYEkEJsYlq/SkywhGGMiT1156Nl9xQ53ht/4vKY0dPuoGHcWnzYSRswLDfZpI9zZfnRcz3+PXsYSgjGm9/HUQXkRlG+Fw1uhfJt73ZgE6g6Hbh+TAKn+4J5zMqSP9Af6Ee558hCIig7HN4kolhCMMeFRc8gF+sNbA4G/MfgfKSakSic2GdJH+at0TmtZpZOUAyLh+iZ9RkQmBOtlZEwEUJ8L7I1Bving+x+bn+UnDYIBo2HYHEgf7Z6n57tHC/g9wnoZGWOOXUMNVBSFnt03Pq8ocr15GkXFuDP6xiA/YHRQ4B8FcSlh+xr9jfUyMsZ0nirUHmp5dt8Y+CtLQrePS3VBPmsSFFwQGvxTh4XthivTMfavY4xxjbiHt0DZRji0MfSxtix02+QhLsCP+FTLqp3ELKvaiWCWEIzpL1Shep8L9Ic+CQ365UWuzr9RylDIGAfjFkPG2NCqndik8H0H060sIRjT13hqoWxzIOAHB//6isB2MYku2OdMhfGXQuY4lwQyxkJ8WvjKb8LGEoIxkUjV3W3bvIrn0Ceun35wl82UPMgcDydc4QJ+pn9JHebG3zHGzxKCMb1ZQ7U7229exXNoY+jgarHJ7sx+yKkw8arA2X7mWPeeMR0QkQnB7kMwfY6nDg5tgAMfwYF1/sf1cGRn0EYCacNdoJ/0JX/AH++Cf0quNeaa42b3IRjTk9TnGnBLP/IHfX8CKNsE6nXbRMdB5gQ3hHJjwM8YBxljrEHXdAm7D8GYnla9v2XgP7g+dNKU9HzIKoSxn4WBk9ykKQPGRMwwyqZvsYRgzPFqqHLVOweaBf/q/YFtErNdsC/8fy4BZE1yVwB2d67pRSwhGNNRPo9r4G0M+o1n/+VFNPXqiUmCrImQf54/8PuDf/KgsBbdmI6whGBMc6puULaQBt6PXKNv49g8Eu3q9AdNdb16sgrdFUD6KOvKaSKWJQTTv6m6fvv71sDeNbBvNex7P3QkzpQ8d5Y/4tMu6GcVusbemITwlduYbmAJwfQvlbth7+qgBLAGag6496JiIfskN1xD9omB6p6EjPCW2ZgeYgnB9F3VpYHA35gEqva49yTa1fWPXgSDp8OgaS4BxMSHt8zGhFFEJgS7Mc20UFsG+94LnPXvXR10U5e4Kp4Rn3KBf/B0dyVgffqNCWE3ppnIU38E9v8n9Mz/8JbA+wNGBwL/oGlujl0brM2YJnZjmolMDTVQ+kHomf+hT2jq5pk6zAX+SVe74D9oKiRmhrXIxkQqSwim9/DWu+6dwWf+B9YFhnRIGuSC/7jF/rP/qda/35guZAnBhE9DNez+FxS/7ZY974K3zr2XkOmCfv55geqflKE2gJsx3cgSguk59Ueg5J9QvNIlgL2rwdfgbuTKORkmfxWGzITB09xk7Bb8jelRlhBM96k9DCX/CFwB7HvfVf9Exbiz/qnfgrwzIHc2xKeHu7TG9HuWEEzXqT4AJX93wX/X21D6IaBuOOfBp8Ap34O8M2HoTBvUzZheyBKCOXZV+wLBv/htN7QzuCEdhsyEmXfCsDNh8AyITQxvWY2JYPX1UFICu3a5ZedOuPRSGDGiaz/HEoLpuCPF/uqflS4JlG1062OTYehsmHAp5J7hGoDtjl9jOsTng717Q4N94/PGZe9eN+xWsEmTLCGYnlS+PfQKoHybWx+XBrmnub7/w86EnCk2oYsxrVCFsrKWQT74dXExeDyh+yUnw7BhbiksDDwfNgyGD4e8PLdNV7OEYBxVd7dvY/AvXhkY+iEhw535n3yDawPIPgmiosNbXmN6gcrKlmfzzQN+dXXoPrGxkJvrAvvs2YEgHxz0MzLC08kuIhOCjWXUReoqoOhV2PYy7FweGPgtMdud+U+/2fUCyppkY/ybfkcVDhyA7dtDl507AwG/rCx0HxEYPNgF9UmT4NxzQ8/shw2DQYMgqpf+OdlYRv1NxS7Y+ifYugx2veXuA0jMguGfckkg70w3EJzdA2D6uMbqnKKi0IAf/LqqKnSfAQNcvX3zKpzG57m5EBfX41+l02wso/5K1XX/3LLMJYL977v1GWNgyjdg9AWuG6hVAZk+6PDhlkE++PWRI6Hbp6XBqFFQUACf+pR7PnJkYEnv47fLWELoi7wNrh2gMQkc2QmIC/yn3+uSwMDx4S6lMcetoqL1M/vG1+XlodunpASC/Jw57jE46A8Y0L8vji0h9BV15a49YOufoOgv7nVMAgyfBzPvcGMC2UBwJsLU17vAvmWLe2we9A8dCt0+KSkQ5GfPDgT6xqCfmdm/A357LCFEslbbA7Kh4CIouABGzLNJYEyvV1MD27a5oN982bnT9dNvlJAQCPIzZrQM+FlZFvCPhyWESBLSHrDMTRIDkDEWptzkksCQU609wPQ6lZWwdWvrQb+4OHTbzExXhz9rFnzxi+756NFuycmxgN+dLCH0dkdtD7jPJYHMceEupTGUl7ce8LdscXfaBsvJcYH+rLPcY+MyerRLCCY8LCH0Ro3tAVuWwfZX/e0Bia4KaOadMPo8SMoJdylNP6Pq6uzbCvoHDoRuP3SoC/Lnntsy6KfZjKa9kiWE3qJip7sC2LIMit8Cn8e1B4z5rOsVNOJT1h5gekRNDWzaBBs2uGXTpkDQP3w4sJ2I63tfUAAXXRQa9PPzu2doBdO9LCGE08GPYeMf/PcHNLYHjHPzBIy+AIbMsPYA020OHoRPPgkE/g0b3Ovt2wMDqUVFucbaMWNcI27jGX5BgWvITUgI5zcwXc0SQk/z1sPmP8KHj7jxghAYOgvO+AmMXmTtAaZL+XxuiIXgwN/4vLQ0sF1iIowb54L+VVfBhAluKSiwoN+fWELoKRU7YO0S+OhRqN4P6fkuCZzwRbs/wBy3+nrYvLnlGf/GjaGDqw0c6AL9BRcEgv748W44ht46vo7pOZYQupP6YPvr8MEjUPSKW5d/nps7eMQ8GzDOdFp5uQv6zc/4t24Frzew3YgRLtifeWZo4M/ODl/ZTe9nCaE7VB+AdY/B2v9xcwgkDYJTvg8nXgdpw8NdOhMB9u+Hjz5qeca/Z09gm9hYGDsWTjwRPv/5QOAfO9YadM2xsYTQVVRh979c28CmP7i2grwz4bQfwZjPuHmFjWmmocEF/bVr4cMPA8u+fYFt0tJcoJ8/353lNwb+UaMgxv6CTReKyP9OvWo+hPpK2PA0fPgwlK51s4kVXgcnfQWyJoa7dKYXOXAgEPAbE8DHH7v6f3DDJk+cCAsWwEknuZmyJkyAIUPs7lzTM2w+hGN1YJ1rG9jwf1B/BLInu7aB8ZdAXEp4ymR6BY/H9d0PPuNfuxZ27w5sM2SIq+o56aTAMnasqwYyprvZfAhdobHL6AcPQ8nfIToexn0eTvqqu2fATuP6nUOHWlb3rF8PdXXu/dhYOOEEOPvsQOA/8UQ3dIMxvY0lhI5otcvoT2HiVZCUFe7SmR7g9bpuncHVPR9+GDowW06OC/hf/3rg7H/8+MiYQcsYsITQtqYuow+7+QXAuoz2E4cPhwb9tWth3To3pAO4htzx412XzsYz/pNOcnPpGhPJLCE0V10a1GW0yLqM9nH19S7ov/tuYNm8OfD+wIEu2H/lK4EqnwkTID4+fGU2prtYQoCgLqMPw6bnAl1GT78XCi60LqN9hKqbcSs4+P/nP4H6/sGDA0M3nHyyC/7Ww8f0J/07IdQf8XcZfSTQZfTEL7suowNPCHfpzHE6fBhWrQoE/1WrAuP3JCbC1Klwww0uCcyY4UbutOBv+rP+mRAOrHdtA8FdRuctsS6jEayhwdX1B5/9b9wYeH/CBFi4MBD8J02yLp7GNNc/E8IHv4J1j1uX0QilCjt2hAb/99+H2lr3fk6OC/pXXOEep0+H9PTwltmYSNA/E8Kpd8Cs/7YuoxGivBxWrw5NAPv3u/cSEmDKFLj++sDZ/4gRlt+NORb9MyGkDAl3CUwbPB43qFtw8P/kk8CELePGwTnnBIL/iSda1Y8xXaV/JgTTa9TXw7//DW++CStWwJo1gf7+WVku6F9ySaDqJyMjvOWNdA0NDRQXF1PbWL9m+rSEhATy8vKI7eBZkyUE06O8XvjgA5cAli+Hv//dTeAi4nr9XHdd4Ox/1Cir+ulqxcXFpKamMnLkSMR+3D5NVTl48CDFxcWMGjWqQ/tYQjDdStX19lm+PHAVUFbm3pswAa6+2o3zc+aZdvbfE2pray0Z9BMiwsCBAykNniu1HZYQTJfbtSuQAJYvh5ISt374cLjwQpcAzjrL3fRlep4lg/6js//WlhDMcTt40J35v/mmWxqHfsjKcoH/7LPdkp9vVUDG9GaWEEynVVa6uv/GBPDhh65qKCXFVf1cf71LBIWFNnG7MZHE/lxNu+rrYeVKuPNOOP10V9d/7rnwy1/CgAHwwx/CO++4uQH+/Gf45jfdOECWDExbHnroISZMmMBll13W4X0OHz7Mww8/3KFtZ82adaxFa9eDDz5IdXV1p/Z56623OO+889p8/5NPPmHmzJnEx8dz//33h7z32muvMW7cOAoKCrj33nuPqcwdZVcIpoW2egJFRbmeQN/5jrsCmD0bkpLCXVpzrG7aDB9Udu0xJ6fAg2Pa3+7hhx/m1Vdf7XDvFwgkhK9+9avtbvvOO+90+Lid4fV6efDBB7n88stJ6sL//JmZmTz00EO89NJLLT7va1/7Gn/961/Jy8tj+vTpLFq0iBNO6J6x1uwczqDqbv56+GH47GchOxumTYNbbnENxNdcAy++6NoKVq2CH/8Y5s2zZGCOzVe+8hW2bdvGokWLuO+++5g1axYnn3wys2bNYqN/AKr169dzyimnMHnyZE488UQ2b97M9773PbZu3crkyZO5+eabqays5Oyzz2bKlCkUFhaybNmyps9ISWl7TLI9e/ZwxhlnMHnyZCZNmsTf//53AJYuXUphYSGTJk3illtuCTnWHXfcwYwZM7jnnnvYvXs3c+fOZe7cuQC88cYbzJw5kylTpnDxxRdTWemy7Guvvcb48eM57bTT+OMf/3jU3yQnJ4fp06e3uF9g1apVFBQUkJ+fT1xcHF/4whdCvmeXU9WIXaZOnarm2Hi9qu+8o/rNb6oOH67q0oLqiBGqV1+t+tRTqrt3h7uUpqt9/PHH4S6CqqqOGDFCS0tLtby8XBsaGlRV9a9//atedNFFqqp6ww036FNPPaWqqnV1dVpdXa1FRUU6ceLEpmM0NDRoeXm5qqqWlpbq6NGj1efzqapqcnJym599//336913362qqh6PRysqKrSkpESHDRum+/fv14aGBp07d66++OKLqqoK6LPPPtui7I2fe/rpp2tlZaWqqt577736gx/8QGtqajQvL083bdqkPp9PL774Yl24cGG7v8udd96pP/3pT5teP/fcc3rNNdc0vX7yySf1a1/7WrvHCdbavzmwRluJqVZl1I/4fO6u4Oeeg+efd9M/xsXB/Plw223WE8j0vPLycq688ko2b96MiNDQ0ADAzJkzueeeeyguLuaiiy5izJiW9VCqyq233srKlSuJioqipKSEffv2MbidqeumT5/O1VdfTUNDAxdeeCGTJ09m+fLlzJkzh+zsbAAuu+wyVq5cyYUXXkh0dDSf/exnWz3Wv//9bz7++GNmz54NQH19PTNnzuSTTz5h1KhRTeW+/PLLWbJkSad/H20csyVId3YbtoTQx/l88K9/BZJASYlLAuec46p+zj/fRgI14XP77bczd+5cXnzxRbZv386cOXMAuPTSS5kxYwavvPIK8+fP59FHHyU/Pz9k36effprS0lLee+89YmNjGTlyZIeG5DjjjDNYuXIlr7zyCldccQU333wzaWlpbW6fkJBAdHR0q++pKvPmzWPp0qUh6z/44IMuCdx5eXns2rWr6XVxcTFDhw497uO2xdoQ+iCfD/7xD/jGN9zNYKedBr/5jWsXeOopN0nMsmVw+eWWDEx4lZeXk5ubC8ATTzzRtH7btm3k5+dz4403smjRItauXUtqaipHjhwJ2TcnJ4fY2FhWrFjBjh07OvSZO3bsICcnh2uvvZZrrrmG999/nxkzZvD2229z4MABvF4vS5cu5cwzz2x1/+BynHrqqfzzn/9ky5YtAFRXV7Np0ybGjx9PUVERW7duBWiRMDpq+vTpbN68maKiIurr6/n973/PokWLjulYHWFXCH2Ezwf//Ke7EnjhBdi92837u2AB/OQncN55cJSTIGPC4rvf/S5XXnklDzzwAGeddVbT+meffZannnqK2NhYBg8ezB133EFmZiazZ89m0qRJLFiwgFtuuYXzzz+fadOmMXnyZMaPH9+hz3zrrbf46U9/SmxsLCkpKTz55JMMGTKEH//4x8ydOxdV5dxzz+WCCy5odf/rrruOBQsWMGTIEFasWMETTzzBJZdcQp1/Lta7776bsWPHsmTJEhYuXEhWVhannXYa69ata7NMe/fuZdq0aVRUVBAVFcWDDz7Ixx9/TFpaGr/61a+YP38+Xq+Xq6++mokTJ3biF+4caa2OKhxEZALwDSALeFNVH2lvn2nTpumaNWu6vWy9ldcbmgT27HHzAyxYABdf7JJAamq4S2l6kw0bNjBhwoRwF8P0oNb+zUXkPVWd1nzbdq8QRGQmcDlwOjAEqAHWAa8AT6lq+VH2fQw4D9ivqpOC1p8D/AKIBh5V1XtVdQPwFRGJAn7b7rfsp7xed1/Ac8/BH/8Ie/e6JHDuuS4JLFxoScAYc2yOmhBE5FVgN7AMuAfYDyQAY4G5wDIReUBV/9TGIZ4AfgU8GXTMaODXwDygGFgtIn9S1Y9FZBHwPf8+xs/rdXcKNyaBffvcJPHBSeAo3a6N6Zc++ugjrrjiipB18fHxvPvuu2EqETz++OP84he/CFk3e/Zsfv3rX4epRKGOWmUkIlmqeuCoB2hnGxEZCfy58QrBf8Vxl6rO97/+PoCq/jhon1dUdWEbx7sOuA5g+PDhUzvakBRpPJ7QJLB/v0sCCxcGkkBycrhLaSKNVRn1P11WZdQY6EUkGahRVZ+IjAXGA6+qakN7CaMVucCuoNfFwAwRmQNcBMQDfzlKmZYAS8C1IXTys3s1jwfefjuQBEpL3d3AjUng3HMtCRhjuk9HexmtBE4XkQzgTWANsBjo+MhUAa11zlVVfQt46xiOF9E8HnjrLZcEXnwxkATOOy+QBGyICGNMT+hoQhBVrRaRa4BfqupPROQ/x/iZxcCwoNd5uHaKfqWoCB58EJ55Bg4ccGf+jUlgwQJLAsaYntfRG9PEX/d/Ga53ERz7PQyrgTEiMkpE4oAvAG01SrdVmPNFZEl5eZsdnHqt996DL3wBCgrgkUfccBEvvOCuDH7/eze4nCUD09fZ8NehjmX466KiImbMmMGYMWNYvHgx9fX1nfsirWltgKPmC3AGLmjf4n+dDzzUgf2WAnuABtyVwTX+9ecCm4CtwG0dKUNrS6QMbufzqf7lL6pz57oB5NLSVL/7XdXi4nCXzPQ3vWVwu3Hjxum2bds6tU/zwe3CwePxhAxu11ErVqw46uB2+/bt01WrVumtt94aMridx+PR/Px83bp1q9bV1emJJ56o69evV1XViy++WJcuXaqqql/+8pf14YcfbvXYXT64naquxLUjNL7eBtzYgf0uaWP9XzhKw3FfUV8PS5fC/ffDunWQl+eeX3ut3TVswu+mm9y8F11p8mRXFXo0wcNfX3755SxbtoyamhoSExN5/PHHGTduHOvXr+dLX/oS9fX1+Hw+XnjhBW6//fam4a/nzZvHnXfeyQUXXEBZWRkNDQ3cfffdTXcXp6SkNA1D3dyePXtYvHgxFRUVeDweHnnkEU4//XSWLl3Kj370I1SVhQsXct999zUd61vf+havv/46CxcubBr+OisrixUrVvDGG29w5513UldXx+jRo3n88cdJSUnhtdde46abbiIrK4spU6Yc9TfJyckhJyeHV155JWR98PDXQNPw1xMmTGD58uU888wzAFx55ZXcddddXH/99e3+Gx1Ne/chLMG1GXzUynvJuIblOlV9+rhK0ceUl8OSJfCLX7jB5AoL4cknYfFiN7CcMf3Zb37zG1577TVWrFhBXFwc3/72t4mJieFvf/sbt956Ky+88AK/+c1v+MY3vsFll11GfX09Xq+Xe++9l3Xr1vGBP4t5PB5efPFF0tLSOHDgAKeeeiqLFi1qd1C5Z555hvnz53Pbbbfh9Xqprq5m9+7d3HLLLbz33ntkZGTw6U9/mpdeeokLL7yQqqoqJk2axA9/+EMAHnvsMVasWEFWVhYHDhzg7rvv5m9/+xvJycncd999PPDAA3z3u9/l2muvZfny5RQUFLB48eJj+q1KSkoYNizQ5JqXl8e7777LwYMHGTBgADExMU3rS0pKjukzgrV3hfAwcLuIFOLuTi7F3Zg2BkgDHgMsGfgVF7sk8D//A0eOuFnFHn3UDS9tQ0qb3qa9M/meYMNfH522Mfx1W+uPV3v3IXwAfF5EUoBpBIau2KCqG4/704+RiJwPnF9QUBCuIoT46CNXFfTMM26amYsvhptvhnauEo3p92z466Nra/jrrKwsDh8+jMfjISYmpsuGxe5QLyNVrVTVt1R1qaq+FM5k4C/Py6p6XXoYx25WhRUr3H0CJ57o5hr46ldhyxbXbmDJwJj22fDXR9fW8Nciwty5c3n++ecB+N3vftfm6Kyd0lpLc6Qs4ehl1NCg+vvfq06d6noM5eSo3n236sGDPV4UYzqtt/Qyauyp88477+iYMWN01qxZ+irq6oMAAB5kSURBVF//9V86YsQIVVX90Y9+pCeccIKedNJJOn/+fD3o/wO75JJLdOLEifqd73xHS0tL9dRTT9WpU6fqNddco+PHj9eioiJVPfoUmk888YROnDhRJ0+erKeddlpTb6enn35aJ02apBMnTtSbb765afvmx3rooYd03LhxOmfOHFVVffPNN3XatGlaWFiohYWFumzZMlVVffXVV3XcuHE6e/ZsveWWW47ay2jPnj2am5urqampmp6errm5uU3Tg77yyis6ZswYzc/Pb5r6U1V169atOn36dB09erR+7nOf09ra2laP3ZleRr1m+Otj0ZPDX1dVwWOPwQMPwPbtMHYsfOc7cMUVbrRRYyKBjWXU/3Tp8NfNDpKsqlXHWb6Isn8//PKX8PDDcOgQzJoFP/85LFoEUTbfnDGmD+lQQhCRWcCjQAowXEROAr6sql/tzsKF06ZN8LOfwe9+5+4nuOAC11DcjTdAGmO6kA1/3XkdqjISkXeBzwF/UtWT/evWadCkNz0pqJfRtZs3b+7SY7/zDvz0p27O4bg4uPJK+Na3YNy4Lv0YY8LCqoz6n85UGXW40kNVdzVb5T224h0/7eJeRj4fvPQSzJ7tlrffhttugx073D0FlgyMMf1BR9sQdvmrjdQ/IN2NwIbuK1bPqK11dxD/7GeuimjkSHjoIbj6apt3wBjT/3Q0IXwFNwdyLm6QujeAr3VXobrboUNupNFf/tJNRzllSmCk0ZhjHcPVGGMiXEcHtzvAsU2G0yvddBP83//BOee4huK5c21oCWOM6VAbgn/uggdE5I8i8qfGpbsL111uvx3WroVXX3XjDVkyMCb8rrrqqqY7bztj5cqVTJkyhZiYmBb7/+53v2PMmDGMGTOG3/3ud11V1D6roxUkLwH/C7wM+LqvOB1zvGMZtTJOljEmQg0fPpwnnniixcQyhw4d4gc/+AFr1qxBRJg6dSqLFi0iIyMjTCXt/TqaEGpV9aFuLUknqOrLwMvTpk27NtxlMSZSlb73Q+rKPu7SY8ZnnED21DuOus327dtZsGABp512Gu+88w65ubksW7YsZJuRI0eyePFiVqxYAbghq9s6ARw5ciQAUc3uFH399deZN28emZmZAMybN4/XXnuNSy5pdZoWQ8e7nf5CRO4UkZkiMqVx6daSGWP6rM2bN/O1r32N9evXM2DAAF544YUW26SlpbFq1SpuuOEGbrrppk5/RmtzCXTFnAF9WUevEAqBK4CzCFQZqf+1MSYCtXcm351GjRrF5MmTAZg6dSrbt29vsU3jmfwll1zCN7/5zU5/Rms33XbFkNR9WUcTwmeAfFXtglmcjTH9XXx8fNPz6OhoampqWmwTHLyPJZDn5eXx1ltvNb0uLi5umm/BtK6jVUYfAgO6syDGGBPs2WefbXqcOXNmp/efP38+b7zxBmVlZZSVlfHGG28wf/78ri5mn9LRK4RBwCcishqoa1ypqou6pVTGmH6vrq6OGTNm4PP5jjrBzOrVq/nMZz5DWVkZL7/8MnfeeSfr168nMzOT22+/nenTpwNwxx13NDUwm9Z1dHC7VqcOUtW3u7xEndCT8yEY0xdEyuB2I0eOZM2aNWRlZYW7KBGvy+dDCHfgb663zalsjDF9wVHbEETkH/7HIyJSEbQcEZGKniliS1092qkxpnfZvn17i6uDe+65h8mTJ4cs99xzT5hK2De1d4WQDKCqqT1QFmOMadNtt93GbbfdFu5i9Gnt9TKK3AmXjTHGdEp7Vwg5IvKttt5U1Qe6uDzGGGPCpL2EEI2bR9lu7zPGmD6uvYSwR1V/2CMlMcYYE1bttSHYlYExpkf05HwI7733HoWFhRQUFHDjjTe2Ou5Rf9ReQji7R0phjDHHqHE+hEsvvTRkfeN8CO+++y6rVq3iBz/4AWVlZQBcf/31LFmyhM2bN7N582Zee+21cBS91zlqlZGqHuqpgnSG3ZhmTBdYcRPs/6Brj5kzGeY+eNRNwj0fwpw5c6ioqGgaH+mLX/wiL730EgsWLOj01+1rOjq4Xa9iN6YZE9nCOR9CSUkJeXl5Ldabjg9uZ4zpa9o5k+9O4ZwPweZJaFtEXiEYYyJb8/kQPB5Pi226Yj6EXbt2Nb0uLi5m6NCh5OXlUVxc3GK9sYRgjOmlums+hCFDhpCamsq///1vVJUnn3ySCy64oKuLH5GsysgY0yt153wIjzzyCFdddRU1NTUsWLDAGpT9OjQfQm9l8yEY0zk2H0L/05n5EKzKyBhjDGBVRsaYXqi1Xkf33HMPzz33XMi6iy++uN8Mie1TaFCo90G9Qlo0xHbxKb0lBGP6GVWNyG6WfXk+BFXwqAv0jQG/+WNDs9r9ggQY0E5C6GyTgCUEY/qRhIQEDh48yMCBAyMyKUQiVfA2BvtmgT74jL956I4C4gTioiA9yj3GCsQHPR79c5WDBw+SkJDQ4bJaQjCmH2nsg19aWhruovQZjWf3Xtyjx58Agh9bO0+PFojxL9H4HyXwGO3P116gxr90VkJCQshd2e2JyIRgYxkZc2xiY2MZNWpUuIsRMVThQANsrw0sO+tgZy3sqnPLgYaW+w2KheEJMCzeLc2fD4oLBPzexLqdGmP6LVUo87hAXxQU9ItqAs+rfKH7DIhpPdA3vs6Nb786pyN8nmo81fvw1OzDW1OKp2aff9mPt2Y/WVNuJSGz8JiO3Va304i8QjDGmI4q94QG+JDAXwtHvKHbp0fDqEQYkwTzMmFUAowMWtKOM2r6PDX+oO6Ce+B542uXAHwNR1rsK9HxxCQOIjoxB/XWH19BWmEJwRgT0Y542g7222vhcLNhklKiA0F+zoBAoG9cNyD22Mrh89bh9Qd0T80+PNX78dbu95/lB4J+q4E+Ko7oxBxiEgcRnz6O6MGnE5M4iJjEHGKSBhGd4B6jYtO6tTOAJQRjTK9W63XBvaiNap2DzQJ+UlQgwM9O8z9PDAT+zBjoTExVnxdvbSme6j00VO9t5czeJQFffXnLnaNiXVBPyCE2vYDEwbOaAn1jAohJHERUXHqv6PVlCcEYE3bVXthWA5trYEuzZVddaC+dhKhAcJ+eFlqlMyoBsmI7HvDV1+ACevVePNV7/I978dQEP98H2qxeSWJcoE/MITZ1FIk5p/ircgY1rY9JHERUfEavCPQdZQnBGNMjjnhgaysBf0sNlDSrDs+KhYJEOGMAjEmE0Ykw2h/0c+IgqgMxtqkKpynQu8eGmr14qnbjqdmLt6aU5p1CJTqRmOQhxCQOIXHQTGKSBhOTNMQ9Jg4mJmkw0fEZiPS9kX8sIRhjuszhhtYD/pYa2Nese+bgOBf052W6x8ZldAfq8X2eWjw1e5ud2e8OOcP31h5ssV9UbGpTYI8fMN4f7IMCftKQbq+n780sIRhjOkwVDnnaDvrN++Tn+oP+eQObBf1ESG0j+qjPS0P1fjxVJTRUlbiz+aoSGqr3uOBfsxdfXVmL/aLi0v3BfgjxmZOCgvxgYpOGEp00iOjY1G74VfoOSwjGmBYqPfBJNWyohk3Ngn5wrx3B9b8vSITPZoUG/fxESIpueWyftw5P1W6qqxsDfgkNjUG/qgRP9R7Q0JbiqLgBxCYPJTZpCIlZU1qc1cckDSYqJql7f5R+wBKCMf2UKuxv8Af+Khf8N1S717vqAttF4eruCxLh0pzQoD8qARKaBX1vfYUL7odLOFxVQkO1C/qeqt00VJXgrW0+bIa43jbJuSRknUxs8nnEJOcSm5zrHpOGEhWb3N0/h8ESgjF9nldhR20g2AcH/7KgE/HkKBifBGcOcI8T/MvoRDewGoCqD2/tQTxVxTQcKKGmqoSKxrP8ahf0m/ezl6g4YpKHEpOcS/LQuaHBPjmXmKTBSNQxdv43XcoSgjF9RK3XddtsDPYbqlwC2FgDtUHDL2THukD/+WyYkBwI/nnxICjeukM0VO6koXInnn07Kassbqra8VTvRn2hXYKiYlP9wT2PxJwZxCYNDQr6eUQnDOyTPXL6IksIxkSYww2B+v3g4F9UC41xX3DVPOOT4OwM/9m+P/hnRNW5M/rKXS7w73fBf1flThoqd6GeqpDPi07IJjY5l/jMiaQM+zQxybnEJA1tOsuPjkvr8d/AdA9LCMb0Qqqwuz5QvROcAPYGnaDHCYxNgimpcOkgf+BPVEbHlBFbvdMf9HfQcMAF/4rKnRyq3kNw33uJjic2ZTixycNIGjST2JRhxKQM96/LIyomsed/ABMWlhCMCbNaL6yvhg8rA8vaqtD6/bRoF+zPyXSP4xPqmcBuBjXswFftP9M/tAuP/yx/b7N6/OiEbGJThrsqnZThxKYMc4+pI4hOyLIqHQNYQjCmx6i6s/sPK+HDqkDw31jtJkEBNw5PYTJcnA1T4g4zkR0M9+4iuXYnnqqdNJT46/ar94D62OffT6LiiE0ZTkzKcBKyp/uDfuOSZ10yTYdEZEKwCXJMb1fvc9U7zc/6S4Nu3BoWD9OSavlS0nZO1G2M9BSRWrMNz4Ei6ouK8NUfBqDOv0QnZLmz/OxpxKaMCDnTj07MsbN8c9xsghxjjlNpK2f9G6oDk6In4uWM+D2cFr2VQi1ieMM2MuqK0MoiPFW7Ca7Pj0kcTGxaPnGpo4hNHUlsauBM387yTVexCXKMOU4en+vCGXzW/2FVoJF3gK+MaRRxYfQ2/kuLGNGwjQG1RURVFYV01YyKSSEqLZ+47GnE5ucTl5ZPbOoo4lJH2g1YJqwsIRjTikMNsLbZWf/6KlBfHSM82xnj3capUsT/020MaygivbaI6Iag8XUkhtjU4S7Y551BXOoof+DP9zfi9s/B00zvZgnB9HtVXlhzBN6tcMuaCh+e6t2M9BSR793GBF8RF/i2kddQRGp9CRJUxROdOMhV7wxaEBT0RxGbMgyJsj8vE1nsf6zpV3zq6vffrYB3y5Wth3bjq9jE6IaNjPVs4hrvZn7YsIU4rW3aJyomhdi0UcRlTSU27XPEpea716kjiYpNCeO3MaZrWUIwfdq+ehf4Pzx4gL0HNuGp2MTw+o2MadjEVzybSdHKpm0lcTCJA8cSl3YZcemueicuLZ/ohGyr4jH9giUE02fUeOGDsnI27t3EwYOb0IqNDK7bxBjPZr7gO9S0XUNsBjHp40jPuIj4AWOJTx9L3ICxRMelh7H0xoSfJQQTkbwN1Wzdv5lt+zZRXraR6CObGFS7mcG+vWT5t6mNSqE6eSwJgz5NetZYUjLGEpc+1hp1jWmDJQTTq/m8dTRUbOPgoU3s2r+J6sMbia/cTHbDTgQYDdRKPPvix3AkcxZkjGFk9jgGZ48lJmmoBX5jOsESgukVVH00HCmi7vAmag5vovTgRurLN5NUXUSUf2CHVGI4EJPPjoRCPsn+LBkZ4xgzaCwTsodTGN3K1FzGmE6xhGB6nPq8NBwpovbQR9QdWkfVwY+oLfuYaK8bdtmHsDt6BFtix7A7fT7R6ePIyRzLhEGjOCM9rs25eI0xx8f+tEy3Up+X+iPbqPMH/9pDH1FXtqFpzP16iefjmBNYF38Rm+InEZ0+gbyBBUzNSOS8NP+kLVbrY0yPsIRguoz6vNRXbA0E/7J11JV9jHqqAfBEJbAz/gT+Hf9ZPkwpZEPsJDIzC5ibEcPZGfD9tJbz8xpjeo4lBHNM1OfxB/91TVU/dWUfo94a9350IoeTT2B9+sW8qYWsip7EtpjRTEiJ4ewB8KUMN3dvmv0PNKbXsD9H0y4X/Le0Evzd3bwSnYgMmEjJkMWsjprEi55C/q2j8Uk0I+LhUxnwgww4KwMGxYX5yxhj2mQJwYRQn4f68s3UHlrXVPVTd3hDIPjHJBGfcQLx+ZewKW4SKyjkhZp8PqmNBh9kRcNZ2fBIhpvLNz/B2gCMiRSWEPoxVaWhcie1pWuoPfhBUPCvA0BikonPOIH0gkuRAZNYG1vI6/Wj+NvhaP5TCVoFKdFwRjpcm+sSQGEyRFkCMCYiWULoR9Tnpb58IzX7V1NT6hZvzX7ADeAWn3kC6QWXE585ieiMQj5kFG8ejuJvZfCvYjfhS6zAzDS4a6RLAKekQqxN1GVMn2AJoQ/zeeuoO7jWBf/9q6k98B4+/+TrMUlDScqZSULOdBKzpxOTVsD66ijeLIM3y+DtYqj0ggAnp8BNeS4BnJYOydYTyJg+yRJCH+JtOEJt6fvUlK6iZv9q6g5+2DRTV1xaASkjziMxezqJOdOJTc6jxguvH4Lnd8Mb6wPz/Y5NhCsGuQQwZwAMjA3jlzLG9BhLCBHMU1NKTelqav1VQHWHN4D6QKKJz5hI+tgrSMw+hcTsaUQnZAJQ64VXDsEfdsDLB91VwMAYWDAQzh7gksCwhDB/MWNMWFhCiBCuAXgHtaVr/G0Aq2g4sh0AiU4gIetkMifeQGLOdBIGnhwyN2+tF14uhedK4U9BSeCSHLg4G+YOgBhrBzCm37OE0EsFGoBX+RuA1wQagOPSScyeRvroL5CQcwoJGROR6NAO/rVeeO2QSwIvH4QjzZLAnAHWGGyMCWUJoZc4egPwEBJzTiXR3wAclz4GkZbRvNYLr5fBH/aHJoHFOfB5SwLGmHb0moQgIhcCC4Ec4Neq+kaYi9St1FtPTelqqve906IBODZtNCnDF/oTwCnEJOe2Oa5/YxJ4br+rDjrihUx/EmisDrIkYIzpiG5NCCLyGHAesF9VJwWtPwf4BRANPKqq96rqS8BLIpIB3A/0uYTgqSmles/bVJW8SfWef+DzVLZoAE7InkpMwsCjHqfWC2/4rwSCk8Dns+HzOZYEjDHHpruvEJ4AfgU82bhCRKKBXwPzgGJgtYj8SVU/9m/yX/73I56qUlf2MVUlb1K1ezl1B9cCSnTiIFJGnEdy7lkkDZoV0gDclsYk8FwpLDsQmgQuzoGzLAkYY45TtyYEVV0pIiObrT4F2KKq2wBE5PfABSKyAbgXeFVV32/rmCJyHXAdwPDhw7uj2MfF56mmeu8/qdq9guqSFXhq9gIQP/AkMgtvIjn3bOIzTujQ1I51PnefwHOl8KcDUOGFDEsCxphuEo42hFxgV9DrYmAG8HXgU0C6iBSo6m9a21lVlwBLAKZNm6bdXNYOaagqpqpkBVUly6nZ9w7qq0dikkkecjrJuWeTNORMYhKzO3SsOh+8cQj+0CwJfM5fHWRJwBjTXcKREFo7NVZVfQh4qKcLcyzU56X24H+oKllOVcly6ss3AhCbMoL0MZeTnDuXxOxTWnQFbUtjEmisDgpOAhdnu5vFLAkYY7pbOBJCMTAs6HUesDsM5egUb3051btXUrV7OVV73sZXVwYSQ2L2NLJOvpXk3LOITc3vUFUQgKobM+jJfaFJ4LPZrkrorAyIsyRgjOlB4UgIq4ExIjIKKAG+AFwahnIclarSULHVJYCS5dSUrgH1EhWfQfKQOa5BeMgZRMeldeq4Hp+7EvjJLvigMpAEGq8ELAkYY8Klu7udLgXmAFkiUgzcqar/KyI3AK/jup0+pqrrO3nc84HzCwoKurS8Pm8dNftXUV2ynKrdy2mo3AlA3IDxZJzwZZKHnk3CwJOQqM4P91nlhcf2wAPFsL0WxifB/46DywZBvCUBY0wvIKq9ol32mEybNk3XrFlzXMfw1JRStXuF6xW05++opwqJjidx0GySc88ieegcYpNzj/n4pfXwqxK3HPLA7DS4ZTgsHGgTyRhjwkNE3lPVac3X95o7lXtS3eFPqNz1BlUlb1J3aC3ghodIG3khSblz3b0BMYnH9Rlba+CBXfDYXqj1wYVZcPMwmJXeFd/AGGO6Xr9MCOWbn6Z889MkZJ3MwBO/TXLuWcQNmNDhBuGjWVMBP90Fz5dCjMAXB8O382B8+/eeGWNMWPXLhJAx8atkFt7U7hARHaXq7iL+yU5YfhjSo+G7w+HGXBgS3yUfYYwx3S4iE8LxNirHJg3pknI0+NwNZD/ZCWurIDcO7h8N1w6BtIj8ZY0x/VlE9m9R1ZdV9br09PBUyFd64MFdUPAuXL4BvApPjIdtp8K3h1kyMMZEJgtdnbC/Hh4qhod3Q5kHzkiHh8fCgkzrMWSMiXyWEDpgczX8bBc8sRfqFT7j7zF0qvUYMsb0IZYQjmJVhWsf+OMBiBO4ajB8axiMTQp3yYwxputFZELorjuVwfUYevWQSwRvl8OAGLh1OHw9DwZ1bKw6Y4yJSNao7Ffvgyf3wolrYOFHsK0Wfj4adp4Kd+dbMjDG9H0ReYXQlY544Ld74OfFUFwHhcnwf+PdnMQ25LQxpj/ptwlhbx08VAIPl0C5181D/NuxMD8TuuCGZWOMiTj9MiHcvR3+ewd41A09ffMwmN65UayNMabP6ZcJYXgCXDMEvpUHBdZjyBhjgAhNCMfby+iLg91ijDEmICKbTcM9dIUxxvRFEZkQjDHGdD1LCMYYYwBLCMYYY/wsIRhjjAEsIRhjjPGzhGCMMQaI0IQgIueLyJLy8vJwF8UYY/oMUdVwl+GYiUgpsOMYd88CDnRhcSKd/R4B9luEst8jVF/4PUaoanbzlRGdEI6HiKxR1WnhLkdvYb9HgP0Woez3CNWXf4+IrDIyxhjT9SwhGGOMAfp3QlgS7gL0MvZ7BNhvEcp+j1B99vfot20IxhhjQvXnKwRjjDFBLCEYY4wB+mlCEJFzRGSjiGwRke+FuzzhIiLDRGSFiGwQkfUi8o1wl6k3EJFoEfmPiPw53GUJNxEZICLPi8gn/v8nM8NdpnARkW/6/07WichSEUkId5m6Wr9LCCISDfwaWACcAFwiIieEt1Rh4wG+raoTgFOBr/Xj3yLYN4AN4S5EL/EL4DVVHQ+cRD/9XUQkF7gRmKaqk4Bo4AvhLVXX63cJATgF2KKq21S1Hvg9cEGYyxQWqrpHVd/3Pz+C+2PPDW+pwktE8oCFwKPhLku4iUgacAbwvwCqWq+qh8NbqrCKARJFJAZIAnaHuTxdrj8mhFxgV9DrYvp5EAQQkZHAycC74S1J2D0IfBfwhbsgvUA+UAo87q9Ce1REksNdqHBQ1RLgfmAnsAcoV9U3wluqrtcfE4K0sq5f970VkRTgBeAmVa0Id3nCRUTOA/ar6nvhLksvEQNMAR5R1ZOBKqBftrmJSAauJmEUMBRIFpHLw1uqrtcfE0IxMCzodR598NKvo0QkFpcMnlbVP4a7PGE2G1gkIttxVYlnichT4S1SWBUDxaraeNX4PC5B9EefAopUtVRVG4A/ArPCXKYu1x8TwmpgjIiMEpE4XMPQn8JcprAQEcHVD29Q1QfCXZ5wU9Xvq2qeqo7E/b9Yrqp97iywo1R1L7BLRMb5V50NfBzGIoXTTuBUEUny/92cTR9sYI8JdwF6mqp6ROQG4HVcT4HHVHV9mIsVLrOBK4CPROQD/7pbVfUvYSyT6V2+DjztP3naBnwpzOUJC1V9V0SeB97H9c77D31wCAsbusIYYwzQP6uMjDHGtMISgjHGGMASgjHGGD9LCMYYYwBLCMYYY/wsIZg+TURURH4W9Po7InJXN3zOVSJSKiIfBC1dNlCgiNwlIt/pquMZ0xpLCKavqwMuEpGsHvisZ1V1ctDSX2/iMhHKEoLp6zy4G4i+2fwNEXlCRD4X9LrS/zhHRN4WkT+IyCYRuVdELhORVSLykYiM7uiH+4+1UkReFJGPReQ3IhLlf+8S//HWich9QfucIyLvi8iHIvJm0OFOEJG3RGSbiNzo3zZZRF7xb7tORBZ3+hcyxq/f3als+qVfA2tF5Ced2OckYAJwCHeH7qOqeop/EqGvAze1ss9iETkt6HXjZDKn4Obe2AG8hrtieQe4D5gKlAFviMiFwD+B3wJnqGqRiGQGHW88MBdIBTaKyCPAOcBuVV0IICLpnfiOxoSwhGD6PFWtEJEncROc1HRwt9WqugdARLYCjUMdf4QLyq15VlVvCF7hhr1hlapu879eCpwGNABvqWqpf/3TuLkHvMBKVS3yl/1Q0OFeUdU6oE5E9gOD/OW533+F8WdV/XsHv58xLViVkekvHgSuAYLH8/fg/xvwD1gWF/ReXdBzX9BrH50/kWo+PozS+jDs+Ne3NZ5McJm8QIyqbsJdZXwE/FhE7uhk2YxpYgnB9Av+M+0/4JJCo+24YApurPvYbvr4U/yj60YBi4F/4CYiOlNEsvzTul4CvA38y79+FECzKqMWRGQoUK2qT+EmcOmvw1ObLmBVRqY/+RkQXKXzW2CZiKwC3sRNAHM8mrchfNX/+C/gXqAQWAm8qKo+Efk+sAJ3VfAXVV0GICLXAX/0J5D9wLyjfGYh8FMR8eGqoa4/zu9g+jEb7dSYbiQic4DvqOp54S6LMe2xKiNjjDGAXSEYY4zxsysEY4wxgCUEY4wxfpYQjDHGAJYQjDHG+FlCMMYYA8D/B1JTAF6/MUAlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "def get_timings(n_epochs, init, per_ep):\n",
    "    #init=48\n",
    "    #eps = n_epochs * 14.25\n",
    "    eps = n_epochs * per_ep\n",
    "    return init+eps\n",
    "\n",
    "fastai_sorted_10 = [48, 14.25]\n",
    "fastai_sorted_100 = [484, 142]\n",
    "nlp_10 = [71, 11.27]\n",
    "nlp_100 = [1290, 323]\n",
    "\n",
    "timings_ls, timing_data = [], []\n",
    "timings = [fastai_sorted_10,fastai_sorted_100, nlp_10, nlp_100]\n",
    "\n",
    "n_eps = list(range(1,20,2))\n",
    "\n",
    "for t in timings: timing_data.append([get_timings(n_epochs=n, init=t[0], per_ep=t[1]) for n in n_eps])\n",
    "\n",
    "colors = ['deepskyblue','blue','goldenrod', 'darkorange']\n",
    "labels = ['fastai_sorted_10','fastai_sorted_100','nlp_10','nlp_100']\n",
    "for i,t in enumerate(timing_data):\n",
    "    plt.plot(t, color=colors[i], label=labels[i])\n",
    "\n",
    "plt.title('Training run duration')\n",
    "plt.xlabel('Num Epochs')\n",
    "plt.ylabel('Time (s)')\n",
    "plt.yscale('log')\n",
    "plt.legend()\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thanks for Reading 😃\n",
    "As always, I would love to hear if you have any comments, thoughts or criticisms, you can find me on Twitter at [@mcgenergy](www.twitter.com/mcgenergy)\n",
    "\n",
    "## [Appendix]\n",
    "\n",
    "## Code:\n",
    "For those curious, you can peek the code used in testing this below 👇"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#hide_collapse\n",
    "# Imports\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from fastai2.basics import *\n",
    "from fastai2.text.all import *\n",
    "# from fastai2.callback.all import *\n",
    "# from fastai2.data.transforms import RandomSplitter\n",
    "from fastai2.text.core import defaults\n",
    "\n",
    "from nlp import load_dataset\n",
    "\n",
    "import spacy,html\n",
    "from spacy.symbols import ORTH\n",
    "\n",
    "import timeit\n",
    "import gc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fastai Testing\n",
    "Init timing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide_collapse\n",
    "#%%timeit -n 1 -r 3\n",
    "\n",
    "# Download data and save as csv\n",
    "# senti_dataset = load_dataset('sentiment140', split='train[:100%]', download_mode='reuse_cache_if_exists')\n",
    "# df = senti_dataset.data.to_pandas()\n",
    "# df.to_csv('sentiment140.csv')\n",
    "\n",
    "# Read data; the first 10% of the sentiment140 dataset, extraced from the `nlp` library and saved as a csv\n",
    "#fn_10pct = 'sentiment140_10pct.csv'\n",
    "fn = 'sentiment140.csv'\n",
    "df = pd.read_csv(fn, index_col=None)\n",
    "\n",
    "# SORT: Calculate text sample lengths\n",
    "df['word_count'] = df['text'].str.split().map(len)\n",
    "\n",
    "res=df['word_count'].values\n",
    "\n",
    "# Create Dataloaders\n",
    "dls = TextDataLoaders.from_csv(path='.', csv_fname=fn, valid_pct=0.2, bs=64, \n",
    "                               text_col='text', label_col='sentiment' , res=res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1 epoch timing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(143.04122078799992, 0.007152061039399996)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#hide_collapse\n",
    "\n",
    "del df, res\n",
    "gc.collect()\n",
    "\n",
    "# Do 1 pass of the training dataloader\n",
    "s = \"\"\"for b in dls.train: pass\n",
    "    \"\"\"\n",
    "\n",
    "time = timeit.timeit(stmt=s, number=1, globals=globals()); time\n",
    "time, time / len(dls.train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HuggingFace `nlp` Datasets Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenizer, Numericalizer and Padding functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide_collapse\n",
    "class SpacyTokenizerNLP():\n",
    "    \"Spacy tokenizer for `lang`\"\n",
    "    def __init__(self, lang='en', special_toks=None, buf_sz=5000):\n",
    "        self.special_toks = ifnone(special_toks, defaults.text_spec_tok)\n",
    "        nlp = spacy.blank(lang, disable=[\"parser\", \"tagger\", \"ner\"])\n",
    "        for w in self.special_toks: nlp.tokenizer.add_special_case(w, [{ORTH: w}])\n",
    "        self.pipe,self.buf_sz = nlp.pipe,buf_sz\n",
    "        \n",
    "    def encodes(self, items):\n",
    "        tmp = [list(doc) for doc in self.pipe(items, batch_size=self.buf_sz)]\n",
    "        return {'tok_text_pre': [list(str(t) for t in l) for l in tmp]}\n",
    "\n",
    "def make_vocab(count, min_freq=3, max_vocab=60000, special_toks=None):\n",
    "    \"Create a vocab of `max_vocab` size from `Counter` `count` with items present more than `min_freq`\"\n",
    "    vocab = [o for o,c in count.most_common(max_vocab) if c >= min_freq]\n",
    "    special_toks = ifnone(special_toks, defaults.text_spec_tok)\n",
    "    for o in reversed(special_toks): #Make sure all special tokens are in the vocab\n",
    "        if o in vocab: vocab.remove(o)\n",
    "        vocab.insert(0, o)\n",
    "    vocab = vocab[:max_vocab]\n",
    "    return vocab + [f'xxfake' for i in range(0, 8-len(vocab)%8)]\n",
    "\n",
    "class NumericalizeNLP(Transform):\n",
    "    \"Reversible transform of tokenized texts to numericalized ids\"\n",
    "    def __init__(self, dsets=None, vocab=None, min_freq=3, max_vocab=60000, special_toks=None, pad_tok=None):\n",
    "        store_attr(self, 'vocab,min_freq,max_vocab,special_toks,pad_tok')\n",
    "        self.vocab, self.special_toks, self.min_freq, self.max_vocab = vocab, special_toks, min_freq, max_vocab\n",
    "        self.o2i = None if vocab is None else defaultdict(int, {v:k for k,v in enumerate(vocab)})\n",
    "\n",
    "        if self.vocab is None:\n",
    "            count = Counter(p for o in dsets for p in o)\n",
    "            self.vocab = make_vocab(count, min_freq=self.min_freq, max_vocab=self.max_vocab, special_toks=self.special_toks)\n",
    "            self.o2i = defaultdict(int, {v:k for k,v in enumerate(self.vocab) if v != 'xxfake'})\n",
    "    \n",
    "    def encodes_nlp(self, o): return TensorText(tensor([self.o2i  [o_] for o_ in o]))\n",
    "    def encodes_nlp(self, b): return {'toks' : [[self.o2i[o_] for o_ in oo] for oo in b['tok_text']]}\n",
    "    \n",
    "# Padding functions\n",
    "def pad_seq(x, max_batch_len, pad_idx):    \n",
    "    pad =  x.new_zeros(max_batch_len-x.size(0))+pad_idx\n",
    "    return torch.cat([x, pad])\n",
    " \n",
    "# Pad up to longest item in the batch and put batch on the GPU\n",
    "def pad_batch(batch=None, pad_token_id=1):\n",
    "    batch_inputs = list()\n",
    "    max_size = max([len(item['toks']) for item in batch])\n",
    "    for item in batch: batch_inputs += [pad_seq(item['toks'], max_size, pad_token_id)]\n",
    "    return torch.stack(batch_inputs).cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data loading and processing functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset sentiment140/sentiment140 (download: 77.59 MiB, generated: 214.21 MiB, total: 291.81 MiB) to /home/morgan/.cache/huggingface/datasets/sentiment140/sentiment140/1.0.0...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset sentiment140 downloaded and prepared to /home/morgan/.cache/huggingface/datasets/sentiment140/sentiment140/1.0.0. Subsequent calls will reuse this data.\n"
     ]
    }
   ],
   "source": [
    "#hide_collapse\n",
    "\n",
    "# Download text, a clean version of the dataset is downloaded (not included in the timings), 'train[:10%]'\n",
    "senti_dataset = load_dataset('sentiment140', split='train', download_mode='reuse_cache_if_exists')\n",
    "\n",
    "spacy_tok = SpacyTokenizerNLP(lang='en', special_toks=defaults.text_spec_tok)\n",
    "\n",
    "def preproc_and_tok(b): return spacy_tok.encodes(list(maps(*defaults.text_proc_rules, b['text'])))\n",
    "\n",
    "def postproc(b): \n",
    "    return {'tok_text': [list(maps(*defaults.text_postproc_rules, _b)) for _b in b['tok_text_pre']]}\n",
    "\n",
    "def get_tok_lengths(example_batch): return {'tok_lens': [len(e) for e in example_batch['toks']]}\n",
    "\n",
    "def prepare_dataset(dataset):\n",
    "    '''\n",
    "        Takes a raw nlp dataset and returns a processed, tokenized, numericalised dataset\n",
    "    '''\n",
    "    # Apply processing rules and tokenize\n",
    "    print('pre-proc and tokenize')\n",
    "    dataset = dataset.map(preproc_and_tok, batched=True)\n",
    "\n",
    "    # Apply post-processing rules \n",
    "    print('post=proc')\n",
    "    dataset = dataset.map(postproc, batched=True)\n",
    "\n",
    "    # Init Numericalizer and create vocab\n",
    "    print('init numericalizer')\n",
    "    numeric = NumericalizeNLP(dsets=dataset['tok_text_pre'], special_toks=defaults.text_spec_tok, pad_tok=1)\n",
    "\n",
    "    # Numericalize\n",
    "    print('numericalizing')\n",
    "    dataset = dataset.map(numeric.encodes_nlp, batched=True)\n",
    "\n",
    "    # Get sample lengths for sorting\n",
    "    dataset=dataset.map(get_tok_lengths, batched=True)\n",
    "    \n",
    "    print('sorting')\n",
    "    # Sort dataset from small to large\n",
    "    dataset = dataset.sort('tok_lens')\n",
    "    \n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Init Timing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/1600 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pre-proc and tokenize\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600/1600 [07:00<00:00,  3.81it/s]\n",
      "  0%|          | 1/1600 [00:00<02:57,  9.03it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "post=proc\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600/1600 [00:27<00:00, 58.42it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "init numericalizer\n",
      "numericalizing\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600/1600 [00:31<00:00, 50.96it/s]\n",
      "100%|██████████| 1600/1600 [00:34<00:00, 46.88it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sorting\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [06:56<00:00, 3842.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "splitting\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1280000/1280000 [04:43<00:00, 4522.59it/s]\n",
      "100%|██████████| 320000/320000 [02:19<00:00, 2298.56it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "setting format\n",
      "init dataloaders\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "#hide_collapse\n",
    "#%%timeit -n 1 -r 1\n",
    "\n",
    "# Do all of the text processing, tokenization and numericalization\n",
    "senti_dataset_f = prepare_dataset(senti_dataset)\n",
    "\n",
    "# Create train and test splits: `.train_test_split` is giving me an error, lets use `.select` instead\n",
    "print('splitting')\n",
    "train_split=int(len(senti_dataset_f)*0.8)\n",
    "train_senti = senti_dataset_f.select(list(range(train_split)))\n",
    "test_senti = senti_dataset_f.select(list(range(train_split, len(senti_dataset_f))))\n",
    "\n",
    "# Format our dataset to outputs torch.Tensor to train a pytorch model\n",
    "print('setting format')\n",
    "columns = ['toks','sentiment']\n",
    "train_senti.set_format(type='torch', columns=columns)\n",
    "test_senti.set_format(type='torch', columns=columns)\n",
    "\n",
    "# Instantiate out PyTorch Dataloaders \n",
    "print('init dataloaders')\n",
    "train_dataloader = torch.utils.data.DataLoader(train_senti, batch_size=64, collate_fn=pad_batch)\n",
    "test_dataloader = torch.utils.data.DataLoader(test_senti, batch_size=64, collate_fn=pad_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Init timings:\n",
    "\n",
    "- pre-proc and tokenize: 374\n",
    "- post-proc : 26\n",
    "- init numericalizer, numericalizing create vocab: 27\n",
    "- get sample lengths: 28\n",
    "- sorting: 416\n",
    "- splitting: 416\n",
    "- setting format: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time `nlp` 1 epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(323.86245587099984, 0.01619312279354999)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#hide_collapse\n",
    "#%%timeit\n",
    "\n",
    "s = \"for b in train_dataloader: pass\"\n",
    "time = timeit.timeit(stmt=s, number=1, globals=globals()); time\n",
    "time, time / (len(train_senti)/64)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}