examples/getting-started-movielens/02-ETL-with-NVTabular.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2021 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\n",
    "# Each user is responsible for checking the content of datasets and the\n",
    "# applicable licenses and determining if suitable for the intended use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_merlin_getting-started-movielens-02-etl-with-nvtabular/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# Getting Started MovieLens: ETL with NVTabular\n",
    "\n",
    "This notebook is created using the latest stable [merlin-hugectr](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-hugectr/tags), [merlin-tensorflow](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-tensorflow/tags), or [merlin-pytorch](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-pytorch/tags) container. \n",
    "\n",
    "## Overview\n",
    "\n",
    "NVTabular is a feature engineering and preprocessing library for tabular data designed to quickly and easily manipulate terabyte scale datasets used to train deep learning based recommender systems.  It provides a high level abstraction to simplify code and accelerates computation on the GPU using the RAPIDS cuDF library.<br><br>\n",
    "\n",
    "Deep Learning models require the input feature in a specific format. Categorical features needs to be continuous integers (0, ..., |C|) to use them with an embedding layer. We will use NVTabular to preprocess the categorical features.<br><br>\n",
    "\n",
    "One other challenge is multi-hot categorical features. A product can have multiple categories assigned, but the number of categories per product varies. For example, a movie can have one or multiple genres:\n",
    "\n",
    "- Father of the Bride Part II: \\[Comedy\\]\n",
    "- Toy Story: \\[Adventure, Animation, Children, Comedy, Fantasy\\]\n",
    "- Jumanji: \\[Adventure, Children, Fantasy\\]\n",
    "\n",
    "One strategy is often to use only the first category or the most frequent ones. However, a better strategy is to use all provided categories per datapoint. [RAPID cuDF](https://github.com/rapidsai/cudf) added list support in its [latest release v0.16](https://medium.com/rapids-ai/two-years-in-a-snap-rapids-0-16-ae797795a5c4) and  NVTabular now supports multi-hot categorical features. <br><br>\n",
    "\n",
    "### Learning objectives\n",
    "\n",
    "In this notebook, we learn how to `Categorify` single-hot and multi-hot categorical input features with NVTabular\n",
    "\n",
    "- Learn NVTabular for using GPU-accelerated ETL (Preprocess and Feature Engineering)\n",
    "- Get familiar with NVTabular's high-level API\n",
    "- Join two dataframes with `JoinExternal` operator\n",
    "- Preprocess single-hot categorical input features with NVTabular\n",
    "- Preprocess multi-hot categorical input features with NVTabular\n",
    "- Use `LambdaOp` for custom row-wise dataframe manipulations with NVTabular"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information about NVTabular, refer to the [documentation](https://nvidia-merlin.github.io/NVTabular/stable/) or the GitHub [repository](https://github.com/NVIDIA-Merlin/NVTabular/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ETL with NVTabular"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/merlin/dtypes/mappings/tf.py:52: UserWarning: Tensorflow dtype mappings did not load successfully due to an error: No module named 'tensorflow'\n",
      "  warn(f\"Tensorflow dtype mappings did not load successfully due to an error: {exc.msg}\")\n",
      "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# External dependencies\n",
    "import os\n",
    "import shutil\n",
    "import numpy as np\n",
    "from nvtabular.ops import *\n",
    "from merlin.schema.tags import Tags\n",
    "\n",
    "import nvtabular as nvt\n",
    "\n",
    "from os import path\n",
    "\n",
    "# Get dataframe library - cudf or pandas\n",
    "from merlin.core.dispatch import get_lib\n",
    "df_lib = get_lib()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define our base input directory, containing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DATA_DIR = os.environ.get(\n",
    "    \"INPUT_DATA_DIR\", os.path.expanduser(\"/workspace/nvt-examples/movielens/data/\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>[Adventure, Animation, Children, Comedy, Fantasy]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>[Adventure, Children, Fantasy]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>[Comedy, Romance]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>[Comedy, Drama, Romance]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>[Comedy]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId                                             genres\n",
       "0        1  [Adventure, Animation, Children, Comedy, Fantasy]\n",
       "1        2                     [Adventure, Children, Fantasy]\n",
       "2        3                                  [Comedy, Romance]\n",
       "3        4                           [Comedy, Drama, Romance]\n",
       "4        5                                           [Comedy]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies = df_lib.read_parquet(os.path.join(INPUT_DATA_DIR, \"movies_converted.parquet\"))\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining our Preprocessing Pipeline\n",
    "The first step is to define the feature engineering and preprocessing pipeline.<br><br>\n",
    "NVTabular has already implemented multiple calculations, called `ops`. An `op` can be applied to a `ColumnGroup` from an overloaded `>>` operator, which in turn returns a new `ColumnGroup`. A `ColumnGroup` is a list of column names as text.<br><br>\n",
    "**Example:**<br>\n",
    "```python\n",
    "features = [ column_name, ...] >> op1 >> op2 >> ...\n",
    "```\n",
    "\n",
    "This may sounds more complicated as it is. Let's define our first pipeline for the MovieLens dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently, our dataset consists of two separate dataframes. First, we use the `JoinExternal` operator to `left-join` the metadata (genres) to our rating dataset.\n",
    "\n",
    "As we process the data, we are also adding tags. They will be helpful to our model during training and inference. Once specified here, we will not have to provide this information down the road."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "CATEGORICAL_COLUMNS = [\"userId\", \"movieId\"]\n",
    "LABEL_COLUMNS = [\"rating\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "userId = [\"userId\"] >> TagAsUserID()\n",
    "movieId = [\"movieId\"] >> TagAsItemID()\n",
    "\n",
    "joined = userId + movieId >> JoinExternal(movies, on=[\"movieId\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data pipelines are **Directed Acyclic Graphs (DAGs)**. We can visualize them with `graphviz`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"310pt\" height=\"404pt\"\n",
       " viewBox=\"0.00 0.00 310.34 404.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 400)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-400 306.34,-400 306.34,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"230.84\" cy=\"-234\" rx=\"71.49\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"230.84\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">TagAsUserID</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>1</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"150.84\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"150.84\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M211.88,-216.41C200.7,-206.62 186.44,-194.15 174.52,-183.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"176.53,-180.83 166.7,-176.88 171.93,-186.1 176.53,-180.83\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>3</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"230.84\" cy=\"-306\" rx=\"66.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"230.84\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">SelectionOp</text>\n",
       "</g>\n",
       "<!-- 3&#45;&gt;0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>3&#45;&gt;0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M230.84,-287.7C230.84,-279.98 230.84,-270.71 230.84,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"234.34,-262.1 230.84,-252.1 227.34,-262.1 234.34,-262.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>2</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"150.84\" cy=\"-90\" rx=\"68.49\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"150.84\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">JoinExternal</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M150.84,-143.7C150.84,-135.98 150.84,-126.71 150.84,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"154.34,-118.1 150.84,-108.1 147.34,-118.1 154.34,-118.1\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>4</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"70.84\" cy=\"-234\" rx=\"70.69\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.84\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">TagAsItemID</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>4&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M89.8,-216.41C100.99,-206.62 115.25,-194.15 127.17,-183.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"129.76,-186.1 134.98,-176.88 125.15,-180.83 129.76,-186.1\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>6</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"150.84\" cy=\"-18\" rx=\"62.29\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"150.84\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">output cols</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;6 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>2&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M150.84,-71.7C150.84,-63.98 150.84,-54.71 150.84,-46.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"154.34,-46.1 150.84,-36.1 147.34,-46.1 154.34,-46.1\"/>\n",
       "</g>\n",
       "<!-- 3_selector -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>3_selector</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"230.84\" cy=\"-378\" rx=\"51.99\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"230.84\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">[&#39;userId&#39;]</text>\n",
       "</g>\n",
       "<!-- 3_selector&#45;&gt;3 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>3_selector&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M230.84,-359.7C230.84,-351.98 230.84,-342.71 230.84,-334.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"234.34,-334.1 230.84,-324.1 227.34,-334.1 234.34,-334.1\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>5</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"70.84\" cy=\"-306\" rx=\"66.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.84\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">SelectionOp</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;4 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M70.84,-287.7C70.84,-279.98 70.84,-270.71 70.84,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"74.34,-262.1 70.84,-252.1 67.34,-262.1 74.34,-262.1\"/>\n",
       "</g>\n",
       "<!-- 5_selector -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>5_selector</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"70.84\" cy=\"-378\" rx=\"59.59\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.84\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">[&#39;movieId&#39;]</text>\n",
       "</g>\n",
       "<!-- 5_selector&#45;&gt;5 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>5_selector&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M70.84,-359.7C70.84,-351.98 70.84,-342.71 70.84,-334.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"74.34,-334.1 70.84,-324.1 67.34,-334.1 74.34,-334.1\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fb0adcdf460>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined.graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Embedding Layers of neural networks require that categorical features are contiguous, incremental Integers: 0, 1, 2, ... , |C|-1. We need to ensure that our categorical features fulfill the requirement.<br>\n",
    "\n",
    "Currently, our genres are a list of Strings. In addition, we should transform the single-hot categorical features userId and movieId, as well.<br>\n",
    "NVTabular provides the operator `Categorify`, which provides this functionality with a high-level API out of the box. In NVTabular release v0.3, list support was added for multi-hot categorical features. Both works in the same way with no need for changes.\n",
    "\n",
    "\n",
    "Next, we will add `Categorify`  for our categorical features (single hot: userId, movieId and multi-hot: genres)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = joined >> Categorify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ratings are on a scale between 1-5. We want to predict a binary target with 1 for ratings `>3` and 0 for  ratings `<=3`. We use the [LambdaOp](https://nvidia-merlin.github.io/NVTabular/stable/api/ops/lambdaop.html) for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings = nvt.ColumnGroup([\"rating\"]) >> LambdaOp(lambda col: (col > 3).astype(\"int8\")) >> AddTags(Tags.TARGET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"709pt\" height=\"548pt\"\n",
       " viewBox=\"0.00 0.00 708.87 548.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 544)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-544 704.87,-544 704.87,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"629.38\" cy=\"-378\" rx=\"71.49\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"629.38\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">TagAsUserID</text>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>3</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"549.38\" cy=\"-306\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"549.38\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;3 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>0&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M610.42,-360.41C599.23,-350.62 584.98,-338.15 573.05,-327.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"575.07,-324.83 565.24,-320.88 570.46,-330.1 575.07,-324.83\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>7</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"629.38\" cy=\"-450\" rx=\"66.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"629.38\" y=\"-446.3\" font-family=\"Times,serif\" font-size=\"14.00\">SelectionOp</text>\n",
       "</g>\n",
       "<!-- 7&#45;&gt;0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>7&#45;&gt;0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M629.38,-431.7C629.38,-423.98 629.38,-414.71 629.38,-406.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"632.88,-406.1 629.38,-396.1 625.88,-406.1 632.88,-406.1\"/>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>1</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"326.38\" cy=\"-162\" rx=\"50.09\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"326.38\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">AddTags</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>2</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"390.38\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"390.38\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M341.22,-144.76C349.82,-135.37 360.75,-123.41 370.12,-113.15\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"372.76,-115.46 376.92,-105.72 367.59,-110.74 372.76,-115.46\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>9</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"231.38\" cy=\"-234\" rx=\"231.26\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"231.38\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">LambdaOp(lambda col: (col &gt; 3).astype(&quot;int8&quot;))</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;1 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>9&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M254.38,-216.05C267.2,-206.61 283.31,-194.74 297.03,-184.63\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"299.24,-187.34 305.22,-178.59 295.09,-181.71 299.24,-187.34\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>11</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"390.38\" cy=\"-18\" rx=\"62.29\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"390.38\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">output cols</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;11 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>2&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M390.38,-71.7C390.38,-63.98 390.38,-54.71 390.38,-46.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"393.88,-46.1 390.38,-36.1 386.88,-46.1 393.88,-46.1\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>6</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"501.38\" cy=\"-162\" rx=\"59.59\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"501.38\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">Categorify</text>\n",
       "</g>\n",
       "<!-- 6&#45;&gt;2 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>6&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M476.74,-145.46C459.45,-134.56 436.29,-119.95 418.31,-108.61\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"419.76,-105.39 409.43,-103.02 416.02,-111.31 419.76,-105.39\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>5</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"549.38\" cy=\"-234\" rx=\"68.49\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"549.38\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">JoinExternal</text>\n",
       "</g>\n",
       "<!-- 3&#45;&gt;5 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>3&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M549.38,-287.7C549.38,-279.98 549.38,-270.71 549.38,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"552.88,-262.1 549.38,-252.1 545.88,-262.1 552.88,-262.1\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>8</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"469.38\" cy=\"-378\" rx=\"70.69\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"469.38\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">TagAsItemID</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;3 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>8&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M488.34,-360.41C499.53,-350.62 513.78,-338.15 525.7,-327.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"528.3,-330.1 533.52,-320.88 523.69,-324.83 528.3,-330.1\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>4</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"231.38\" cy=\"-306\" rx=\"66.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"231.38\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">SelectionOp</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;9 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>4&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M231.38,-287.7C231.38,-279.98 231.38,-270.71 231.38,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"234.88,-262.1 231.38,-252.1 227.88,-262.1 234.88,-262.1\"/>\n",
       "</g>\n",
       "<!-- 4_selector -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>4_selector</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"231.38\" cy=\"-378\" rx=\"50.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"231.38\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">[&#39;rating&#39;]</text>\n",
       "</g>\n",
       "<!-- 4_selector&#45;&gt;4 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>4_selector&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M231.38,-359.7C231.38,-351.98 231.38,-342.71 231.38,-334.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"234.88,-334.1 231.38,-324.1 227.88,-334.1 234.88,-334.1\"/>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M537.76,-216.05C532.04,-207.71 525.03,-197.49 518.69,-188.25\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"521.43,-186.06 512.89,-179.79 515.66,-190.02 521.43,-186.06\"/>\n",
       "</g>\n",
       "<!-- 7_selector -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>7_selector</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"629.38\" cy=\"-522\" rx=\"51.99\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"629.38\" y=\"-518.3\" font-family=\"Times,serif\" font-size=\"14.00\">[&#39;userId&#39;]</text>\n",
       "</g>\n",
       "<!-- 7_selector&#45;&gt;7 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>7_selector&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M629.38,-503.7C629.38,-495.98 629.38,-486.71 629.38,-478.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"632.88,-478.1 629.38,-468.1 625.88,-478.1 632.88,-478.1\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>10</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"469.38\" cy=\"-450\" rx=\"66.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"469.38\" y=\"-446.3\" font-family=\"Times,serif\" font-size=\"14.00\">SelectionOp</text>\n",
       "</g>\n",
       "<!-- 10&#45;&gt;8 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>10&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M469.38,-431.7C469.38,-423.98 469.38,-414.71 469.38,-406.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"472.88,-406.1 469.38,-396.1 465.88,-406.1 472.88,-406.1\"/>\n",
       "</g>\n",
       "<!-- 10_selector -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>10_selector</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"469.38\" cy=\"-522\" rx=\"59.59\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"469.38\" y=\"-518.3\" font-family=\"Times,serif\" font-size=\"14.00\">[&#39;movieId&#39;]</text>\n",
       "</g>\n",
       "<!-- 10_selector&#45;&gt;10 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>10_selector&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M469.38,-503.7C469.38,-495.98 469.38,-486.71 469.38,-478.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"472.88,-478.1 469.38,-468.1 465.88,-478.1 472.88,-478.1\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fb0add30730>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = cat_features + ratings\n",
    "(output).graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize our NVTabular `workflow`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow = nvt.Workflow(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, the `Op`s in our `Workflow` will require measurements of statistical properties of our data in order to be leveraged. For example, the `Normalize` op requires measurements of the dataset mean and standard deviation, and the `Categorify` op requires an accounting of all the categories a particular feature can manifest. However, we frequently need to measure these properties across datasets which are too large to fit into GPU memory (or CPU memory for that matter) at once.\n",
    "\n",
    "NVTabular solves this by providing the `Dataset` class, which breaks a set of parquet or csv files into into a collection of `cudf.DataFrame` chunks that can fit in device memory. The main purpose of this class is to abstract away the raw format of the data, and to allow other NVTabular classes to reliably materialize a `dask_cudf.DataFrame` collection (and/or collection-based iterator) on demand. Under the hood, the data decomposition corresponds to the construction of a [dask_cudf.DataFrame](https://docs.rapids.ai/api/cudf/stable/) object.  By representing our dataset as a lazily-evaluated [Dask](https://www.dask.org/) collection, we can handle the calculation of complex global statistics (and later, can also iterate over the partitions while feeding data into a neural network). `part_size` defines the size read into GPU-memory at once."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now instantiate dataset iterators to loop through our dataset (which we couldn't fit into GPU memory). HugeCTR expect the categorical input columns as `int64` and continuous/label columns as `float32` We need to enforce the required HugeCTR data types, so we set them in a dictionary and give as an argument when creating our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_dtypes = {}\n",
    "\n",
    "for col in CATEGORICAL_COLUMNS:\n",
    "    dict_dtypes[col] = np.int64\n",
    "\n",
    "for col in LABEL_COLUMNS:\n",
    "    dict_dtypes[col] = np.float32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = nvt.Dataset([os.path.join(INPUT_DATA_DIR, \"train.parquet\")])\n",
    "valid_dataset = nvt.Dataset([os.path.join(INPUT_DATA_DIR, \"valid.parquet\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our datasets, we'll apply our `Workflow` to them and save the results out to parquet files for fast reading at train time. Similar to the `scikit learn` API, we collect the statistics of our train dataset with `.fit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 692 ms, sys: 354 ms, total: 1.05 s\n",
      "Wall time: 1.06 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<nvtabular.workflow.workflow.Workflow at 0x7fb2d7fc42b0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "workflow.fit(train_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We clear our output directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure we have a clean output path\n",
    "if path.exists(os.path.join(INPUT_DATA_DIR, \"train\")):\n",
    "    shutil.rmtree(os.path.join(INPUT_DATA_DIR, \"train\"))\n",
    "if path.exists(os.path.join(INPUT_DATA_DIR, \"valid\")):\n",
    "    shutil.rmtree(os.path.join(INPUT_DATA_DIR, \"valid\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We transform our workflow with `.transform`. We are going to add `'userId', 'movieId', 'genres'` columns to `_metadata.json`, because this json file will be needed for HugeCTR training to obtain the required information from all the rows in each parquet file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 0 ns, sys: 1e+03 ns, total: 1e+03 ns\n",
      "Wall time: 2.86 µs\n"
     ]
    }
   ],
   "source": [
    "%time\n",
    "workflow.transform(train_dataset).to_parquet(\n",
    "    output_path=os.path.join(INPUT_DATA_DIR, \"train\"),\n",
    "    shuffle=nvt.io.Shuffle.PER_PARTITION,\n",
    "    cats=[\"userId\", \"movieId\", \"genres\"],\n",
    "    labels=[\"rating\"],\n",
    "    dtypes=dict_dtypes,\n",
    "    write_hugectr_keyset=True  # only needed if using this ETL Notebook for training with HugeCTR\n",
    "                               # should be removed otherwise to speed up computation\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1 µs, sys: 0 ns, total: 1 µs\n",
      "Wall time: 2.62 µs\n"
     ]
    }
   ],
   "source": [
    "%time\n",
    "workflow.transform(valid_dataset).to_parquet(\n",
    "    output_path=os.path.join(INPUT_DATA_DIR, \"valid\"),\n",
    "    shuffle=False,\n",
    "    cats=[\"userId\", \"movieId\", \"genres\"],\n",
    "    labels=[\"rating\"],\n",
    "    dtypes=dict_dtypes,\n",
    "    write_hugectr_keyset=True  # only needed if using this ETL Notebook for training with HugeCTR\n",
    "                               # should be removed otherwise to speed up computation\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next notebooks, we will train a deep learning model. Our training pipeline requires information about the data schema to define the neural network architecture. We will save the NVTabular workflow to disk so that we can restore it in the next notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>tags</th>\n",
       "      <th>dtype</th>\n",
       "      <th>is_list</th>\n",
       "      <th>is_ragged</th>\n",
       "      <th>properties.num_buckets</th>\n",
       "      <th>properties.freq_threshold</th>\n",
       "      <th>properties.max_size</th>\n",
       "      <th>properties.start_index</th>\n",
       "      <th>properties.cat_path</th>\n",
       "      <th>properties.domain.min</th>\n",
       "      <th>properties.domain.max</th>\n",
       "      <th>properties.domain.name</th>\n",
       "      <th>properties.embedding_sizes.cardinality</th>\n",
       "      <th>properties.embedding_sizes.dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>userId</td>\n",
       "      <td>(Tags.CATEGORICAL)</td>\n",
       "      <td>DType(name='int64', element_type=&lt;ElementType....</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>.//categories/unique.userId.parquet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>162541.0</td>\n",
       "      <td>userId</td>\n",
       "      <td>162542.0</td>\n",
       "      <td>512.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>movieId</td>\n",
       "      <td>(Tags.CATEGORICAL)</td>\n",
       "      <td>DType(name='int64', element_type=&lt;ElementType....</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>.//categories/unique.movieId.parquet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>56658.0</td>\n",
       "      <td>movieId</td>\n",
       "      <td>56659.0</td>\n",
       "      <td>512.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>genres</td>\n",
       "      <td>(Tags.CATEGORICAL)</td>\n",
       "      <td>DType(name='int64', element_type=&lt;ElementType....</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>.//categories/unique.genres.parquet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>genres</td>\n",
       "      <td>21.0</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>rating</td>\n",
       "      <td>(Tags.TARGET)</td>\n",
       "      <td>DType(name='int8', element_type=&lt;ElementType.I...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "[{'name': 'userId', 'tags': {<Tags.CATEGORICAL: 'categorical'>}, 'properties': {'num_buckets': None, 'freq_threshold': 0, 'max_size': 0, 'start_index': 0, 'cat_path': './/categories/unique.userId.parquet', 'domain': {'min': 0, 'max': 162541, 'name': 'userId'}, 'embedding_sizes': {'cardinality': 162542, 'dimension': 512}}, 'dtype': DType(name='int64', element_type=<ElementType.Int: 'int'>, element_size=64, element_unit=None, signed=True), 'is_list': False, 'is_ragged': False}, {'name': 'movieId', 'tags': {<Tags.CATEGORICAL: 'categorical'>}, 'properties': {'num_buckets': None, 'freq_threshold': 0, 'max_size': 0, 'start_index': 0, 'cat_path': './/categories/unique.movieId.parquet', 'domain': {'min': 0, 'max': 56658, 'name': 'movieId'}, 'embedding_sizes': {'cardinality': 56659, 'dimension': 512}}, 'dtype': DType(name='int64', element_type=<ElementType.Int: 'int'>, element_size=64, element_unit=None, signed=True), 'is_list': False, 'is_ragged': False}, {'name': 'genres', 'tags': {<Tags.CATEGORICAL: 'categorical'>}, 'properties': {'num_buckets': None, 'freq_threshold': 0, 'max_size': 0, 'start_index': 0, 'cat_path': './/categories/unique.genres.parquet', 'domain': {'min': 0, 'max': 20, 'name': 'genres'}, 'embedding_sizes': {'cardinality': 21, 'dimension': 16}}, 'dtype': DType(name='int64', element_type=<ElementType.Int: 'int'>, element_size=64, element_unit=None, signed=True), 'is_list': True, 'is_ragged': True}, {'name': 'rating', 'tags': {<Tags.TARGET: 'target'>}, 'properties': {}, 'dtype': DType(name='int8', element_type=<ElementType.Int: 'int'>, element_size=8, element_unit=None, signed=True), 'is_list': False, 'is_ragged': False}]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workflow.save(os.path.join(INPUT_DATA_DIR, \"workflow\"))\n",
    "workflow.output_schema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the pre-processing outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look on the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['/workspace/nvt-examples/movielens/data/train/part_0.parquet'],\n",
       " ['/workspace/nvt-examples/movielens/data/valid/part_0.parquet'])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "\n",
    "TRAIN_PATHS = sorted(glob.glob(os.path.join(INPUT_DATA_DIR, \"train\", \"*.parquet\")))\n",
    "VALID_PATHS = sorted(glob.glob(os.path.join(INPUT_DATA_DIR, \"valid\", \"*.parquet\")))\n",
    "TRAIN_PATHS, VALID_PATHS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that genres are a list of Integers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>genres</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1691</td>\n",
       "      <td>332</td>\n",
       "      <td>[1]</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1001</td>\n",
       "      <td>154</td>\n",
       "      <td>[2, 6]</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>967</td>\n",
       "      <td>245</td>\n",
       "      <td>[3, 2]</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150851</td>\n",
       "      <td>622</td>\n",
       "      <td>[3, 5, 7, 15]</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>39553</td>\n",
       "      <td>1146</td>\n",
       "      <td>[3, 1]</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId         genres  rating\n",
       "0    1691      332            [1]     1.0\n",
       "1    1001      154         [2, 6]     1.0\n",
       "2     967      245         [3, 2]     0.0\n",
       "3  150851      622  [3, 5, 7, 15]     1.0\n",
       "4   39553     1146         [3, 1]     1.0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_lib.read_parquet(TRAIN_PATHS[0])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "You can read the API documentation for the Operators and classes used in this notebook:\n",
    "\n",
    "- [JoinExternal](https://nvidia-merlin.github.io/NVTabular/stable/generated/nvtabular.ops.JoinExternal.html#nvtabular.ops.JoinExternal)\n",
    "- [Categorify](https://nvidia-merlin.github.io/NVTabular/stable/generated/nvtabular.ops.Categorify.html#nvtabular.ops.Categorify)\n",
    "- [LambdaOp](https://nvidia-merlin.github.io/NVTabular/stable/generated/nvtabular.ops.LambdaOp.html#nvtabular.ops.LambdaOp)\n",
    "- [Workflow](https://nvidia-merlin.github.io/NVTabular/stable/generated/nvtabular.workflow.workflow.Workflow.html#nvtabular.workflow.workflow.Workflow)\n",
    "- [Dataset](https://nvidia-merlin.github.io/core/stable/api/merlin.io.html#merlin.io.Dataset)\n",
    "\n",
    "The next step for learning to use Merlin for creating a recommender system is to train a model.\n",
    "Refer to [Training with TensorFlow](./03-Training-with-TF.ipynb),  [Training with HugeCTR](./03-Training-with-HugeCTR.ipynb), or [Training with PyTorch](./03-Training-with-PyTorch.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "merlin": {
   "containers": [
    "nvcr.io/nvidia/merlin/merlin-hugectr:latest",
    "nvcr.io/nvidia/merlin/merlin-tensorflow:latest",
    "nvcr.io/nvidia/merlin/merlin-pytorch:latest"
   ]
  },
  "vscode": {
   "interpreter": {
    "hash": "ac3998f1d5f7dccf01e3ccfbb15b61acbc5b8a94bc76660c3c9107b3b4437593"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}