examples/getting-started-movielens/03-Training-with-TF.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2021 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\n",
    "# Each user is responsible for checking the content of datasets and the\n",
    "# applicable licenses and determining if suitable for the intended use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_merlin_getting-started-movielens-03-training-with-tf/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# Getting Started MovieLens: Training with TensorFlow\n",
    "\n",
    "This notebook is created using the latest stable [merlin-tensorflow-training](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-tensorflow-training/tags) container.\n",
    "\n",
    "## Overview\n",
    "\n",
    "In this notebook, we will train a Merlin Models model implementing the Deep and Cross Network (DCN) architecture.\n",
    "\n",
    "Merlin Models streamlines the training process and thus despite using a fairly elaborate deep learning architecture, we will only need to write a few lines of code!\n",
    "\n",
    "Additionally, to accelerate the training, we will leverage the Merlin Dataloader.\n",
    "\n",
    "The [following notebooks](https://github.com/NVIDIA-Merlin/models/tree/stable/examples) provide a great overview of the concepts in Merlin Models. To learn more about the Merlin Dataloader, please take a look [at its repository](https://github.com/NVIDIA-Merlin/dataloader).\n",
    "\n",
    "### Learning objectives\n",
    "This notebook explains, how to use the Merlin dataloader to accelerate TensorFlow training.\n",
    "\n",
    "1. Use **Merlin Dataloader** with TensorFlow Keras model.\n",
    "2. Export the model for performing inference on the Triton Inference Server.\n",
    "\n",
    "### MovieLens25M\n",
    "\n",
    "The [MovieLens25M](https://grouplens.org/datasets/movielens/25m/) is a popular dataset for recommender systems and is used in academic publications. The dataset contains 25M movie ratings for 62,000 movies given by 162,000 users. Many projects use only the user/item/rating information of MovieLens, but the original dataset provides metadata for the movies, as well. For example, which genres a movie has.\n",
    "\n",
    "In this notebook we will train a Merlin Models model (Deep Cross Network) to predict the rating a user is likely to give a movie. To ensure we utilize our hardware to the fullest, we will leverage the Merlin Dataloder. It will allow us to load data in a highly optmized way and will ensure that our GPU is utilized to maximum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# External dependencies\n",
    "import os\n",
    "import glob\n",
    "os.environ[\"TF_GPU_ALLOCATOR\"]=\"cuda_malloc_async\"\n",
    "\n",
    "import nvtabular as nvt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define our base input directory, containing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DATA_DIR = os.environ.get(\n",
    "    \"INPUT_DATA_DIR\", os.path.expanduser(\"/workspace/nvt-examples/movielens/data/\")\n",
    ")\n",
    "# path to save the models\n",
    "MODEL_DIR = os.environ.get(\"MODEL_DIR\", os.path.expanduser(\"/workspace/nvt-examples/models\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-01-20 11:26:39.373230: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-01-20 11:26:40.514976: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:40.515413: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:40.515595: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:40.727125: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-01-20 11:26:40.728141: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:40.728351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:40.728508: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:41.475009: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:41.475231: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:41.475394: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-01-20 11:26:41.475508: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0\n",
      "2023-01-20 11:26:41.475570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1637] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 24576 MB memory:  -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:08:00.0, compute capability: 7.5\n",
      "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "\n",
    "import nvtabular as nvt\n",
    "from nvtabular.ops import *\n",
    "from merlin.schema.tags import Tags\n",
    "from merlin.models.utils.example_utils import workflow_fit_transform, save_results\n",
    "\n",
    "from merlin.schema.tags import Tags\n",
    "\n",
    "import merlin.models.tf as mm\n",
    "from merlin.io.dataset import Dataset\n",
    "\n",
    "import tensorflow as tf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read our train and validation set that we created in previous notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'rating'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_ds = nvt.Dataset(f'{INPUT_DATA_DIR}/train', engine='parquet', dtypes={'rating': np.int8})\n",
    "valid_ds = nvt.Dataset(f'{INPUT_DATA_DIR}/valid', engine='parquet', dtypes={'rating': np.int8})\n",
    "\n",
    "# I am modifying the schema here as we will not use the `genres` column for training\n",
    "train_ds.schema = train_ds.schema.remove_col('genres')\n",
    "valid_ds.schema = valid_ds.schema.remove_col('genres')\n",
    "\n",
    "# specifying the target column\n",
    "target_column = train_ds.schema.select_by_tag(Tags.TARGET).column_names[0]\n",
    "target_column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training our model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now train our model. The process will be extremely streamlined as this is what Merlin Models was designed to facilitate.\n",
    "\n",
    "Only few lines of code are needed to carry out the training!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = mm.DCNModel(\n",
    "    train_ds.schema,\n",
    "    depth=2,\n",
    "    deep_block=mm.MLPBlock([64, 32]),\n",
    "    prediction_tasks=mm.BinaryOutput(target_column),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specifying Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 16 * 1024\n",
    "LR = 0.03"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training our model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During training, we pass our dataset to the `fit` function of the model and everything is taken care of for us.\n",
    "\n",
    "Internally, `Merlin Dataloader` is used to feed the data in a highly optimized way to our model during training.\n",
    "\n",
    "The DCN-V2 is an architecture proposed as an improvement upon the original [DCN model](https://arxiv.org/pdf/1708.05123.pdf). The explicit feature interactions of the inputs are learned through cross layers, and then combined with a deep network to learn complementary implicit interactions. The overall model architecture is depicted in Figure below, with two ways to combine the cross network with the deep network: (1) stacked and (2) parallel. The output of the embbedding layer is the concatenation of all the embedded vectors and the normalized dense features: x<sub>0</sub> = [x<sub>embed,1</sub>; . . . ; x<sub>embed,𝑛</sub>; 𝑥<sub>dense</sub>].\n",
    "\n",
    "![DCN](./imgs/DCN.png)\n",
    "\n",
    "<a href=\"https://arxiv.org/abs/2008.13535\">Image Source: DCN V2 paper</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1221/1221 [==============================] - 9s 6ms/step - loss: 0.6609 - auc: 0.5281 - regularization_loss: 0.0000e+00 - loss_batch: 0.6609 - val_loss: 0.6588 - val_auc: 0.5626 - val_regularization_loss: 0.0000e+00 - val_loss_batch: 0.6537\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f7b5959ed30>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opt = tf.keras.optimizers.legacy.Adagrad(learning_rate=LR)\n",
    "model.compile(optimizer=opt, run_eagerly=False, metrics=[tf.keras.metrics.AUC()])\n",
    "model.fit(train_ds, validation_data=valid_ds, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the model for inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to save the model for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from merlin.systems.dag.ensemble import Ensemble\n",
    "from merlin.systems.dag.ops.workflow import TransformWorkflow\n",
    "from merlin.systems.dag.ops.tensorflow import PredictTensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create the serving operator that will use to predict using our model on TIS (the Triton Inference Server) and write it to disk along with the config files to load onto the server in the subsequent notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Unsupported signature for serialization: ((Prediction(outputs={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/outputs/rating/binary_output')}, targets={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/targets/rating/binary_output')}, sample_weight={'rating/binary_output': None}, features=None, negative_candidate_ids=None), <tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f7b58732c40>), {}).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:Function `_wrapped_model` contains input name(s) movieId, userId with unsupported characters which will be renamed to movieid, userid in the SavedModel.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Unsupported signature for serialization: ((Prediction(outputs={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/outputs/rating/binary_output')}, targets={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/targets/rating/binary_output')}, sample_weight={'rating/binary_output': None}, features=None, negative_candidate_ids=None), <tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f7b58732c40>), {}).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Unsupported signature for serialization: ((Prediction(outputs={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/outputs/rating/binary_output')}, targets={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/targets/rating/binary_output')}, sample_weight={'rating/binary_output': None}, features=None, negative_candidate_ids=None), <tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f7b58732c40>), {}).\n",
      "WARNING:absl:Found untraced functions such as train_compute_metrics, model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, dense_6_layer_call_fn, dense_6_layer_call_and_return_conditional_losses while saving (showing 5 of 47). These functions will not be directly callable after loading.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /tmp/tmp5eyrbewk/model.savedmodel/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /tmp/tmp5eyrbewk/model.savedmodel/assets\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Unsupported signature for serialization: ((Prediction(outputs={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/outputs/rating/binary_output')}, targets={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/targets/rating/binary_output')}, sample_weight={'rating/binary_output': None}, features=None, negative_candidate_ids=None), <tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f7b58732c40>), {}).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Unsupported signature for serialization: ((Prediction(outputs={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/outputs/rating/binary_output')}, targets={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/targets/rating/binary_output')}, sample_weight={'rating/binary_output': None}, features=None, negative_candidate_ids=None), <tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f7b58732c40>), {}).\n",
      "WARNING:absl:Function `_wrapped_model` contains input name(s) movieId, userId with unsupported characters which will be renamed to movieid, userid in the SavedModel.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Unsupported signature for serialization: ((Prediction(outputs={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/outputs/rating/binary_output')}, targets={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/targets/rating/binary_output')}, sample_weight={'rating/binary_output': None}, features=None, negative_candidate_ids=None), <tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f7b58732c40>), {}).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Unsupported signature for serialization: ((Prediction(outputs={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/outputs/rating/binary_output')}, targets={'rating/binary_output': TensorSpec(shape=(None, 1), dtype=tf.float32, name='outputs/targets/rating/binary_output')}, sample_weight={'rating/binary_output': None}, features=None, negative_candidate_ids=None), <tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f7b58732c40>), {}).\n",
      "WARNING:absl:Found untraced functions such as train_compute_metrics, model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, dense_6_layer_call_fn, dense_6_layer_call_and_return_conditional_losses while saving (showing 5 of 47). These functions will not be directly callable after loading.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /workspace/nvt-examples/models/ensemble/0_predicttensorflowtriton/1/model.savedmodel/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /workspace/nvt-examples/models/ensemble/0_predicttensorflowtriton/1/model.savedmodel/assets\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    }
   ],
   "source": [
    "from merlin.systems.dag.ensemble import Ensemble\n",
    "\n",
    "serving_operators = ['userId', 'movieId'] >>  PredictTensorflow(model)\n",
    "ensemble = Ensemble(serving_operators, train_ds.schema.remove_by_tag(Tags.TARGET).remove_col('genres'))\n",
    "\n",
    "export_path = os.path.join(MODEL_DIR, \"ensemble\")\n",
    "os.makedirs(export_path)\n",
    "\n",
    "ens_conf, node_confs = ensemble.export(export_path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "merlin": {
   "containers": [
    "nvcr.io/nvidia/merlin/merlin-tensorflow-training:latest"
   ]
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}