models/image_classification/imagenet_validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validation script for Imagenet models\n",
    "\n",
    "## Overview\n",
    "Use this notebook to verify the accuracy of a trained ONNX model on the validation set of ImageNet dataset.\n",
    "\n",
    "## Models Support in This Demo\n",
    "\n",
    "* SqueezeNet\n",
    "* VGG\n",
    "* ResNet\n",
    "* MobileNet\n",
    "\n",
    "## Prerequisites\n",
    "Dependencies:\n",
    "* Protobuf compiler - `sudo apt-get install protobuf-compiler libprotoc-dev` (required for ONNX. This will work for any linux system. For detailed installation guidelines head over to [ONNX documentation](https://github.com/onnx/onnx#installation))\n",
    "* ONNX - `pip install onnx`\n",
    "* MXNet - `pip install mxnet-cu90mkl --pre -U` (tested on this version GPU, can use other versions. `--pre` indicates a pre build of MXNet which is required here for ONNX version compatibility. `-U` uninstalls any existing MXNet version allowing for a clean install)\n",
    "* numpy - `pip install numpy`\n",
    "* matplotlib - `pip install matplotlib`\n",
    "* gluoncv - `pip install gluoncv` (for ImageNet data preparation)\n",
    "\n",
    "In order to do validate accuracy with a python script: \n",
    "* Generate the script : In Jupyter Notebook browser, go to File -> Download as -> Python (.py)\n",
    "* Run the script: `python imagenet_validation.py`\n",
    "\n",
    "The ImageNet dataset must be downloaded and extracted in the required directory structure. Refer to the guidelines in [imagenet_prep](imagenet_prep.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import dependencies\n",
    "Verify that all dependencies are installed using the cell below. Continue if no errors encountered, warnings can be ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import mxnet as mx\n",
    "import numpy as np\n",
    "from mxnet import gluon, nd\n",
    "from mxnet.gluon.data.vision import transforms\n",
    "from gluoncv.data import imagenet\n",
    "from collections import namedtuple\n",
    "import multiprocessing\n",
    "from mxnet.contrib.onnx._import.import_model import import_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set context, paths and parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determine and set context\n",
    "if max(mx.test_utils.list_gpus())>=0:\n",
    "    ctx = [mx.gpu(0)]\n",
    "else:\n",
    "    ctx = [mx.cpu()]\n",
    "\n",
    "# path to imagenet dataset folder\n",
    "data_dir = '/home/ubuntu/imagenet/img_dataset/'\n",
    "\n",
    "# batch size (set to 1 for cpu)\n",
    "batch_size = 128\n",
    "\n",
    "# number of preprocessing workers\n",
    "num_workers = multiprocessing.cpu_count()\n",
    "\n",
    "# path to ONNX model file\n",
    "model_path = 'squeezenet1.1.onnx'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import ONNX model\n",
    "Import a model from ONNX to MXNet symbols and params using `import_model`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "sym, arg_params, aux_params = import_model(model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define evaluation metrics\n",
    "top1 and top 5 accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define evaluation metrics\n",
    "acc_top1 = mx.metric.Accuracy()\n",
    "acc_top5 = mx.metric.TopKAccuracy(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess images\n",
    "For each image-> resize to 256x256, take center crop of 224x224, normalize image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define image transforms\n",
    "normalize = transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "transform_test = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.CenterCrop(224),\n",
    "    transforms.ToTensor(),\n",
    "    normalize\n",
    "])\n",
    "\n",
    "# Load and process input\n",
    "val_data = gluon.data.DataLoader(\n",
    "    imagenet.classification.ImageNet(data_dir, train=False).transform_first(transform_test),\n",
    "    batch_size=batch_size, shuffle=False, num_workers=num_workers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load network for validation\n",
    "Use `mx.mod.Module` to define the network architecture and bind the parameter values using `mod.set_params`. `mod.bind` tells the network the shape of input and labels to expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load module\n",
    "mod = mx.mod.Module(symbol=sym, context=ctx, label_names=None)\n",
    "mod.bind(for_training=False, data_shapes=[('data', (1,3,224,224))], \n",
    "         label_shapes=mod._label_shapes)\n",
    "mod.set_params(arg_params, aux_params, allow_missing=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute evaluations\n",
    "Perform forward pass over each batch and generate evaluations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 / 390] batches done\n",
      "[50 / 390] batches done\n",
      "[100 / 390] batches done\n",
      "[150 / 390] batches done\n",
      "[200 / 390] batches done\n",
      "[250 / 390] batches done\n",
      "[300 / 390] batches done\n",
      "[350 / 390] batches done\n"
     ]
    }
   ],
   "source": [
    "# Compute evaluations\n",
    "Batch = namedtuple('Batch', ['data'])\n",
    "acc_top1.reset()\n",
    "acc_top5.reset()\n",
    "num_batches = int(50000/batch_size)\n",
    "print('[0 / %d] batches done'%(num_batches))\n",
    "# Loop over batches\n",
    "for i, batch in enumerate(val_data):\n",
    "    # Load batch\n",
    "    data = gluon.utils.split_and_load(batch[0], ctx_list=ctx, batch_axis=0)\n",
    "    label = gluon.utils.split_and_load(batch[1], ctx_list=ctx, batch_axis=0)\n",
    "    # Perform forward pass\n",
    "    mod.forward(Batch([data[0]]))\n",
    "    outputs=mod.get_outputs()\n",
    "    # Update accuracy metrics\n",
    "    acc_top1.update(label, outputs)\n",
    "    acc_top5.update(label, outputs)\n",
    "    if (i+1)%50==0:\n",
    "        print('[%d / %d] batches done'%(i+1,num_batches))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print results\n",
    "top1 and top5 accuracy of the model on the validation set are shown in the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top-1 accuracy: 0.56336, Top-5 accuracy: 0.7912\n"
     ]
    }
   ],
   "source": [
    "# Print results\n",
    "_, top1 = acc_top1.get()\n",
    "_, top5 = acc_top5.get()\n",
    "print(\"Top-1 accuracy: {}, Top-5 accuracy: {}\".format(top1, top5))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}