homeworks/homework7.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Homework 7 - Berkeley STAT 157\n",
    "\n",
    "**Your name: XX, SID YY, teammates A,B,C** (Please add your name, SID and teammates to ease Ryan and Rachel to grade.)\n",
    "\n",
    "**Please submit your homework through [gradescope](http://gradescope.com/)**\n",
    "\n",
    "Handout 4/2/2019, due 4/9/2019 by 4pm.\n",
    "\n",
    "This homework deals with fine-tuning for computer vision. In this task, we attempt to identify 120 different breeds of dogs. The data set used in this competition is actually a subset of the ImageNet data set. Different from the images in the CIFAR-10 data set used in the previous homework, the images in the ImageNet data set are higher and wider and their dimensions are inconsistent. Again, you need to use GPU.\n",
    "\n",
    "The dataset is available at [Kaggle](https://www.kaggle.com/c/dog-breed-identification). The rule is similar to homework 6: \n",
    "\n",
    "- work as a team\n",
    "- submit your results into Kaggle\n",
    "- take a screen shot of your best score and insert it below\n",
    "- the top 3 teams/individuals will be awarded with 500 dollar AWS credits\n",
    "\n",
    "First, import the packages or modules required for the competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "import d2l\n",
    "import math\n",
    "from mxnet import autograd, gluon, init, nd\n",
    "from mxnet.gluon import model_zoo, nn\n",
    "from mxnet.gluon import data as gdata, loss as gloss, utils as gutils\n",
    "import os\n",
    "import shutil\n",
    "import time\n",
    "import zipfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtain and Organize the Data Sets\n",
    "\n",
    "The competition data is divided into a training set and testing set. The training set contains 10,222 images and the testing set contains 10,357 images. The images in both sets are in JPEG format. These images contain three RGB channels (color) and they have different heights and widths. There are 120 breeds of dogs in the training set, including Labradors, Poodles, Dachshunds, Samoyeds, Huskies, Chihuahuas, and Yorkshire Terriers.\n",
    "\n",
    "### Download the Data Set\n",
    "\n",
    "After logging in to Kaggle, we can click on the \"Data\" tab on the dog breed identification competition webpage shown in Figure 9.17 and download the training data set \"train.zip\", the testing data set \"test.zip\", and the training data set labels \"label.csv.zip\". After downloading the files, place them in the three paths below:\n",
    "\n",
    "* kaggle_dog/train.zip\n",
    "* kaggle_dog/test.zip\n",
    "* kaggle_dog/labels.csv.zip\n",
    "\n",
    "\n",
    "To make it easier to get started, we provide a small-scale sample of the data set mentioned above, \"train_valid_test_tiny.zip\". If you are going to use the full data set for the Kaggle competition, you will also need to change the `demo` variable below to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "1"
    }
   },
   "outputs": [],
   "source": [
    "# If you use the full data set downloaded for the Kaggle competition, \n",
    "# change the variable below to False.\n",
    "demo = True\n",
    "data_dir = './kaggle_dog'\n",
    "if demo:\n",
    "    if not os.path.exists(data_dir):\n",
    "        os.mkdir(data_dir)\n",
    "    gutils.download('https://github.com/d2l-ai/d2l-en/raw/master/data/kaggle_dog/train_valid_test_tiny.zip', \n",
    "                    data_dir)\n",
    "    zipfiles = ['train_valid_test_tiny.zip']\n",
    "else:\n",
    "    zipfiles = ['train.zip', 'test.zip', 'labels.csv.zip']\n",
    "for f in zipfiles:\n",
    "    with zipfile.ZipFile(data_dir + '/' + f, 'r') as z:\n",
    "        z.extractall(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Organize the Data Set\n",
    "\n",
    "Next, we define the `reorg_train_valid` function to segment the validation set from the original Kaggle competition training set.  The parameter `valid_ratio` in this function is the ratio of the number of examples of each dog breed in the validation set to the number of examples of the breed with the least examples (66) in the original training set. After organizing the data, images of the same breed will be placed in the same folder so that we can read them later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reorg_train_valid(data_dir, train_dir, input_dir, valid_ratio, idx_label):\n",
    "    # The number of examples of the least represented breed in the training set.\n",
    "    min_n_train_per_label = (\n",
    "        collections.Counter(idx_label.values()).most_common()[:-2:-1][0][1])\n",
    "    # The number of examples of each breed in the validation set.\n",
    "    n_valid_per_label = math.floor(min_n_train_per_label * valid_ratio)\n",
    "    label_count = {}\n",
    "    for train_file in os.listdir(os.path.join(data_dir, train_dir)):\n",
    "        idx = train_file.split('.')[0]\n",
    "        label = idx_label[idx]\n",
    "        d2l.mkdir_if_not_exist([data_dir, input_dir, 'train_valid', label])\n",
    "        shutil.copy(os.path.join(data_dir, train_dir, train_file),\n",
    "                    os.path.join(data_dir, input_dir, 'train_valid', label))\n",
    "        if label not in label_count or label_count[label] < n_valid_per_label:\n",
    "            d2l.mkdir_if_not_exist([data_dir, input_dir, 'valid', label])\n",
    "            shutil.copy(os.path.join(data_dir, train_dir, train_file),\n",
    "                        os.path.join(data_dir, input_dir, 'valid', label))\n",
    "            label_count[label] = label_count.get(label, 0) + 1\n",
    "        else:\n",
    "            d2l.mkdir_if_not_exist([data_dir, input_dir, 'train', label])\n",
    "            shutil.copy(os.path.join(data_dir, train_dir, train_file),\n",
    "                        os.path.join(data_dir, input_dir, 'train', label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `reorg_dog_data` function below is used to read the training data labels, segment the validation set, and organize the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "2"
    }
   },
   "outputs": [],
   "source": [
    "def reorg_dog_data(data_dir, label_file, train_dir, test_dir, input_dir,\n",
    "                   valid_ratio):\n",
    "    # Read the training data labels.\n",
    "    with open(os.path.join(data_dir, label_file), 'r') as f:\n",
    "        # Skip the file header line (column name).\n",
    "        lines = f.readlines()[1:]\n",
    "        tokens = [l.rstrip().split(',') for l in lines]\n",
    "        idx_label = dict(((idx, label) for idx, label in tokens))\n",
    "    reorg_train_valid(data_dir, train_dir, input_dir, valid_ratio, idx_label)\n",
    "    # Organize the training set.\n",
    "    d2l.mkdir_if_not_exist([data_dir, input_dir, 'test', 'unknown'])\n",
    "    for test_file in os.listdir(os.path.join(data_dir, test_dir)):\n",
    "        shutil.copy(os.path.join(data_dir, test_dir, test_file),\n",
    "                    os.path.join(data_dir, input_dir, 'test', 'unknown'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we are using a small data set, we set the batch size to 1. During actual training and testing, we would use the entire Kaggle Competition data set and call the `reorg_dog_data` function to organize the data set. Likewise, we would need to set the `batch_size` to a larger integer, such as 128."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "3"
    }
   },
   "outputs": [],
   "source": [
    "if demo:\n",
    "    # Note: Here, we use a small data set and the batch size should be set\n",
    "    # smaller. When using the complete data set for the Kaggle competition, \n",
    "    # we can set the batch size to a larger integer.\n",
    "    input_dir, batch_size = 'train_valid_test_tiny', 1\n",
    "else:\n",
    "    label_file, train_dir, test_dir = 'labels.csv', 'train', 'test'\n",
    "    input_dir, batch_size, valid_ratio = 'train_valid_test', 128, 0.1\n",
    "    reorg_dog_data(data_dir, label_file, train_dir, test_dir, input_dir,\n",
    "                   valid_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image Augmentation\n",
    "\n",
    "The size of the images in this section are larger than the images in the previous section. Here are some more image augmentation operations that might be useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "4"
    }
   },
   "outputs": [],
   "source": [
    "transform_train = gdata.vision.transforms.Compose([\n",
    "    # Randomly crop the image to obtain an image with an area of 0.08 to 1 \n",
    "    # of the original area and height to width ratio between 3/4 and 4/3.\n",
    "    # Then, scale the image to create a new image with a height and width \n",
    "    # of 224 pixels each.\n",
    "    gdata.vision.transforms.RandomResizedCrop(224, scale=(0.08, 1.0),\n",
    "                                              ratio=(3.0/4.0, 4.0/3.0)),\n",
    "    gdata.vision.transforms.RandomFlipLeftRight(),\n",
    "    # Randomly change the brightness, contrast, and saturation.\n",
    "    gdata.vision.transforms.RandomColorJitter(brightness=0.4, contrast=0.4,\n",
    "                                              saturation=0.4),\n",
    "    # Add random noise.\n",
    "    gdata.vision.transforms.RandomLighting(0.1),\n",
    "    gdata.vision.transforms.ToTensor(),\n",
    "    # Standardize each channel of the image.\n",
    "    gdata.vision.transforms.Normalize([0.485, 0.456, 0.406],\n",
    "                                      [0.229, 0.224, 0.225])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During testing, we only use definite image preprocessing operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_test = gdata.vision.transforms.Compose([\n",
    "    gdata.vision.transforms.Resize(256),\n",
    "    # Crop a square of 224 by 224 from the center of the image.\n",
    "    gdata.vision.transforms.CenterCrop(224),\n",
    "    gdata.vision.transforms.ToTensor(),\n",
    "    gdata.vision.transforms.Normalize([0.485, 0.456, 0.406],\n",
    "                                      [0.229, 0.224, 0.225])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the Data Set\n",
    "\n",
    "As in the previous section, we can create an `ImageFolderDataset` instance to read the data set containing the original image files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    }
   },
   "outputs": [],
   "source": [
    "train_ds = gdata.vision.ImageFolderDataset(\n",
    "    os.path.join(data_dir, input_dir, 'train'), flag=1)\n",
    "valid_ds = gdata.vision.ImageFolderDataset(\n",
    "    os.path.join(data_dir, input_dir, 'valid'), flag=1)\n",
    "train_valid_ds = gdata.vision.ImageFolderDataset(\n",
    "    os.path.join(data_dir, input_dir, 'train_valid'), flag=1)\n",
    "test_ds = gdata.vision.ImageFolderDataset(\n",
    "    os.path.join(data_dir, input_dir, 'test'), flag=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we create a `DataLoader` instance, just like in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_iter = gdata.DataLoader(train_ds.transform_first(transform_train),\n",
    "                              batch_size, shuffle=True, last_batch='keep')\n",
    "valid_iter = gdata.DataLoader(valid_ds.transform_first(transform_test),\n",
    "                              batch_size, shuffle=True, last_batch='keep')\n",
    "train_valid_iter = gdata.DataLoader(train_valid_ds.transform_first(\n",
    "    transform_train), batch_size, shuffle=True, last_batch='keep')\n",
    "test_iter = gdata.DataLoader(test_ds.transform_first(transform_test),\n",
    "                             batch_size, shuffle=False, last_batch='keep')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Model\n",
    "\n",
    "The data set for this competition is a subset of the ImageNet data set. Therefore, we can use the approach discussed in the [\"Fine Tuning\"](fine-tuning.md) section to select a model pre-trained on the entire ImageNet data set and use it to extract image features to be input in the custom small-scale output network. Gluon provides a wide range of pre-trained models. Here, we will use the pre-trained ResNet-34 model. Because the competition data set is a subset of the pre-training data set, we simply reuse the input of the pre-trained model's output layer, i.e. the extracted features. Then, we can replace the original output layer with a small custom output network that can be trained, such as two fully connected layers in a series. Different from the experiment in the [\"Fine Tuning\"](fine-tuning.md) section, here, we do not retrain the pre-trained model used for feature extraction. This reduces the training time and the memory required to store model parameter gradients.\n",
    "\n",
    "You must note that, during image augmentation, we use the mean values and standard deviations of the three RGB channels for the entire ImageNet data set for normalization. This is consistent with the normalization of the pre-trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "6"
    }
   },
   "outputs": [],
   "source": [
    "def get_net(ctx):\n",
    "    finetune_net = model_zoo.vision.resnet34_v2(pretrained=True)\n",
    "    # Define a new output network.\n",
    "    finetune_net.output_new = nn.HybridSequential(prefix='')\n",
    "    finetune_net.output_new.add(nn.Dense(256, activation='relu'))\n",
    "    # There are 120 output categories.\n",
    "    finetune_net.output_new.add(nn.Dense(120))\n",
    "    # Initialize the output network.\n",
    "    finetune_net.output_new.initialize(init.Xavier(), ctx=ctx)\n",
    "    # Distribute the model parameters to the CPUs or GPUs used for computation.\n",
    "    finetune_net.collect_params().reset_ctx(ctx)\n",
    "    return finetune_net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When calculating the loss, we first use the member variable `features` to obtain the input of the pre-trained model's output layer, i.e. the extracted feature. Then, we use this feature as the input for our small custom output network and compute the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = gloss.SoftmaxCrossEntropyLoss()\n",
    "\n",
    "def evaluate_loss(data_iter, net, ctx):\n",
    "    l_sum, n = 0.0, 0\n",
    "    for X, y in data_iter:\n",
    "        y = y.as_in_context(ctx)\n",
    "        output_features = net.features(X.as_in_context(ctx))\n",
    "        outputs = net.output_new(output_features)\n",
    "        l_sum += loss(outputs, y).sum().asscalar()\n",
    "        n += y.size\n",
    "    return l_sum / n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Training Functions\n",
    "\n",
    "We will select the model and tune hyper-parameters according to the model's performance on the validation set. The model training function `train` only trains the small custom output network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "7"
    }
   },
   "outputs": [],
   "source": [
    "def train(net, train_iter, valid_iter, num_epochs, lr, wd, ctx, lr_period,\n",
    "          lr_decay):\n",
    "    # Only train the small custom output network.\n",
    "    trainer = gluon.Trainer(net.output_new.collect_params(), 'sgd',\n",
    "                            {'learning_rate': lr, 'momentum': 0.9, 'wd': wd})\n",
    "    for epoch in range(num_epochs):\n",
    "        train_l_sum, n, start = 0.0, 0, time.time()\n",
    "        if epoch > 0 and epoch % lr_period == 0:\n",
    "            trainer.set_learning_rate(trainer.learning_rate * lr_decay)\n",
    "        for X, y in train_iter:\n",
    "            y = y.as_in_context(ctx)\n",
    "            output_features = net.features(X.as_in_context(ctx))\n",
    "            with autograd.record():\n",
    "                outputs = net.output_new(output_features)\n",
    "                l = loss(outputs, y).sum()\n",
    "            l.backward()\n",
    "            trainer.step(batch_size)\n",
    "            train_l_sum += l.asscalar()\n",
    "            n += y.size\n",
    "        time_s = \"time %.2f sec\" % (time.time() - start)\n",
    "        if valid_iter is not None:\n",
    "            valid_loss = evaluate_loss(valid_iter, net, ctx)\n",
    "            epoch_s = (\"epoch %d, train loss %f, valid loss %f, \"\n",
    "                       % (epoch + 1, train_l_sum / n, valid_loss))\n",
    "        else:\n",
    "            epoch_s = (\"epoch %d, train loss %f, \"\n",
    "                       % (epoch + 1, train_l_sum / n))\n",
    "        print(epoch_s + time_s + ', lr ' + str(trainer.learning_rate))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and Validate the Model\n",
    "\n",
    "Now, we can train and validate the model. The following hyper-parameters can be tuned. For example, we can increase the number of epochs. Because `lr_period` and `lr_decay` are set to 10 and 0.1 respectively, the learning rate of the optimization algorithm will be multiplied by 0.1 after every 10 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "9"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading /Users/muli/.mxnet/models/resnet34_v2-9d6b80bb.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/resnet34_v2-9d6b80bb.zip...\n",
      "epoch 1, train loss 5.320887, valid loss 4.771201, time 8.65 sec, lr 0.01\n"
     ]
    }
   ],
   "source": [
    "ctx, num_epochs, lr, wd = d2l.try_gpu(), 1, 0.01, 1e-4\n",
    "lr_period, lr_decay, net = 10, 0.1, get_net(ctx)\n",
    "net.hybridize()\n",
    "train(net, train_iter, valid_iter, num_epochs, lr, wd, ctx, lr_period,\n",
    "      lr_decay)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classify the Testing Set and Submit Results on Kaggle\n",
    "\n",
    "After obtaining a satisfactory model design and hyper-parameters, we use all training data sets (including validation sets) to retrain the model and then classify the testing set. Note that predictions are made by the output network we just trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "8"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, train loss 5.090321, time 18.22 sec, lr 0.01\n"
     ]
    }
   ],
   "source": [
    "net = get_net(ctx)\n",
    "net.hybridize()\n",
    "train(net, train_valid_iter, None, num_epochs, lr, wd, ctx, lr_period,\n",
    "      lr_decay)\n",
    "\n",
    "preds = []\n",
    "for data, label in test_iter:\n",
    "    output_features = net.features(data.as_in_context(ctx))\n",
    "    output = nd.softmax(net.output_new(output_features))\n",
    "    preds.extend(output.asnumpy())\n",
    "ids = sorted(os.listdir(os.path.join(data_dir, input_dir, 'test/unknown')))\n",
    "with open('submission.csv', 'w') as f:\n",
    "    f.write('id,' + ','.join(train_valid_ds.synsets) + '\\n')\n",
    "    for i, output in zip(ids, preds):\n",
    "        f.write(i.split('.')[0] + ',' + ','.join(\n",
    "            [str(num) for num in output]) + '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After executing the above code, we will generate a \"submission.csv\" file. The format of this file is consistent with the Kaggle competition requirements. \n",
    "\n",
    "## Hints to Improve Your Results\n",
    "\n",
    "* You should download the whole data set from Kaggle and switch to `demo=False`. \n",
    "* Try to increase the `batch_size` (batch size) and `num_epochs` (number of epochs).\n",
    "* Try a deeper pre-trained model, you may find models from [gluoncv](https://gluon-cv.mxnet.io/model_zoo/classification.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}