deakin_ai_challenge_training.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4Fj2O3Au9xdS"
   },
   "outputs": [],
   "source": [
    "# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
    "#  Copyright (c) 2021. Mohamed Reda Bouadjenek, Deakin University              +\n",
    "#           Email:  reda.bouadjenek@deakin.edu.au                              +\n",
    "#                                                                              +\n",
    "#  Licensed under the Apache License, Version 2.0 (the \"License\");             +\n",
    "#   you may not use this file except in compliance with the License.           +\n",
    "#    You may obtain a copy of the License at:                                  +\n",
    "#                                                                              +\n",
    "#                 http://www.apache.org/licenses/LICENSE-2.0                   +\n",
    "#                                                                              +\n",
    "#    Unless required by applicable law or agreed to in writing, software       +\n",
    "#    distributed under the License is distributed on an \"AS IS\" BASIS,         +\n",
    "#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  +\n",
    "#    See the License for the specific language governing permissions and       +\n",
    "#    limitations under the License.                                            +\n",
    "#                                                                              +\n",
    "# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notebook author:** [Mohamed Reda Bouadjenek](https://rbouadjenek.github.io/), Lecturer of Applied Artificial Intelligence, \n",
    "\n",
    "**Institution:** Deakin University, School of Information Technology, Faculty of Sci Eng & Built Env\n",
    "\n",
    "**Adress:** Locked Bag 20000, Geelong, VIC 3220\n",
    "\n",
    "**Phone:** +61 3 522 78380\n",
    "\n",
    "**Email:** reda.bouadjenek@deakin.edu.au\n",
    "\n",
    "<img style=\"float: left;\" src=\"images/deakin2.png\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "**Welcome to the Notebook for the Deakin Simpsons Challenge 2021!**\n",
    "\n",
    "![](images/Simpsons_cast.png)\n",
    "\n",
    "\n",
    "\n",
    "This Notebook allows you to build a classification model for The Deakin Simpsons challenge 2021.\n",
    "\n",
    "The **Deakin Simpsons challenge 2021** is a computer vision competition for which the goal is to recognize Simpsons characters individually in images using machine learning/deep learning. The challenge is designed to provide students with the opportunity to work as team members, to compete with each other, and to enhance the student learning experience by improving their AI modeling, problem-solving, and team-working skills.\n",
    " \n",
    "\n",
    "\n",
    "As participants, your goal is to build a machine learning/deep learning model to automatically recognize the following Simpsons characters:\n",
    "\n",
    " \n",
    "1. [Abraham grampa simpson](https://en.wikipedia.org/wiki/Grampa_Simpson)\n",
    "2. [Apu nahasapeemapetilon](https://en.wikipedia.org/wiki/Apu_Nahasapeemapetilon)\n",
    "3. [Bart simpson](https://en.wikipedia.org/wiki/Bart_Simpson)\n",
    "4. [Charles montgomery burns](https://en.wikipedia.org/wiki/Mr._Burns)\n",
    "5. [chief wiggum](https://en.wikipedia.org/wiki/Chief_Wiggum)\n",
    "6. [Comic book guy](https://en.wikipedia.org/wiki/Comic_Book_Guy)\n",
    "7. [Edna krabappel](https://en.wikipedia.org/wiki/Edna_Krabappel)\n",
    "8. [Homer simpson](https://en.wikipedia.org/wiki/Homer_Simpson)\n",
    "9. [Kent brockman](https://en.wikipedia.org/wiki/Kent_Brockman)\n",
    "10. [Krusty the clown](https://en.wikipedia.org/wiki/Krusty_the_Clown)\n",
    "11. [Lenny leonard](https://simpsons.fandom.com/wiki/Lenny_Leonard)\n",
    "12. [Lisa simpson](https://en.wikipedia.org/wiki/Lisa_Simpson)\n",
    "13. [Marge simpson](https://en.wikipedia.org/wiki/Marge_Simpson)\n",
    "14. [Mayor quimby](https://en.wikipedia.org/wiki/Mayor_Quimby)\n",
    "15. [Milhouse van houten](https://en.wikipedia.org/wiki/Milhouse_Van_Houten)\n",
    "16. [Moe szyslak](https://en.wikipedia.org/wiki/Moe_Szyslak)\n",
    "17. [Ned flanders](https://en.wikipedia.org/wiki/Ned_Flanders)\n",
    "18. [Nelson muntz](https://en.wikipedia.org/wiki/Nelson_Muntz)\n",
    "19. [Principal skinner](https://en.wikipedia.org/wiki/Principal_Skinner)\n",
    "20. [Sideshow bob](https://en.wikipedia.org/wiki/Sideshow_Bob)\n",
    "\n",
    "\n",
    "To achieve this taks, you will be given a data set that consists of 19,548 images to train your model and to tune your hyperparameters. However, feel free to extend it by collecting new images or by using data augmentation techniques.\n",
    "\n",
    "Once you have built your model, you will have to submit it on the [CodaLab](https://competitions.codalab.org/competitions/27191?secret_key=f0a7cc3e-7f78-4bb1-8564-95bc2fadafa5) platform to be evaluated. \n",
    "We evaluate the performance of your model using the [Accuracy](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)  on a private test set that we have directly collected and labeled from TV show episodes.\n",
    "Once the evaluation completed, your entry will appear on the leaderboard to see your performance against other competitors.\n",
    "\n",
    "\n",
    "In the following, we will take you through  a 6-step process to build a simple model to perform this task as follows:\n",
    "\n",
    "1. `Setup the environment:` Thie first step consists of setting the environement and downloading the data.\n",
    "2. `Preprocessing:` The second step is a preprocessing step that consists of resizing, plitting, and piping the input data.\n",
    "3. `Exploring the data:` The third step consists of a simple data exploration step where you will see samples of the data and some statistics to help you in understanding the data.\n",
    "4. `Designing the model:` The forth step consists of designing an architecture for the task.\n",
    "5. `Traning:` The fifth step consists of starting the training process.\n",
    "6. `Monitoring:` The sixth step consists of monitoring the traning process to investigate possible overfitting.\n",
    "7. `Submission:` The seventh and last step will take you through the submission process.\n",
    "\n",
    "\n",
    "**References:**\n",
    "\n",
    "- [The Simpsons characters recognition and detection using Keras](https://medium.com/alex-attia-blog/the-simpsons-character-recognition-using-keras-d8e1796eae36)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup the environment\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, it is important to mention that in order to submit you model to the leaderbord, you need to generate it and save it using  <span style=\"color:red;font-weight: bold;\">TensorFlow 2.2.0</span> and not  <span style=\"color:red;font-weight: bold;text-decoration: line-through;\">TensorFlow 2.3.0</span>. Therefore, please first run the following cell to install the appropriate <span style=\"color:red;font-weight: bold;\">TensorFlow version (2.2.0)</span>. You may need to restart your kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fusNkGWA9qF4"
   },
   "outputs": [],
   "source": [
    "# Run this to install the appropriate tensorflow package\n",
    "!pip install tensorflow==2.2.0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the appropriate TensorFlow version installed, you need now to load all the required packages for this Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sbbWh-959k8g"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix, balanced_accuracy_score, accuracy_score, classification_report\n",
    "from tensorflow.keras import models, layers, optimizers\n",
    "from tensorflow.python.keras.saving import hdf5_format\n",
    "from keras.preprocessing.image import ImageDataGenerator, DirectoryIterator\n",
    "import h5py, itertools, collections\n",
    "import itertools\n",
    "\n",
    "##################\n",
    "# Verifications:\n",
    "#################\n",
    "print('GPU is used.' if len(tf.config.list_physical_devices('GPU')) > 0 else 'GPU is NOT used.')\n",
    "print(\"Tensorflow version: \" + tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, please run the following cell to download the dataset that you will use to build your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "r62rx-yX97GE"
   },
   "outputs": [],
   "source": [
    "# Download dataset:\n",
    "!wget http://206.12.93.90:8080/simpson_dataset/simpsons_train.tar.gz \n",
    "# Unzip the dataset:\n",
    "!tar -xzvf simpsons_train.tar.gz > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing\n",
    "\n",
    "\n",
    "We use the Simpson character data available in [kaggle](https://www.kaggle.com/alexattia/the-simpsons-characters-dataset). \n",
    "\n",
    "This dataset is composed of 20 folders (one for each character) with 400-2000 images in each folder. The total number of images is 19,548.\n",
    "\n",
    "For reading these images, we use `DirectoryIterator` in `tf.keras.preprocessing.image` that is an iterator capable of reading images from a directory on disk and is capable to extract labels. We also use `ImageDataGenerator` to split this dataset into training and validation set, this later is used to tune the hyperparameters of our model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DpgROocq9k8k"
   },
   "outputs": [],
   "source": [
    "'''\n",
    "    Split train and validation.\n",
    "'''\n",
    "# We define the size of input images to 128x128 pixels.\n",
    "image_size = (128, 128)\n",
    "# We define the batch size\n",
    "batch_size = 64\n",
    "\n",
    "# Create an image generator with a fraction of images reserved for validation:\n",
    "image_generator = ImageDataGenerator(validation_split=0.1)\n",
    "\n",
    "# Now, we create a training data iterator by creating batchs of images of the same size as \n",
    "# defined previously, i.e., each image is resized in a 64x64 pixels format.\n",
    "train_ds =  DirectoryIterator(\n",
    "    \"dataset/simpsons_train/\",\n",
    "    image_generator,\n",
    "    class_mode='categorical',\n",
    "    seed=1337,\n",
    "    target_size=image_size,\n",
    "    batch_size=batch_size,\n",
    "    subset = 'training',\n",
    ")\n",
    "\n",
    "# Similarly, we create a validation data iterator by creating batchs of images of the same size as \n",
    "# defined previously, i.e., each image is resized in a 64x64 pixels format.\n",
    "val_ds = DirectoryIterator(\n",
    "    \"dataset/simpsons_train/\",\n",
    "    image_generator,\n",
    "    class_mode='categorical',\n",
    "    seed=1337,\n",
    "    target_size=image_size,\n",
    "    batch_size=batch_size,\n",
    "    subset = 'validation',\n",
    "    shuffle=False\n",
    ")\n",
    "\n",
    "# We save the list of classes (labels).\n",
    "class_names = list(train_ds.class_indices.keys())\n",
    "\n",
    "# We also save the number of labels.\n",
    "num_classes = train_ds.num_classes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the data\n",
    "\n",
    "Now, we do data exploration to show you samples of the images and their labels and some statistics to help you in understanding the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HjIlDsQW9k8m",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "###############################################\n",
    "#### Show distribution of images per class.\n",
    "###############################################\n",
    "counter=collections.Counter(train_ds.labels)\n",
    "v = [ [class_names[item[0]],item[1]]  for item in counter.items()]\n",
    "df = pd.DataFrame(data=v, columns=['index','value'])\n",
    "g = sns.catplot(x='index', y= 'value',  data=df, kind='bar', \n",
    "                legend=False,height=4,aspect=4,saturation=1)\n",
    "(g.despine(top=False,right=False))\n",
    "plt.xlabel(\"Classes\")\n",
    "plt.ylabel(\"#images\")\n",
    "plt.title(\"Distribution of images per class\")\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.show()\n",
    "\n",
    "#####################################\n",
    "######### Show sample of images.\n",
    "#####################################\n",
    "plt.figure(figsize=(16, 16))\n",
    "images = []\n",
    "labels = []\n",
    "for itr in train_ds.next():\n",
    "    for i in range(25):\n",
    "        if len(images) < 25:\n",
    "            images.append(itr[i].astype(\"uint8\"))\n",
    "        else:\n",
    "            labels.append(list(itr[i]).index(1))\n",
    "\n",
    "for i in range(len(images)):\n",
    "    ax = plt.subplot(5, 5, i + 1)\n",
    "    plt.imshow(images[i])\n",
    "    plt.title(class_names[labels[i]].replace('_',' ') +' ('+str(int(labels[i]))+')')\n",
    "    plt.axis(\"off\")\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Designing the model\n",
    "\n",
    "\n",
    "We now design the architecture for the task. The artchitecture below consists of:\n",
    "1. `Rescaling layer:` whose role is to normalize the input data to values between 0 and 1. This will help in speed up the training process.\n",
    "2. `CNN block 1:` which consists of a Convolution layer followed by a max pulling.\n",
    "3. `CNN block 2:` which consists of a Convolution layer followed by a max pulling.\n",
    "4. `CNN block 3:` which consists of a Convolution layer followed by a max pulling.\n",
    "5. `CNN block 4:` which consists of a Convolution layer followed by a max pulling.\n",
    "6. `Flatten layer:` whose role is just to flatten the volume.\n",
    "7. `Dense layers`: one dense layer followed by a classification layer with a softmax activation function.\n",
    "\n",
    "Please note that you will have to design your own model if you want to beat the baseline and be at the top of the leaderboard!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Dhx9ZVzlV9CE"
   },
   "outputs": [],
   "source": [
    "# Defining the model here:\n",
    "model = models.Sequential()\n",
    "model.add(keras.Input(shape=image_size + (3,))) \n",
    "model.add(layers.experimental.preprocessing.Rescaling(1./255))\n",
    "# CNN block 1\n",
    "model.add(layers.Conv2D(4, kernel_size=3, activation='relu'))\n",
    "model.add(layers.MaxPooling2D(pool_size=(2, 2), padding='same'))\n",
    "# # CNN block 2     \n",
    "model.add(layers.Conv2D(8, kernel_size=3, activation='relu'))\n",
    "model.add(layers.MaxPooling2D(pool_size=(2, 2), padding='same'))\n",
    "# # CNN block 3       \n",
    "model.add(layers.Conv2D(16, kernel_size=3, activation='relu'))\n",
    "model.add(layers.MaxPooling2D(pool_size=(2, 2), padding='same'))\n",
    "# # CNN block 4       \n",
    "model.add(layers.Conv2D(32, kernel_size=3, activation='relu'))\n",
    "model.add(layers.MaxPooling2D(pool_size=(2, 2), padding='same'))\n",
    "#Dense part\n",
    "model.add(layers.Flatten())\n",
    "model.add(layers.Dense(32, activation='relu'))\n",
    "model.add(layers.Dense(num_classes, activation='softmax'))\n",
    "\n",
    "# Print a summary of the model\n",
    "model.summary()\n",
    "\n",
    "# Compiling the model by defininf an optimizer, a loss function, \n",
    "# and the metrics to be used for monitoring the traning.\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "              loss='CategoricalCrossentropy',\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Traning\n",
    "\n",
    "Let's now starting the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VtOTspgk9k8r",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Start the trining by defining the number of epochs to train, the traing set and the validation set.\n",
    "history = model.fit(\n",
    "    train_ds, epochs=6, \n",
    "    validation_data=val_ds,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the created model. It is important to note that we are saving with the model:\n",
    "1. The `class_names`: which is important for submission as we will submit class names and not indexes.\n",
    "2. The `image_size`: which is important to resize the test images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "with h5py.File('model.h5', mode='w') as f:\n",
    "    hdf5_format.save_model_to_hdf5(model, f)\n",
    "    f.attrs['class_names'] = class_names\n",
    "    f.attrs['image_size'] = image_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monitoring and analysis of the model\n",
    "\n",
    "The next step consists of monitoring the traning process to investigate possible overfitting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9ObvEJffXxiC",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "acc = history.history['accuracy']\n",
    "val_acc = history.history['val_accuracy']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.figure(figsize=(15,4))\n",
    "\n",
    "ax1 = plt.subplot(1, 2, 1)\n",
    "plt.plot(epochs, loss, label='Training loss')\n",
    "plt.plot(epochs, val_loss, label='Validation loss')\n",
    "plt.fill_between(epochs, loss,val_loss,color='g',alpha=.1)\n",
    "\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "ax2 = plt.subplot(1, 2, 2)\n",
    "plt.plot(epochs, acc, label='Training accuracy')\n",
    "plt.plot(epochs, val_acc, label='Validation accuracy')\n",
    "plt.fill_between(epochs, acc,val_acc,color='g',alpha=.1)\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the `classification_report` function below to build a text report showing the main classification metrics for your model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_ds.reset()\n",
    "val_ds.shuffle = False\n",
    "val_ds.next()\n",
    "y_prob = model.predict(val_ds)\n",
    "y_pred = y_prob.argmax(axis=-1)\n",
    "y_true = val_ds.labels\n",
    "print(classification_report(y_true, y_pred, target_names=class_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the  next cell to create a confusion matrix function `plot_confusion_matrix`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ozFeX6Xu9k8w"
   },
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                        normalize=False,\n",
    "                        title='Confusion matrix',\n",
    "                        cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    vmax = cm.max()\n",
    "    if normalize:\n",
    "        title = 'Confusion matrix (normalized)'\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        cm = [[int(j*100) for j in i ] for i in cm]\n",
    "        cm =np.array(cm)\n",
    "        vmax = 100\n",
    "        \n",
    "    plt.figure(figsize=(8,8))\n",
    "\n",
    "    im = plt.imshow(cm, interpolation='nearest', cmap=cmap, vmin=0.0, vmax=vmax)\n",
    "    plt.title(title)\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=90)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "    \n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, cm[i, j],\n",
    "            horizontalalignment=\"center\",\n",
    "            color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    plt.colorbar(im,fraction=0.046, pad=0.04)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now create the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hc5KeKBT9k8y",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "val_ds.reset()\n",
    "val_ds.shuffle = False\n",
    "val_ds.next()\n",
    "y_prob = model.predict(val_ds)\n",
    "y_pred = y_prob.argmax(axis=-1)\n",
    "y_true = val_ds.labels\n",
    "cnf_matrix = confusion_matrix(y_true, y_pred)\n",
    "plot_confusion_matrix(cm=cnf_matrix, classes=class_names, title='Confusion Matrix', normalize=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a look at the confusion matrix, and observe what classes are confused with each other.\n",
    "\n",
    "Let's now have a look at example of predictions made by your model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_ds.reset()\n",
    "val_ds.shuffle = True\n",
    "#####################################\n",
    "######### Show sample of images.\n",
    "#####################################\n",
    "plt.figure(figsize=(16, 16))\n",
    "images = []\n",
    "labels_pred = []\n",
    "labels_true = []\n",
    "for itr in val_ds.next():\n",
    "    for i in range(25):\n",
    "        if len(images) < 25:\n",
    "            images.append(itr[i].astype(\"uint8\"))\n",
    "            y_proba = model.predict(np.array([itr[i]]))\n",
    "            y_pred = np.argmax(y_proba,axis=1)[0]\n",
    "            labels_pred.append(y_pred)\n",
    "        else:\n",
    "            labels_true.append(list(itr[i]).index(1))\n",
    "    \n",
    "\n",
    "for i in range(len(images)):\n",
    "    ax = plt.subplot(5, 5, i + 1)\n",
    "    plt.imshow(images[i])    \n",
    "    title = 'Pred: ' + class_names[labels_pred[i]].replace('_',' ') +'\\n' +'True: ' + class_names[labels_true[i]].replace('_',' ') \n",
    "    \n",
    "    \n",
    "    plt.title(title,fontsize= 11, pad=5)\n",
    "    plt.axis(\"off\")\n",
    "#     plt.subplots_adjust(left=None, bottom=0.1, right=None, top=1, wspace=None, hspace=None)\n",
    "\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe where are the errors!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to submit the model to CodaLab?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulation, you have sucessfully created your model and you need now to submit it to CodaLab!\n",
    "\n",
    "CodaLab is an open-source web-based platform that enables researchers, developers, and data scientists to organize and participate to data science ana machine learning competitions.\n",
    "\n",
    "Submitting your model to CodaLab is very simple. You need to follow the following steps:\n",
    "\n",
    "1. Mount your Google Drive by runing this code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Copy the model to your Google Drive by runing the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp model.h5 'drive/My Drive/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Now, go to your Google Drive and you will find a file called `model.h5`.\n",
    "\n",
    "\n",
    "\n",
    "4. Copy the model file `model.h5` you have created to the directory `deakin_ai_challenge_submission`.\n",
    "\n",
    "\n",
    "\n",
    "5. Zip `deakin_ai_challenge_submission/` to generate `deakin_ai_challenge_submission.zip`.\n",
    "\n",
    "\n",
    "6. Submit `deakin_ai_challenge_submission.zip` to CodaLab  following these steps.\n",
    "\n",
    "\n",
    "6. Your submission will appear here. Just wait until it is processed (it may take time)!\n",
    "\n",
    "\n",
    "***Watch the video below or click [here](https://www.youtube.com/watch?v=UVP4yATavFM):***\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('UVP4yATavFM',  width=1000, height=600)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulation on building your model and submitting to CodaLab! We hope that your model will achieve a high accuracy on the testset.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Acknowledgment\n",
    "\n",
    "\n",
    "**Author:** [Mohamed Reda Bouadjenek](https://rbouadjenek.github.io/), Lecturer of Applied Artificial Intelligence, \n",
    "\n",
    "**Institution:** Deakin University, School of Information Technology, Faculty of Sci Eng & Built Env\n",
    "\n",
    "**Adress:** Locked Bag 20000, Geelong, VIC 3220\n",
    "\n",
    "**Phone:** +61 3 522 78380\n",
    "\n",
    "**Email:** reda.bouadjenek@deakin.edu.au\n",
    "\n",
    "**www.deakin.edu.au**\n",
    "\n",
    "<div>\n",
    "<img style=\"float: left;\" src=\"images/deakin2.png\" width=\"200\" >\n",
    "</div>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "<div>  <a href=\"https://twitter.com/DeakinAI2021\" > <img style=\"float: left;\" src=\"https://irisconnect.com/uk/wp-content/uploads/sites/3/2020/12/twitter-Follow-us-button.png\" width=\"200\" > </a>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "deakin_ai_challenge_training.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.7"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}