01_materials/labs/lab_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutions\n",
    "\n",
    "In this lab, we'll look in detail at convolutions and how they can be used to process images. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading and opening images\n",
    "\n",
    "We'll use the `skimage` library to read and process images. It's a library dedicated to image processing, which is part of the `scikit-learn` family."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from skimage.io import imread"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Helper function to download images from our GitHub repo\n",
    "\n",
    "def download_image(image_name):\n",
    "    url = \"https://raw.githubusercontent.com/UofT-DSI/deep_learning/main/notebooks/images/\" + image_name\n",
    "    import requests\n",
    "    r = requests.get(url)\n",
    "    with open(image_name, 'wb') as f:\n",
    "        f.write(r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "download_image(\"bumblebee.png\")\n",
    "sample_image = imread(\"bumblebee.png\")\n",
    "sample_image= sample_image.astype(\"float32\")\n",
    "\n",
    "size = sample_image.shape\n",
    "print(\"sample image shape: \", sample_image.shape)\n",
    "\n",
    "plt.imshow(sample_image.astype('uint8'));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A simple convolution filter\n",
    "\n",
    "Before we start working on training any models, let's look at applying a convolution filter to an image. We'll use the `Conv2D` layer from Keras to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Conv2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "conv = Conv2D(filters=3, kernel_size=(5, 5), padding=\"same\",\n",
    "              input_shape=(None, None, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember: in Keras, `None` is used as a marker for tensor dimensions with dynamic size. In this case `batch_size`, `width` and `height` are all dynamic: they can depend on the input. This is a neat feature of convolutional neural networks: the same model can be used to process images of any size, because all we have to do is slide the convolutional filter across the image as much as necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_image.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_in = np.expand_dims(sample_image, 0)\n",
    "img_in.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_out = conv(img_in) # Apply the convolutional filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output is a tensorflow Eager Tensor - a special data structure that is used to represent the result of operations in TensorFlow. It is not a numpy array, but it can be converted to one using the `.numpy()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np_img_out = img_out[0].numpy()\n",
    "print(type(np_img_out))\n",
    "print(np_img_out.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(10, 5))\n",
    "ax0.imshow(sample_image.astype('uint8'))\n",
    "ax1.imshow(np_img_out.astype('uint8'));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, our convolutional filter was initialized randomly, so our output doesn't contain any specific meaning. Each pixel is a random combination of the pixels in the input image, in a 5x5 window."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's instead take a look at a convolutional feature with a clear purpose. We can build a kernel ourselves, by defining a function which will be passed to `Conv2D` Layer.\n",
    "We'll create an array with 1/25 for filters, with each channel seperated. Before you move to the next cell, can you guess what this filter will do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_kernel(shape=(5, 5, 3, 3), dtype=None):\n",
    "    array = np.zeros(shape=shape, dtype=\"float32\")\n",
    "    array[:, :, 0, 0] = 1 / 25\n",
    "    array[:, :, 1, 1] = 1 / 25\n",
    "    array[:, :, 2, 2] = 1 / 25\n",
    "    return array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now we can use this function to initialize a `Conv2D` layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "conv = Conv2D(filters=3, kernel_size=(5, 5), padding=\"same\",\n",
    "           input_shape=(None, None, 3), kernel_initializer=my_kernel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(10, 5))\n",
    "ax0.imshow(img_in[0].astype('uint8'))\n",
    "\n",
    "img_out = conv(img_in)\n",
    "np_img_out = img_out[0].numpy()\n",
    "ax1.imshow(np_img_out.astype('uint8'));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully you can tell what this filter does! \n",
    "\n",
    "**Exercise**\n",
    "- There are a number of settings when we define our Conv2D layer. Try changing the following parameters to get a sense of how they impact the result:\n",
    "- kernel_size: try different sizes\n",
    "- padding: try 'valid' instead of 'same' (hint: this may change the size of the output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working on edge detection on Grayscale image\n",
    "\n",
    "Using a grayscale image, let's build an \"edge detector\" using a convolutional filter. Some filters pre-date the deep learning era and are still used today. For example, the Sobel filter is used to detect edges in images. These easy-to-compute filters were used in the early days of computer vision and are still useful now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# convert image to greyscale\n",
    "grey_sample_image = sample_image.mean(axis=2)\n",
    "\n",
    "# add the channel dimension even if it's only one channel so\n",
    "# to be consistent with Keras expectations.\n",
    "grey_sample_image = grey_sample_image[:, :, np.newaxis]\n",
    "\n",
    "# matplotlib does not like the extra dim for the color channel\n",
    "# when plotting gray-level images. Let's use squeeze:\n",
    "plt.imshow(np.squeeze(grey_sample_image.astype(np.uint8)),\n",
    "           cmap=plt.cm.gray);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**\n",
    "- Build an edge detector using `Conv2D` on greyscale image by defining the kernel inside `my_kernel`.\n",
    "- You may experiment with several kernels to find a way to detect edges. The following article contains specific examples of kernels that you can use:\n",
    "- https://en.wikipedia.org/wiki/Kernel_(image_processing)\n",
    "- Try different kernels and see the impact on the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_kernel(shape=(3, 3, 1, 1), dtype=None):\n",
    "    array = np.array([[0, 0, 0],\n",
    "                      [0, 1, 0],\n",
    "                      [0, 0, 0]]) # Replace with your kernel\n",
    "    array = array.reshape(*shape) # Reshape if needed\n",
    "    return array\n",
    "\n",
    "conv = Conv2D(filters=1, kernel_size=(3, 3), padding=\"same\",\n",
    "              input_shape=(None, None, 1), kernel_initializer=my_kernel)\n",
    "\n",
    "img_in = np.expand_dims(grey_sample_image, 0) # Reshape into a batch of size 1\n",
    "img_out = conv(img_in) # Apply the convolutional filter\n",
    "np_img_out = img_out[0].numpy() # Convert to numpy array\n",
    "\n",
    "fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(10, 5))\n",
    "ax0.imshow(np.squeeze(grey_sample_image.astype(np.uint8)),\n",
    "           cmap=plt.cm.gray)\n",
    "ax1.imshow(np_img_out.astype(np.uint8), cmap=plt.cm.gray);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pooling and strides with convolutions\n",
    "\n",
    "**Exercise**\n",
    "- Use `MaxPool2D` to apply a 2x2 max pool with strides 2 to the image. What is the impact on the shape of the image?\n",
    "- Use `AvgPool2D` to apply an average pooling.\n",
    "- Is it possible to compute a max pooling and an average pooling with well chosen kernels?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import MaxPool2D, AvgPool2D\n",
    "\n",
    "# You can use `img_in` from above as input to the pooling layers\n",
    "\n",
    "plt.imshow(np.squeeze(grey_sample_image.astype(np.uint8)),\n",
    "           cmap=plt.cm.gray);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Loading a JPEG file as a numpy array\n",
    "\n",
    "Let's use [scikit-image](http://scikit-image.rg) to load the content of a JPEG file into a numpy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from skimage.io import imread\n",
    "\n",
    "download_image('laptop.jpeg')\n",
    "image = imread('laptop.jpeg')\n",
    "plt.imshow(image);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Resizing images, handling data types and dynamic ranges\n",
    "\n",
    "While convolutions can handle inputs of any size, it is often useful to resize images to a fixed size. This is particularly important for training deep learning models:\n",
    "\n",
    "- for **image classification**, most networks expect a specific **fixed input size**;\n",
    "\n",
    "- for **object detection** and instance segmentation, networks have more flexibility but the image should have **approximately the same size as the training set images**.\n",
    "\n",
    "Furthermore **large images can be much slower to process** than smaller images (the number of pixels varies quadratically with the height and width)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from skimage.transform import resize\n",
    "\n",
    "lowres_image = resize(image, (50, 50), mode='reflect', anti_aliasing=True)\n",
    "lowres_image.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.imshow(lowres_image, interpolation='nearest');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "The values of the pixels of the low resolution image are computed from by combining the values of the pixels in the high resolution image. The result is therefore represented as floating points. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Using a pretrained model\n",
    "\n",
    "Objectives:\n",
    "\n",
    "- Load a pre-trained ResNet50 pre-trained model using Keras Zoo\n",
    "- Use the model to classify an image\n",
    "- Use the model to classify an image from the webcam\n",
    "\n",
    "Let's start with loading ResNet50, a well-established method for image classification. The ResNet50 \"application\" takes two key parameters here: firstly, `include_top` indicates whether we want to include the last layer of the network (the classification layer) or not. Secondly, `weights` indicates whether we want to load the weights of a model that has been pre-trained on ImageNet or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.applications.resnet50 import ResNet50\n",
    "\n",
    "model = ResNet50(include_top=True, weights='imagenet')\n",
    "model.compile(optimizer='sgd', loss='categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Classification of an image\n",
    "\n",
    "**Exercise**\n",
    "- Reshape the `laptop` image to the shape `(224, 224, 3)` using `resize` from `skimage.transform`\n",
    "- Use `preprocess_input` from `tensorflow.keras.applications.imagenet_utils` to preprocess the image\n",
    "- Use `predict` to classify the image\n",
    "\n",
    "Documentation for each method:\n",
    "- [resize](https://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.resize)\n",
    "- [preprocess_input](https://www.tensorflow.org/api_docs/python/tf/keras/applications/resnet/preprocess_input)\n",
    "- [predict](https://www.tensorflow.org/api_docs/python/tf/keras/Model#predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.applications.imagenet_utils import preprocess_input\n",
    "from tensorflow.keras.applications.imagenet_utils import decode_predictions\n",
    "\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "##  Taking snapshots from the webcam\n",
    "\n",
    "For this section, we will take an image from your laptop webcam and classify it. If you feel uncomfortable doing this section, you can skip it and use a photo of your choice from the web instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "\n",
    "def camera_grab(camera_id=0, fallback_filename=None):\n",
    "    camera = cv2.VideoCapture(camera_id)\n",
    "    try:\n",
    "        # take 10 consecutive snapshots to let the camera automatically tune\n",
    "        # itself and hope that the contrast and lighting of the last snapshot\n",
    "        # is good enough.\n",
    "        for i in range(10):\n",
    "            snapshot_ok, image = camera.read()\n",
    "        if snapshot_ok:\n",
    "            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "        else:\n",
    "            print(\"WARNING: could not access camera\")\n",
    "            if fallback_filename:\n",
    "                image = imread(fallback_filename)\n",
    "    finally:\n",
    "        camera.release()\n",
    "    return image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "image = camera_grab(camera_id=0, fallback_filename='laptop.jpeg')\n",
    "plt.imshow(image)\n",
    "print(\"dtype: {}, shape: {}, range: {}\".format(\n",
    "    image.dtype, image.shape, (image.min(), image.max())))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Apply the same preprocessing as before and classify the image. What are your results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}