Lab_10_(part_2).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lab 10 (part 2)",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "REaV1GblZovs",
        "colab_type": "text"
      },
      "source": [
        "#Visualizing and understanding ConvNets (part 2)\n",
        "In this part of the lab we consider reconstruction-based filter isualization.\n",
        "\n",
        "The content is inspired by the following:\n",
        "\n",
        "Blog post: https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html\n",
        "\n",
        "Source: https://github.com/keras-team/keras/blob/master/examples/conv_filter_visualization.py\n",
        "\n",
        "\n",
        "**Before we start - remember to set runtime to GPU**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VSIpfu6wUG1r",
        "colab_type": "text"
      },
      "source": [
        "**NOTE:** In case you have trouble running Keras/TensorFlow in Colab, try one of the following:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fZ01fAojUGCV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Try this\n",
        "#!pip install --upgrade tensorflow==1.8.0\n",
        "\n",
        "# ... or this\n",
        "%tensorflow_version 1.x\n",
        "\n",
        "# Check TensorFlow version\n",
        "#import tensorflow as tf\n",
        "#print(tf.__version__)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LDD-J8deAV3p",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "Run this code block:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "77TqsuoOAQU8",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "#Visualization of the filters of VGG16, via gradient ascent in input space.\n",
        "\n",
        "This script can run on CPU in a few minutes.\n",
        "\n",
        "Results example: ![Visualization](http://i.imgur.com/4nj4KjN.jpg)\n",
        "\"\"\"\n",
        "from __future__ import print_function\n",
        "\n",
        "import time\n",
        "import numpy as np\n",
        "from PIL import Image as pil_image\n",
        "from keras.preprocessing.image import save_img\n",
        "from keras import layers\n",
        "from keras.applications import vgg16\n",
        "from keras import backend as K\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def normalize(x):\n",
        "    \"\"\"utility function to normalize a tensor.\n",
        "\n",
        "    # Arguments\n",
        "        x: An input tensor.\n",
        "\n",
        "    # Returns\n",
        "        The normalized input tensor.\n",
        "    \"\"\"\n",
        "    return x / (K.sqrt(K.mean(K.square(x))) + K.epsilon())\n",
        "\n",
        "\n",
        "def deprocess_image(x):\n",
        "    \"\"\"utility function to convert a float array into a valid uint8 image.\n",
        "\n",
        "    # Arguments\n",
        "        x: A numpy-array representing the generated image.\n",
        "\n",
        "    # Returns\n",
        "        A processed numpy-array, which could be used in e.g. imshow.\n",
        "    \"\"\"\n",
        "    # normalize tensor: center on 0., ensure std is 0.25\n",
        "    x -= x.mean()\n",
        "    x /= (x.std() + K.epsilon())\n",
        "    x *= 0.25\n",
        "\n",
        "    # clip to [0, 1]\n",
        "    x += 0.5\n",
        "    x = np.clip(x, 0, 1)\n",
        "\n",
        "    # convert to RGB array\n",
        "    x *= 255\n",
        "    if K.image_data_format() == 'channels_first':\n",
        "        x = x.transpose((1, 2, 0))\n",
        "    x = np.clip(x, 0, 255).astype('uint8')\n",
        "    return x\n",
        "\n",
        "\n",
        "def process_image(x, former):\n",
        "    \"\"\"utility function to convert a valid uint8 image back into a float array.\n",
        "       Reverses `deprocess_image`.\n",
        "\n",
        "    # Arguments\n",
        "        x: A numpy-array, which could be used in e.g. imshow.\n",
        "        former: The former numpy-array.\n",
        "                Need to determine the former mean and variance.\n",
        "\n",
        "    # Returns\n",
        "        A processed numpy-array representing the generated image.\n",
        "    \"\"\"\n",
        "    if K.image_data_format() == 'channels_first':\n",
        "        x = x.transpose((2, 0, 1))\n",
        "    return (x / 255 - 0.5) * 4 * former.std() + former.mean()\n",
        "\n",
        "\n",
        "def visualize_layer(model,\n",
        "                    layer_name,\n",
        "                    step=1.,\n",
        "                    epochs=15,\n",
        "                    upscaling_steps=9,\n",
        "                    upscaling_factor=1.2,\n",
        "                    output_dim=(412, 412),\n",
        "                    filter_range=(0, None)):\n",
        "    \"\"\"Visualizes the most relevant filters of one conv-layer in a certain model.\n",
        "\n",
        "    # Arguments\n",
        "        model: The model containing layer_name.\n",
        "        layer_name: The name of the layer to be visualized.\n",
        "                    Has to be a part of model.\n",
        "        step: step size for gradient ascent.\n",
        "        epochs: Number of iterations for gradient ascent.\n",
        "        upscaling_steps: Number of upscaling steps.\n",
        "                         Starting image is in this case (80, 80).\n",
        "        upscaling_factor: Factor to which to slowly upgrade\n",
        "                          the image towards output_dim.\n",
        "        output_dim: [img_width, img_height] The output image dimensions.\n",
        "        filter_range: Tupel[lower, upper]\n",
        "                      Determines the to be computed filter numbers.\n",
        "                      If the second value is `None`,\n",
        "                      the last filter will be inferred as the upper boundary.\n",
        "    \"\"\"\n",
        "\n",
        "    def _generate_filter_image(input_img,\n",
        "                               layer_output,\n",
        "                               filter_index):\n",
        "        \"\"\"Generates image for one particular filter.\n",
        "\n",
        "        # Arguments\n",
        "            input_img: The input-image Tensor.\n",
        "            layer_output: The output-image Tensor.\n",
        "            filter_index: The to be processed filter number.\n",
        "                          Assumed to be valid.\n",
        "\n",
        "        #Returns\n",
        "            Either None if no image could be generated.\n",
        "            or a tuple of the image (array) itself and the last loss.\n",
        "        \"\"\"\n",
        "        s_time = time.time()\n",
        "\n",
        "        # we build a loss function that maximizes the activation\n",
        "        # of the nth filter of the layer considered\n",
        "        if K.image_data_format() == 'channels_first':\n",
        "            loss = K.mean(layer_output[:, filter_index, :, :])\n",
        "        else:\n",
        "            loss = K.mean(layer_output[:, :, :, filter_index])\n",
        "\n",
        "        # we compute the gradient of the input picture wrt this loss\n",
        "        grads = K.gradients(loss, input_img)[0]\n",
        "\n",
        "        # normalization trick: we normalize the gradient\n",
        "        grads = normalize(grads)\n",
        "\n",
        "        # this function returns the loss and grads given the input picture\n",
        "        iterate = K.function([input_img], [loss, grads])\n",
        "\n",
        "        # we start from a gray image with some random noise\n",
        "        intermediate_dim = tuple(\n",
        "            int(x / (upscaling_factor ** upscaling_steps)) for x in output_dim)\n",
        "        if K.image_data_format() == 'channels_first':\n",
        "            input_img_data = np.random.random(\n",
        "                (1, 3, intermediate_dim[0], intermediate_dim[1]))\n",
        "        else:\n",
        "            input_img_data = np.random.random(\n",
        "                (1, intermediate_dim[0], intermediate_dim[1], 3))\n",
        "        input_img_data = (input_img_data - 0.5) * 20 + 128\n",
        "\n",
        "        # Slowly upscaling towards the original size prevents\n",
        "        # a dominating high-frequency of the to visualized structure\n",
        "        # as it would occur if we directly compute the 412d-image.\n",
        "        # Behaves as a better starting point for each following dimension\n",
        "        # and therefore avoids poor local minima\n",
        "        for up in reversed(range(upscaling_steps)):\n",
        "            # we run gradient ascent for e.g. 20 steps\n",
        "            for _ in range(epochs):\n",
        "                loss_value, grads_value = iterate([input_img_data])\n",
        "                input_img_data += grads_value * step\n",
        "\n",
        "                # some filters get stuck to 0, we can skip them\n",
        "                if loss_value <= K.epsilon():\n",
        "                    return None\n",
        "\n",
        "            # Calculate upscaled dimension\n",
        "            intermediate_dim = tuple(\n",
        "                int(x / (upscaling_factor ** up)) for x in output_dim)\n",
        "            # Upscale\n",
        "            img = deprocess_image(input_img_data[0])\n",
        "            img = np.array(pil_image.fromarray(img).resize(intermediate_dim,\n",
        "                                                           pil_image.BICUBIC))\n",
        "            input_img_data = np.expand_dims(\n",
        "                process_image(img, input_img_data[0]), 0)\n",
        "\n",
        "        # decode the resulting input image\n",
        "        img = deprocess_image(input_img_data[0])\n",
        "        e_time = time.time()\n",
        "        print('Costs of filter {:3}: {:5.0f} ( {:4.2f}s )'.format(filter_index,\n",
        "                                                                  loss_value,\n",
        "                                                                  e_time - s_time))\n",
        "        return img, loss_value\n",
        "\n",
        "    def _draw_filters(filters, n=None):\n",
        "        \"\"\"Draw the best filters in a nxn grid.\n",
        "\n",
        "        # Arguments\n",
        "            filters: A List of generated images and their corresponding losses\n",
        "                     for each processed filter.\n",
        "            n: dimension of the grid.\n",
        "               If none, the largest possible square will be used\n",
        "        \"\"\"\n",
        "        if n is None:\n",
        "            n = int(np.floor(np.sqrt(len(filters))))\n",
        "\n",
        "        # the filters that have the highest loss are assumed to be better-looking.\n",
        "        # we will only keep the top n*n filters.\n",
        "        filters.sort(key=lambda x: x[1], reverse=True)\n",
        "        filters = filters[:n * n]\n",
        "\n",
        "        # build a black picture with enough space for\n",
        "        # e.g. our 8 x 8 filters of size 412 x 412, with a 5px margin in between\n",
        "        MARGIN = 5\n",
        "        width = n * output_dim[0] + (n - 1) * MARGIN\n",
        "        height = n * output_dim[1] + (n - 1) * MARGIN\n",
        "        stitched_filters = np.zeros((width, height, 3), dtype='uint8')\n",
        "\n",
        "        # fill the picture with our saved filters\n",
        "        for i in range(n):\n",
        "            for j in range(n):\n",
        "                img, _ = filters[i * n + j]\n",
        "                width_margin = (output_dim[0] + MARGIN) * i\n",
        "                height_margin = (output_dim[1] + MARGIN) * j\n",
        "                stitched_filters[\n",
        "                    width_margin: width_margin + output_dim[0],\n",
        "                    height_margin: height_margin + output_dim[1], :] = img\n",
        "\n",
        "        # save the result to disk\n",
        "        save_img('vgg_{0:}_{1:}x{1:}.png'.format(layer_name, n), stitched_filters)\n",
        "        plt.imshow(stitched_filters)\n",
        "\n",
        "    # this is the placeholder for the input images\n",
        "    assert len(model.inputs) == 1\n",
        "    input_img = model.inputs[0]\n",
        "\n",
        "    # get the symbolic outputs of each \"key\" layer (we gave them unique names).\n",
        "    layer_dict = dict([(layer.name, layer) for layer in model.layers[1:]])\n",
        "\n",
        "    output_layer = layer_dict[layer_name]\n",
        "    assert isinstance(output_layer, layers.Conv2D)\n",
        "\n",
        "    # Compute to be processed filter range\n",
        "    filter_lower = filter_range[0]\n",
        "    filter_upper = (filter_range[1]\n",
        "                    if filter_range[1] is not None\n",
        "                    else len(output_layer.get_weights()[1]))\n",
        "    assert(filter_lower >= 0\n",
        "           and filter_upper <= len(output_layer.get_weights()[1])\n",
        "           and filter_upper > filter_lower)\n",
        "    print('Compute filters {:} to {:}'.format(filter_lower, filter_upper))\n",
        "\n",
        "    # iterate through each filter and generate its corresponding image\n",
        "    processed_filters = []\n",
        "    for f in range(filter_lower, filter_upper):\n",
        "        img_loss = _generate_filter_image(input_img, output_layer.output, f)\n",
        "\n",
        "        if img_loss is not None:\n",
        "            processed_filters.append(img_loss)\n",
        "\n",
        "    print('{} filter processed.'.format(len(processed_filters)))\n",
        "    # Finally draw and store the best filters to disk\n",
        "    _draw_filters(processed_filters)\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    # the name of the layer we want to visualize\n",
        "    # (see model definition at keras/applications/vgg16.py)\n",
        "    LAYER_NAME = 'block5_conv1'\n",
        "\n",
        "    # build the VGG16 network with ImageNet weights\n",
        "    vgg = vgg16.VGG16(weights='imagenet', include_top=False)\n",
        "    print('Model loaded.')\n",
        "    vgg.summary()\n",
        "\n",
        "    # example function call\n",
        "    i = 10 # Index of ilfter to visualize\n",
        "    visualize_layer(vgg, LAYER_NAME,\n",
        "                    filter_range=(i, i+1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5uukAQ0oFZ2F",
        "colab_type": "text"
      },
      "source": [
        "##Questions\n",
        "1. The graph is changed in order to derive the filter response image. How is the graph modified?\n",
        "2. How is the loss calculated? What is being maximized/minimized?\n",
        "3. Try visualizing other filters from another layers by changing `LAYER_NAME` and `filter_range`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PPKunlzLGZOT",
        "colab_type": "text"
      },
      "source": [
        "## Better implementation\n",
        "Below is a better implementation from here\n",
        "\n",
        "Source: https://github.com/raghakot/keras-vis\n",
        "\n",
        "Homepage: https://raghakot.github.io/keras-vis/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ml1KOBhgGfrj",
        "colab_type": "text"
      },
      "source": [
        "Try it out (refer to the notebook for explanations: https://github.com/raghakot/keras-vis/blob/master/examples/vggnet/activation_maximization.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JFi0zcJQ1V6h",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install scipy==0.19.1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8nSMD_f3Gxfd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install git+https://github.com/raghakot/keras-vis.git -U"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D8wZnOgwGxu2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.applications import VGG16\n",
        "from vis.utils import utils\n",
        "from keras import activations\n",
        "\n",
        "# Build the VGG16 network with ImageNet weights\n",
        "model = VGG16(weights='imagenet', include_top=True)\n",
        "\n",
        "# Utility to search for layer index by name. \n",
        "# Alternatively we can specify this as -1 since it corresponds to the last layer.\n",
        "layer_idx = utils.find_layer_idx(model, 'predictions')\n",
        "\n",
        "# Swap softmax with linear\n",
        "model.layers[layer_idx].activation = activations.linear\n",
        "model = utils.apply_modifications(model)\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EvZJv5AGG9cG",
        "colab_type": "text"
      },
      "source": [
        "Last layer visualization (i.e., fully connected layer):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zoup1ylMG4RZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from vis.visualization import visualize_activation\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "%matplotlib inline\n",
        "plt.rcParams['figure.figsize'] = (18, 6)\n",
        "\n",
        "# 20 is the imagenet category for 'ouzel'\n",
        "img = visualize_activation(model, layer_idx, filter_indices=20)\n",
        "plt.imshow(img)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ntaw1pkoHE_0",
        "colab_type": "text"
      },
      "source": [
        "Try more iterations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rdbaoWzkHGbO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 20 is the imagenet category for 'ouzel'\n",
        "img = visualize_activation(model, layer_idx, filter_indices=20, max_iter=500, verbose=True)\n",
        "plt.imshow(img)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "usU6lcCJHIhX",
        "colab_type": "text"
      },
      "source": [
        "Try with Jitter (what is it?):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FOEtNYZ9HPFT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from vis.input_modifiers import Jitter\n",
        "\n",
        "# 20 is the imagenet category for 'ouzel'\n",
        "# Jitter 16 pixels along all dimensions to during the optimization process.\n",
        "img = visualize_activation(model, layer_idx, filter_indices=20, max_iter=500, input_modifiers=[Jitter(16)])\n",
        "plt.imshow(img)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SLZ81mF3HVi6",
        "colab_type": "text"
      },
      "source": [
        "Filter visualization (for convolutional layers)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VkDsI3zYHYaY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from vis.visualization import get_num_filters\n",
        "import numpy as np\n",
        "# The name of the layer we want to visualize\n",
        "# You can see this in the model definition.\n",
        "layer_name = 'block1_conv2'\n",
        "layer_idx = utils.find_layer_idx(model, layer_name)\n",
        "\n",
        "# Visualize all filters in this layer.\n",
        "filters = np.arange(get_num_filters(model.layers[layer_idx]))\n",
        "\n",
        "# Generate input image for each filter.\n",
        "vis_images = []\n",
        "for idx in filters:\n",
        "    img = visualize_activation(model, layer_idx, filter_indices=idx)\n",
        "    \n",
        "    # Utility to overlay text on image.\n",
        "    img = utils.draw_text(img, 'Filter {}'.format(idx))    \n",
        "    vis_images.append(img)\n",
        "\n",
        "# Generate stitched image palette with 8 cols.\n",
        "stitched = utils.stitch_images(vis_images, cols=8)    \n",
        "plt.axis('off')\n",
        "plt.imshow(stitched)\n",
        "plt.title(layer_name)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "01-ookNsIH3U",
        "colab_type": "text"
      },
      "source": [
        "###Grad-CAM\n",
        "If time permits, you may also want to try out Grad-CAM from the same repo:\n",
        "\n",
        "https://github.com/raghakot/keras-vis/blob/master/examples/vggnet/attention.ipynb"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_QYgZqSiIPEu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}