lectures/lecture-6/lecture-6.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 6: Vision Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Previous lecture:  Modelling sequences\n",
    "- Modelling sequences\n",
    "- Recurrent neural networks (RNN) and their variants: RNN, GRU, LSTM\n",
    "- Concept of attention\n",
    "- Transformer architecture\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Todays lecture\n",
    "- Original idea of VIT\n",
    "- Beit (3 versions)\n",
    "- Cait\n",
    "- Cross-ViT\n",
    "- SWIN\n",
    "- Transformers for object detection\n",
    "- DINO\n",
    "- MobileVIT (3 versions)\n",
    "- XCiT\n",
    "- MLP-Mixer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformers in vision\n",
    "\n",
    "Transormer-based models have revolutionized NLP models. The idea to apply them to images seems to be natural --- only after it has been proposed in the paper\n",
    "\n",
    "\n",
    "[An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby](https://arxiv.org/abs/2010.11929)\n",
    "\n",
    "What is the main idea?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Patches\n",
    "\n",
    "The main task is to replace an image by a sequence.\n",
    "\n",
    "\n",
    "The most natural idea to do so is **patches**.\n",
    "\n",
    "The image is split into patches (blocks). The blocks are processed by a very simple 1-layer convolutional neural networks, \n",
    "\n",
    "the patches are flattened and **positional encodings** are added to each pixel.\n",
    "\n",
    "For classification, and additional **classification** head is added.\n",
    "\n",
    "It is better to see...\n",
    "\n",
    "<img src='patches.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Architecture of VIT\n",
    "\n",
    "When we have our **sequence**, we can just apply the transformer on top to do classication.\n",
    "\n",
    "<img src='architecture.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## General scheme of VIT\n",
    "\n",
    "<img src='source.gif'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training original VIT\n",
    "\n",
    "The model is pretrained on **large datasets** and then fine-tuned on downstream tasks.\n",
    "\n",
    "Position embeddings are **learnable**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training on downstream tasks\n",
    "\n",
    "For training on downstream tasks, the prediction head for the original classification task is **removed** and replaced by $D \\times K$ layer where $K$ is the number of output classes.\n",
    "\n",
    "We can also train on **higher resolution**. How can we train on a higher resolution?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training on higher resolution\n",
    "\n",
    "Fully-convolutional neural networks and be applied to arbitrary resolution images. \n",
    "\n",
    "VIT can also be applied to images of arbitrary resolutions. \n",
    "\n",
    "The patch size is the same, thus different resolution will lead to larger **sequence lengths**, and transformer can be applied to sequences of variable length.\n",
    "\n",
    "The only small problem is **positional encodings**, but they can be **interpolated** for new image sizes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Types of original VIT models\n",
    "\n",
    "The original VIT models differ in the sizes of the layers and number of them\n",
    "\n",
    "<img src='vit-types.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## VIT-numbers\n",
    "\n",
    "The numbers from the original VIT paper compared with ResNet baselines.\n",
    "\n",
    "The JFT dataset has 303M images, I21K (Imagenet 21k) has 14M images.\n",
    "\n",
    "<img src='vit-results.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Self-supervised pre-training\n",
    "\n",
    "One can also do **self-supervised pretraining**: similar to BERT, we can mask some of the patches and try to reconstruct them. \n",
    "\n",
    "The original VIT paper reported 79.9% accuracy on ImageNet (2% more compared to training from scratch, but 4% less behind supervised pretraing.\n",
    "\n",
    "This was later explicitly done in [BEiT: BERT Pre-Training of Image Transformers](https://arxiv.org/abs/2106.08254)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Bert pretraining of image transformers\n",
    "\n",
    "The image has two views: patches and visual tokens. The visual tokens are **highly non-trivial**: they are based on latent codes of **discrete variational autoencoders**\n",
    "\n",
    "<img src='beit-scheme.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## BeIT version 2\n",
    "\n",
    "The BeIT version 2 uses **visual token masking**\n",
    "\n",
    "[https://arxiv.org/abs/2208.06366](https://arxiv.org/abs/2208.06366)\n",
    "\n",
    "<img src='bert-v2.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## BeIT version 3\n",
    "\n",
    "[Image as a Foreign Language: BEIT Pretraining for\n",
    "All Vision and Vision-Language Tasks](https://arxiv.org/pdf/2208.10442v2.pdf) has reached SOTA on many classical tasks:\n",
    "\n",
    "- object detection (COCO) \n",
    "- semantic segmentation (ADE20K) \n",
    "- image classification (ImageNet)\n",
    "- visual reasoning (NLVR2)\n",
    "- visual question answering (VQAv2)\n",
    "- image captioning (COCO)\n",
    "- cross-modal retrieval (Flickr30K, COCO).\n",
    "\n",
    "The idea is to use masked pretraining!\n",
    "\n",
    "<img src='beit-3-pretraining.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## DEIT transformer\n",
    "\n",
    "The next milestone is data-efficient image transformer (DeIT) proposed in [Training data-efficient image transformers & distillation through attention](https://proceedings.mlr.press/v139/touvron21a)\n",
    "\n",
    "One of the conclusions of VIT paper was that **vision transformers do not generalize well when trained on insufficient amount of data** which is not **true**\n",
    "\n",
    "The DEIT paper showed that transformers can be trained competitively on the same data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## DEIT training efficiency\n",
    "\n",
    "The model is trained on ImageNet, Deit-B has the same architecture as VIT-B but in **data-starving regime**, trained on several days on one node with 4 GPUs.\n",
    "\n",
    "The training had several innovations and tricks, and the most interesting part: **distillation** (learning from another neural network).\n",
    "\n",
    "<img src='deit-flops.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Distillation \n",
    "\n",
    "We have one model $f_1(x)$ (called teacher), and we want that another model (student) learns from such predictions. \n",
    "\n",
    "The idea of DeIT paper (and it seems to be the main contribution of it) is the idea of **distillation through attention**.\n",
    "\n",
    "<img src='deit-distill.png' width=80%>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## CaIT transformer\n",
    "\n",
    "Another innovation from the same group [Going deeper with Image Transformers](https://arxiv.org/pdf/2103.17239.pdf) proposes the architectural modifications:\n",
    "\n",
    "\n",
    "- Put class token **deeper** into the architecture\n",
    "<img src='cait-cls.png'  width=30% >\n",
    "- **LayerScale:** in the residual connection, the weights are multiplied by very small numbers at initialization (starting layers are almost identical). These numbers are **learnable**.\n",
    "\n",
    "  $$x'_l = x_l + \\mathrm{diag}(\\lambda_{l,1}, \\ldots, \\lambda_{l, d}) \\mathrm{SA}(\\eta(x_l)).$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Cross-VIT: yet another attention\n",
    "\n",
    "[Cross-VIT paper](https://arxiv.org/abs/2103.14899) proposes a multiscale variant (later to be replaced with so-called SWIN transformer which we will discuss later).\n",
    "\n",
    "The idea is to process patches of different sizes.\n",
    "\n",
    "This can be done by **parallel branches**. But how can we exchange information between those branches?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Cross-VIT: scheme\n",
    "\n",
    "The Cross-VIT comes from cross-attention, the concept we have not yet discussed. \n",
    "\n",
    "It is attention between two different token sets (compared to **self-attention**).\n",
    "\n",
    "<img src='cross-vit.png' width=40%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SWIN transformer\n",
    "\n",
    "Next architecture step is [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV 2021](https://openaccess.thecvf.com/content/ICCV2021/papers/Liu_Swin_Transformer_Hierarchical_Vision_Transformer_Using_Shifted_Windows_ICCV_2021_paper.pdf)\n",
    "\n",
    "**Challenge 1:** One of the main differences between NLP and CV is **scale**: different objects have different sizes.\n",
    "\n",
    "In VIT, all tokens have the same size.\n",
    "\n",
    "**Challenge 2:** The token size. If we go to higher resolutions, the self-attention will scale quadratically with the number of tokens (patches), making computations infeasible.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SWIN architecture\n",
    "The idea of SWIN is to process coarser resolution at deeper layers\n",
    "\n",
    "<img src='swin.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SWIN-scheme of the model\n",
    "\n",
    "The model has the following scheme (from the original paper).\n",
    "The main difference is in self-attention: it is only computed in **local windows**, \n",
    "removing the quadratic complexity effectively.\n",
    "<img src='swin-scheme.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Two self-attention modules in SWIN\n",
    "\n",
    "SWIN transformer has two different attention.\n",
    "\n",
    "**First** is **W-MSA** (window multi-head self-attention), where attention is computed within a local window.\n",
    "\n",
    "**Second** is **SW-MSA** (shifted window multi-head self-attention), where attention is computed within a window shifted by half-of-the size)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Architecture variants\n",
    "\n",
    "The variants considered in the original paper:\n",
    "\n",
    "$ \\text{Swin-T}: C = 96, \\text{layer numbers} = \\{2, 2, 6, 2\\} $ \n",
    "\n",
    "$ \\text{Swin-S}: C = 96, \\text{layer numbers} = \\{2, 2, 18, 2\\} $ \n",
    "\n",
    "$ \\text{Swin-B}: C = 128, \\text{layer numbers} = \\{2, 2, 18, 2\\} $ \n",
    "\n",
    "$ \\text{Swin-L}: C = 192, \\text{layer numbers} = \\{2, 2, 18, 2\\} $ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SWIN: numbers\n",
    "\n",
    "The numbers from the paper:\n",
    "<img src='swin-results.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformer for other tasks\n",
    "\n",
    "One can directly change the backbone from CNN to SWIN and leave everything 'as it is'. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## End-to-end detection with transformers: loss\n",
    "\n",
    "The first application of transformers was in [End-to-End Object Detection with Transformers](https://arxiv.org/pdf/2005.12872.pdf).\n",
    "\n",
    "Which proposed **DEtection TRansformer (DETR)** architecture.\n",
    "\n",
    "\n",
    "Let us denote by $y$ the ground truth set of objects, $\\hat{y} = \\{\\hat{y}_i\\}_{i=1}^N$ the set of predictions.\n",
    "\n",
    "Then, we match the sets\n",
    "\n",
    "$$\\hat{\\sigma} = \\arg \\min_{\\sigma \\in \\Pi_N} \\sum_{i=1}^N \\mathcal{L}_{\\mathrm{match}}(y_i, \\hat{y}_{\\sigma_i})$$\n",
    "\n",
    "The optimal matching can be computed by i.e. Hungarian algorithm (optimal assignment problem).\n",
    "\n",
    "For the matching selected, the loss is computed. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## DETR: architecture\n",
    "\n",
    "The architecture is combined CNN+Encoder/decoder transformer\n",
    "\n",
    "<img src='detr-architecture.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Semantic segmentation with transformers\n",
    "\n",
    "[Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective\n",
    "with Transformers](https://openaccess.thecvf.com/content/CVPR2021/papers/Zheng_Rethinking_Semantic_Segmentation_From_a_Sequence-to-Sequence_Perspective_With_Transformers_CVPR_2021_paper.pdf)\n",
    "\n",
    "Building upon idea of of fully-connected-networks, we can design a transformer-based architecture for segmentation.\n",
    "<img src='setr.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## DINO\n",
    "\n",
    "[DINO paper](https://openaccess.thecvf.com/content/ICCV2021/papers/Caron_Emerging_Properties_in_Self-Supervised_Vision_Transformers_ICCV_2021_paper.pdf)\n",
    "\n",
    "**Di**stillation with **No** labels: self-supervised contrastive learning. \n",
    "\n",
    "Replaces **masking** with training on **image augmentations**.\n",
    "\n",
    "You can also learn meaningful embeddings with **masked autoencoders** see [Masked Autoencoders are scalable vision learners](https://openaccess.thecvf.com/content/CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.html)\n",
    "\n",
    "How and why it is done we will cover in the corresponding lecture.\n",
    "\n",
    "Now we can move to making transformers more efficient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Mobile versions: MobileVIT\n",
    "\n",
    "VIT-B version is much heavier:\n",
    "\n",
    "ViT-B/16 vs. MobileNetv3: 86 vs. 7.5 million parameters\n",
    "\n",
    "Seem to be more difficult to optimize.\n",
    "\n",
    "Paper [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://arxiv.org/abs/2110.02178) proposes a light-weight architecture for transformers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Key idea of MobileVIT\n",
    "\n",
    "The key idea of MobileVIT is **leave the spatial structure of the patches**.\n",
    "\n",
    "I.e., we have \n",
    "\n",
    "1. Convolutions\n",
    "2. Transformer on patches (no flattening of patches!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Architecture of Mobile-VIT\n",
    "The full architecture of original MobileVIT (compared to VIT) is shown here:\n",
    "\n",
    "<img src='mobile-vit.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Numbers of original MobileVIT\n",
    "\n",
    "<img src='mobilevit-numbers.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Evolution of MobileVIT architecture\n",
    "The MobileVIT has evolved to [version2](https://arxiv.org/abs/2206.02680). \n",
    "\n",
    "The main idea to make $\\mathcal{O}(k)$ attention which has linear complexity in the number of tokens.\n",
    "\n",
    "<img src='mobilevitv2-attention.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MobileVIT-version3\n",
    "\n",
    "Soon, the third version of the block has been proposed in [this paper](https://arxiv.org/abs/2209.15159).\n",
    "\n",
    "This is deep learning engineering: 1x1 conv. layer instead of 3, depthwise convolutional layer (remember 3x3 case).\n",
    "\n",
    "<img src='mobilevitv3.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## XCiT: Cross-Covariance Image Transformers\n",
    "\n",
    "\n",
    "[XCiT: Cross-Covariance Image Transformers](https://arxiv.org/abs/2106.09681) paper proposes another way of tackling the problem with attention\n",
    "\n",
    "\n",
    "The idea is interesting: take attention along **feature dimension**:\n",
    "\n",
    "<img src='xcit.png'>\n",
    "\n",
    "Very fast in practice. Not sure if it was not combined with other innovations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Do we need attention at all?\n",
    "\n",
    "The goal of **attention** is to mix the information between token. But does the actual multi-head attention matter?\n",
    "\n",
    "The [MLP-Mixer](https://proceedings.neurips.cc/paper/2021/hash/cba0a4ee5ccd02fda0fe3f9a3e7b89fe-Abstract.html) architecture shows that for sequences of fixed size we can replace everything by fully-connected layers!\n",
    "\n",
    "<img src='mlp-mixer.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summary\n",
    "\n",
    "- Original idea of VIT\n",
    "- Beit (3 versions)\n",
    "- Cait\n",
    "- Cross-ViT\n",
    "- Transformers for object detection\n",
    "- DINO\n",
    "- MobileVIT (3 versions)\n",
    "- XCiT\n",
    "- MLP-Mixer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Next lecture: Graph neural networks\n",
    "\n",
    "- Deep learning on graphs\n",
    "- Basic models\n",
    "- Applications"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Слайд-шоу",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "nav_menu": {},
  "rise": {
   "scroll": true,
   "theme": "sky",
   "transition": "zoom"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}