demo/denas/DENAS_SUMMARY.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e293d57f",
   "metadata": {},
   "source": [
    "# AIOK DE-NAS DEMO\n",
    "\n",
    "DE-NAS extends AIOK optimized models with Neural Architecture Search Technology. It is a multi-model, hardware-aware, train-free NAS named DE-NAS to construct compact (low computation complexity) model architectures for target platform directly. Firstly, DE-NAS constructs compact neural architecture directly from designed search spaces for multiple domains. Then, a hardware-aware search strategy is used based on given budget to determine the best network. Finally, hardware-aware train-free scoring method is designed to evaluate the candidate network’s performance rather than acquire accuracy of each candidate after its training.\n",
    "\n",
    "# Content \n",
    "* [DE-NAS Architecture and Componements](#DE-NAS-Architecture-and-Componements)\n",
    "* [Multi-Model Unified Search Space](#Multi-Model-Unified-Search-Space)\n",
    "    * [CNN-based Search Space](#CNN-based-Search-Space)\n",
    "    * [Transformer-based Search Space](#Transformer-based-Search-Space)\n",
    "* [Hardware-Aware Unified Search Framework](#Hardware-Aware-Unified-Search-Framework)\n",
    "    * [Random Search](#Random-Search)\n",
    "    * [Evolutionary Search](#Evolutionary-Search)\n",
    "    * [Bayesian Search (SigOpt-based)](#Bayesian-Search-(SigOpt-based))\n",
    "* [Train-Free Score](#Train-Free-Score)\n",
    "* [Try DE-NAS](#Try-DE-NAS)\n",
    "    * [DE-NAS on CV Domain](#DE-NAS-on-CV-Domain)\n",
    "    * [DE-NAS on NLP Domain](#DE-NAS-on-NLP-Domain)\n",
    "    * [DE-NAS on ASR Domain](#DE-NAS-on-ASR-Domain)\n",
    "    * [DE-NAS on HF Domain](#DE-NAS-on-HF-Domain)\n",
    "\n",
    "# DE-NAS Architecture and Componements\n",
    "\n",
    "- **Multi-Model Unified Search Space:** DE-NAS supports search spaces including Compute Vision (CV), Natural Language Processing (NLP), and Automatic Speech Recognition (ASR). Meanwhile, the search space utilizes Intel optimized neural operators, and incorporates prior knowledge. Another key innovation is that it is the first of its kind to share a unified transformer-based search space among different models. We leveraged a unified transformer structure for transformer-based search space for CV, NLP and ASR domains.\n",
    "   \n",
    "- **Hardware-Aware Unified Search Framework:** DE-NAS provides hardware-aware unifed search framework, and comes with three types of built-in search strategies: Random Search / Evolutionary Search / Bayesian Search, which can cover popular domains of CV/NLP/ASR and enhance DE-NAS with SigOpt-based Bayesian search strategy as an advanced feature.\n",
    "    \n",
    "- **Train-Free Evaluation:** DE-NAS provides a novel zero-cost metric called DE-Score, which combines network expressivity, network complexity, network diversity, network saliecny and network latency. Computation of DE-Score only takes a few forward inferences without training, making it extremely fast, lightweight and data-free. \n",
    "\n",
    "<img src=\"./img/denas_framework.jpg\" width=\"80%\">\n",
    "<center>DE-NAS Architecture</center>\n",
    "\n",
    "# Multi-Model Unified Search Space\n",
    "\n",
    "## CNN-based Search Space\n",
    "\n",
    "- Supports CV Domain\n",
    "- ResNet like skeleton: consists of residual blocks and bottleneck blocks, number of layers, convolutional layer kernel size, number of filters,\n",
    "\n",
    "<img src=\"./img/denas_cnn.png\" width=\"50%\">\n",
    "<center>CNN-based Search Space and Supernet</center>\n",
    "\n",
    "## Transformer-based Search Space \n",
    "\n",
    "- Supports CV, NLP, ASR Domain \n",
    "- Embedding dimension, number of heads, query/key/value dimension, MLP ratio, network depth.\n",
    "    \n",
    "<img src=\"./img/denas_transformer.png\" width=\"50%\">  \n",
    "<center>Transformer-based Search Space and Supernet</center>\n",
    "\n",
    "# Hardware-Aware Unified Search Framework\n",
    "\n",
    "DE-NAS employs a unified search framework for different search strategies. It uses factory design pattern in the DE-NAS search framework, which abstracts the domain-independent search process, and provides three built-in search strategies: Random Search / Evolutionary Search / Bayesian Search that can be easily turned on/off with a switch. Bayesian search algorithm was implemented by leveraging SigOpt API.\n",
    "\n",
    "<img src=\"./img/denas_search_framework.jpg\" width=\"50%\">\n",
    "<center>DE-NAS Unified Search Framework</center>\n",
    "\n",
    "## Random Search\n",
    "Random Search is the basic search strategy for DE-NAS, which maintains a fixed-size max-heap to store the randomly generated structures. For each randomly generated structure, we compute the hardware-aware train-free score and insert it into the heap. After a given number of iterations, the top of heap is returned as optimal structure.\n",
    "\n",
    "## Evolutionary Search\n",
    "\n",
    "<img src=\"./img/denas_ea.png\" width=\"30%\">\n",
    "<center>Evolutionary Search</center>\n",
    "\n",
    "- Initialization: Initialize super network architecture with search space, and initialize population pool for neural network architectures.\n",
    "- Selection: In the selection step, the population pool is sorted according to the calculated hardware-aware train-free score and a new fixed-size population pool is generated.\n",
    "- Crossover: In the crossover step, two network architecture sequences are selected from the population pool and perform cross over to generate new sequences.\n",
    "- Mutation: In the mutation step, network architecture sequences of the population pool are randomly flipped.\n",
    "\n",
    "## Bayesian Search (SigOpt-based)\n",
    "\n",
    "<img src=\"./img/sigopt_based_bayesian_search.jpg\" width=\"50%\">\n",
    "<center>SigOpt-based Bayesian Search</center>\n",
    "\n",
    "# Train-Free Score\n",
    "\n",
    "The train-free score uses an innovative, zero-cost “proxy” to predict model accuracy instead of full training and validation. It used a novel zero-cost metric combined Gaussian complexity based on network expressivity, NTK score based on network complexity, nuclear norm score based on network diversity, Synflow score based on network saliency, and latency score. The computation of DE-Score only takes a few forward inferences other than iterative training, making it extremely fast, lightweight, and data-free. The overall DE-Score was calculated as following equation:\n",
    "\n",
    "$DE_{score}=(D_{EXP}+D_{COM}+D_{DIV}+D_{SAL} )∙D_{LAT}$\n",
    "\n",
    "# Try DE-NAS\n",
    "\n",
    "## DE-NAS on CV Domain\n",
    "* [CV CNN](computer_vision/DENAS_CNN_DEMO.ipynb) - DE-NAS for CV CNN domain\n",
    "* [CV ViT](computer_vision/DENAS_ViT_DEMO.ipynb) - DE-NAS for CV ViT domain\n",
    "\n",
    "## DE-NAS on NLP Domain\n",
    "* [NLP](bert/DENAS_BERT_DEMO.ipynb) - DE-NAS for NLP domain\n",
    "\n",
    "## DE-NAS on ASR Domain\n",
    "* [ASR](asr/DENAS_ASR_DEMO.ipynb) - DE-NAS for ASR domain\n",
    "\n",
    "## DE-NAS on HF Domain\n",
    "* [HF](hf/DENAS_HF_DEMO.ipynb) - DE-NAS for HF domain"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch]",
   "language": "python",
   "name": "conda-env-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "61a57a4b5406d2de388e2f91097d4e4bcd7d5f4a46f53a795aa28a02eed27fc5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}