Y2023_M02_D17_how_to_quantize_and_bit_flip.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-03-01 18:38:14.928339: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-03-01 18:38:15.061468: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2023-03-01 18:38:15.061492: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2023-03-01 18:38:15.867374: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
      "2023-03-01 18:38:15.867465: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
      "2023-03-01 18:38:15.867474: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n"
     ]
    }
   ],
   "source": [
    "import qkeras\n",
    "import tensorflow as tf\n",
    "from qkeras.quantizers import quantized_bits\n",
    "import fkeras as fk\n",
    "from fkeras.utils import quantize_and_bitflip, quantize_and_bitflip_deterministic\n",
    "from tabulate import tabulate\n",
    "import numpy as np\n",
    "np.random.seed(1029384756)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How Does FKeras Inject Faults?\n",
    "The purpose of this notebook is to explain the process FKeras uses to inject faults (i.e., bit flips) via the `fkeras.utils.quantize_and_bitflip` function. We'll begin by defining exactly what we mean by \"fault injection\" in the context of FKeras and then we will work through the process of creating the necessary code/functions to make a generalized `quantize_and_bitflip` function.\n",
    "\n",
    "## What sort of faults are we injecting? \n",
    "In FKeras, a single fault injection maps onto the action of flipping a single bit in one of the parameters (e.g., weights, biases, inputs, etc.) of a Keras/QKeras model. Let's assume we have a QKeras model with a CONV2D layer and we want to inject a single fault into one of the weights in that CONV2D Layer. \n",
    "\n",
    "Before we inject the fault, let's review the quantization that QKeras applies to weight matrix. Assume the following is a representation of the CONV2D layer's original (unquantized) weight matrix.\n",
    "\n",
    "<!-- $$\\begin{bmatrix} 0.0 & -2.5 \\\\ -0.1258 & 1.879\\end{bmatrix}$$ -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.42638122 0.0285622  0.14838862]\n",
      " [0.61286693 0.7276006  0.31245513]]\n",
      "[[0.9670236  0.44825718 0.42079282]\n",
      " [0.31039814 0.27730493 0.7789492 ]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-03-01 18:38:17.042809: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2023-03-01 18:38:17.042854: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2023-03-01 18:38:17.042892: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fabricant): /proc/driver/nvidia/version does not exist\n",
      "2023-03-01 18:38:17.043422: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "weights_u = tf.convert_to_tensor(np.random.rand(2,2,3))\n",
    "for w in weights_u.numpy():\n",
    "    print(w)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "QKeras allows users to quantize the weights of a model layer via the use of a QKeras quantizer. The quantizer relies on a fixed-point representation in order to achieve the quantization. The following cell instantiates a QKeras quantizer which will quantize the weights to values that are representable by a fixed-point representation made up of 5 bits with 1 sign-bit, 1 integer bit, and 3 fractional bits. *It should be noted that QKeras will steal a bit from the fractional bits for the sign bit if `keep_negative=1`*.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_quantizer_kn1 = quantized_bits(bits=5, integer=1, keep_negative=1, alpha=1)\n",
    "test_quantizer_kn0 = quantized_bits(bits=5, integer=1, keep_negative=1, alpha=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fixed-point representation defined in the quantizer can represent the following values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.   ,  0.125,  0.25 ,  0.375,  0.5  ,  0.625,  0.75 ,  0.875,\n",
       "        1.   ,  1.125,  1.25 ,  1.375,  1.5  ,  1.625,  1.75 ,  1.875,\n",
       "       -2.   , -1.875, -1.75 , -1.625, -1.5  , -1.375, -1.25 , -1.125,\n",
       "       -1.   , -0.875, -0.75 , -0.625, -0.5  , -0.375, -0.25 , -0.125],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_quantizer_kn1.range()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we apply the quantizer to the original weight matrix, we get the following quantized weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.375 0.    0.125]\n",
      " [0.625 0.75  0.25 ]]\n",
      "[[1.    0.5   0.375]\n",
      " [0.25  0.25  0.75 ]]\n"
     ]
    }
   ],
   "source": [
    "weights_q = test_quantizer_kn1(weights_u)\n",
    "for w in weights_q.numpy():\n",
    "    print(w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weight (0, 0, 0): 0.4263812245  -->  0.375\n",
      "Weight (0, 0, 1): 0.0285622043  -->  0.000\n",
      "Weight (0, 0, 2): 0.1483886153  -->  0.125\n",
      "Weight (0, 1, 0): 0.6128669344  -->  0.625\n",
      "Weight (0, 1, 1): 0.7276006002  -->  0.750\n",
      "Weight (0, 1, 2): 0.3124551335  -->  0.250\n",
      "Weight (1, 0, 0): 0.9670235976  -->  1.000\n",
      "Weight (1, 0, 1): 0.4482571768  -->  0.500\n",
      "Weight (1, 0, 2): 0.4207928178  -->  0.375\n",
      "Weight (1, 1, 0): 0.3103981442  -->  0.250\n",
      "Weight (1, 1, 1): 0.2773049277  -->  0.250\n",
      "Weight (1, 1, 2): 0.7789492049  -->  0.750\n"
     ]
    }
   ],
   "source": [
    "for i in range(weights_q.numpy().shape[0]):\n",
    "    for j in range(weights_q.numpy().shape[1]):\n",
    "        for k in range(weights_q.numpy().shape[2]):\n",
    "            print(f\"Weight {(i,j,k)}: {weights_u.numpy()[i][j][k]:.10f}  -->  {weights_q.numpy()[i][j][k]:.3f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output from the previous cells show that, after the application of the quantizer, each weight in the weight matrix gets mapped to one of the representable values of the quantizer. Since we want to inject faults (i.e., bit flips) after this quantization occurs, we simply need a way of mapping from one of the quantizer's representable values to another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 00000 :  0.000\n",
      " 00001 :  0.125\n",
      " 00010 :  0.250\n",
      " 00011 :  0.375\n",
      " 00100 :  0.500\n",
      " 00101 :  0.625\n",
      " 00110 :  0.750\n",
      " 00111 :  0.875\n",
      " 01000 :  1.000\n",
      " 01001 :  1.125\n",
      " 01010 :  1.250\n",
      " 01011 :  1.375\n",
      " 01100 :  1.500\n",
      " 01101 :  1.625\n",
      " 01110 :  1.750\n",
      " 01111 :  1.875\n",
      " 10000 : -2.000\n",
      " 10001 : -1.875\n",
      " 10010 : -1.750\n",
      " 10011 : -1.625\n",
      " 10100 : -1.500\n",
      " 10101 : -1.375\n",
      " 10110 : -1.250\n",
      " 10111 : -1.125\n",
      " 11000 : -1.000\n",
      " 11001 : -0.875\n",
      " 11010 : -0.750\n",
      " 11011 : -0.625\n",
      " 11100 : -0.500\n",
      " 11101 : -0.375\n",
      " 11110 : -0.250\n",
      " 11111 : -0.125\n"
     ]
    }
   ],
   "source": [
    "for i, rv in enumerate(test_quantizer_kn1.range()):\n",
    "    rv_str = f\"{rv:2.3f}\"\n",
    "    print(f\" {i:05b} : {rv_str:>6}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "quant_config = test_quantizer_kn1.get_config()\n",
    "scaling_exponent = quant_config[\"bits\"] - quant_config[\"integer\"] - quant_config[\"keep_negative\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])\n",
      "tf.Tensor(\n",
      "[[[-0.375  0.    -0.125]\n",
      "  [-0.625 -0.75  -0.25 ]]\n",
      "\n",
      " [[-1.    -0.5   -0.375]\n",
      "  [-0.25  -0.25  -0.75 ]]], shape=(2, 2, 3), dtype=float32)\n",
      "\n",
      "\n",
      "array([-8,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0])\n",
      "tf.Tensor(\n",
      "[[[ 0.625  0.    -0.125]\n",
      "  [-0.625 -0.75  -0.25 ]]\n",
      "\n",
      " [[-1.    -0.5   -0.375]\n",
      "  [-0.25  -0.25  -0.75 ]]], shape=(2, 2, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "flbi_regions = [\n",
    "    fk.utils.FaultyLayerBitRegion(0, 0, 1.0),\n",
    "    fk.utils.FaultyLayerBitRegion(1, 1, 1.0),\n",
    "    ]\n",
    "\n",
    "print(fk.utils.quantize_and_bitflip_deterministic_v3(weights_u*-1, test_quantizer_kn1, []          , []))\n",
    "print()\n",
    "print()\n",
    "print(fk.utils.quantize_and_bitflip_deterministic_v3(weights_u*-1, test_quantizer_kn1, flbi_regions, []))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01000\n",
      "10000\n"
     ]
    }
   ],
   "source": [
    "print(f\"{8:05b}\")\n",
    "print(f\"{16:05b}\")\n",
    "\n",
    "curr_val = 8\n",
    "\n",
    "sign_mask = 2**(i_qbits - i_keep_negative)\n",
    "rval_mask = sign_mask -1\n",
    "for i in range(mask_array.shape[0]):\n",
    "    mask_array[i] = (mask_array[i] & rval_mask) - (mask_array[i] & sign_mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00100 10000 10100\n",
      "4 16 20\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'i_tensor' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_6326/3917596832.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{a} {b} {result}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mi_tensor\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mi_scaling_exp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'i_tensor' is not defined"
     ]
    }
   ],
   "source": [
    "#0.5 <--> -1.5\n",
    "a = int(0.5 * 2**(scaling_exponent))\n",
    "b = int(0b10000)\n",
    "\n",
    "result = a ^ b\n",
    "\n",
    "print(f\"{a:05b} {b:05b} {result:05b}\")\n",
    "\n",
    "print(f\"{a} {b} {result}\")\n",
    "\n",
    "i_tensor * (2**-i_scaling_exp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.5"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(int(0.5 * 2**scaling_exponent) ^ -16)*2**-scaling_exponent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int(0.5 * 2**scaling_exponent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(int(0.5 * 2**scaling_exponent) ^ int(0b10000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-12"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(int(0.5 * 2**scaling_exponent) ^ -int(0b10000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.5"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(int(0.5 * 2**scaling_exponent) ^ int(0b10000))*2**-scaling_exponent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-12"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(int(0.5 * 2**scaling_exponent) ^ -int(0b10000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.5"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(int(0.5 * 2**scaling_exponent) ^ -int(0b10000))*2**-scaling_exponent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'-10000'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"{-16:05b}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'-0b1100'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bin(int(-1.5 * 2**(scaling_exponent)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0b10100'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bin(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'-0b1100'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bin(-12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int(0b11100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'i_tensor' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_6326/1931551785.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mog_dtype\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mi_tensor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mi_tensor\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mi_tensor\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mi_scaling_exp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mi_tensor\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi_tensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mint64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m i_tensor = tf.bitwise.bitwise_xor(\n\u001b[1;32m      5\u001b[0m     \u001b[0mi_tensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgen_mask_tensor_random\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi_tensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi_ber\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi_qbits\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'i_tensor' is not defined"
     ]
    }
   ],
   "source": [
    "    og_dtype = i_tensor.dtype\n",
    "    i_tensor = i_tensor * (2**i_scaling_exp)\n",
    "    i_tensor = tf.cast(i_tensor, tf.int64)\n",
    "    i_tensor = tf.bitwise.bitwise_xor(\n",
    "        i_tensor, gen_mask_tensor_random(i_tensor, i_ber, i_qbits)\n",
    "    )\n",
    "    i_tensor = tf.cast(i_tensor, og_dtype)\n",
    "    i_tensor = i_tensor * (2**-i_scaling_exp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[-1.625,  0.   ,  0.125],\n",
       "        [ 0.625,  0.75 ,  0.25 ]],\n",
       "\n",
       "       [[ 1.   ,  0.5  ,  0.375],\n",
       "        [ 0.25 ,  0.25 ,  0.75 ]]], dtype=float32)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def full_tensor_quantize_and_bit_flip_manual(i_tensor, i_scaling_exp, i_ber, i_qbits, i_mask_tensor):\n",
    "    og_dtype = i_tensor.dtype\n",
    "    i_tensor = i_tensor * (2**i_scaling_exp)\n",
    "    i_tensor = tf.cast(i_tensor, tf.int64)\n",
    "    i_tensor = tf.bitwise.bitwise_xor(\n",
    "        i_tensor, i_mask_tensor\n",
    "    )\n",
    "    i_tensor = tf.cast(i_tensor, og_dtype)\n",
    "    i_tensor = i_tensor * (2**-i_scaling_exp)\n",
    "\n",
    "    return i_tensor\n",
    "\n",
    "manual_mask_tensor = np.zeros((2,2,3), dtype=np.int64)\n",
    "manual_mask_tensor[0,0,0] = -16\n",
    "manual_mask_tensor = tf.convert_to_tensor( manual_mask_tensor, dtype=np.int64)\n",
    "\n",
    "full_tensor_quantize_and_bit_flip_manual(weights_q, scaling_exponent, 1/60, quant_config[\"bits\"], manual_mask_tensor).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[0.375 0.    0.125]\n",
      "  [0.625 0.75  0.25 ]]\n",
      "\n",
      " [[1.    0.5   0.375]\n",
      "  [0.25  0.25  0.75 ]]]\n",
      "####################\n",
      "[[[0.25  0.    0.125]\n",
      "  [0.625 0.75  0.25 ]]\n",
      "\n",
      " [[1.    0.5   0.375]\n",
      "  [0.25  0.25  0.75 ]]]\n",
      "####################\n",
      "[[[0.25  0.    0.125]\n",
      "  [0.625 0.75  0.25 ]]\n",
      "\n",
      " [[1.    0.5   0.375]\n",
      "  [0.25  0.25  0.75 ]]]\n",
      "####################\n",
      "[[[0.25  0.    0.125]\n",
      "  [0.625 0.75  0.25 ]]\n",
      "\n",
      " [[1.    0.5   0.375]\n",
      "  [0.25  0.25  0.75 ]]]\n"
     ]
    }
   ],
   "source": [
    "print(weights_q.numpy())\n",
    "print(\"####################\")\n",
    "print(fk.utils.full_tensor_quantize_and_bit_flip_deterministic(weights_q, scaling_exponent, 1/60, quant_config[\"bits\"]).numpy())\n",
    "print(\"####################\")\n",
    "print(fk.utils.full_tensor_quantize_and_bit_flip_deterministic_v2(weights_q, scaling_exponent, 1/60, quant_config[\"bits\"], quant_config[\"keep_negative\"]).numpy())\n",
    "print(\"####################\")\n",
    "print(fk.utils.full_tensor_quantize_and_bit_flip_deterministic_v2(weights_q, scaling_exponent, 1/60, quant_config[\"bits\"], quant_config[\"keep_negative\"]).numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2, 3), dtype=float32, numpy=\n",
       "array([[[-0.390625, -0.015625, -0.140625],\n",
       "        [-0.640625, -0.765625, -0.265625]],\n",
       "\n",
       "       [[-1.015625, -0.515625, -0.390625],\n",
       "        [-0.265625, -0.265625, -0.765625]]], dtype=float32)>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fk.utils.quantize_and_bitflip_deterministic_v2(weights_u, test_quantizer_kn1, [], [60/60], i_keep_negative=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'quant_config' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_19371/1259183298.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen_mask_tensor_deterministic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweights_q\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mquant_config\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"bits\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen_mask_tensor_random\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweights_q\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mquant_config\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"bits\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'quant_config' is not defined"
     ]
    }
   ],
   "source": [
    "print(fk.utils.gen_mask_tensor_deterministic(weights_q,60/60,quant_config[\"bits\"]))\n",
    "print(fk.utils.gen_mask_tensor_random(weights_q,60/60,quant_config[\"bits\"]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[7 7 7]\n",
      "  [7 7 7]]\n",
      "\n",
      " [[3 3 3]\n",
      "  [3 3 3]]], shape=(2, 2, 3), dtype=int64)\n",
      "tf.Tensor(\n",
      "[[[7 7 7]\n",
      "  [7 7 7]]\n",
      "\n",
      " [[3 3 3]\n",
      "  [3 3 3]]], shape=(2, 2, 3), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "def gen_mask_tensor_deterministic_v2(i_tensor, i_ber, i_qbits, i_keep_negative=0):\n",
    "    # S: Generate the mask array (default value is 0)\n",
    "    mask_array = np.full(i_tensor.shape, 0).flatten()\n",
    "\n",
    "    # S: Determine the number of bits in region\n",
    "    num_rbits = mask_array.size * i_qbits\n",
    "\n",
    "    # S: Determine the number of faults to inject\n",
    "    num_rfaults = int(num_rbits * i_ber)\n",
    "\n",
    "    # S: Inject faults\n",
    "    faults_injected = 0\n",
    "    while faults_injected < num_rfaults:\n",
    "        # print(mask_array)\n",
    "        mask_array[faults_injected % mask_array.size] = mask_array[\n",
    "            faults_injected % mask_array.size\n",
    "        ] + 2 ** (faults_injected // mask_array.size)\n",
    "        faults_injected = faults_injected + 1\n",
    "\n",
    "    sign_mask = 2**(i_qbits - i_keep_negative)\n",
    "    rval_mask = sign_mask -1\n",
    "    for i in range(mask_array.shape[0]):\n",
    "        mask_array[i] = (mask_array[i] & rval_mask) - (mask_array[i] & sign_mask)\n",
    "\n",
    "    return tf.convert_to_tensor(np.reshape(mask_array, i_tensor.shape), dtype=tf.int64)\n",
    "\n",
    "print(fk.utils.gen_mask_tensor_deterministic(weights_q,30/60,quant_config[\"bits\"]))\n",
    "print(         gen_mask_tensor_deterministic_v2(weights_q,30/60,quant_config[\"bits\"], 1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[1 0 0]\n",
      "  [0 0 0]]\n",
      "\n",
      " [[0 0 0]\n",
      "  [0 0 0]]], shape=(2, 2, 3), dtype=int64)\n",
      "tf.Tensor(\n",
      "[[[1 0 0]\n",
      "  [0 0 0]]\n",
      "\n",
      " [[0 0 0]\n",
      "  [0 0 0]]], shape=(2, 2, 3), dtype=int64)\n",
      "[8 0 0 0 0 0 0 0 0 0 0 0]\n",
      "8\n",
      "8\n",
      "0\n",
      "8\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "\n",
      "16\n",
      "[8 0 0 0 0 0 0 0 0 0 0 0]\n",
      "tf.Tensor(\n",
      "[[[8 0 0]\n",
      "  [0 0 0]]\n",
      "\n",
      " [[0 0 0]\n",
      "  [0 0 0]]], shape=(2, 2, 3), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "# Goal\n",
    "# I want to be able to flip a specific set of bits in a particular layer\n",
    "# In order to accomplish this, I need to be able to generate a mask tensor\n",
    "# that will mask the bits that I want to flip for that layer.\n",
    "\n",
    "def gen_mask_tensor_deterministic_v3(i_tensor, i_flbirs, i_qbits, i_keep_negative=0):\n",
    "    # S: Generate the mask array (default value is 0)\n",
    "    mask_array = np.full(i_tensor.shape, 0).flatten()\n",
    "\n",
    "    # S: Inject faults\n",
    "    for curr_flbir in i_flbirs:\n",
    "        #S: Get the start LBI and end LBI\n",
    "        s_lbi, e_lbi, rber = (curr_flbir.start_lbi, curr_flbir.end_lbi, curr_flbir.ber)\n",
    "\n",
    "        #S: Get the weight bit index representation of the LBI\n",
    "        ### TODO: This code assumes a region of a single bit (s_lbi == e_lbi)\n",
    "        ### Update this code to be more general.\n",
    "        s_wbi = fk.utils.lb_index_to_wb_index(s_lbi,i_qbits)\n",
    "        \n",
    "        #S: Flip the bit of indicated weight\n",
    "        mask_array[s_wbi[0]] = mask_array[s_wbi[0]] | (1 << s_wbi[1])\n",
    "\n",
    "\n",
    "    print(mask_array)\n",
    "\n",
    "    sign_mask = 2**(i_qbits - i_keep_negative)\n",
    "    rval_mask = sign_mask - 1\n",
    "    for i in range(mask_array.shape[0]):\n",
    "        print(mask_array[i])\n",
    "        print((mask_array[i] & rval_mask))\n",
    "        print((mask_array[i] & sign_mask))\n",
    "        mask_array[i] = (mask_array[i] & rval_mask) - (mask_array[i] & sign_mask)\n",
    "        print(mask_array[i])\n",
    "        print()\n",
    "    print(sign_mask)\n",
    "    print(mask_array)\n",
    "\n",
    "    return tf.convert_to_tensor(np.reshape(mask_array, i_tensor.shape), dtype=tf.int64)\n",
    "\n",
    "print(fk.utils.gen_mask_tensor_deterministic(weights_q,1/60,quant_config[\"bits\"]))\n",
    "print(         gen_mask_tensor_deterministic_v2(weights_q,1/60,quant_config[\"bits\"], 1))\n",
    "\n",
    "flbi_regions = [\n",
    "    fk.utils.FaultyLayerBitRegion(1, 0, 1.0),\n",
    "    # fk.utils.FaultyLayerBitRegion(1, 1, 1.0),\n",
    "    ]\n",
    "print(         gen_mask_tensor_deterministic_v3(weights_q,flbi_regions,quant_config[\"bits\"], 1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 4)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fk.utils.lb_index_to_wb_index(0,quant_config[\"bits\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "8 & 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00000\n",
      "00001\n",
      "00011\n",
      "00011\n",
      "01011\n",
      "11011\n"
     ]
    }
   ],
   "source": [
    "entry_in_mask_array = 0\n",
    "print(f\"{entry_in_mask_array:05b}\")\n",
    "entry_in_mask_array = entry_in_mask_array | 1 << 0\n",
    "print(f\"{entry_in_mask_array:05b}\")\n",
    "entry_in_mask_array = entry_in_mask_array | 1 << 1\n",
    "print(f\"{entry_in_mask_array:05b}\")\n",
    "entry_in_mask_array = entry_in_mask_array | 1 << 2\n",
    "print(f\"{entry_in_mask_array:05b}\")\n",
    "entry_in_mask_array = entry_in_mask_array | 1 << 3\n",
    "print(f\"{entry_in_mask_array:05b}\")\n",
    "entry_in_mask_array = entry_in_mask_array | 1 << 4\n",
    "print(f\"{entry_in_mask_array:05b}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01000\n",
      "10000\n"
     ]
    }
   ],
   "source": [
    "print(f\"{8:05b}\")\n",
    "print(f\"{16:05b}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fastml-science-ecoder-env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "bc4ffff1fad6309fa5ea941019981774ea7b1c2483487be824a2978c2501dfe6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}