A) RoadMap 1 - Torch Main 1 - Basic Tensor functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Roadmap 1 - Pytorch Main Module\n",
    "\n",
    "    1.1.  torch.is_tensor   - returns true if object is a tensor\n",
    "    1.2.  torch.tensor      - Constructs a tensor with data\n",
    "    1.3.  torch.from_numpy  - From numpy array to Tensor \n",
    "    \n",
    "    \n",
    "    2.1.  torch.zeros       - Returns a tensor filled with the scalar value 0\n",
    "    2.2.  torch.ones        - Returns a tensor filled with the scalar value 1\n",
    "    2.3.  torch.empty       - Returns an empty tensor.\n",
    "    \n",
    "    3.1.  torch.arange      - Returns a 1-D tensor of size ⌊end−start]/[step⌋ with values from the interval\n",
    "    3.2.  torch.linspace    - Returns a one-dimensional tensor of steps equally spaced points between start and end\n",
    "    3.3.  torch.logspace    - Returns a one-dimensional tensor of steps points logarithmically spaced \n",
    "    \n",
    "    4.1.  torch.eye         - Returns a 2-D tensor with ones on the diagonal and zeros elsewhere.\n",
    "    4.2.  torch.full        - Returns a tensor of size size filled with fill_value\n",
    "    \n",
    "    5.1.  torch.cat         - Concatenates the given sequence of seq tensors in the given dimension.\n",
    "    5.2.  torch.masked_select - Returns a new 1-D tensor which indexes the input tensor according to \n",
    "                                the binary mask mask which is a ByteTensor.\n",
    "    5.3.  torch.nonzero     - Returns a tensor containing the indices of all non-zero elements of input. \n",
    "    5.4.  torch.where       - Return a tensor of elements selected from either x or y, depending on condition\n",
    "    \n",
    "    6.1.  torch.reshape     - Returns a reshaped tensor\n",
    "    6.2.  torch.squeeze     - Returns a tensor with all the dimensions of input of size 1 removed.\n",
    "    6.3.  torch.take        - Returns a new tensor with the elements of input at the given indices.\n",
    "    6.4.  torch.t           - Expects input to be a matrix (2-D tensor) and transposes dimensions 0 and 1.\n",
    "    6.5.  torch.transpose   - Returns a tensor that is a transposed version of input.\n",
    "    6.6.  torch.unsqueeze   - Returns a new tensor with a dimension of size one inserted at the specified position.\n",
    "    \n",
    "    7.1.  torch.normal      - Returns a tensor of random numbers drawn from separate normal distributions \n",
    "    7.2.  torch.rand        - Returns a tensor filled with random numbers from a uniform distribution on the\n",
    "                                interval [0,1)\n",
    "    7.3.  torch.randint     - Returns a tensor filled with random integers generated uniformly between low \n",
    "                              (inclusive) and high (exclusive).\n",
    "    7.4.  torch.randn       - Returns a tensor filled with random numbers from a normal distribution with mean 0 \n",
    "                              and variance 1 (also called the standard normal distribution).\n",
    "    \n",
    "    8.1.  torch.save        - Saves an object to a disk file.\n",
    "    8.2.  tensor.load       - Loads an object saved with torch.save() from a file.\n",
    "    8.3.  torch.no_grad()   - Disable grad\n",
    "    8.4.  torch.enable_grad()  - Enable grad\n",
    "    8.5.  torch.is_nan      - Check if any nan value are there in tensor\n",
    "    8.6.  torch.device()    - Switch between cpu, gpu\n",
    "    8.7.  torch.autograd.profiler.profile - Context manager that manages autograd profiler state and holds a summary of results.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra Blog Resources\n",
    "\n",
    "1. https://www.kdnuggets.com/2018/05/wtf-tensor.html\n",
    "\n",
    "2. https://www.datacamp.com/community/tutorials/investigating-tensors-pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is x a tensor:  False\n"
     ]
    }
   ],
   "source": [
    "# Checking Tensor\n",
    "\n",
    "'''\n",
    "1.1 torch.is_tensor - returns true if object is a tensor\n",
    "'''\n",
    "\n",
    "x = np.matrix('1 2; 3 4')\n",
    "value = torch.is_tensor(x)\n",
    "print(\"Is x a tensor: \", value)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy to tensor. <class 'numpy.matrix'> <class 'torch.Tensor'> \n",
      "\n",
      "List to tensor. <class 'list'> <class 'torch.Tensor'> \n",
      "\n",
      "Scalar to tensor. <class 'float'> <class 'torch.Tensor'> \n",
      "\n",
      "Empty List to tensor. <class 'list'> <class 'torch.Tensor'> \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Torch Tensor\n",
    "\n",
    "'''\n",
    "1.2. torch.tensor - Constructs a tensor with data\n",
    "        - data (array_like) – Can be a list, tuple, NumPy ndarray, scalar, and other types.\n",
    "'''\n",
    "\n",
    "# Numpy to Tensor\n",
    "x = np.matrix('1 2; 3 4')\n",
    "x_tensor = torch.tensor(x)\n",
    "print(\"Numpy to tensor.\", type(x), type(x_tensor), \"\\n\")\n",
    "\n",
    "\n",
    "# List to Tensor\n",
    "x = [1, 2, 3, 4]\n",
    "x_tensor = torch.tensor(x)\n",
    "print(\"List to tensor.\", type(x), type(x_tensor), \"\\n\")\n",
    "\n",
    "\n",
    "# Scalar to Tensor\n",
    "x = 3.142\n",
    "x_tensor = torch.tensor(x)\n",
    "print(\"Scalar to tensor.\", type(x), type(x_tensor), \"\\n\")\n",
    "\n",
    "\n",
    "# Empty list to Tensor\n",
    "x = []\n",
    "x_tensor = torch.tensor(x)\n",
    "print(\"Empty List to tensor.\", type(x), type(x_tensor), \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type of x:  <class 'numpy.ndarray'>\n",
      "Type of x_t:  <class 'torch.Tensor'>\n",
      "Type of x_n:  <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "# From numpy array to Tensor \n",
    "# From Tensor to numpy array\n",
    "\n",
    "'''\n",
    "1.3 torch.from_numpy() - Numpy to Tensor\n",
    "\n",
    "    numpy() - Tensor to Numpy\n",
    "\n",
    "'''\n",
    "\n",
    "\n",
    "x = np.array([1, 2, 3])\n",
    "print(\"Type of x: \", type(x))\n",
    "x_t = torch.from_numpy(x)\n",
    "print(\"Type of x_t: \", type(x_t))\n",
    "x_n = x_t.numpy()\n",
    "print(\"Type of x_n: \", type(x_n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t1 =  tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.]])\n",
      "x_t2 =  tensor([0., 0., 0., 0., 0.])\n",
      "x_t2.shape =  torch.Size([4, 2, 3])\n"
     ]
    }
   ],
   "source": [
    "# Zeros\n",
    "\n",
    "'''\n",
    "2.1. torch.zeros - Returns a tensor filled with the scalar value 0, with the shape defined by the variable argument\n",
    "        - sizes - Mention the ND array size\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "#rows - 2, columns - 3\n",
    "x_t1 = torch.zeros(2, 3)\n",
    "print(\"x_t1 = \", x_t1)\n",
    "\n",
    "x_t2 = torch.zeros(5)\n",
    "print(\"x_t2 = \", x_t2)\n",
    "\n",
    "#depth - 4, rows - 2, columns - 3\n",
    "x_t2 = torch.zeros(4, 2, 3)\n",
    "print(\"x_t2.shape = \", x_t2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t1 =  tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]])\n",
      "x_t2 =  tensor([1., 1., 1., 1., 1.])\n"
     ]
    }
   ],
   "source": [
    "# Ones\n",
    "\n",
    "'''\n",
    "2.2. torch.ones - Returns a tensor filled with the scalar value 1, with the shape defined by the variable argument \n",
    "        - sizes - Mention the ND array size\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_t1 = torch.ones(2, 3)\n",
    "print(\"x_t1 = \", x_t1)\n",
    "\n",
    "x_t2 = torch.ones(5)\n",
    "print(\"x_t2 = \", x_t2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t1 =  tensor([[2.2076e+24, 4.5670e-41, 2.3426e-38],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00]])\n"
     ]
    }
   ],
   "source": [
    "# Empty\n",
    "\n",
    "'''\n",
    "2.3. torch.empty - Returns a empty tensor\n",
    "        - sizes - Mention the ND array size\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_t1 = torch.empty(2, 3)\n",
    "print(\"x_t1 = \", x_t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t1 =  tensor([2.0000, 2.5000, 3.0000, 3.5000, 4.0000, 4.5000, 5.0000, 5.5000, 6.0000,\n",
      "        6.5000, 7.0000, 7.5000, 8.0000, 8.5000, 9.0000, 9.5000])\n"
     ]
    }
   ],
   "source": [
    "# Arrange\n",
    "\n",
    "'''\n",
    "3.1. torch.arrange - Returns a 1-D tensor of size ⌊end−start]/[step⌋ with values from the interval \n",
    "    [start, end) taken with common difference step beginning from start.\n",
    "        - start (Number) – the starting value for the set of points. Default: 0.\n",
    "        - end (Number) – the ending value for the set of points\n",
    "        - step (Number) – the gap between each pair of adjacent points. Default: 1.\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_t1 = torch.arange(start=2, end=10, step=0.5)\n",
    "print(\"x_t1 = \", x_t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t1 =  tensor([-10.0000,  -7.7778,  -5.5556,  -3.3333,  -1.1111,   1.1111,   3.3333,\n",
      "          5.5556,   7.7778,  10.0000])\n"
     ]
    }
   ],
   "source": [
    "# Linespace\n",
    "\n",
    "'''\n",
    "3.2. torch.linspace - Returns a one-dimensional tensor of steps equally spaced points between start and end\n",
    "        - start (Number) – the starting value for the set of points. Default: 0.\n",
    "        - end (Number) – the ending value for the set of points\n",
    "        - step (Number) – the gap between each pair of adjacent points. Default: 1.\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_t1 = torch.linspace(start=-10, end=10, steps=10)\n",
    "print(\"x_t1 = \", x_t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t1 =  tensor([1.0000e-10, 1.0000e-05, 1.0000e+00, 1.0000e+05, 1.0000e+10])\n"
     ]
    }
   ],
   "source": [
    "# Logspace\n",
    "\n",
    "'''\n",
    "3.3. torch.logspace - Returns a one-dimensional tensor of steps points logarithmically spaced \n",
    "    between 10^start and 10^end\n",
    "        - start (Number) – the starting value for the set of points. Default: 0.\n",
    "        - end (Number) – the ending value for the set of points\n",
    "        - step (Number) – the gap between each pair of adjacent points. Default: 1.\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_t1 = torch.logspace(start=-10, end=10, steps=5)\n",
    "print(\"x_t1 = \", x_t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t1 =  tensor([[1., 0., 0., 0.],\n",
      "        [0., 1., 0., 0.],\n",
      "        [0., 0., 1., 0.]])\n"
     ]
    }
   ],
   "source": [
    "# Eye tensor\n",
    "\n",
    "'''\n",
    "4.1. torch.eye - Returns a 2-D tensor with ones on the diagonal and zeros elsewhere.\n",
    "        - n (int) – the number of rows\n",
    "        - m (int, optional) – the number of columns with default being n\n",
    "        - out (Tensor, optional) – the output tensor\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_t1 = torch.eye(m=4, n=3)\n",
    "print(\"x_t1 = \", x_t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t1 =  tensor([[3.1416, 3.1416, 3.1416],\n",
      "        [3.1416, 3.1416, 3.1416]])\n"
     ]
    }
   ],
   "source": [
    "# Full tensor\n",
    "\n",
    "'''\n",
    "4.2. torch.full - Returns a tensor of size size filled with fill_value\n",
    "        - size (int...) – a list, tuple, or torch.Size of integers defining the shape of the output tensor.\n",
    "        - fill_value – the number to fill the output tensor with.\n",
    "        - out (Tensor, optional) – the output tensor\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_t1 = torch.full((2, 3), 3.141592)\n",
    "print(\"x_t1 = \", x_t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t =  tensor([[1, 2],\n",
      "        [3, 4]])\n",
      "x_t_concat =  tensor([[1, 2],\n",
      "        [3, 4],\n",
      "        [1, 2],\n",
      "        [3, 4],\n",
      "        [1, 2],\n",
      "        [3, 4]])\n"
     ]
    }
   ],
   "source": [
    "# Cat\n",
    "\n",
    "'''\n",
    "5.1. torch.cat - Concatenates the given sequence of seq tensors in the given dimension. \n",
    "    All tensors must either have the same shape (except in the concatenating dimension) or be empty.\n",
    "        - seq (sequence of Tensors) \n",
    "        - dim (int, optional) – the dimension over which the tensors are concatenated\n",
    "        - out (Tensor, optional) – the output tensor\n",
    "'''\n",
    "\n",
    "x = np.matrix('1 2; 3 4')\n",
    "x_t = torch.tensor(x)\n",
    "x_t_concat = torch.cat((x_t, x_t, x_t), 0)\n",
    "\n",
    "print(\"x_t = \", x_t)\n",
    "print(\"x_t_concat = \", x_t_concat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x =  tensor([[ 0.6490,  0.5250,  1.3039,  0.3746],\n",
      "        [-0.5571, -0.4512,  0.7614,  0.5499],\n",
      "        [-0.2040,  0.1652,  0.0700, -1.1196]])\n",
      "mask =  tensor([[ True,  True,  True,  True],\n",
      "        [False, False,  True,  True],\n",
      "        [False,  True,  True, False]])\n",
      "masked_selection =  tensor([0.6490, 0.5250, 1.3039, 0.3746, 0.7614, 0.5499, 0.1652, 0.0700])\n"
     ]
    }
   ],
   "source": [
    "# Masked Tensor\n",
    "\n",
    "'''\n",
    "5.2. torch.masked_select - Returns a new 1-D tensor which indexes the input tensor according to \n",
    "    the binary mask mask which is a ByteTensor.\n",
    "        - input (Tensor) – the input data\n",
    "        - mask (ByteTensor) – the tensor containing the binary mask to index with\n",
    "        - out (Tensor, optional) – the output tensor   \n",
    "'''\n",
    "\n",
    "x = torch.randn(3, 4)\n",
    "mask = x.ge(0.0)\n",
    "\n",
    "print(\"x = \", x)\n",
    "print(\"mask = \", mask)\n",
    "\n",
    "masked_selection = torch.masked_select(x, mask)\n",
    "print(\"masked_selection = \", masked_selection)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t =  tensor([1, 1, 1, 0, 1])\n",
      "Non zero indices -  tensor([[0],\n",
      "        [1],\n",
      "        [2],\n",
      "        [4]])\n"
     ]
    }
   ],
   "source": [
    "# NonZero\n",
    "\n",
    "'''\n",
    "5.3. torch.nonzero - Returns a tensor containing the indices of all non-zero elements of input. \n",
    "    Each row in the result contains the indices of a non-zero element in input\n",
    "        - input (Tensor) – the input tensor\n",
    "        - out (LongTensor, optional) – the output tensor containing indices\n",
    "'''\n",
    "\n",
    "x_t = torch.tensor([1, 1, 1, 0, 1])\n",
    "x_nonzero = torch.nonzero(x_t)\n",
    "print(\"x_t = \", x_t)\n",
    "print(\"Non zero indices - \", x_nonzero)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x =  tensor([[-0.4996,  0.0618],\n",
      "        [-1.1040,  1.0288],\n",
      "        [-1.4798,  0.1900]])\n",
      "y =  tensor([[1., 1.],\n",
      "        [1., 1.],\n",
      "        [1., 1.]])\n",
      "z =  tensor([[1.0000, 0.0618],\n",
      "        [1.0000, 1.0288],\n",
      "        [1.0000, 0.1900]])\n"
     ]
    }
   ],
   "source": [
    "# Conditioned output\n",
    "\n",
    "'''\n",
    "5.4. torch.where - Return a tensor of elements selected from either x or y, depending on condition\n",
    "        - condition (ByteTensor) – When True (nonzero), yield x, otherwise yield y\n",
    "        - x (Tensor) – values selected at indices where condition is True\n",
    "        - y (Tensor) – values selected at indices where condition is False\n",
    "'''\n",
    "\n",
    "x = torch.randn(3, 2)\n",
    "y = torch.ones(3, 2)\n",
    "\n",
    "z = torch.where(x > 0, x, y)\n",
    "\n",
    "print(\"x = \", x)\n",
    "print(\"y = \", y)\n",
    "print(\"z = \", z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t =  tensor([0, 1, 2, 3, 4, 5])\n",
      "x_t_reshaped =  tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n"
     ]
    }
   ],
   "source": [
    "# Reshape\n",
    "\n",
    "'''\n",
    "6.1. torch.reshape - Returns a tensor with the same data and number of elements as input, \n",
    "                but with the specified shape.\n",
    "        - input (Tensor) – the tensor to be reshaped\n",
    "        - shape (tuple of python:ints) – the new shape\n",
    "'''\n",
    "\n",
    "x_t = torch.arange(6)\n",
    "x_t_reshaped = torch.reshape(x_t, (2,3))\n",
    "\n",
    "print(\"x_t = \", x_t)\n",
    "print(\"x_t_reshaped = \", x_t_reshaped)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial shape -  torch.Size([2, 1, 2, 1, 2])\n",
      "Final shape -  torch.Size([3, 2])\n"
     ]
    }
   ],
   "source": [
    "# Squeeze\n",
    "\n",
    "'''\n",
    "6.2. torch.squeeze - Returns a tensor with all the dimensions of input of size 1 removed.\n",
    "        - For example, if input is of shape: (A×1×B×C×1×D) then the out tensor will be of shape: (A×B×C×D)\n",
    "        - input (Tensor) – the input tensor\n",
    "        - dim (int, optional) – if given, the input will be squeezed only in this dimension\n",
    "        - out (Tensor, optional) – the output tensor\n",
    "'''\n",
    "\n",
    "x_t = torch.zeros(2, 1, 2, 1, 2)\n",
    "print(\"Initial shape - \", x_t.size())\n",
    "\n",
    "x_t_squeezed = torch.squeeze(x_t)\n",
    "print(\"Final shape - \", x_t_squeezed.size())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t =  tensor([[-0.4493, -0.6883,  0.3534],\n",
      "        [-0.1650,  1.6260, -1.1655]])\n",
      "x_t_transposed =  tensor([[-0.4996, -1.1040, -1.4798],\n",
      "        [ 0.0618,  1.0288,  0.1900]])\n"
     ]
    }
   ],
   "source": [
    "# 2D Transpose\n",
    "\n",
    "'''\n",
    "6.3. torch.t - Expects input to be a matrix (2-D tensor) and transposes dimensions 0 and 1.\n",
    "        - input (Tensor) – the input tensor\n",
    "'''\n",
    "\n",
    "x_t = torch.randn(2, 3)\n",
    "x_t_transposed = torch.t(x)\n",
    "\n",
    "print(\"x_t = \", x_t)\n",
    "print(\"x_t_transposed = \", x_t_transposed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracted element =  tensor([4, 5, 8])\n"
     ]
    }
   ],
   "source": [
    "# Take\n",
    "\n",
    "'''\n",
    "6.4. torch.take - Returns a new tensor with the elements of input at the given indices. \n",
    "    The input tensor is treated as if it were viewed as a 1-D tensor. \n",
    "    The result takes the same shape as the indices.\n",
    "        - input (Tensor) – the input tensor\n",
    "        - indices (LongTensor) – the indices into tensor\n",
    "'''\n",
    "\n",
    "src = torch.tensor([4, 3, 5, 6, 7, 8])\n",
    "taken_indexes = torch.take(src, torch.tensor([0, 2, 5]))\n",
    "\n",
    "print(\"Extracted element = \", taken_indexes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_t =  tensor([[[ 0.4075, -1.1647],\n",
      "         [ 1.0240,  0.4893]],\n",
      "\n",
      "        [[ 1.1354,  0.0303],\n",
      "         [ 0.6570,  0.4725]],\n",
      "\n",
      "        [[ 0.9163,  0.4061],\n",
      "         [-1.4334,  1.2190]]])\n"
     ]
    }
   ],
   "source": [
    "# N-D Transpose\n",
    "\n",
    "'''\n",
    "6.5. torch.transpose - Returns a tensor that is a transposed version of input. \n",
    "    The given dimensions dim0 and dim1 are swapped.\n",
    "        - input (Tensor) – the input tensor\n",
    "        - dim0 (int) – the first dimension to be transposed\n",
    "        - dim1 (int) – the second dimension to be transposed\n",
    "'''\n",
    "\n",
    "x = torch.randn(2, 3, 2)\n",
    "x_t = torch.transpose(x, 0, 1)\n",
    "\n",
    "print(\"x_t = \", x_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial size:  torch.Size([4])\n",
      "Final size:  torch.Size([1, 4])\n"
     ]
    }
   ],
   "source": [
    "# Unsqueeze\n",
    "\n",
    "'''\n",
    "6.6. torch.unsqueeze - Returns a new tensor with a dimension of size one inserted at the specified position.\n",
    "        - input (Tensor) – the input tensor\n",
    "        - dim (int) – the index at which to insert the singleton dimension\n",
    "        - out (Tensor, optional) – the output tensor\n",
    "'''\n",
    "\n",
    "x_t = torch.tensor([1, 2, 3, 4])\n",
    "x_t_unsqueezed = torch.unsqueeze(x_t, 0)\n",
    "\n",
    "print(\"Initial size: \", x_t.size())\n",
    "print(\"Final size: \", x_t_unsqueezed.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal_random_tensor =  tensor([ 2.8009,  2.0180,  4.0506,  3.0206,  3.9917,  5.8660,  7.3965,  7.7849,\n",
      "         9.2433, 10.0327])\n"
     ]
    }
   ],
   "source": [
    "# Normal random number\n",
    "\n",
    "'''\n",
    "7.1. torch.normal - Returns a tensor of random numbers drawn from separate normal distributions \n",
    "        whose mean and standard deviation are given.\n",
    "            - mean (Tensor) – the tensor of per-element means\n",
    "            - std (Tensor) – the tensor of per-element standard deviations\n",
    "            - out (Tensor, optional) – the output tensor\n",
    "\n",
    "'''\n",
    "\n",
    "normal_random_tensor = torch.normal(mean=torch.arange(1., 11.), std=torch.arange(1, 0, -0.1))\n",
    "print(\"normal_random_tensor = \", normal_random_tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_rand =  tensor([[0.8648, 0.1956, 0.6620],\n",
      "        [0.1343, 0.4898, 0.4056]])\n"
     ]
    }
   ],
   "source": [
    "# Random num\n",
    "\n",
    "'''\n",
    "7.2. torch.rand - Returns a tensor filled with random numbers from a uniform distribution on the interval [0,1)\n",
    "        - sizes (int...) – a sequence of integers defining the shape of the output tensor. \n",
    "            Can be a variable number of arguments or a collection like a list or tuple.\n",
    "'''\n",
    "\n",
    "x_rand = torch.rand(2, 3)\n",
    "print(\"x_rand = \", x_rand)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_randint =  tensor([[ -8,   0],\n",
      "        [  8, -10]])\n"
     ]
    }
   ],
   "source": [
    "# Random Integer\n",
    "\n",
    "'''\n",
    "7.3. torch.randint - Returns a tensor filled with random integers generated uniformly between low \n",
    "    (inclusive) and high (exclusive).\n",
    "        - low (int, optional) – Lowest integer to be drawn from the distribution. Default: 0.\n",
    "        - high (int) – One above the highest integer to be drawn from the distribution.\n",
    "        - size (tuple) – a tuple defining the shape of the output tensor.\n",
    "        - out (Tensor, optional) – the output tensor\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_randint = torch.randint(low=-10, high=10, size=(2,2))\n",
    "print(\"x_randint = \", x_randint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_randn =  tensor([[-1.5862, -0.6674, -1.2721],\n",
      "        [ 0.5390,  0.9478, -1.0593]])\n"
     ]
    }
   ],
   "source": [
    "# Random normal\n",
    "\n",
    "'''\n",
    "7.4. torch.randn - Returns a tensor filled with random numbers from a normal distribution with \n",
    "    mean 0 and variance 1 (also called the standard normal distribution).\n",
    "        - sizes (int...) – a sequence of integers defining the shape of the output tensor. Can be a variable number of arguments or a collection like a list or tuple.\n",
    "        - out (Tensor, optional) – the output tensor\n",
    "        - dtype - Data type.\n",
    "'''\n",
    "\n",
    "x_randn = torch.randn(2, 3)\n",
    "print(\"x_randn = \", x_randn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving a tensor\n",
    "\n",
    "'''\n",
    "8.1. torch.save - Saves an object to a disk file.\n",
    "        - obj – saved object\n",
    "        - f – a file-like object (has to implement write and flush) or a string containing a file name\n",
    "        - pickle_module – module used for pickling metadata and objects\n",
    "        - pickle_protocol – can be specified to override the default protocol\n",
    "\n",
    "'''\n",
    "\n",
    "x = torch.tensor([0, 1, 2, 3, 4])\n",
    "torch.save(x, 'tensor.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'torch.Tensor'>\n",
      "tensor([0, 1, 2, 3, 4])\n"
     ]
    }
   ],
   "source": [
    "# Loading a saved tensor\n",
    "\n",
    "'''\n",
    "8.2. tensor.load - Loads an object saved with torch.save() from a file.\n",
    "'''\n",
    "\n",
    "x = torch.load('tensor.pt')\n",
    "print(type(x))\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x requires grad =  True\n",
      "y requires grad =  True\n",
      "z requires grad =  False\n"
     ]
    }
   ],
   "source": [
    "# Adding and disabling gradient\n",
    "\n",
    "'''\n",
    "8.3. torch.no_grad()      - Disable grad\n",
    "    torch.enable_grad()  - Enable grad\n",
    "'''\n",
    "\n",
    "x = torch.zeros(1, requires_grad=True)\n",
    "print(\"x requires grad = \", x.requires_grad)\n",
    "\n",
    "y = x*3\n",
    "print(\"y requires grad = \", y.requires_grad)\n",
    "\n",
    "with torch.no_grad():\n",
    "    z = x * 2\n",
    "print(\"z requires grad = \", z.requires_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_in =  tensor([1., nan, 2.])\n",
      "x_out =  tensor([False,  True, False])\n"
     ]
    }
   ],
   "source": [
    "# Checking if NAN value is there in tensor\n",
    "\n",
    "'''\n",
    "8.4. torch.is_nan - Returns a new tensor with boolean elements representing if each element is NaN or not.\n",
    "'''\n",
    "\n",
    "x_in = torch.tensor([1, float('nan'), 2])\n",
    "x_out = torch.isnan(x_in)\n",
    "\n",
    "print(\"x_in = \", x_in)\n",
    "print(\"x_out = \", x_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set computation device\n",
    "\n",
    "'''\n",
    "8.5. torch.device\n",
    "\n",
    "\n",
    "Ex\n",
    ">>> torch.device('cuda:0')\n",
    "device(type='cuda', index=0)\n",
    "\n",
    ">>> torch.device('cpu')\n",
    "device(type='cpu')\n",
    "\n",
    ">>> torch.device('cuda')  # current cuda device\n",
    "device(type='cuda')\n",
    "'''\n",
    "\n",
    "torch.device('cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  \n",
      "Name                                 Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  Input Shapes                         \n",
      "-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  \n",
      "pow                                  23.09%           99.058us         23.09%           99.058us         99.058us         NaN              0.000us          0.000us          1                []                                   \n",
      "torch::autograd::GraphRoot           1.06%            4.551us          1.06%            4.551us          4.551us          NaN              0.000us          0.000us          1                []                                   \n",
      "PowBackward0                         41.67%           178.772us        41.67%           178.772us        178.772us        NaN              0.000us          0.000us          1                []                                   \n",
      "pow                                  17.16%           73.595us         17.16%           73.595us         73.595us         NaN              0.000us          0.000us          1                []                                   \n",
      "mul                                  11.73%           50.299us         11.73%           50.299us         50.299us         NaN              0.000us          0.000us          1                []                                   \n",
      "mul                                  2.53%            10.866us         2.53%            10.866us         10.866us         NaN              0.000us          0.000us          1                []                                   \n",
      "torch::autograd::AccumulateGrad      2.76%            11.830us         2.76%            11.830us         11.830us         NaN              0.000us          0.000us          1                []                                   \n",
      "-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  \n",
      "Self CPU time total: 428.971us\n",
      "CUDA time total: 0.000us\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Profiling - 1\n",
    "\n",
    "'''\n",
    "8.6. torch.autograd.profiler.profile - Context manager that manages autograd profiler state and holds a summary of results.\n",
    "        - enabled (bool, optional) – Setting this to False makes this context manager a no-op. Default: True.\n",
    "        - use_cuda (bool, optional) – Enables timing of CUDA events as well using the cudaEvent API. \n",
    "            Adds approximately 4us of overhead to each tensor operation. \n",
    "'''\n",
    "\n",
    "x = torch.randn((1, 1), requires_grad=True)\n",
    "with torch.autograd.profiler.profile() as prof:\n",
    "    y = x ** 2\n",
    "    y.backward()\n",
    "\n",
    "print(prof)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Author - Tessellate Imaging - https://www.tessellateimaging.com/\n",
    "\n",
    "## Monk Library - https://github.com/Tessellate-Imaging/monk_v1\n",
    "\n",
    "    Monk is an opensource low-code tool for computer vision and deep learning\n",
    "\n",
    "### Monk features\n",
    "- low-code\n",
    "- unified wrapper over major deep learning framework - keras, pytorch, gluoncv\n",
    "- syntax invariant wrapper\n",
    "\n",
    "\n",
    "### Enables\n",
    "- to create, manage and version control deep learning experiments\n",
    "- to compare experiments across training metrics\n",
    "- to quickly find best hyper-parameters\n",
    "\n",
    "\n",
    "### At present it only supports transfer learning, but we are working each day to incorporate\n",
    "- GUI based custom model creation\n",
    "- various object detection and segmentation algorithms\n",
    "- deployment pipelines to cloud and local platforms\n",
    "- acceleration libraries such as TensorRT\n",
    "- preprocessing and post processing libraries\n",
    "\n",
    "## To contribute to Monk AI or Pytorch RoadMap repository raise an issue in the git-repo or dm us on linkedin \n",
    " - Abhishek - https://www.linkedin.com/in/abhishek-kumar-annamraju/\n",
    " - Akash - https://www.linkedin.com/in/akashdeepsingh01/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}