Temas/Módulo 04 - Introducción a las redes neuronales/Cuadernos/Activation_Functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<img src=\"../Imagenes/logo-final-ap.png\"  width=\"80\" height=\"80\" align=\"left\"/> \n",
    "</figure>\n",
    "\n",
    "# <span style=\"color:blue\"><left>Aprendizaje Profundo</left></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:red\"><center>Funciones de Activación</center></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Autores</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Alvaro Mauricio Montenegro Díaz, ammontenegrod@unal.edu.co\n",
    "2. Daniel Mauricio Montenegro Reyes, dextronomo@gmail.com "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Asesora Medios y Marketing digital</span>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Maria del Pilar Montenegro, pmontenegro88@gmail.com "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Asistentes</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Oleg Jarma, ojarmam@unal.edu.co \n",
    "6. Laura Lizarazo, ljlizarazore@unal.edu.co "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Referencias</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "1. https://theclevermachine.wordpress.com/2014/09/6/\n",
    "1. https://en.wikipedia.org/wiki/Activation_function\n",
    "1. https://www.geeksforgeeks.org/activation-functions-neural-networks/\n",
    "1. https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0\n",
    "1. https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Contenido</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Introducción](#Introducción)\n",
    "* [Funciones de activación clásicas](#Funciones-de-activación-clásicas)\n",
    "* [Estado del arte. Variaciones de la función de activación de ReLu](#Estado-del-arte.-Variaciones-de-la-función-de-activación-de-ReLu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Introducción</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Como vimos anteriormente, la red neuronal tiene neuronas que trabajan en consonancia con sus pesos, su sesgo y su respectiva función de activación. En el modo de entrenamiento, se actualizan los pesos y sesgos de las neuronas sobre la base del error en la salida. Este proceso se conoce como `retropropagación` (backpropagation). Las funciones de activación permiten la retropropagación y los gradientes se suministran junto con el error para actualizar los pesos y sesgos.\n",
    "\n",
    "La función de activación de un nodo define la salida de ese nodo dada una entrada o un conjunto de entradas. La función de activación decide si una neurona debe activarse o no calculando la suma ponderada y agregando más sesgo con ella. \n",
    "\n",
    "**El propósito de la función de activación es introducir no linealidad en la salida de una neurona**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ¿Porqué necesitamos funciones de activación? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Una red neuronal sin una función de activación es esencialmente un modelo de regresión lineal clásico. La función de activación realiza la transformación no lineal de la entrada, lo que la hace capaz de aprender y realizar tareas más complejas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Funciones de activación clásicas</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En la entrada, una neurona artificial calcula una \"suma ponderada\" de su entrada y agrega un sesgo. Como antes, sea $ z = \\sum_i x_iw_i + b $, $ z  \\in \\mathcal {R} $. Una vez calculado el valor $ z $, la neurona artificial decide si se debe \"disparar\" o no, basándose en el valor de $ z $."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Función Step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lo primero que nos viene a la mente es ¿qué tal una función de activación basada en umbrales? Si el valor de $ z $ está por encima de cierto valor, declare que se ha `activado`. Si es menor que el umbral, diga que no.\n",
    "El umbral más común es 0. La *función de activación Step* se define como\n",
    "\n",
    "$$\n",
    "f(x) = Step(x) \\begin{cases}\n",
    "1, &\\text{ if } x \\ge 0,\\\\\n",
    "0, &\\text{ en otro caso }\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "Observe la figura 1. Obviamente, $f'(x) = 0$ for $x\\ne 0$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAASNElEQVR4nO3df4wcZ33H8fc3/pGYYs4BX4KxXWxUB9m1KgVOUShtSJsAlwjZpQJqS4hfES5t05ZCqwalSqPwTwG1EVUDwYWIgoAk0EIsZGJcGhSpakIuwQlxjJMjCeRs1zmCc6TKxfHF3/6xY2tZ7/nm7Nk7+8n7Ja0888yzM997dvzZudnZm8hMJEmnvzNmuwBJUjMMdEkqhIEuSYUw0CWpEAa6JBVi7mxtePHixblixYrZ2rwknZbuvffen2dmf7dlsxboK1asYGhoaLY2L0mnpYj46WTLPOUiSYUw0CWpEAa6JBXCQJekQhjoklSIKa9yiYibgLcBT2bm2i7LA/g0cDnwLPC+zLyv6UIl6XS3a98Ytz+4nz1Pj7N00QIG157L6iV9ja2/zhH6F4HB4yy/DFhVPTYBnz35siSpLLv2jbH5zscYGz/Ekr6zGBs/xOY7H2PXvrHGtjHlEXpm3hkRK47TZT3wpWz9Hd67ImJRRCzJzH0N1SjNmK/e/TNu27FntstQgUYOPMvEC4eZO+cM+heeyZvOOweA2x/c39hRehPn0JcCT7TNj1Rtx4iITRExFBFDo6OjDWxaatZtO/bw0L5fznYZKtDBicPMOSN+pW3hWXPZ8/R4Y9to4pui0aWt610zMnMzsBlgYGDAO2volLRmycu45Y/fMNtlqDDXb3+YsfFD9C2Yd7TtmecmWLpoQWPbaOIIfQRY3ja/DNjbwHolqRiDa89lbPwQY+OHOJx5dHpw7bmNbaOJQN8CvCdaLgTGPH8uSb9q9ZI+Nl20kr4F89g39hx9C+ax6aKVjV7lUueyxa8BFwOLI2IE+HtgHkBm3ghspXXJ4jCtyxbf31h1klSQ1Uv6Gg3wTnWuctk4xfIE/qyxiiRJJ8RvikpSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVIhagR4RgxGxOyKGI+KqLst/PSLuiIgfRsQDEXF586VKko5nykCPiDnADcBlwBpgY0Ss6ej2d8CtmXk+sAH4TNOFSpKOr84R+gXAcGY+mpnPAzcD6zv6JPCyaroP2NtciZKkOuoE+lLgibb5kaqt3bXAuyNiBNgK/Hm3FUXEpogYioih0dHREyhXkjSZOoEeXdqyY34j8MXMXAZcDnw5Io5Zd2ZuzsyBzBzo7++ffrWSpEnVCfQRYHnb/DKOPaVyBXArQGb+D3AWsLiJAiVJ9dQJ9HuAVRGxMiLm0/rQc0tHn58BlwBExGpage45FUmaQVMGemZOAFcC24BdtK5m2RkR10XEuqrbR4EPRsT9wNeA92Vm52kZSVIPza3TKTO30vqws73tmrbph4A3NluaJGk6/KaoJBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKkStQI+IwYjYHRHDEXHVJH3eFREPRcTOiPhqs2VKkqYyd6oOETEHuAF4MzAC3BMRWzLzobY+q4CPAW/MzAMRcU6vCpYkdVfnCP0CYDgzH83M54GbgfUdfT4I3JCZBwAy88lmy5QkTaVOoC8FnmibH6na2p0HnBcR/x0Rd0XEYLcVRcSmiBiKiKHR0dETq1iS1FWdQI8ubdkxPxdYBVwMbAQ+HxGLjnlS5ubMHMjMgf7+/unWKkk6jjqBPgIsb5tfBuzt0ue2zDyUmY8Bu2kFvCRphtQJ9HuAVRGxMiLmAxuALR19vgX8HkBELKZ1CubRJguVJB3flIGemRPAlcA2YBdwa2bujIjrImJd1W0b8FREPATcAfxNZj7Vq6IlScea8rJFgMzcCmztaLumbTqBj1QPSdIs8JuiklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVolagR8RgROyOiOGIuOo4/d4RERkRA82VKEmqY8pAj4g5wA3AZcAaYGNErOnSbyHwF8DdTRcpSZpanSP0C4DhzHw0M58HbgbWd+n3ceCTwHMN1idJqqlOoC8FnmibH6najoqI84Hlmfnt460oIjZFxFBEDI2Ojk67WEnS5OoEenRpy6MLI84Argc+OtWKMnNzZg5k5kB/f3/9KiVJU6oT6CPA8rb5ZcDetvmFwFrg+xHxOHAhsMUPRiVpZtUJ9HuAVRGxMiLmAxuALUcWZuZYZi7OzBWZuQK4C1iXmUM9qViS1NWUgZ6ZE8CVwDZgF3BrZu6MiOsiYl2vC5Qk1TO3TqfM3Aps7Wi7ZpK+F598WZKk6fKbopJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQtQI9IgYjYndEDEfEVV2WfyQiHoqIByLiexHx6uZLlSQdz5SBHhFzgBuAy4A1wMaIWNPR7YfAQGb+FvAN4JNNFypJOr46R+gXAMOZ+WhmPg/cDKxv75CZd2Tms9XsXcCyZsuUJE2lTqAvBZ5omx+p2iZzBfCdbgsiYlNEDEXE0OjoaP0qJUlTqhPo0aUtu3aMeDcwAHyq2/LM3JyZA5k50N/fX79KSdKU5tboMwIsb5tfBuzt7BQRlwJXA2/KzIPNlCdJqqvOEfo9wKqIWBkR84ENwJb2DhFxPvA5YF1mPtl8mZKkqUwZ6Jk5AVwJbAN2Abdm5s6IuC4i1lXdPgW8FPh6ROyIiC2TrE6S1CN1TrmQmVuBrR1t17RNX9pwXZKkafKbopJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFWJunU4RMQh8GpgDfD4z/6Fj+ZnAl4DXA08Bf5SZjzdbKuzaN8btD+5nz9PjLF20gMG157J6SV/Tm9GL1K59Y4wceJaDE4e5fvvD7l867Ux5hB4Rc4AbgMuANcDGiFjT0e0K4EBm/gZwPfCJpgvdtW+MzXc+xtj4IZb0ncXY+CE23/kYu/aNNb0pvQgd2b8mXjjM/Dnh/qXTUp0j9AuA4cx8FCAibgbWAw+19VkPXFtNfwP4l4iIzMymCr39wf30LZjHjicOMPrMQQAmXjjMDx57imVnv6SpzehFauTAs0y8cJgDzx6if+GZ9C2YB7T2O4/Sdbqocw59KfBE2/xI1da1T2ZOAGPAKzpXFBGbImIoIoZGR0enVeiep8dZeNavvv/MOSM4OHF4WuuRujk4cZg5ZwT9C8/kta98GQALz5rLnqfHZ7kyqb46R+jRpa3zyLtOHzJzM7AZYGBgYFpH70sXLWBs/BBvOu+co21j44foWzCPv3rzedNZlXSM67c/fHR/OuKZ5yZYumjBLFYlTU+dI/QRYHnb/DJg72R9ImIu0Af8ookCjxhcey5j44cYGz/E4cyj04Nrz21yM3qRcv9SCeoE+j3AqohYGRHzgQ3Alo4+W4D3VtPvAP6ryfPnAKuX9LHpopX0LZjHvrHn6Fswj00XrfT8phrh/qUSTHnKJTMnIuJKYButyxZvysydEXEdMJSZW4AvAF+OiGFaR+YbelHs6iV9/gdTz7h/6XRX6zr0zNwKbO1ou6Zt+jngnc2WJkmaDr8pKkmFMNAlqRAGuiQVwkCXpEJEw1cX1t9wxCjw0xN8+mLg5w2W0xTrmh7rmr5TtTbrmp6TqevVmdnfbcGsBfrJiIihzByY7To6Wdf0WNf0naq1Wdf09KouT7lIUiEMdEkqxOka6Jtnu4BJWNf0WNf0naq1Wdf09KSu0/IcuiTpWKfrEbokqYOBLkmFOGUDPSLeGRE7I+JwRAx0LPtYRAxHxO6IeOskz18ZEXdHxCMRcUv1p3+brvGWiNhRPR6PiB2T9Hs8In5U9Rtquo4u27s2Iva01Xb5JP0GqzEcjoirZqCuT0XEjyPigYj4ZkQsmqTfjIzXVD9/RJxZvcbD1b60ole1tG1zeUTcERG7qv3/L7v0uTgixtpe32u6rasHtR33dYmWf67G64GIeN0M1PTatnHYERG/jIgPd/SZsfGKiJsi4smIeLCt7eURsb3Kou0RcfYkz31v1eeRiHhvtz5TysxT8gGsBl4LfB8YaGtfA9wPnAmsBH4CzOny/FuBDdX0jcCf9LjefwSumWTZ48DiGRy7a4G/nqLPnGrsXgPMr8Z0TY/regswt5r+BPCJ2RqvOj8/8KfAjdX0BuCWGXjtlgCvq6YXAg93qeti4NsztT/VfV2Ay4Hv0LqD2YXA3TNc3xzgf2l98WZWxgu4CHgd8GBb2yeBq6rpq7rt98DLgUerf8+ups+e7vZP2SP0zNyVmbu7LFoP3JyZBzPzMWCY1o2sj4qIAH6f1g2rAf4N+INe1Vpt713A13q1jR44evPvzHweOHLz757JzO9m656zAHfRuvvVbKnz86+nte9Aa1+6pHqteyYz92XmfdX0M8Aujr2H76lqPfClbLkLWBQRS2Zw+5cAP8nME/0G+knLzDs59m5t7fvRZFn0VmB7Zv4iMw8A24HB6W7/lA3046hz0+pXAE+3hUe3Pk36XWB/Zj4yyfIEvhsR90bEph7W0e7K6tfemyb5Fa/OOPbSB2gdzXUzE+PV2M3Pe6U6xXM+cHeXxW+IiPsj4jsR8ZszVNJUr8ts71MbmPygajbG64hzM3MftN6wgXO69Glk7Grd4KJXIuI/gVd2WXR1Zt422dO6tJ3QTavrqFnjRo5/dP7GzNwbEecA2yPix9U7+Qk7Xl3AZ4GP0/qZP07rdNAHOlfR5bknfQ1rnfGKiKuBCeArk6ym8fHqVmqXtp7tR9MVES8F/h34cGb+smPxfbROK/xf9fnIt4BVM1DWVK/LbI7XfGAd8LEui2drvKajkbGb1UDPzEtP4Gl1blr9c1q/7s2tjqy69WmkxmjdFPsPgdcfZx17q3+fjIhv0vp1/6QCqu7YRcS/At/usqjOODZeV/Vhz9uAS7I6edhlHY2PVxfTufn5SPTo5ufdRMQ8WmH+lcz8j87l7QGfmVsj4jMRsTgze/pHqGq8Lj3Zp2q6DLgvM/d3Lpit8WqzPyKWZOa+6hTUk136jNA613/EMlqfH07L6XjKZQuwoboCYSWtd9oftHeoguIOWjeshtYNrCc74j9ZlwI/zsyRbgsj4tciYuGRaVofDD7YrW9TOs5bvn2S7dW5+XfTdQ0Cfwusy8xnJ+kzU+N1Stz8vFN1jv4LwK7M/KdJ+rzyyLn8iLiA1v/jp3pcV53XZQvwnupqlwuBsSOnGmbApL8lz8Z4dWjfjybLom3AWyLi7OoU6VuqtumZiU9+T/DT4rfTetc6COwHtrUtu5rWFQq7gcva2rcCr6qmX0Mr6IeBrwNn9qjOLwIf6mh7FbC1rY77q8dOWqceej12XwZ+BDxQ7UxLOuuq5i+ndRXFT2aormFa5wl3VI8bO+uayfHq9vMD19F6wwE4q9p3hqt96TUzMEa/Q+tX7Qfaxuly4ENH9jPgymps7qf14fJvz0BdXV+XjroCuKEazx/RdnVaj2t7Ca2A7mtrm5XxovWmsg84VOXXFbQ+d/ke8Ej178urvgPA59ue+4FqXxsG3n8i2/er/5JUiNPxlIskqQsDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXi/wH25fJi0VZ8pAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = (-10,0, 10)\n",
    "y = (0,0,1)\n",
    "\n",
    "plt.step(x, y , where='pre')\n",
    "plt.plot(x, y , 'C0o', alpha=0.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figura 1.** * Función de activación Step.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activación Lineal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La función de activación lineal está definida por $ f (x) = ax $. Tenga en cuenta que $ f '(x) = a $, para todos $ x \\in \\mathcal {R} $. Tenga en cuenta que $ - \\infty \\le f (x) \\le \\infty $.\n",
    "\n",
    "La función de activación lineal se utiliza en un solo lugar, es decir, la capa de salida. El efecto de esta función de activación es simplemente escalar los valores provenientes de la última capa oculta, para que se ajusten a los valores objetivo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dfbzUc/7/8cdLZW0KKY6lyGJtFyjnKBcr9au1tK7W2t2sdRkHsS42i3X1pWXlMiw2F7nKViSJtFLpIJJKtbrQloqOqKTUoajO6/fH+xPTmFOn08x8Zs4877fb3M7MfD7z+bzeZ2Y+r3l/3u/P+23ujoiIyJbaJu4AREQkPymBiIhIjSiBiIhIjSiBiIhIjSiBiIhIjSiBiIhIjSiBFBAzu9HMnk7TtorM7A0zW2Vmd1XzNQvMrEua9u9mtm90v6+ZXZ+O7eYKM9vTzCrMrE4Gtn2amb2a7u1G277QzBZHsTfOxD6q2O81ZvZotvYngRJIjMysm5lNMLOvzGxJdL+HmVncsVVDKfA5sIO790xeaGZPmNnN2QjE3S9w979nY1+Zkpxc3f1jd2/g7uu3crvNo2RbN2Hb/3b3o7dmu1Xsqx5wN3B0FPuydO8j2k9HMytPfM7d/+Hu52Zif1vLzA41s1Fm9oWZLTWzwWb2k7jjSgclkJiYWU/gXuAOYDegCLgAOALYtorXpP3X6FbYC5jpBXolqgX6/mysCNgOmBF3IDmmEfAw0JzwvVkFPB5nQGnj7rpl+QbsCHwF/HYz6z0B/AsYEa3fBfg1MAVYCSwEbkxYvznghNrBIuBToGfC8huBZ4GnCB/iGUDJJvZ/ODAR+DL6e3hCXGuBb4EKoEvS60qTlr8UPb8AuAL4b7TNZ4DtEl53HDAVWAG8DRy4idgc2Dchnpuj+x2BcqAnsCT6H5yd8LofAXcCHwOLgb7Aj6NljYDhwFJgeXS/acJry4BbgLeA1Rv2nxTX1cCH0f93JvCbpOXnAbMSlh8M9Acqo21WAFcmvJd1gW7ApKTtXA68GN3f1Gfi42g7FdHtMOAsYNzm3ueEMv89KvMq4FWgSYpy/4zwGd2wr9cSy5C0vXOj+2cB46L3YzkwHzg2Yd2dCQfaRdHyF4Dto/9TZUKZdid8tp9OeO0JhM/3imifLRKWLWATn8Okz8oXwAEJz+0a7X+Xrfj+Hwysiu5vS/jM/zl6XCf6X98Q93GqWmWJO4BCvAHHAOsSv1hVrPdE9AE/glBb3I5wgDwgenwg4SB4UrT+hi/swOiLdgDhYNglWn4jsAboGn1QbwXeqWLfO0df2tMJB7FTo8eNE2K7eTOx35z03ALg3egLvzPhQHpBtOxgwgG/fRTbmdH6P6pi+5tKIOuAXkC9qKxfA42i5fcAL0b7bwi8BNwaLWsM/BaoHy0bDLyQsM8ywgG5VfQ/qZcirt9F5dsG+APhoPqThGWfAIcABuwL7JXwv+mSsJ0N72XdKJ5VwH4JyycC3RLKvLnPROJB/CyiBFKN97mMkBB/Bvw4ety7ivdko31Vse8yNk4gawlJtQ5wISFZWLT8ZcLBvVH0Xh6VUN7ypH3fSJRA+D6Z/TJ63ZXAXGDbzX0OU5TpQeC2hMeX8v0Poj0JCaqq2x+r2OZlJHzvgNbR/7wFcC3wDlAn7uNUtY5lcQdQiDfgT8BnSc+9HX3oVgMdoueeAJ7azLbuAfpE9zd8YX+esPx2oF90/0ZgdMKylsDqKrZ7OvBu0nPjgbMSYqtJAvlTUmx9o/v/Av6etP7sDQeNFNvfVAJZzcYHrSXAoYSD9lfAPgnLDgPmV7GPNsDyhMdlQK8tfK+nAidG90cCl1ax3gKqSCDR46eJfpUC+xESSv0t+ExUlUA29z6XAdclLOsBvFLFfpNjTrXvMjZOIHMTltWP1t8N+AmhltEoxX46sukEcj3wbMKybQiJu+PmPocp9tWeUKvbJno8Cfj9lnwGkrZ3IKFWc2TS8z2BDwiJZL+abj/bN53DjccyoElSw+bh7r5TtCzxfVmY+EIza29mY6PGuC8J7SZNkraf+JqPCL+0Nvgs4f7XwHaJcSTYPXptoo+APaouVrUk779BdH8voKeZrdhwA5qxcezVtczd16XYzy6Eg9TkhH28Ej2PmdU3s4fM7CMzWwm8AeyU1Pa00fuRzMzOMLOpCdtvzffvTzPCr/maGECoHQD8kVAz+jraZ3U+E1Wpzvtc1XuWDt9te0N5ou03A75w9+U12OZGZXL3SsL7tsVlcvcJhB8dR5nZzwm1xhdrEBNRr8H/EH5EvJm0+ElCwh3h7nNqsv04KIHEYzzwDXBiNdb1pMcDCB/gZu6+I+EcfnKvrWYJ9/cknBbYUosIB/VEexJ+yVVHctybsxC4xd13SrjVd/eBW7idTfmcUDtplbCPHd19w8GjJ7A/0N7ddwA6RM8n/n+rLJeZ7QU8AlxMOAW0EzA94fULgX2qePnm/l+vEn50tCEkkgEJyzb1mdjcdrf2fd6Ur6K/9ROe262ar10I7GxmO6VYtkVlino1NqPmZXqScNbgdOA5d18TbXdDV+uqbqclxLAXMJpQy+6fYh8PEtrcfmVmv6hhnFmnBBIDd18B3AQ8aGanmFkDM9smOjhsv5mXNyT8MltjZu0Iv0aTXR/9mm4FnE04j7ylRgA/M7M/mlldM/sD4ZTX8Gq+fjHw0y3Y3yPABdGvaTOz7c3s12bWcAvjrlL0S/QRoI+Z7QpgZnuY2a+iVRoSEswKM9sZ+L8t3MX2hIPb0mjbZxNqIBs8ClxhZsVRGfeNDiywmf9XVKN6jtBrb2dgVMLiTX0mlhJOBVW17a19n6vk7ksJB+0/mVkdMzuHqhNo8ms/Jfxaf9DMGplZPTPbkNAXA43NbMcqXv4s8Gsz6xx1Le5J+MH2dg2L0h/4DSGJPJUQ44au1lXd/g3hM0boVPCAu/dN3riZnQ4UE07pXQI8aWbprOVljBJITNz9duAvhAa+JYQvxUPAVWz6g94D6GVmq4AbCF+WZK8TGg3HAHe6+xZfNOahD/9xhC/fsijO49z982puoh/QMjqV80I19jeJ0Jh6P+E88FzCFyrdroq2/U50mmo0odYBoe3gx4SayjuE01vV5u4zgbsINczFhIbttxKWDyb04hpAaMN4gZAMIHRouC76f11RxS4GEHriDU46RVflZyI6LXQL8Fa07UOTYt7a93lzzgP+Gm27FVt2ED+d0Mj+AeE7clkU8weEjiLzojJtdJrT3WcTDvb/JLyXxwPHu/u3NSmAu5cD7xF+HCSfeqqOcwkJ/P8SaygQajGEz90Z7l7h7gMI7Sx9ahJrtm3o7SC1gJk1J3SFrJd0gBGRrWBmjwGL3P26uGPJJakaT0VEJBL9MDsZaBtvJLlHp7BERKpgZn8ndIS4w93nxx1PrtEpLBERqRHVQEREpEYKqg2kSZMm3rx587jD2CJfffUV22+/uZ69tYvKXBhU5vwxefLkz919l+TnCyqBNG/enEmTJsUdxhYpKyujY8eOcYeRVSpzYVCZ84eZJY9WAOgUloiI1JASiIiI1IgSiIiI1EhBtYGksnbtWsrLy1mzZk3coaS04447MmvWrLRuc7vttqNp06bUq1cvrdsVkcJS8AmkvLychg0b0rx5cywHpyJftWoVDRumbTxB3J1ly5ZRXl7O3nvvnbbtikjhifUUlpk9ZmZLzGx6wnM7RxPQz4n+NqritWdG68wxszNrGsOaNWto3LhxTiaPTDAzGjdunLM1LhHJH3G3gTxBmN410dXAGHffjzCa7NXJL0oYars90I4wymXKRFMdhZI8Nii08opIZsSaQNz9DcL0jolOJEzgQvT3pBQv/RUwyt03zFg2ih8mIhERmT4drrkGMjBsVS62gRRFk8ng7p9umPgnyR5sPLVoOVVMtWpmpUApQFFREWVlZRst33HHHVm1alUaws6M9evXp4yve/fuTJkyhXr16lFcXMy99967RY3ia9as+cH/IldUVFTkbGyZojIXhmyW2dauZc8BA9jr6adZt/32TG7Thm92TXU43QpxT8pOmAd4esLjFUnLl6d4zV+B6xIeXw/03Ny+iouLPdnMmTN/8FwuWblyZcrnX375Za+srPTKykrv1q2bP/jgg1u03Vwu99ixY+MOIetU5sKQtTK/+65769bu4H7aae5Ll27V5oBJnuKYGncbSCqLzewnANHfJSnWKWfjeb+bUrN5v2N3/fXXc++99373+Nprr+W+++7b7Ou6du2KmWFmtGvXjvLycgAuueQSevXqBcDIkSPp0KEDlZWVmQleRHLL11/DFVfAoYfC8uUwfDg8/TQ0aZKR3eXiKawXgTOB3tHfYSnWGQn8I6Hh/Gjgb1u958sug6lTt3ozG2nTBu65p8rF3bt35+STT+bSSy+lsrKSQYMG8dprr9GmTRsAKisr2Wab7/P8gAEDaNmy5XeP165dS//+/b9LQr179+aQQw7hyCOP5JJLLmHEiBEbvV5EaqmxY+G88+DDD+H88+G222DHqqaNT49YE4iZDQQ6Ak3MrJzQs6o38KyZdQc+Bn4XrVsCXODu57r7F9FELxOjTfVy9+TG+LzQvHlzGjduzJQpU1i8eDFt27Zlr732YmqUyDZ3HUiPHj3o0KEDRx55JAD169fnkUceoUOHDvTp04d99tknK+UQkZh8+SVceSU8/DDss09IJFkasDHWBOLup1axqHOKdScRJqff8Pgx4LG0BrSJmkImnXvuuTzxxBN89tlnnHPOOaxateq7hLCpGshNN93E0qVLeeihhzba3vvvv0/jxo1ZtCgvz+qJSHW99BJceCF8+mk4dXXTTVC/ftZ2n4unsArOb37zG2644QbWrl3LgAEDqFOnzmZrII8++igjR45kzJgxGyWYjz76iLvuuospU6bQtWtXTjrpJNq3b5+1sohIFixdCpdcAoMGQevWMHQoHHJI1sPQyfEcsO2229KpUyd+//vfU6dOnWq95oILLmDx4sUcdthhtGnThl69euHudO/enTvvvJPdd9+dfv36ce655+qqc5Hawh0GDIAWLWDIEOjVCyZPjiV5gGogOaGyspJ33nmHwYMHV/s169atS/n86NGjv7tfXFzM+++/v9XxiUgOWLgwnK56+WVo3x769YNWrWINSTWQmM2cOZN9992Xzp07s99++8UdjojkmspK6Ns3JIuxY6FPH3jrrdiTB6gGEruWLVsyb968uMMQkVw0Z07omvv669C5c+hp9dOfxh3Vd1QDgQ1XsxeMQiuvSN5Ztw7uuAMOPDBcm9avH4walVPJA5RA2G677Vi2bFnBHFQ9mg9ku+22izsUEUll2rRwJfmVV8Ixx8DMmXDOOZCDo2gX/Cmspk2bUl5eztKlS+MOJaU1a9ak/WC/YUZCEckh33wDN98MvXvDzjvDs8/CKafkZOLYoOATSL169XJ6Zr6ysjLatm0bdxgikknvvBNqGbNmwemnh4byxo3jjmqzCv4UlohIbL76Ci6/HA4/HCoqYMQIeOqpvEgeoBqIiEg8xowJPazmz4cePcKpq02Me5eLVAMREcmmFSvg3HOhSxeoWxfeeAMeeCDvkgcogYiIZE2TceOgZUt44gm46qrQ4yoaODUf6RSWiEimLV4Mf/4zrQcPhoMOCqPoFhfHHdVWUw1ERCRT3KF//1DrGDaMed27w8SJtSJ5gBKIiEhmfPwxdO0KZ5wBP/85TJvGx3/6E9SrF3dkaZOTCcTM9jezqQm3lWZ2WdI6Hc3sy4R1bogrXhGR71RWhkbxVq3gzTfhvvtCQ/nPfx53ZGmXk20g7j4baANgZnWAT4ChKVZ9092Py2ZsIiJVmj079LAaNw5++csw+GHz5nFHlTE5WQNJ0hn40N0/ijsQEZGU1q0L13EcdBBMnw6PPw4jR9bq5AFguT6IoJk9Brzn7vcnPd8RGAKUA4uAK9x9RorXlwKlAEVFRcWDBg3KeMzpVFFRQYMGDeIOI6tU5sJQW8rcYO5c9r/9dhrOmcPSDh2Yc+mlfLvzzinXzdcyd+rUabK7l/xggbvn7A3YFvgcKEqxbAegQXS/KzBnc9srLi72fDN27Ni4Q8g6lbkw5H2ZV692v+Ya9zp13IuK3J97brMvydcyA5M8xTE1109hHUuofSxOXuDuK929Iro/AqhnZk2yHaCIFKC33oI2beAf/wiDH86cCb/9bdxRZV2uJ5BTgYGpFpjZbmZhnGMza0coy7IsxiYihaaiAi65JFw9vno1vPJKaO+o4pRVbZeTvbAAzKw+8Evg/ITnLgBw977AKcCFZrYOWA10i6paIiLpN2pUGPzw44/h4ovhllvycvyqdMrZBOLuXwONk57rm3D/fuD+5NeJiKTVF19Az55h/Kr99w/XdhxxRNxR5YRcP4UlIhKf558Pw5D07w/XXBPmJ1fy+E7O1kBERGLz2WfhNNWQIdC2bWjraNMm7qhyjmogIiIbuIdTVS1bwvDh4eLACROUPKqgGoiICMCCBVBaGhrLf/ELePTR0OYhVVINREQKW2Ul/POf0Lo1jB8fBkJ8/XUlj2pQDURECtesWWHww7ffhmOOgb59Ya+94o4qb6gGIiKFZ+3acBV5mzbwwQfw1FMwYoSSxxZSDURECst778E554T5yE85Be6/H4qK4o4qL6kGIiKFYfVquPpqaNcuzFH+/PMweLCSx1ZQDUREar833wxtHf/7X6h93HknNGoUd1R5TzUQEam9Vq2Ciy6CDh3g229DF91+/ZQ80kQJRERqp1deCV1z//UvuOyyMFNgly5xR1WrKIGISO2ybBmceSYceyxsv32Yu6NPn3Bf0koJRERqB/fQKN6yJQwYANddB1OmwGGHxR1ZraVGdBHJf59+Cj16wAsvQHExvPoqHHRQ3FHVeqqBiEj+cofHHoMWLUKbx+23wzvvKHlkSc7WQMxsAbAKWA+sc/eSpOUG3At0Bb4GznL397Idp4jEZP78MPjh6NGhl9Ujj8DPfhZ3VAUlZxNIpJO7f17FsmOB/aJbe+Bf0V8Rqc3Wrw9Xj19zDdSpE3pZlZbCNjqhkm25nkA25UTgqWge9HfMbCcz+4m7fxp3YCKSITNnQvfu4TRV165h8MNmzeKOqmBZOP7mHjObDywHHHjI3R9OWj4c6O3u46LHY4Cr3H1S0nqlQClAUVFR8aBBg7IRftpUVFTQoEGDuMPIKpW5MGxJmW3tWvYcOJC9+vdnff36zPnzn1nSuTOYZTjK9MrX97lTp06Tk5sRAHD3nLwBu0d/dwWmAR2Slr8M/CLh8RigeFPbLC4u9nwzduzYuEPIOpW5MFS7zBMnuh9wgDu4d+vmvmRJRuPKpHx9n4FJnuKYmrMnDd19UfR3CTAUaJe0SjmQWHdtCizKTnQiknFffw1XXgnt24eLA4cNg4EDYZdd4o5MIjmZQMxsezNruOE+cDQwPWm1F4EzLDgU+NLV/iFSO7z+euiKe8cdYRDEmTPhhBPijkqS5GojehEwNPTUpS4wwN1fMbMLANy9LzCC0IV3LqEb79kxxSoi6bJyJVx1VWgc/+lP4bXXoFOnuKOSKuRkAnH3ecAPrgSKEseG+w5clM24RCSDXn4ZLrgAFi2Cnj2hVy+oXz/uqGQTcvIUlogUkKVL4bTT4LjjYKedYPz4MF+HkkfOUwIRkXi4w6BBYfDDwYPhxhth8uQwY6DkhZw8hSUitdwnn9D6uuvg7bdDwujXL8zdIXlFNRARyR73MGZVy5Y0mjwZ7r47JBElj7ykBCIi2TF3LnTuHMatKi5mYr9+cPnlYTwryUtKICKSWevXw113wYEHhjaORx6BMWNYs8cecUcmW0ltICKSOdOnwznnwMSJcPzxYeRcJY5aQzUQEUm/b78NvaoOPjjM2zFwYBiKRMmjVlENRETS6913Q61jxoxwfcc990CTJnFHJRmgGoiIpMdXX4UryA87DL78EoYPh6efVvKoxVQDEZGt99prcN55MG8eXHgh9O4NO+wQd1SSYaqBiEjNffll6JbbuXOYUrasDB58UMmjQCiBiEjNvPRSGIakXz+44gqYNg2OOiruqCSLlEBEZMssWQKnnhrm52jcGCZMCPN2aPDDgqMEIiLV4w7//neodQwZEoZbnzQJSn44VbYUhpxLIGbWzMzGmtksM5thZpemWKejmX1pZlOj2w1xxCpSMBYuDBcC/ulPsN9+MHUqXH89bLtt3JFJjHKxF9Y6oKe7vxdNazvZzEa5+8yk9d509+NiiE+kcFRWwsMPh7nJ16+HPn3gz3/W+FUC5GACieY1/zS6v8rMZgF7AMkJREQyac6cMB/5G29Aly4hkey9d9xRSQ6xMDNsbjKz5sAbQGt3X5nwfEdgCFAOLAKucPcZVWyjFCgFKCoqKh40aFBmg06ziooKGjRoEHcYWaUyx8vWr6fps8/S/Ikn8Hr1mNujB58deyyYpXU/uVTmbMnXMnfq1Gmyu/+wscvdc/IGNAAmAyenWLYD0CC63xWYU51tFhcXe74ZO3Zs3CFkncoco6lT3YuL3cH9pJPcP/kkY7vKmTJnUb6WGZjkKY6pOdeIDmBm9Qg1jH+7+/PJy919pbtXRPdHAPXMTOMliNTUN9+ERvGSktBgPngwPP887L573JFJDsu5NhAzM6AfMMvd765ind2Axe7uZtaO0JtsWRbDFKk9xo+H7t1h1iw444wwS2DjxnFHJXkg5xIIcARwOvC+mU2NnrsG2BPA3fsCpwAXmtk6YDXQLapmiUh1VVTAddfBffdB06bwn//AMcfEHZXkkZxLIO4+Dthka5273w/cn52IRGqh0aPD4IcLFsBFF8Gtt0LDhnFHJXkmJ9tARCRDVqwIp6t++UuoVy900b3/fiUPqRElEJFC8cILYRiSJ5+Eq68Ogx8eeWTcUUkey7lTWCKSZosXh6vHBw+Ggw4Ko+gWF8cdldQCqoGI1Fbu0L9/qHUMGwa33AITJyp5SNqoBiJSG338MZx/PrzyChx+eJiz4+c/jzsqqWVUAxGpTSorw4yArVrBm2+GLrpvvqnkIRmhGohIbTF7dhj8cNw4OPpoeOghaN487qikFlMNRCTfrV0LvXuHBvIZM+CJJ8KpKyUPyTDVQETy2ZQp4bqOKVPg5JPhgQdgt93ijkoKhGogIvlozRq49lo45BBYtAieey5MM6vkIVmkGohIvnnrrVDrmD0bzjorDH7YqFHcUUkBUg1EJF9UVIQLAo88MtRARo6Exx9X8pDYKIGI5INXXw1dcx94AC6+GKZPDz2tRGKkBCKSy774As4+G371K6hf//trO/JwWlSpfZRARHLVkCFhGJL+/eGaa0JPqyOOiDsqke+oEV0k13z2WThNNWQItG0brulo0ybuqER+IGdrIGZ2jJnNNrO5ZnZ1iuU/MrNnouUTzKx59qMUSSP3cBFgixYwfHiY5Ondd5U8JGflZAIxszrAA8CxQEvgVDNrmbRad2C5u+8L9AFuy26UIumz3Wefhelkzz4bWrcOc3VcfTXU1UkCyV2bTSDNWpRc3KxFSbb7CbYD5rr7PHf/FhgEnJi0zonAk9H954DOZrbJqXBFcs769fDPf3LI2WfD22+HXlavvw777x93ZCKbVZ2fN7sBE5u1KHkPeAwYuXDWJM9sWOwBLEx4XA60r2odd19nZl8CjYHPE1cys1KgFKCoqIiysrIMhZwZFRUVeRfz1iqUMtf/6CP2v+MOdpwxg2UHH8yHV17JN0VFYZrZAlAo73Oi2lbmzSaQhbMmXdesRcn1wNHA2cD9zVqUPAv0Wzhr0ocZiitVTSI5aVVnHdz9YeBhgJKSEu/YseNWB5dNZWVl5FvMW6vWl3ntWrj9dujVK3THfeopZjZtSsdOneKOLKtq/fucQm0rc7XaQKIax2fRbR3QCHiuWYuS2zMUVznQLOFxU2BRVeuYWV1gR+CLDMUjkh6TJ4fxq667Dk46CWbOhNNPB519lTxUnTaQS5q1KJkM3A68BRywcNakC4Fi4LcZimsisJ+Z7W1m2wLdgBeT1nkRODO6fwrwmrtn+tSaSM2sXh0axdu3D3OUDx0KzzwDRUVxRyZSY9VpA2kCnLxw1qSPEp9cOGtSZbMWJcdlIqioTeNiYCRQB3jM3WeYWS9gkru/CPQD+pvZXELNo1smYhHZam+8ESZ6mjMnDIJ4xx0av0pqheq0gdywiWWz0hvO99x9BDAi6bkbEu6vAX6Xqf2LbLWVK+FvfwtTzDZvDqNGQZcucUclkjY5eR2ISN77z3/C9Rz/+hdcdlkY/FDJQ2oZJRCRdFq2DM44A7p2DT2s3noL+vSB7bePOzKRtFMCEUkHd3j22TAMycCBcP31YfDDww6LOzKRjNE4CSJba9Ei6NEDhg2DkhIYPRoOPDDuqEQyTjUQkZpyh379wpDrI0eG3lXjxyt5SMFQDUSkJubNg/POg9deg6OOgkcfhX33jTsqkaxSDURkS6xfD/fcAwccABMnQt++IYkoeUgBUg1EpLpmzAgXAk6YAL/+dUgeTZvGHZVIbFQDEdmcb78NAx+2bQsffggDBsBLLyl5SMFTDURkUyZODLWO99+HU0+Fe++FXXaJOyqRnKAaiEgqX38Nf/0rHHoofPEFvPhiqHkoeYh8RzUQkWRlZWHwww8/hNLSMHfHjjvGHZVIzlENRGSDL7+E88+HDRM7vfYaPPSQkodIFZRARACGD4dWrcL1HD17wn//+30iEZGUlECksC1dCqedBscfH+boGD8e7rwT6tePOzKRnKcEIoXJPcwI2KoVDB4MN90Upptt1y7uyETyRk41opvZHcDxwLfAh8DZ7r4ixXoLgFXAemCdu5dkM07Jc598AhdeGK7laNcujGfVunXcUYnknVyrgYwCWrv7gcD/gL9tYt1O7t5GyUOqrbISHn44DH44ejTcfTe8/baSh0gN5VQCcfdX3X1d9PAdQJf6SnrMnQudO4deVsXF4cLAyy+HOnXijkwkb5m7xx1DSmb2EvCMuz+dYtl8YDngwEPu/vAmtlMKlAIUFRUVDxo0KEMRZ0ZFRQUNGjSIO4ysSmuZ16+n6ZAh7P3YY3jdunx44YV82rUrmKVn+2mi97kw5GuZO+5LCzcAAA9oSURBVHXqNDnl2R53z+oNGA1MT3E7MWGda4GhRAkuxTZ2j/7uCkwDOlRn38XFxZ5vxo4dG3cIWZe2Mr//vvshh7iD+wknuJeXp2e7GaD3uTDka5mBSZ7imJr1RnR377Kp5WZ2JnAc0DkKPNU2FkV/l5jZUKAd8Ea6Y5U89c03cOut8I9/wE47hSlm//CHnKt1iOS7nGoDMbNjgKuAE9z96yrW2d7MGm64DxxNqMGIhKHWi4tDt9w//AFmzoRu3ZQ8RDIgpxIIcD/QEBhlZlPNrC+Ame1uZiOidYqAcWY2DXgXeNndX4knXMkZX30Ff/kLHHZYGJJk+HDo3x+aNIk7MpFaK6euA3H3lNO6Raesukb35wEHZTMuyXGvvRaml503L1zf0bs37LBD3FGJ1Hq5VgMRqb4VK0Li6NwZttkmjKL74INKHiJZogQi+enFF8MwJI89BldeGQY/POqouKMSKShKIJJfliwJjeInnhjaNyZMgNtugx//OO7IRAqOEojkB3d4+mlo0QKGDoW//x0mTYISjWQjEpecakQXSWnhQrjgAhgxIkwx269fGM9KRGKlGojkrspK6Ns3tHWUlcG998K4cUoeIjlCNRDJTXPmhHnJ33gDunQJo+juvXfcUYlIAtVAJLesW0ezgQPhwANDz6rHHoNXX1XyEMlBqoFI7pg2Dbp3Z5/Jk+Gkk8I1HT/5SdxRiUgVVAOR+H3zDVx/fehRtXAhM268EZ5/XslDJMcpgUi8xo+Htm3h5pvh1FNh5kyWHnWUBj8UyQNKIBKPigq47DI44ogwEOJ//gNPPQWNG8cdmYhUk9pAJPtGjYLSUliwAC66KMzd0bBh3FGJyBZSDUSyZ/ly6N4djj4att02dNG9/34lD5E8pQQi2TF0aLgA8Mkn4eqrYepUOPLIuKMSka2QcwnEzG40s0+iCaWmmlnXKtY7xsxmm9lcM7s623FKNS1eDL//PZx8Muy2G7z7bjhlpcEPRfJeziWQSB93bxPdRiQvNLM6wAPAsUBL4FQz0/gWucQ9NIq3aAHDhoVeVu++CwcfHHdkIpImuZpANqcdMNfd57n7t8Ag4MSYY5INPv4YunaFM88MCWTaNLj2WqhXL+7IRCSNzN3jjmEjZnYjcBawEpgE9HT35UnrnAIc4+7nRo9PB9q7+8UptlcKlAIUFRUVDxo0KKPxp1tFRQUNGjSIO4zqqaxk92HD+Okjj2DuzCst5ZMTTwyzBW6BvCpzmqjMhSFfy9ypU6fJ7v7DuRPcPes3YDQwPcXtRKAIqEOoHd0CPJbi9b8DHk14fDrwz83tt7i42PPN2LFj4w6hej74wP0Xv3AH96OPdp8/v8abypsyp5HKXBjytczAJE9xTI3lOhB371Kd9czsEWB4ikXlQLOEx02BRWkITbbU2rVw551w001Qvz48/ng4daUryUVqvZxrAzGzxAGQfkOomSSbCOxnZnub2bZAN+DFbMQnCaZMgfbt4Zpr4PjjYeZMOOssJQ+RApFzCQS43czeN7P/Ap2AywHMbHczGwHg7uuAi4GRwCzgWXefEVfABWfNmpA0DjkEFi2CIUNg8ODQTVdECkbODWXi7qdX8fwioGvC4xHAD7r4SoaNGxeuJv/f/+Dss+Guu6BRo7ijEpEY5GINRHLRqlVw8cXh6vFvvoGRI8NkT0oeIgVLCUQ2b+RIaN06TPB0ySUwfXoYz0pECpoSiFTtiy9Cj6pjjgk9rMaNg3vvhTzsxy4i6acEIqk991y4inzAgHAV+ZQpcPjhcUclIjkk5xrRJWaffhraOp5/PoxbNXIktGkTd1QikoNUA5HAPVwE2LIlvPwy9O4NEyYoeYhIlVQDkTAzYGlpmCnwyCPh0UfhZz+LOyoRyXGqgRSy9evhvvtCD6vx4+GBB6CsTMlDRKpFNZBCNWtWuCBw/Hg49ljo2xf23DPuqEQkj6gGUmjWroVbbgltG7NnQ//+oc1DyUNEtpBqIIVk8mQ45xz473/DNLP//CfsumvcUYlInlINpBCsXg1XXQXt2sHSpfDCC/DMM0oeIrJVVAOp7d54A849F+bMCW0ed94JO+0Ud1QiUguoBlJbrVwJPXrAUUfBunUwenTonqvkISJpogRSG40YEbrm9u0Ll18O778PnTvHHZWI1DI6hVWbfP55SBhPPx2uKH/7bTj00LijEpFaKqcSiJk9A+wfPdwJWOHuPxhLw8wWAKuA9cA6dy/JWpC5yD3MCHjxxbB8OdxwQ5gx8Ec/ijsyEanFciqBuPsfNtw3s7uALzexeid3/zzzUeW4RYtCW8ewYVBSAmPGwAEHxB2ViBSAnGwDMTMDfg8MjDuWnOUO/fqFU1UjR8Ltt4erypU8RCRLzN3jjuEHzKwDcHdVp6bMbD6wHHDgIXd/eBPbKgVKAYqKiooHDRqUgYgzp6KiggZJEzhtt2gR+991F43ee48VBx3E7L/+ldV77BFThOmXqsy1ncpcGPK1zJ06dZqc8njs7lm9AaOB6SluJyas8y+g5ya2sXv0d1dgGtChOvsuLi72fDN27NjvH6xb53733e7167s3bOjet6/7+vWxxZYpG5W5QKjMhSFfywxM8hTH1Ky3gbh7l00tN7O6wMlA8Sa2sSj6u8TMhgLtgDfSGWfOmTEjXAg4YQL8+tehi27TpnFHJSIFLBfbQLoAH7h7eaqFZra9mTXccB84mlCDqZVs7Vro1QvatoUPPwxTzL70kpKHiMQup3phRbqR1HhuZrsDj7p7V6AIGBra2akLDHD3V7IeZTZMnEjx+efD/Plw6qlw772wyy5xRyUiAuRgAnH3s1I8twjoGt2fBxyU5bCy6+uvw7UcffpQb+ed4cUX4fjj445KRGQjOZdACl5ZWRj88MMPobSUd48/niOPOy7uqEREfiAX20AK05dfwvnnQ6dO4fFrr8FDD7E+D7v8iUhhUALJBS+9BK1ahdFye/YMEz5tSCQiIjlKCSROS5fCH/8IJ5wAjRqFK8nvvBPq1487MhGRzVICiYM7DBwYhiF57jm46aYw3Wy7dnFHJiJSbWpEz7bycrjwQhg+PCSMfv3C3B0iInlGNZBsqayEhx4KtY4xY+Duu8N8HUoeIpKnVAPJhrlz4bzzQhfdTp3gkUdgn33ijkpEZKuoBpJJ69bBXXfBgQfCe++FxDFmjJKHiNQKqoFkyvvvh8EPJ04MvawefBBq0ZDrIiKqgaTbN9/A//0fHHwwLFgAzzwDL7yg5CEitY5qIOk0YUKodcyYAaedBvfcA02axB2ViEhGqAaSDl99BX/5Cxx2GKxcCS+/DE8/reQhIrWaaiBba8yY0MNq/nzo0QNuvRV22CHuqEREMk41kJpasSIkji5doG5deP11eOABJQ8RKRhKIDUxbFi4IPDxx+HKK2HaNOjQIe6oRESyKpYEYma/M7MZZlZpZiVJy/5mZnPNbLaZ/aqK1+9tZhPMbI6ZPWNm22Yl8CVLoFs3OOkk2HXX0Gh+223w4x9nZfciIrkkrhrIdOBk4I3EJ82sJWFK21bAMcCDZlYnxetvA/q4+37AcqB7RqN1D43iLVrA0KHw97+H6zuKizO6WxGRXBZLAnH3We4+O8WiE4FB7v6Nu88H5gIbDVFrYTL0/wc8Fz31JHBSxoJduxaOOw5OPx323x+mToXrroN69TK2SxGRfJBrvbD2AN5JeFwePZeoMbDC3ddtYp3vmFkpUApQVFREWVnZFge1T/36rLn4Yj456SRYvDjcsqSioqJGMeczlbkwqMz5L2MJxMxGA7ulWHStuw+r6mUpnvMarPP9AveHgYcBSkpKvGPHjlWtWrXoNftt+Su3WllZGTWKOY+pzIVBZc5/GUsg7t6lBi8rB5olPG4KLEpa53NgJzOrG9VCUq0jIiIZlmvdeF8EupnZj8xsb8KP/ncTV3B3B8YCp0RPnQlUVaMREZEMiasb72/MrBw4DHjZzEYCuPsM4FlgJvAKcJG7r49eM8LMdo82cRXwFzObS2gT6ZftMoiIFLpYGtHdfSgwtIpltwC3pHi+a8L9eST1zhIRkezKtVNYIiKSJ5RARESkRpRARESkRpRARESkRiz0ii0MZrYU+CjuOLZQE8K1L4VEZS4MKnP+2Mvdd0l+sqASSD4ys0nuXrL5NWsPlbkwqMz5T6ewRESkRpRARESkRpRAct/DcQcQA5W5MKjMeU5tICIiUiOqgYiISI0ogYiISI0ogeQRM7vCzNzMmsQdS6aZ2R1m9oGZ/dfMhprZTnHHlClmdoyZzTazuWZ2ddzxZJqZNTOzsWY2y8xmmNmlcceUDWZWx8ymmNnwuGNJFyWQPGFmzYBfAh/HHUuWjAJau/uBwP+Av8UcT0aYWR3gAeBYoCVwqpm1jDeqjFsH9HT3FsChwEUFUGaAS4FZcQeRTkog+aMPcCWbmL63NnH3VxPmvX+HMPNkbdQOmOvu89z9W2AQcGLMMWWUu3/q7u9F91cRDqp7xBtVZplZU+DXwKNxx5JOSiB5wMxOAD5x92lxxxKTc4D/xB1EhuwBLEx4XE4tP5gmMrPmQFtgQryRZNw9hB+AlXEHkk6xTCglP2Rmo4HdUiy6FrgGODq7EWXepsrs7sOida4lnPL4dzZjyyJL8VxB1DLNrAEwBLjM3VfGHU+mmNlxwBJ3n2xmHeOOJ52UQHKEu3dJ9byZHQDsDUwzMwinct4zs3bu/lkWQ0y7qsq8gZmdCRwHdPbae8FSOdAs4XFTYFFMsWSNmdUjJI9/u/vzcceTYUcAJ5hZV2A7YAcze9rd/xRzXFtNFxLmGTNbAJS4ez6O6FltZnYMcDdwlLsvjTueTDGzuoROAp2BT4CJwB/dfUasgWWQhV9CTwJfuPtlcceTTVEN5Ap3Py7uWNJBbSCSq+4HGgKjzGyqmfWNO6BMiDoKXAyMJDQmP1ubk0fkCOB04P9F7+3U6Ne55BnVQEREpEZUAxERkRpRAhERkRpRAhERkRpRAhERkRpRAhERkRpRAhERkRpRAhERkRrRUCYiMWnWouQQoB9hRN46wLvAHxbOmjQ91sBEqkkXEorEqFmLkpsJ4yP9GChfOGvSrTGHJFJtqoGIxKsXYfyrNcAlMcciskXUBiISr52BBoRxv7aLORaRLaIEIhKvh4HrCfOd3BZzLCJbRAlEJCbNWpScAaxbOGvSAKA3cEizFiX/L+awRKpNjegiIlIjqoGIiEiNKIGIiEiNKIGIiEiNKIGIiEiNKIGIiEiNKIGIiEiNKIGIiEiN/H8ftp3QcGgRHQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "x = np.linspace(-5,5,100)\n",
    "y = 2*x\n",
    "plt.plot(x, y, '-r', label='y=2x')\n",
    "plt.title('Graph of the linear activation function y=2x')\n",
    "plt.xlabel('x', color='#1C2833')\n",
    "plt.ylabel('y', color='#1C2833')\n",
    "plt.legend(loc='upper left')\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 4.** Linear activation function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Funciones de activación tipo sigmoide"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Son funciones cuyo gráfico tiene una  forma de \"S\" alargada. Hay varios tipos de funciones sigmoideas. Las más utilizadas son las funciones logísticas y tangentes hiperbólicas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Función logística (sigmoid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Esta función es definida por\n",
    "\n",
    "$$\n",
    "f(x) = sigmoid(x) =\\frac{1}{{1+ e^{-x}}},\n",
    "$$\n",
    "para $x\\in \\mathcal{R}$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ejercicio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verifique que\n",
    "\n",
    "1. $0 \\le f(x)\\le 1$.\n",
    "2. $f'(x)= f(x)*(1-f(x))$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Por lo general, $ f (x) $ se usa en la capa de salida de una clasificación binaria, donde el resultado es 0 o 1, ya que el valor de la función sigmoidea se encuentra entre 0 y 1, por lo que el resultado se puede predecir fácilmente como 1 si el valor es mayor. que 0,5 y 0 en caso contrario."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZgcZbn+8e+TmcySWbIz2RfIQkLCliEJoBgUZAd/yGGPgkoUxQPKqigieg4cFBEUgaCCyL6JEcMOA4iEJXtCEhKy75lMZjL7+vz+qA4Ow0zSyfRM9XJ/ritXuqeqq563uvvut9+qrjJ3R0REEl+XsAsQEZHYUKCLiCQJBbqISJJQoIuIJAkFuohIklCgi4gkCQV6gjGzG83soRgtq8DM3jSzcjO7LcrHrDaz42K0fjezETFYzvNm9vV9eNznzWxZe9ffHvtae5TLrjCz/TtguaPNbG7kdfPfsV7+btY7JNKmtM5aZ6JJD7uARGdm5wI/AMYBlcAq4C/A3R7/B/lPA4qB/NZqNbMHgPXu/pPOLmxvuPtJ0cxnZg6MdPcVkce9BYzuyNparP9GYIS7X7jrb9HWHsWyi4CH3P2PzZadG4tlt+IaoMjdD+ug5QNB5wH4lru/AuDua4GOalNSUA+9HczsSuAO4FdAP6AA+A5wNJDRxmPiqXcxFPgwAT54JL4MBRaHXYS0wt31bx/+Ad0JeuRf3cN8DwB3AzMj8x8HnALMBXYC64Abm80/DHCC3vNGYBNwZbPpNwJPAA8C5QRvrMLdrP8o4H2gLPL/Uc3qqgfqgArguBaPm9Zi+j8if18NXAUsiCzzcSCr2eNOBeYBpcC/gYN3U5sT9Fh3bc8HgW3AGuAnQJfItDTgNoJvE6uAyyKPTY9MLyLoyQGMAN6I1FYMPB75+5uRx1RG2nMOMIXgG8iuegYDz0Rq2A78vo26JwLvRNq4Cfg9kNFs+kHAy0AJsAX4MXBiZFvWR9Y/v3ntQGZkeeOaLacvUA3sB/QEnovUtiNye1Bkvv8BGoGayLJ/v5fb9yLgX8CvI8teBZzURttfa7GuUc23f/PltXievwMsjyz/LsCaTb8EWELwev4QOBz4K9AUaX8FwbeCYS2e9wHAjMh2XgFcsq/vk2T5F3oBifov8gZt2PXi2s18DxCEy9EE34iyIkEyPnL/4Mib/iuR+Xe9aB8FciLzbSMSuJEXag1wMkHQ3QzMamPdvSJvoKkEw2vnRe73blbbL/dQ+y9b/G018F7kzdQr8kb8TmTa4cBWYFKktq9H5s9sY/nNA+dB4O9AXmQbfAR8MzLtO5E3+iCCYHuFtgP9UeD6Ztv6c62tL3J/CpFAj9Q7H7g9st0/9dgWdU8AJke26bDINrgiMi2PyIdwZBl5wKRmz91DLZbVvPY/A//TbNr3gBcit3sDXwW6RZb5JPBsa8vZh+17EcEHzSWR7XApQWfC2mj/p9bVyv2L+GygPwf0AIYQvJ5PjEz7L2ADcARgBB/IQ5u91o5rtpxhLZ73N4A/RLbzoZHlfmlv3yfJ9E9DLvuuD1Ds7g27/mBm/zazUjOrNrNjms37d3d/292b3L3G3YvcfWHk/gKCEPpCi+X/3N0r3X0hcD9BGO/yL3ef6e6NBD2ZQ9qo8RRgubv/1d0b3P1RYClwWvuazp3uvtHdS4B/ELyZIAiEe939XXdvdPe/ALUE4demyDDUOcCP3L3c3VcT9MinRmY5G7jD3de7+w7glt0srp5gSGBAZFv/K8o2TST4kLo6st3bfKy7z3b3WZFtuhq4l/88f6cCm939tsgyyt393ShreIRPP8/nR/6Gu29396fdvcrdywl65S1fM62KYvsCrHH3+yKvqb8A/QmGEGPlFncv9WAc/HX+85r5FnCru7/vgRXuviaKNg0GPgdcG9nO84A/tmhTtO+TpKFA33fbgT5m9smOZXc/yt17RKY137brmj/QzCaZ2etmts3Mygh6oH1aLL/5Y9YQhM0um5vdrgKymtfRzIDIY5tbAwxsu1lRabn+XTuqhgJXRj7USs2slGAYY0DLBbTQh2CfQ/Nam9c5gE9vj09tzxauIejpvWdmi83sG3tY9y6DCUKtYU8zmtkoM3vOzDab2U7gf/nP8zcY+DjKdbb0GpAdeX0MJQi9v0XW2c3M7jWzNZF1vgn0iHKfzJ62LzR7Tt29KnIzljsg23rN7Ov2GgCURD7cdmmzTez+fZI0FOj77h2C3ucZUczbcqfjIwRjf4PdvTtwD0EINTe42e0hBF+B99ZGgpBtbgjBV9xo7O3O0nUEQwY9mv3rFvlmsDvF/Kdn3VqdmwiGW3Zpvm0+XbD7Zne/xN0HAN8G/hDloZHrgCFRvuHvJvimM9Ld8wnGyHc9f+uAA9oqb3cLdfcmgnHf8wh65881C6wrCY7ImRRZ565vgLvWu7tl72n7tlclwVDQLv324rH7ur02Ar3MLK/Z32LZpoSkQN9H7l4K/JwgMM4ys1wz62JmhxKMwe5OHkHvosbMJhK8eVv6aaRXdhBwMcHOx701ExhlZuebWbqZnQOMJRjPjMYWYG+OY74P+E6kh2lmlmNmp7R4031G5CvxE8D/mFlepHf6Q2DX8fZPAJeb2UAz6wFc29ayzOy/zGxX+O8gCIXGKNrzHsEHxy2RurPM7Og25s0j2KFdYWYHEow57/Ic0M/MrjCzzEh7JjVb/zAz29377hGC4ZELIrebr7MaKDWzXsDPWjyuzbZFsX3bax5wZuT1OgL45l489o/AVWY2IfKaGRGpD3bfpnUEO91vjjxXB0fW+/C+NyPxKdDbwd1vJXhjXEOwM3ALwXjqtQQvtrZ8F7jJzMqBGwjebC29QbDn/lXg1+7+0j7Ut51gTPdKgmGga4BT3b04ykX8CRgbGT55Nor1fUAwjv57gjBdQbCDLBrfJ+jprSQ44uIRgp2EEHxQvERwZM1cgg+qBv4T1M0dAbxrZhUE34Iud/dVkWk3An+JtOfsFrU3EuxbGAGsBdYTBGtrriL4EC6P1PbJh22kR318ZFmbCY7sODYy+cnI/9vNbE5rC46Mt1cSDCk832zSb4Fsgt72LOCFFg+9AzjLzHaY2Z2tLHp327e9bic4gmcLwfh71KHq7k8S7A94hGB7Pkuwsx2CHZk/iTxfV7Xy8PMIdpRuJBia+pm7v7yPbUgK5q5DkOOJmQ0jOGysazTjuanIzE4C7nH3lsNJIilNPXSJe2aWbWYnR4aNBhIMN/wt7LpE4o0CXRKBEeyv2EEw5LKEYKhKRJrRkIuISJJQD11EJEmEdpB9nz59fNiwYWGtfp9VVlaSk7OnoxKTS6q1OdXaC2pzIpk9e3axu/dtbVpogT5s2DA++OCDsFa/z4qKipgyZUrYZXSqVGtzqrUX1OZEYmZtnhpBQy4iIklCgS4ikiQU6CIiSUKBLiKSJPYY6Gb2ZzPbamaL2phuZnanma0wswVmdnjsyxQRkT2Jpof+AMHVedpyEjAy8m8awalFRUSkk+0x0N39TYJr9rXlDODByNVGZhGcdL9/rAoUEZHoxOI49IF8+goy6yN/29RyRjObRtCLp6CggKKiohisvnNVVFQkZN3tkWptTrX2gtocC+5OXSNUNTjVDVDT4NQ0QnWDU9Pg1DZCTaNT2wCH7JfG/t2judjU3olFoLe80g60caURd58OTAcoLCz0RDyoP1F/jNAeqdbmVGsvqM0tNTY52ytrKS6vo6Syju2VteyorKOkqp4dlXWUVtdTWlVHWXU9ZdX1lNc0sLO6noam6M6NdcT40UyZHPuzP8ci0Nfz6UuCDWLfLpcmItLhKmsb2FBazYJtDWx6by2bymrYUlbDlvIatuysZVt5DSWVdbSWzWbQPbsrPbK70qNbBj27ZTCsdw7ds7uSn51OXlZX8rLSyc1MJy8rnZyMdHIyg/s5menkZKaRlZ5Gly6t9YPbLxaBPgO4zMweAyYBZe7+meEWEZHO4O6UVNaxqriSlcWVrNleyZrtVawtqWJdSRU7qur/M/PshZhBn9xM+uVnMbBHFocO7k7f3Ez65mXSOzeT3jkZ9M7NpFdOBt2zu5LWQWEcC3sMdDN7FJhCcIX79QQXF+gK4O73EFwO7GSCy41VEVz/UkSkw5VU1rF0006WbC5nxdZylm+pYPnWCsqq/xPa6V2MgT2zGdKrG+PG92dgj2wG9cxmy6qlnDzlSArys+ialhw/ydljoLv7eXuY7sD3YlaRiEgriitqmb+ulAXry1i0oYxFG8vYsrP2k+m9cjIYuV8upx7cnwP65jK8bw7De+cwqGc26a0EdlHpcgb17NaZTehwoZ1tUUSkLU1NzvKtFby3uoQPVpcwd20pa0uqgGAce0TfXI46oA9j++czpn8+o/vl0TcvM+Sqw6dAF5HQuTuriit5++PtvL28mFmrtlMaGesuyM/k8CE9uXDyEA4d3JODBuSTk6noao22ioiEoqa+kXc+3s7ry7by+rKtrCupBmBgj2yOG1PApOG9mDS8N4N7ZWMWvzsi44kCXUQ6zc6ael5bspUXF2+maNk2qusbye6axtEjejPtmAP4/Ig+DO3dTQG+jxToItKhauobeXXJVmbM38Dry7ZR19BE37xMzjx8IF8+qB+Thvciq2vsfzWZihToIhJz7s7cdaU8+cF6npu/kfLaBvrmZXL+xCGcdkh/Dhvcs8N+XJPKFOgiEjPlNfX8be4GHpq1ho+2VJDVtQsnj+/PVw8fxOT9e8f1j3KSgQJdRNptVXElf/7XKp6es56qukYOHtSdm88cz6kH9ycvq2vY5aUMBbqI7LPZa0q4942VvLxkC127dOH0QwcwdfJQDhncI+zSUpICXUT2irsza2UJd766nHdWbqdHt658/9gRTD1ymH7cEzIFuohE7YPVJdz64jLeW1VC37xMfnrqWM6bOJhuGYqSeKBnQUT2aNnmcn714lJeWbKVvnmZ3HjaWM6dOESHG8YZBbqItKmkso7bXlrGo++tJScjnatPGM3FRw9TjzxO6VkRkc9obHIemrWG37z8ERW1DXztyGFc/qWR9MzJCLs02Q0Fuoh8yqINZfzomYUs3FDG50b04YbTxjKqIC/ssiQKCnQRAaC6rpHbXlrGn99eRe/cTO46/3BOHt9P51VJIAp0EWH5jkZ+fudbrCqu5PxJQ7j2xAPpnq0fBCUaBbpICqttaOT2l5dz77s1DOiRzaOXTObIA3qHXZbsIwW6SIpaVVzJ9x+dw6INOzlmUDp/uOQYcnXhiISmZ08kBT07dwPX/20h6WldmD51AhnblirMk0ByXOpaRKJS29DIj55ZyBWPz2PsgHyev/zzfPmgfmGXJTGij2SRFLGprJrvPDSH+etKuXTKAVx5/CjS09SnSyYKdJEU8P7qEi59aDbVdY3cc+HhnDiuf9glSQdQoIskuWfmrOe6pxcysGdwFMtI/UgoaSnQRZJUU5Nz+ysf8bvXVnDk/r25+8LD6dFNP91PZgp0kSRU19DENU/N59l5Gzm7cBC//Mp4MtI1Xp7sFOgiSaaytoFLH57Dmx9t46ovj+J7x47Qz/dThAJdJImUVNZx8QPvs3B9KbecOZ5zJw4JuyTpRAp0kSSxtbyGC+57l7UlVdxz4QQdX56CFOgiSWBzWQ3n3zeLTWU13H/xERx1QJ+wS5IQKNBFEtyG0mrOv28W2yvqePCbEzliWK+wS5KQRLXb28xONLNlZrbCzK5rZfoQM3vdzOaa2QIzOzn2pYpIS5vKqjlv+ixKKuv4q8I85e0x0M0sDbgLOAkYC5xnZmNbzPYT4Al3Pww4F/hDrAsVkU/bNWYehPkkDhvSM+ySJGTR9NAnAivcfaW71wGPAWe0mMeB/Mjt7sDG2JUoIi2VVNZx4R/f/WTM/NDBPcIuSeKAufvuZzA7CzjR3b8VuT8VmOTulzWbpz/wEtATyAGOc/fZrSxrGjANoKCgYMJjjz0Wq3Z0moqKCnJzc8Muo1OlWpvjvb3VDc4t79WwsaKJH0zIYmzvtHYvM97b3BEStc3HHnvsbHcvbG1aNDtFW/tFQstPgfOAB9z9NjM7EvirmY1z96ZPPch9OjAdoLCw0KdMmRLF6uNLUVERiVh3e6Ram+O5vbUNjXzjgfdZX1HNfV8r5IsHFsRkufHc5o6SjG2OZshlPTC42f1BfHZI5ZvAEwDu/g6QBei4KZEYamxyfvjEfN5esZ1fnXVwzMJckkc0gf4+MNLMhptZBsFOzxkt5lkLfAnAzMYQBPq2WBYqksrcnZv+sZh/LtjE9SeP4czDB4VdksShPQa6uzcAlwEvAksIjmZZbGY3mdnpkdmuBC4xs/nAo8BFvqfBeRGJ2p/+tYq/vLOGSz4/nEuO2T/sciRORfXDInefCcxs8bcbmt3+EDg6tqWJCMALizbzPzOXcNK4fvzopDFhlyNxTOfTFIlj89eVcsXjczlkUA9uP+dQunTRWROlbQp0kTi1qayabz34AX1yM7nva4VkdW3/4YmS3BToInGopr6Rb/91NlW1Dfzp60fQNy8z7JIkAejkXCJxxt257ukFLFhfxvSpExjdT9cAleiohy4SZ6a/uZJn523kyuNH6ZzmslcU6CJx5O0VxfzfC0s5ZXx/LvviiLDLkQSjQBeJExtLq/n+o3M5oG8ut551sK4DKntNgS4SB+oamvjuw3OorW/k7gsnkJOp3Vuy9/SqEYkDv/znh8xbV8ofLjicEfsl3hkAJT6ohy4SsucWbOTByM/6Tx7fP+xyJIEp0EVCtGZ7Jdc9vZDDhvTgmhMPDLscSXAKdJGQ1DY0ctkjc+li8LvzDqNrmt6O0j4aQxcJyS3PL2XhhjLunTqBQT27hV2OJAF1CURC8OqSLdz/9mouOmoYJ+jHQxIjCnSRTra1vIZrnlrAgf3y+NHJGjeX2NGQi0gncneufnIBFbUNPDptMpnpOoOixI566CKd6IF/r+aNj7Zx/SljGFWgk25JbCnQRTrJR1vKufn5pXzxwP2YOnlo2OVIElKgi3SCuoYmfvD4PPIy03WeFukwGkMX6QS/e205izfuZPrUCfTJ1cUqpGOohy7Sweas3cFdr6/grAmDdH5z6VAKdJEOVF3XyJVPzKd/92xuOG1s2OVIktOQi0gHuvXFpawqruSRSyaRn9U17HIkyamHLtJB3ltVwgP/Xs3XjxzKUQf0CbscSQEKdJEOUF3XyDVPzWdQz2ydRVE6jYZcRDrAr19axurtVTxyySRdfUg6jXroIjH2weoS/vz2KqZO1lCLdC4FukgM1dQ3cu3TCxjQPZtrT9JQi3QufRcUiaHfv7aCj7dV8uA3JpKroRbpZOqhi8TIhxt3cs8bH/PVwwdxzKi+YZcjKUiBLhIDDY1NXPv0Anp068pPTx0TdjmSoqIKdDM70cyWmdkKM7uujXnONrMPzWyxmT0S2zJF4tuf317Fwg1l3HTGOHp0ywi7HElRexzkM7M04C7geGA98L6ZzXD3D5vNMxL4EXC0u+8ws/06qmCReLOupIrfvPwRx40p4KRxOleLhCeaHvpEYIW7r3T3OuAx4IwW81wC3OXuOwDcfWtsyxSJT+7O9c8uIs2MX3zlIJ0WV0IVzW74gcC6ZvfXA5NazDMKwMzeBtKAG939hZYLMrNpwDSAgoICioqK9qHkcFVUVCRk3e2Ram3em/a+s7GBNz+q5cIxGSyb+y7LOra0DpNqzzEkZ5ujCfTWuhzeynJGAlOAQcBbZjbO3Us/9SD36cB0gMLCQp8yZcre1hu6oqIiErHu9ki1Nkfb3h2VdVz5mzc4dHAPfj71KNK6JG7vPNWeY0jONkcz5LIeGNzs/iBgYyvz/N3d6919FbCMIOBFktbNzy+hrLqem88cn9BhLskjmkB/HxhpZsPNLAM4F5jRYp5ngWMBzKwPwRDMylgWKhJP3l25nSc+WM+3Pr8/Y/rnh12OCBBFoLt7A3AZ8CKwBHjC3Reb2U1mdnpktheB7Wb2IfA6cLW7b++ookXCVNfQxPXPLmJQz2wu/5K+iEr8iOq3ye4+E5jZ4m83NLvtwA8j/0SS2vQ3P2bF1gruv/gIsjPSwi5H5BP6pajIXlhdXMnvXlvBKeP7c+xo/dxC4osCXSRK7s5P/76IrmlddH1QiUsKdJEo/XPhJt5aXsxVXx5FQX5W2OWIfIYCXSQK5TX13PSPDxk3MJ+pRw4LuxyRVumEzSJRuO2lj9hWUct9XyvUMecSt9RDF9mDRRvKePCd1VwwaQiHDO4RdjkibVKgi+xGY1Nw8q1eORlcfYIuKSfxTYEushuPvb+W+etKuf6UMXTP7hp2OSK7pUAXaUNxRS23vrCMScN78ZVDB4ZdjsgeKdBF2nDL80uprG3gl18Zp/OcS0JQoIu04r1VJTw1ez2XHLM/Iwvywi5HJCoKdJEWGpqcnz67iIE9svn+F0eEXY5I1HQcukgLr6xpYNmWKu6dOoFuGXqLSOJQD12kmc1lNTy7oo4vHrgfXx5bEHY5IntFgS7SzC/++SGNDjeepgs+S+JRoItEvPnRNv65YBOn7t+VIb27hV2OyF7TAKEIUNvQyM9mLGZY726cNDzsakT2jXroIsD0N1ayqriSm84YR0aahlokMSnQJeWt3V7F718PrkJ0zKi+YZcjss8U6JLS3J2fzVhEehfjp6fqKkSS2BToktJeXLyF15dt4wfHj6Jfd12FSBKbAl1SVmVtAzf9YzEH9svj60cNC7sckXZToEvKuvPV5Wwsq+GXXxlH1zS9FSTx6VUsKWnZ5nL+9K9VnFM4mMJhvcIuRyQmFOiScpqanJ88u5C8rHSuO0lXIZLkoUCXlPPUnPW8v3oHPzp5DD1zMsIuRyRmFOiSUkoq67h55hKOGNaTsw4fFHY5IjGlQJeUcvPMJZTXNPDLr4ynSxf9IlSSiwJdUsasldt5MnIVotH9dBUiST4KdEkJtQ2NXP+3hQzulc1/f3Fk2OWIdAidbVFSwvQ3VvLxtkruv/gIsjPSwi5HpENE1UM3sxPNbJmZrTCz63Yz31lm5mZWGLsSRdpnVXElv4ucfOvY0fuFXY5Ih9ljoJtZGnAXcBIwFjjPzD5zFiMzywP+G3g31kWK7Ct358fPLCQzvQs3nKaTb0lyi6aHPhFY4e4r3b0OeAw4o5X5fgHcCtTEsD6Rdnlq9nreWbmd6046kIJ8nXxLkls0Y+gDgXXN7q8HJjWfwcwOAwa7+3NmdlVbCzKzacA0gIKCAoqKiva64LBVVFQkZN3tkaht3lnr3PivKkb26EL/qpUUFa2K6nGJ2t72UJuTQzSB3trBuv7JRLMuwO3ARXtakLtPB6YDFBYW+pQpU6IqMp4UFRWRiHW3R6K2+YrH5lLXVM0fLv4cIwuiP0wxUdvbHmpzcohmyGU9MLjZ/UHAxmb384BxQJGZrQYmAzO0Y1TC9PqyrTw7byOXfuGAvQpzkUQWTaC/D4w0s+FmlgGcC8zYNdHdy9y9j7sPc/dhwCzgdHf/oEMqFtmDitoGrn9mISP2y+V7XxwRdjkinWaPge7uDcBlwIvAEuAJd19sZjeZ2ekdXaDI3vrVC0vZtLOG//vqeDLTdcy5pI6ofljk7jOBmS3+dkMb805pf1ki++b91SU8OGsNXz9yGBOG6jznklr0039JGjX1jVz79AIGdM/m6hNGh12OSKfTT/8lafz2leWs3FbJg9+YSE6mXtqSetRDl6Qwb10p09/8mHMKB3PMqL5hlyMSCgW6JLya+kaufnI+BflZXH/qmLDLEQmNvpdKwrvz1eUs31rBAxcfQX5W17DLEQmNeuiS0OatK+XeN1dyduEgpuhMipLiFOiSsKrrGvnhE/MoyMvk+lN0JkURDblIwvq/F5ayclslD39rEt2zNdQioh66JKS3VxTzwL9Xc9FRwzh6RJ+wyxGJCwp0SThl1fVc/eR89u+bw7UnHhh2OSJxQ0MuknBu+PsitpTX8vSlR+n6oCLNqIcuCeXZuRv4+7yNXP6lkRw6uEfY5YjEFQW6JIx1JVX85NlFHDGsJ987VqfFFWlJgS4JoaGxiSsen4cBvzn7UNK6tHYhLZHUpjF0SQh3vraC2Wt2cMe5hzK4V7ewyxGJS+qhS9z794pifvfacs48fCBnHDow7HJE4pYCXeJacUUtlz8+j/375PCLM8aFXY5IXNOQi8StpibnB4/Po6y6Xuc4F4mCeugSt+5+42PeWl7Mz04by5j++WGXIxL3FOgSl/61vJjbXlrGaYcM4PyJQ8IuRyQhKNAl7mwsrea/H5vLAX1zueXM8ZjpEEWRaCjQJa7UNjRy6cNzqGto4p6pEzRuLrIX9G6RuPLzf3zI/HWl3H3B4RzQNzfsckQSinroEjf+OmsNj7y7lm9/YX9OGt8/7HJEEo4CXeLCrJXb+fmMxRw7ui/XnKBT4orsCwW6hG5dSRXffXgOQ3p3447zDtN5WkT2kQJdQlVeU8+3/vIB9Y1N3Pe1QvKzdCk5kX2lQJfQ1Dc28d2H5/DxtgruvmCCdoKKtJOOcpFQuDs/m7GYt5YXc8uZ4/ncSF0XVKS91EOXUEx/cyWPvLuWS6ccwLn6JahITCjQpdM9PXs9Nz+/lFMP7s/VXx4ddjkiSSOqQDezE81smZmtMLPrWpn+QzP70MwWmNmrZjY09qVKMnh96VaueXoBR4/ozW1nH0IXHdEiEjN7DHQzSwPuAk4CxgLnmdnYFrPNBQrd/WDgKeDWWBcqiW/2mh1c+vBsxvTP454LJ5CZnhZ2SSJJJZoe+kRghbuvdPc64DHgjOYzuPvr7l4VuTsLGBTbMiXRLdpQxsX3v0dBfhb3XzSRPB2eKBJz0RzlMhBY1+z+emDSbub/JvB8axPMbBowDaCgoICioqLoqowjFRUVCVl3e7S3zRvKm7j5vWoy0ozLDnIWz34ndsV1AD3HqSEZ2xxNoLc2yOmtzmh2IVAIfKG16e4+HZgOUFhY6FOmTImuyjhSVFREItbdHu1p88ptFVx17yy6ZWXy+LePZHifnNgW1wH0HKeGZGxzNIG+Hhjc7P4gYGPLmczsOOB64AvuXhub8iSRfbytgvPvm4W788i0yQkR5iKJLJox9PeBkWY23BHQcesAAAoQSURBVMwygHOBGc1nMLPDgHuB0919a+zLlETz0ZZyzrl3Fg2NzsOXTGLEfnlhlySS9PYY6O7eAFwGvAgsAZ5w98VmdpOZnR6Z7VdALvCkmc0zsxltLE5SwIcbd3Lu9Fl0MXj825M5sJ+uByrSGaL66b+7zwRmtvjbDc1uHxfjuiRBzV5Twjce+IBuGWk8comGWUQ6k34pKjHz2tItXPDHd+nZrStPJMgOUJFkopNzSUw8NXs91z69gLH987n/4iPok5sZdkkiKUeBLu3i7vz2leXc8epyjh7Rm3unFpKrCzuLhELvPNlnNfWNXPPUAmbM38hZEwbxv/9vPBnpGsUTCYsCXfbJ1vIaLn1oDrPX7OCaE0dz6RcOwEwn2hIJkwJd9trsNTu49KHZlNc08IcLDufk8f3DLklEUKDLXnB3Hnp3LTf9YzEDemTzl29MZEx/HWMuEi8U6BKVnTX1/PiZhTy3YBPHju7Lb885jO7ddMZEkXiiQJc9WlnayE/vfIuNpTVcfUIwXq4LU4jEHwW6tKmhsYm7iz7mt+/W0K97Nk98ezIThvYKuywRaYMCXVq1YmsFVz4xj/nry5jUL43p0z6vIRaROKdAl0+pb2zivrdWcscry+mWkcZd5x9OTskyhblIAlCgyyfmrN3Bj59ZyNLN5Zx4UD9u+spB7JeXRVHRsrBLE5EoKNCF4opabntpGY+9v45++Vnc97VCjh9bEHZZIrKXFOgprK6hiQffWc0dry6nuq6Rbx49nCuOH6VzsYgkKL1zU1BTk/OPBRu57aWPWFtSxRdG9eWnp45lxH65YZcmIu2gQE8h7s7ry7by6xc/4sNNOzmwXx73X3QEU0b31XlYRJKAAj0FuDsvf7iFO19bzqINOxncK5vbzzmEMw4ZqB8IiSQRBXoSq21o5O/zNvKnt1axbEs5Q3t349azDub/HTaQrmk6za1IslGgJ6Gt5TU8/t46Hpy1hm3ltRzYL4/fnH0Ipx8ygHQFuUjSUqAniaYmZ9aq7Tzy7lpeWLSZhibnmFF9uf3s/Tl6RG+NkYukAAV6gltXUsUzczbw1Jx1rCupJj8rnYuOGsYFk4fqIs0iKUaBnoC2ltcwc8EmZszfyJy1pQAcPaI3Vx4/mhMO6kd2RlrIFYpIGBToCWLN9kpeWryFFxdvZvbaHbjDgf3yuPqE0Zx+yAAG9+oWdokiEjIFepyqqW/kg9U7KFq2ldeWbWXltkoAxvTP5/IvjeTk8f0ZVZAXcpUiEk8U6HGitqGRhevLeHdVCW+vKOaDNTuoa2giI70Lk/fvzYWThnLcmAKG9FZPXERap0APydbyGuatLWXuulLmrNnBvHWl1DY0AcFQytTJQzl6RG8m79+bbhl6mkRkz5QUHczd2VBazdJN5Xy4aScLN5SxaEMZm8pqAEjvYowdkM+Fk4dyxLBeHDGsJ71zM0OuWkQSkQI9Rhoam9hQWs3K4ko+3lrBiq0VLN9awUdbyimvafhkvv375jBxeC/GD+zOYUN6cNCA7mR11VEpItJ+CvQouTsllXWsLmvkhUWbWL+jmnUlVawpqWLt9irW7aiivtE/mb93TgYj9svljEMHcGC/fMb0z2d0vzydmlZEOkzKp0tDYxM7quopqayjuKKWbeW1FFfUsrW8li07a9hcVsPmnTVsKquhLjLGzTtzAMjNTGdIr26M7pfHlw/qx/59c9i/Tw7D++Ro2EREOl1UgW5mJwJ3AGnAH939lhbTM4EHgQnAduAcd18d21Jb5+7UNjRRUdtAZW0D5TXBv4raBnZW17Ozpp6d1Q2UVtdRVl1PWVU9O6rqKN31f3U97p9dbmZ6FwrysyjIz2T8wO6ccFA/+uVnsWPDx5zwuSMY1DOb7tld9ZN6EYkbewx0M0sD7gKOB9YD75vZDHf/sNls3wR2uPsIMzsX+D/gnI4o+PH313Lvmyupqm2ksq6BqrpGGptaSeQWcjPT6Z7dle7ZXemZ05UBPbLp2S2DXjkZ9M4N/u+Tm0nfvEz65GaSn5XealgXFa1h3MDuHdE0EZF2iaaHPhFY4e4rAczsMeAMoHmgnwHcGLn9FPB7MzP31vq+7dMrJ5Ox/fPJyUinW2Ya3TLSyMlMJzcznZyMdPKy0snNSicvsyv52enkZ3UlLytdZxkUkaRne8pcMzsLONHdvxW5PxWY5O6XNZtnUWSe9ZH7H0fmKW6xrGnANICCgoIJjz32WCzb0ikqKirIzU2tS7WlWptTrb2gNieSY489dra7F7Y2LZoeemuDxC0/BaKZB3efDkwHKCws9ClTpkSx+vhSVFREItbdHqnW5lRrL6jNySKacYj1wOBm9wcBG9uax8zSge5ASSwKFBGR6EQT6O8DI81suJllAOcCM1rMMwP4euT2WcBrHTF+LiIibdvjkIu7N5jZZcCLBIct/tndF5vZTcAH7j4D+BPwVzNbQdAzP7cjixYRkc+K6jh0d58JzGzxtxua3a4B/iu2pYmIyN7QsXwiIklCgS4ikiQU6CIiSWKPPyzqsBWbbQPWhLLy9ukDFO9xruSSam1OtfaC2pxIhrp739YmhBboicrMPmjrV1rJKtXanGrtBbU5WWjIRUQkSSjQRUSShAJ9700Pu4AQpFqbU629oDYnBY2hi4gkCfXQRUSShAJdRCRJKNDbwcyuMjM3sz5h19KRzOxXZrbUzBaY2d/MrEfYNXUUMzvRzJaZ2Qozuy7sejqamQ02s9fNbImZLTazy8OuqbOYWZqZzTWz58KuJVYU6PvIzAYTXGd1bdi1dIKXgXHufjDwEfCjkOvpEM2un3sSMBY4z8zGhltVh2sArnT3McBk4Hsp0OZdLgeWhF1ELCnQ993twDW0cmWmZOPuL7l7Q+TuLIKLnCSjT66f6+51wK7r5yYtd9/k7nMit8sJAm5guFV1PDMbBJwC/DHsWmJJgb4PzOx0YIO7zw+7lhB8A3g+7CI6yEBgXbP760mBcNvFzIYBhwHvhltJp/gtQYesKexCYimq86GnIjN7BejXyqTrgR8DX+7cijrW7trr7n+PzHM9wVf0hzuztk4U1bVxk5GZ5QJPA1e4+86w6+lIZnYqsNXdZ5vZlLDriSUFehvc/bjW/m5m44HhwHwzg2D4YY6ZTXT3zZ1YYky11d5dzOzrwKnAl5L48oLRXD836ZhZV4Iwf9jdnwm7nk5wNHC6mZ0MZAH5ZvaQu18Ycl3tph8WtZOZrQYK3T0Rz9oWFTM7EfgN8AV33xZ2PR0lcoHzj4AvARsIrqd7vrsvDrWwDmRBr+QvQIm7XxF2PZ0t0kO/yt1PDbuWWNAYukTj90Ae8LKZzTOze8IuqCNEdvzuun7uEuCJZA7ziKOBqcAXI8/tvEjPVRKQeugiIklCPXQRkSShQBcRSRIKdBGRJKFAFxFJEgp0EZEkoUAXEUkSCnQRkSTx/wFcMFWz7ygLiwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "def logistic(x):\n",
    "    return (1/(1+np.exp(-x)))\n",
    "x = np.linspace(-5,5,100)\n",
    "plt.title('Graph of the logistic activation function')\n",
    "plt.plot(x,logistic(x) )\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Función de activacion Tangente hiperbólica"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La activación que funciona casi siempre mejor que la función sigmoidea es la función *tanh* también conocida como función tangente hiperbólica . En realidad, es una versión matemáticamente modificada de la función sigmoidea. Ambos son similares y pueden obtenerse una de la otra entre sí.\n",
    "\n",
    "\n",
    "La función *Tanh* está definida por\n",
    "\n",
    "$$\n",
    "f(x) = tanh(x) = \\frac{2}{{1+ e^{-2x}}}-1,\n",
    "$$\n",
    "para  $x\\in \\mathcal{R}$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ejercicio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verifique that\n",
    "\n",
    "1. $-1 \\le f(x)\\le 1$.\n",
    "2. $f'(x)= 1 -f(x)^2$.\n",
    "3. $tanh(x) = 2 sigmoid(2x) -1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tanh se usa generalmente en capas ocultas de una red neuronal, ya que sus valores se encuentran entre -1 a 1, por lo que la media de la capa oculta resulta ser 0 o muy cerca de ella,  lo que ayuda a centrar los datos al acercar la media a 0. Esto facilita mucho el aprendizaje de la siguiente capa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEICAYAAABS0fM3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZwV9ZX//9fplaVR9gbZVZDFBaWDWxIBRXGJmEQTdXQw32RIvo7J5BtNoskkYzTOz0wyWSYxi2McnWhETQZllIiotNFEFFRkFdlEmqXZoRvo7d7z+6OqzeVye4F7u6tv3/fz8biPW8unqk7Vrapz61ObuTsiIpK78qIOQEREoqVEICKS45QIRERynBKBiEiOUyIQEclxSgQiIjlOiaAdmdmdZvZIhsZVamZ/NrMqM/v3Vg7zvpldlInpZ5NMLvd0mVm5mX3hGIf98Pczs2+Z2QOZje6o46k2sxPbYLwfM7PVmR5vOO5PmtmmMPYz22IaTUz378zs+faa3tHK6URgZtea2etmdsDMtofNN5uZRR1bK8wEdgLHufutyT3N7CEz+35bTdzM3MxObqvxH0Ucw8NYCqKOpT25+7+6+zEllGORKoG5e4m7r8/AuA9bl9z9FXc/Jd3xNuFHwC1h7G+3xQRSrZPu/qi7X9wW08uEnE0EZnYr8DPgh8AAoBT4EnA+UNTEMPntFmDLhgErXXcEZgUL5Oz21oEMA1ZEHUSH4+459wGOBw4An26h3EPAr4C5YfmLgMuBt4H9wCbgzoTywwEn+Le+BdgK3JrQ/07gCeC/gSqCFbKsmemfBywC9oXf5yXEVQ/UAdXARUnDzUzq/79h9/eB24Cl4TgfB7okDHcFsATYC/wVOL2JuP4czueBcPyfBXoBzwA7gD1h8+CEYcqBu4G/hPP+PNA3of/fAxuBXcB3wlgvCvvlAbcD68L+TwC9w34fhLFUh59zU8Tb5HIHvg78Man8z4GfJsT9/wFvhMvs6cZph/3PCZfVXuAdYFLSPN8TzvMh4ORWjO/KML69YdkxCf0Sl8mdwCMJ/T6aEMcm4KYmfrvPAavC5bAe+GJS/+nhOrA/XN7TwnmIATXhMv5FWNbDeToH2AbkJ4znk8DSsHki8FoY21bgF0BRM+vSJKAiYVxjwmWxN1w2VyZto/cBz4bz9DpwUor5Lg7H3zitdYnzkDS+74fNk4AK4FZgexj75xLKdgX+nWC93Qe8GnY7Yp0EbgJebWnbbs220ib7xLYceUf9hCt3A1DQQrmHwh/qfIKdUZdw5TgtbD8dqASuCssPD1eAx4DuYbkdHL7x1gCXAfkEO4SFTUy7N8EO9UagALgubO+TvMI2E/v3k7q9T7ADOiEc/yrgS2G/s8KV/ewwthlh+eImxp+8AfUBPg10A3oATwJPJa3c64BR4cZSDtwb9hsbbjAfJTga+xFBImtcbl8FFgKDCTbo3wCPJS3zJn/L5pY7MJBgx9AzbC8Il8OEhLg3A6eGv+kfCXfAwCCCxHRZuD5MDdv7JQz7ATAuHG9hC+MbFcYyNSz7DWAtf9tpvk+KRAAMJdhhXBcO1wcY38SyuBw4CTDgAuAgcFbYbyLB+j41nJ9BwOiEeflCU+tA+NtOTej3JHB72DyBIFkUhL/XKuCrzaxLkwgTQTg/a4FvEawbU8J5PSVhPd8dxl4APArMamZdSJ5WS4mgAbgrjOOycHn1CvvfFy6XQQTr1XkE6+dwktZJEhIBLW/b5TSxrbTZPrEtR95RP8ANwLakbo3/pg4BH09YKf67hXH9FPhJ2Ny4AoxO6P9vwG/D5juBFxL6jQUONTHeG4E3krq9RvhPj2NPBDckxfbrsPlXwN1J5VcDFzQx/sM2oBT9xwN7EtrLgX9OaL8ZeC5s/i7hjj1s70ZwNNO401sFXJjQfyBBomjcsbQmETS53IE/Af8QNl9BUOWWGPe9ScPWEWz43wR+lzStecCMhGHvSurf3Pi+AzyR0C+PIGlMSvj9UiWCO4DZx7gtPAX8U9j8G8J1OUW5cppPBN8HHgybexAktGFNjOurifEmr0scngg+RnC0kZfQ/zHCI3GC9fyBhH6XAe82M79HmwgOcfgOfTtBUssL+52RYhpHrJMcngha2rbLaWJbaatPrtZZ7gL6Jp3MOc/de4b9EpfLpsQBzexsM1tgZjvMbB/BeYW+SeNPHGYjwT/wRtsSmg8CXZo40XlCOGyijQT/PtKRPP2SsHkYcKuZ7W38AEM4PPYmmVk3M/uNmW00s/0Eh/w9k86rNDXtE0hYZu5+kOB3aDQMmJ0Q1yqCqorS1sTWxLQTl/vDBH8OCL9/lzRs8u9ZSPCbDwOuSVpmHyVIVKmGbWl8h/3m7h4Py7b0mw8h+AfZIjO71MwWmtnuMN7L+Nv62+rxpPB74FNmVgx8CnjL3TeG0xxlZs+Y2bZw3fhXjtxmmnICsClcFo2St4Om1qtM2OXuDSnG35eghuBYlldrtu22nKcj5GoieA2oJagPbYkntf8emAMMcffjgV8THGYnGpLQPJTgfMHR2kKwo0k0lOAfYmskx92STcA97t4z4dPN3R9r5fC3AqcAZ7v7ccDHw+6tuQJrK0G1TzCAWVeC6o3E2C5Niq2Lu2/m6OczlaeA083sVIIjgkeT+if/nvUEV2xtIjgiSIyru7vfm1A+VXxNje+w3zy8em0ILf/mmwiqe5oV7qT/SFD1Vhr+8ZnL336j5sbT7HJ295UEO7NLgesJtpNGvwLeBUaG68a3aN16AcEyGZJ0ov1otoOWHCQ4Am00oJXD7SSobky1vFpaJ9PdtjMuJxOBu+8Fvgf80syuNrMSM8szs/EE9bbN6QHsdvcaM5tIsNIn+074D3kcwcm5x48hzLnAKDO73swKzOyzBNUIz7Ry+ErgaK7x/k/gS+ERj5lZdzO73Mx6tHL8PQgOlfeaWW/gX45i2n8APmFm55lZEcFvk7ij+DVwj5kNAzCzfmbWmMR3AHGObl4P4+41YQy/Jzhk/yCpyA1mNtbMuhHUF//B3WPAI2Hcl5hZvpl1MbNJZjaY5jU1vieAy83sQjMrJEiutQTVls15FLjIzD4Trit9wnU5WRFBHfYOoMHMLgUSL2n8LfC5cPp5ZjbIzEaH/VqzPv0e+ArBn4AnE7r3IDj5XB2O7/8mDdfcuF8nqGb6hpkVmtkk4BPArBZiaa0lwPXh7zeN4LxJi8IjlAeBH5vZCeHw54bJtqV1Mt1tO+NyMhEAuPu/AV8jOCG3nWBl/A1BvW9zG97NwF1mVkVQt/1EijIvE5zgehH4kbsf9Y0k7r6L4N/prQTVJN8ArnD3na0cxW+BsWGVxVOtmN5i4B8IrujYE8Z/UzOD3Ak8HI7/MwTnSroS/FNaCDzXyjhx9xXAlwk27q0EJwO3E+wEIbjMdw7wfLjcFxKc1G6sRroH+EsYyzmtnW6ShwlO7idXCxF2e4jgcL0Lwc4Od99EcFT5LYKNfxPBVUgtbVdNjW81QdXUzwmW4yeAT7h7XXMjCxPXZQTrym6CndsZKcpVhdN6guA3vp5guTb2f4Pgj8tPCE4av8zf/rn+DLjazPaY2X80EcpjBPXqLyWtp7eF06oi+MOR/MfoTg5flxJjriO4kupSgmXyS+Dv3f3dJmI4Wv9EsJz3An9HcHTYWrcBywiu+tkN/IDgXEaz62QGtu2Ms/BkhGSAmQ0HNgCFSfWKchTMrIRgwxzp7hvaaZpDCaovBrj7/oTu5QQnZSO9i1ekLeXsEYF0LGb2ibA6rTtBHfYygqtk2mPaeQRHh7MSk4BIrsip2/KlQ5tOUGViwGLgWm+Hw9Uw8VQSnOic1tbTE+mIVDUkIpLjVDUkIpLjsrJqqG/fvj58+PCowzgqBw4coHv3lq5M7Vw0z7lB85w93nzzzZ3u3i+5e1YmguHDh7N48eKowzgq5eXlTJo0Keow2pXmOTdonrOHmSXf0QyoakhEJOcpEYiI5DglAhGRHKdEICKS45QIRERyXEYSgZk9aMHL35c30d/M7D/MbK2ZLTWzsxL6zTCzNeFnRibiERGR1svUEcFDNH97/qXAyPAzk+D55CQ8rvhsglfN/YuZ9cpQTCIi0goZuY/A3f8cPnmzKdMJXvnowEIz62lmAwkeWTvf3XcDmNl8goTS2pehiEgOcnfqYnFqG+LUJXzqY3HqYnEaYk59LE5D3InFg+ZY2Bx3/7C7Ox92Cz58+B28xvHwdgja126o411bR+MTehwn8Wk9jWXd//aWmsSyJHU7Yv4On9nD+s04bzh9SoqPccml1l43lA3i8NfzVYTdmup+BDObSXA0QWlpKeXl5W0SaFuprq7OupjTpXnODccyz/VxZ1+ts7/O2V/rVNc7B+rhQL1zoN451ACHGpyaBqemAWpjTm0M6mJObRzqY5l5NV1aVmfqlQgtS3xL04C6zZxQktnTu+2VCFK9ls6b6X5kR/f7gfsBysrKPNvu6svWOxHToXnODanmubYhxsZdB1m/4wDv7zpAxZ6DbN5ziC17a6isqmHvwfqU4zKD47oU0qNLASXFBZT0KKB/cQHdi/PpWlhAt6J8uhbl06Ugj+LCfIoL8igqyKMoP/guzG/8GIX5eRTkGQX5eeTnGQV5Rn7CJ8+MPONvzXlhuxkYGI3lgmaMoNmMV195hY9//GMYhtnh8QOHdTeCYRqbE8uR0C9K7ZUIKjj8Pa2DCd7bWUFQPZTYvbydYhKRDDhQ77z83g6WVexl1dYqVm3bz/s7DxBP+Et3fNdCBvXsypDe3Zg4ojelxxXTv0cX+pQU0at7EX26F9GzaxE9uhSQlxf9jrElXQqMbkVZ+YSelNprTuYAt5jZLIITw/vcfauZzQP+NeEE8cXAHe0Uk4gcg30H63lt/U5eWbOTNzbsZs32g8AbAAzt3Y0xA3twxWkDOal/CSP6dmd43+4c16Uw2qClWRlJBGbW+K7SvmZWQXAlUCGAu/+a4GXNlxG8B/cgwXtRcffdZnY3wTs/Ae5qPHEsIh3H5r2HeG75Np5bvpU3N+4h7tC9KJ+PjOjNacfX8ukLzuL0wcfTQzv8rJSpq4aua6G/A//YRL8HgQczEYeIZM7BugaeXbqVxxdtYvHGPQCMHtCDW6aM5GMj+zJ+SE8K8/MoLy/n/JP7RhytpKPzVHKJSEZU7DnIA69s4I9vVlBV28CJ/brz9UtO4fLTBjK8b/Y9g19apkQgIgCs21HNLxes4+klmwG44vSBXDdxKBNH9O4QV7ZI21EiEMlxew7U8dMX3uOR1z+gMN+48dxh/MPHTuSEnl2jDk3aiRKBSI6Kx53fLdzIj+e/R1VNPdefPZSvXjSKvhm+a1U6PiUCkRy0afdBbnvyHV7fsJuPntyX71wxllMG9Ig6LImIEoFIDnF3nnyzgu/NWYGZ8W9Xn841EwbrHECOUyIQyRF1DXH+Zc4KHnvjA845sTc/uuYMBvfqFnVY0gEoEYjkgJ3Vtdz8yFu88f5ubp50ErdefAr5WfAoB2kfSgQindyGnQe44YHX2Vldy8+uHc/08Skf8Cs5TIlApBNbu72K6/7zdWJx58kvncvpg3tGHZJ0QEoEIp3Uqq37ueGB18nLMx6feQ4jS3VVkKSml9eLdELBkcBCCvPzlASkRToiEOlktlfVMOPBRRTk5fH4F89hWB89H0iapyMCkU7kQG0Dn39oMbsP1PFfN31ESUBaRYlApJOIxZ2vPPY2K7bs4xfXn8lpg4+POiTJEqoaEukkfvbCe7z47nbuvupULhxTGnU4kkUyckRgZtPMbLWZrTWz21P0/4mZLQk/75nZ3oR+sYR+czIRj0iueXXNTn6+YC3XTBjMjecMizocyTJpHxGYWT5wHzCV4GX0i8xsjruvbCzj7v8vofyXgTMTRnHI3cenG4dIrtpeVcNXH1/CSf1K+N70cVGHI1koE0cEE4G17r7e3euAWcD0ZspfBzyWgemK5LxY3PnqrCVU1dRz3/Vn0a1Itb1y9Cx4nXAaIzC7Gpjm7l8I228Eznb3W1KUHQYsBAa7eyzs1gAsARqAe939qSamMxOYCVBaWjph1qxZacXd3qqrqykpKYk6jHaleW57z22oZ9bqOj43rogLhkTz4nj9ztlj8uTJb7p7WXL3TPx9SPXkqqayy7XAHxqTQGiou28xsxOBl8xsmbuvO2KE7vcD9wOUlZX5pEmT0gy7fZWXl5NtMadL89y2Nu0+yNMv/pkLR/fnuzeURfYoaf3O2S8TVUMVwJCE9sHAlibKXktStZC7bwm/1wPlHH7+QERScHe+NXsZeQZ3X3Wq3icgaclEIlgEjDSzEWZWRLCzP+LqHzM7BegFvJbQrZeZFYfNfYHzgZXJw4rI4Wa/vZlX1uzkm5eO1ruFJW1pVw25e4OZ3QLMA/KBB919hZndBSx298akcB0wyw8/KTEG+I2ZxQmS0r2JVxuJyJF2Vddy9zMrOWtoT244W5eKSvoycomBu88F5iZ1+25S+50phvsrcFomYhDJFT96/j2qahr4wadPJ08vl5EM0CMmRLLIe5VVPL7oA248d5ieKCoZo0QgkkX+de4qSooL+MqUkVGHIp2IEoFIlnhlzQ7KV+/gy1NG0qt7UdThSCeiRCCSBWJx555nVzG4V1f+/jydIJbMUiIQyQKz397Mu9uq+Oa00RQX5EcdjnQySgQiHVxDLM7PX1rDuBOO44rTB0YdjnRCSgQiHdz/Lt3Cxl0H+cqFI3UHsbQJJQKRDiwWd37+0lpGD+jBVL1sRtqIEoFIBzZ32VbW7zjALVNO1s1j0maUCEQ6qHjc+cVLazmpX3cuPVXnBqTtKBGIdFDPr6xkdWUVX54yknwdDUgbUiIQ6aD+85X1DO3dTVcKSZtTIhDpgN7ZtJc3N+7hpvOGU5CvzVTaltYwkQ7ov/6ygZLiAq4pGxx1KJIDlAhEOpjK/TU8s3Qr15QNpkeXaN5DLLklI4nAzKaZ2WozW2tmt6fof5OZ7TCzJeHnCwn9ZpjZmvAzIxPxiGSzRxZuJObOTecNjzoUyRFpv5jGzPKB+4CpBO8vXmRmc1K8aexxd78ladjewL8AZQQvvH8zHHZPunGJZKOa+hiPvv4BF44uZVif7lGHIzkiE0cEE4G17r7e3euAWcD0Vg57CTDf3XeHO//5wLQMxCSSleYs2cLuA3X8n/OHRx2K5JBMJIJBwKaE9oqwW7JPm9lSM/uDmQ05ymFFcsIjr29kVGkJ557UJ+pQJIdk4p3Fqe508aT2/wUec/daM/sS8DAwpZXDBhMxmwnMBCgtLaW8vPyYA45CdXV11sWcLs3z0dm4P8bSihr+bnQRL7/8cmYDa0P6nbNfJhJBBTAkoX0wsCWxgLvvSmj9T+AHCcNOShq2PNVE3P1+4H6AsrIynzRpUqpiHVZ5eTnZFnO6NM9H5ztPLaeoYBNf/8wF9OyWPW8g0++c/TJRNbQIGGlmI8ysCLgWmJNYwMwSb428ElgVNs8DLjazXmbWC7g47CaSUw7VxXhqyWYuO3VAViUB6RzSPiJw9wYzu4VgB54PPOjuK8zsLmCxu88BvmJmVwINwG7gpnDY3WZ2N0EyAbjL3XenG5NItpm7bCtVNQ1cO3Fo1KFIDspE1RDuPheYm9TtuwnNdwB3NDHsg8CDmYhDJFvNWvQBI/p25+wRvaMORXKQ7iwWidja7VUsen8Pn/3IEL2BTCKhRCASsccXbaIgz/j0WXqukERDiUAkQg2xOLPf3sKFY/rTr0dx1OFIjlIiEInQX9btYmd1LZ88U0cDEh0lApEIzX6rguO6FDB5dL+oQ5EcpkQgEpEDtQ3MW1HJ5aefQHFBftThSA5TIhCJyLwV2zhUH+NTZ+nxWhItJQKRiMx+ezODe3VlwtBeUYciOU6JQCQC2/fX8Je1O/nkmYPIy9O9AxItJQKRCMx5Zwtxh+njVS0k0VMiEInAU0s2c9qg4zm5f0nUoYgoEYi0t/d3HmD55v1cecYJUYciAigRiLS7Z5dtBeCy0we2UFKkfSgRiLSzZ5Zu5ayhPRnUs2vUoYgASgQi7WrdjmpWbd3P5aerWkg6DiUCkXb07NKgWujy01QtJB1HRhKBmU0zs9VmttbMbk/R/2tmttLMlprZi2Y2LKFfzMyWhJ85ycOKdCbPLt3KR4b3YsDxXaIOReRDaScCM8sH7gMuBcYC15nZ2KRibwNl7n468Afg3xL6HXL38eHnynTjEemo1lRWsbqyiitULSQdTCaOCCYCa919vbvXAbOA6YkF3H2Bux8MWxcCeuau5Jxnlm7FDC49dUDUoYgcJhPvLB4EbEporwDObqb854E/JbR3MbPFBC+2v9fdn0o1kJnNBGYClJaWUl5enk7M7a66ujrrYk6X5vlwTy48yKieeax8ayEr2zesNqXfOftlIhGkelCKpyxodgNQBlyQ0Hmou28xsxOBl8xsmbuvO2KE7vcD9wOUlZX5pEmT0g68PZWXl5NtMadL8/w3a7dXseW5P/O9C8cy6bzh7R5XW9LvnP0yUTVUAQxJaB8MbEkuZGYXAd8GrnT32sbu7r4l/F4PlANnZiAmkQ5l3opKAC4Zp2oh6XgykQgWASPNbISZFQHXAodd/WNmZwK/IUgC2xO69zKz4rC5L3A+dKqjZhEAnlu+jTOH9tTVQtIhpZ0I3L0BuAWYB6wCnnD3FWZ2l5k1XgX0Q6AEeDLpMtExwGIzewdYQHCOQIlAOpWKPQdZtnkf03Q0IB1UJs4R4O5zgblJ3b6b0HxRE8P9FTgtEzGIdFSqFpKOTncWi7Sx55ZvZfSAHgzv2z3qUERSUiIQaUPbq2pYvHEPl56qR0pIx6VEINKG5q+sxB2m6SYy6cCUCETa0HPLtzGib3dGlepNZNJxKRGItJF9h+p5bd0uLh5XipleUC8dlxKBSBspX72dhrjraiHp8JQIRNrI8ysr6dejmPGDe0YdikizlAhE2kBtQ4zyd7dz0ZhS8vJULSQdmxKBSBv467pdHKiLcfHY0qhDEWmREoFIG5i/spLuRfmce1KfqEMRaZESgUiGxePO/JWVTDqlP10K86MOR6RFSgQiGbakYi87qmqZqmohyRJKBCIZNn9lJQV5xuRT+kcdikirKBGIZNjzK7Zxzol9OL5bYdShiLSKEoFIBm2tjrNuxwFVC0lWUSIQyaC3dzQAcJESgWSRjCQCM5tmZqvNbK2Z3Z6if7GZPR72f93Mhif0uyPsvtrMLslEPCJRebsyxrgTjmNQz65RhyLSamknAjPLB+4DLgXGAteZ2dikYp8H9rj7ycBPgB+Ew44leMfxOGAa8MtwfCJZZ2d1LWv3xlUtJFknE0cEE4G17r7e3euAWcD0pDLTgYfD5j8AF1rwOMbpwCx3r3X3DcDacHwiWeelVdtxUCKQrJOJdxYPAjYltFcAZzdVxt0bzGwf0CfsvjBp2EGpJmJmM4GZAKWlpZSXl2cg9PZTXV2ddTGnK9fm+bG3auhd7Gxf/RY73sud5wvl2u8MnW+eM5EIUq3x3soyrRk26Oh+P3A/QFlZmU+aNOkoQoxeeXk52RZzunJpng/VxVj14vN8dGAhkydPjjqcdpVLv3OjzjbPmagaqgCGJLQPBrY0VcbMCoDjgd2tHFakw3tlzQ5q6uOcVZqJ/1Yi7SsTiWARMNLMRphZEcHJ3zlJZeYAM8Lmq4GX3N3D7teGVxWNAEYCb2QgJpF2NX9lJcd1KWBUL12RLdkn7b8vYZ3/LcA8IB940N1XmNldwGJ3nwP8Fvidma0lOBK4Nhx2hZk9AawEGoB/dPdYujGJtKdY3Hnp3e1MHt2fgrx9UYcjctQychzr7nOBuUndvpvQXANc08Sw9wD3ZCIOkSi89cEedh2oC64W2q1EINlHx7EiaZq/spKi/DwuGNUv6lBEjokSgUga3J15K7Zx7kl96NFFD5mT7KREIJKGNdur2bjrIBeP001kkr2UCETS8PyKbQBMHaNEINlLiUAkDfNXVjJ+SE/6H9cl6lBEjpkSgcgx2rrvEO9U7FO1kGQ9JQKRY/TCykoALh47IOJIRNKjRCByjJ5fWcmJfbtzcv+SqEMRSYsSgcgx2F9Tz8L1u5iqaiHpBJQIRI7BS6u2Ux9zVQtJp6BEIHIMnlu+jdLjijlzSM+oQxFJmxKByFE6VBej/L3tXDJuAHl5ufMCGum8lAhEjtLL7wXvHpg2TtVC0jkoEYgcpXkrttGrWyETR/SOOhSRjFAiEDkKdQ1xXlhVydSxpRTka/ORzkFrsshR+Ou6nVTVNDDtVFULSeeRViIws95mNt/M1oTfvVKUGW9mr5nZCjNbamafTej3kJltMLMl4Wd8OvGItLV5K7ZRUlzAeSf1jToUkYxJ94jgduBFdx8JvBi2JzsI/L27jwOmAT81s8Rr7r7u7uPDz5I04xFpM7G48/yKSiaP7k+XwvyowxHJmHQTwXTg4bD5YeCq5ALu/p67rwmbtwDbAb3KSbLOovd3s+tAna4Wkk7H3P3YBzbb6+49E9r3uPsR1UMJ/ScSJIxx7h43s4eAc4FawiMKd69tYtiZwEyA0tLSCbNmzTrmuKNQXV1NSUluPZOms83zf6+s5dXNDfx8cjeKC1LfP9DZ5rk1NM/ZY/LkyW+6e9kRPdy92Q/wArA8xWc6sDep7J5mxjMQWA2ck9TNgGKCBPHdluJxdyZMmODZZsGCBVGH0O460zzXN8R8wt3P+82Pvtlsuc40z62lec4ewGJPsU8taCmDuPtFTfUzs0ozG+juW81sIEG1T6pyxwHPAv/s7gsTxr01bKw1s/8CbmspHpEovLFhNzur67jitIFRhyKScemeI5gDzAibZwBPJxcwsyJgNvDf7v5kUr+B4bcRnF9YnmY8Im3imWVb6VaUz6RT+kcdikjGpZsI7gWmmtkaYGrYjpmVmdkDYZnPAB8HbkpxmeijZrYMWAb0Bb6fZjwiGdcQi/Pc8m1cNKaUrkW6Wkg6nxarhprj7ruAC1N0Xwx8IWx+BHikieGnpDN9kfbw2vpd7D5Qx+Wnq1pIOifdWSzSgmeXbqWkuIALRumqZ+mclAhEmlEfi/Pcim1MHVuqm8ik01IiEGnGq2t3svdgPR+kJq4AAA6YSURBVJfpaiHpxJQIRJrx1Nub6dmtUNVC0qkpEYg0obq2gXkrtnH5aQMpKtCmIp2X1m6RJsxbvo2a+jifOmtQ1KGItCklApEmzH57M0N6d+WsoU0+PkukU1AiEEmhcn8Nf1m3k0+OH0Rw47tI56VEIJLC00s24w5XnalqIen8lAhEUpj99hbOGNKTE/tl36OGRY6WEoFIkne37WfV1v18SkcDkiOUCESSPLGogsJ84wo9W0hyhBKBSILahhj/83YFU8eW0qekOOpwRNqFEoFIgnkrKtl7sJ5rPzI06lBE2o0SgUiCWW98wOBeXfnoyX2jDkWk3aSVCMyst5nNN7M14XfKO2/MLJbwUpo5Cd1HmNnr4fCPh28zE4nExl0H+Ou6XXy2bAh5ebp3QHJHukcEtwMvuvtI4MWwPZVD7j4+/FyZ0P0HwE/C4fcAn08zHpFjNmvRJvIMrikbEnUoIu0q3UQwHXg4bH6Y4L3DrRK+p3gK8IdjGV4kk+pjcf7wZgVTRvdnwPFdog5HpF2lmwhK3X0rQPjd1Ju9u5jZYjNbaGaNO/s+wF53bwjbKwBduC2ReHHVdnZU1fJZnSSWHNTiO4vN7AVgQIpe3z6K6Qx19y1mdiLwUvjC+v0pynkzccwEZgKUlpZSXl5+FJOPXnV1ddbFnK5smuefvXGI3l2MvG0rKd++6pjHk03znCma507A3Y/5A6wGBobNA4HVrRjmIeBqwICdQEHY/VxgXmumO2HCBM82CxYsiDqEdpct87xyyz4f9s1n/Ffla9MeV7bMcyZpnrMHsNhT7FPTrRqaA8wIm2cATycXMLNeZlYcNvcFzgdWhkEtCJNCk8OLtLWH/vI+XQrzuPYjOkksuSndRHAvMNXM1gBTw3bMrMzMHgjLjAEWm9k7BDv+e919Zdjvm8DXzGwtwTmD36YZj8hR2VVdy+wlm/n0WYPp2U1XL0tuavEcQXPcfRdwYYrui4EvhM1/BU5rYvj1wMR0YhBJx2NvfEBdQ5zPnT886lBEIqM7iyVn1TXE+d3CjXx8VD9O7t8j6nBEIqNEIDnrT8u3Urm/VkcDkvOUCCQnxePOLxesY2T/Ei4Y2S/qcEQipUQgOen5lZWsrqzilikn67lCkvOUCCTnuDs/f2kNI/p254rTT4g6HJHIKRFIznnp3e2s2LKfmyedRL6OBkSUCCS3uDv/8dJahvTuylV6J7EIoEQgOebPa3byzqa93DzpZArztfqLgBKB5JB43PnRvNUM6tmVT581OOpwRDoMJQLJGU+/s5llm/dx2yWjKCrQqi/SSFuD5ISa+hg/fG41pw06nuln6NyASCIlAskJv311A1v21fCty8bovgGRJEoE0untrK7lV+XruGhMKeee1CfqcEQ6HCUC6fT+/fn3OFQf447LRkcdikiHpEQgndqi93fz2BsfcNN5wzmpX0nU4Yh0SEoE0mnV1Me4/Y9LGdyrK7dePCrqcEQ6rLQSgZn1NrP5ZrYm/O6VosxkM1uS8Kkxs6vCfg+Z2YaEfuPTiUck0S8XrGXdjgPc88nT6FaU1juYRDq1dI8IbgdedPeRwIth+2HcfYG7j3f38cAU4CDwfEKRrzf2d/clacYjAsC72/bzy/J1fOrMQVwwSo+ZFmlOuolgOvBw2PwwcFUL5a8G/uTuB9OcrkiTahtifP3JpRzXtZB/vmJs1OGIdHjm7sc+sNled++Z0L7H3Y+oHkro/xLwY3d/Jmx/CDgXqCU8onD32iaGnQnMBCgtLZ0wa9asY447CtXV1ZSU5NbJyqjm+dFVtczf2MCXzyxmQmn7Vgnpd84N2TrPkydPftPdy47o4e7NfoAXgOUpPtOBvUll9zQznoHADqAwqZsBxQRHFN9tKR53Z8KECZ5tFixYEHUI7S6KeX5u+VYf9s1n/M45y9t92u76nXNFts4zsNhT7FNb/Lvk7hc11c/MKs1soLtvNbOBwPZmRvUZYLa71yeMe2vYWGtm/wXc1lI8Ik3ZtPsgX3/yHU4ffDx3XDom6nBEska65wjmADPC5hnA082UvQ54LLFDmDwwMyM4v7A8zXgkRx2qi/GPv38Ld/jFdWfpoXIiRyHdreVeYKqZrQGmhu2YWZmZPdBYyMyGA0OAl5OGf9TMlgHLgL7A99OMR3JQLO58ZdbbLNu8jx9/djxD+3SLOiSRrJLWmTR33wVcmKL7YuALCe3vA0c88tHdp6QzfRF35+5nVjJ/ZSXfu3IcU8eWRh2SSNbR8bNktd++uoGH/vo+X/joCGacNzzqcESykhKBZK0HX93A959dxWWnDeBbl+nksMix0n33kpV+/fI67v3Tu0wbN4CffvZMvWNAJA1KBJJV3J3/eHEtP3nhPT5xxgn8+DNn6CX0ImlSIpCsUVMf447/WcbstzfzqbMG8cOrzyBfRwIiaVMikKywbV8NX/zdYt6p2MetU0dxy5STCW4/EZF0KRFIh7fg3e18449LOVjbwG9unMAl4wZEHZJIp6JEIB1WdW0D9zy7ksfe2MQppT145PNnc8qAHlGHJdLpKBFIh+PuzFtRyfefXcnmvYf44gUn8rWpoyguyI86NJFOSYlAOpSVW/Zz9zMreW39Lkb2L+GJL57LR4b3jjoskU5NiUA6hGUV+/hl+VqeW7GNnl0LuXv6OK6bOJQCXRoq0uaUCCQyDbE45at38PBr7/PKmp306FLAzZNOYubHTuL4boVRhyeSM5QIpN2tqazi6SVbePLNTVTur6Vfj2K+Me0UbjhnGMd1UQIQaW9KBNLmGmJx3qnYy4J3d/Cn5VtZt+MAeQYXjOrHXdOHMmV0f90dLBIhJQLJuJr6GCu27ONPG+p5ZONiXl+/i6raBvIMzjmxDzedN5xLxg2g/3Fdog5VRFAikDTE4s6WvYdYv/MAayqrWLW1ine37ee9yirqYw7A0N5VXHHGCXz05L6cd1IfenUvijhqEUmWViIws2uAO4ExwMTwhTSpyk0DfgbkAw+4e+ObzEYAs4DewFvAje5el05Mkhl1DXF2H6hj14FadlTVsn1/Ldurati8t4bNew9RsecgFXsOUdcQ/3CYfj2KGTPwOD4/sh9nDe3JwYqVXHXJ5AjnQkRaI90jguXAp4DfNFXAzPKB+wheZVkBLDKzOe6+EvgB8BN3n2VmvwY+D/wqzZg6nXjcibkTi4cfd2IxpyHuNMTjNMSc+lic+vC7tiH+4XddQ5zahhg19XEO1ceorY9xsC7GgboGDtbGqK5toKqmgeraevYdamD/oXr2HaqnurYhZSy9uxcxqGdXTintwUVjSjmxb3dG9O3OSf1L6FtSfFjZ8h3vtsfiEZE0pfuqylVASw//mgisdff1YdlZwHQzWwVMAa4Pyz1McHTRZong27OX8fqG3R+2u3vKct5EiycM09jZHTxscw8+ieP3sHtNbS1Ff3nhw/bGfnF34nHHPWwOv935cOffFooK8uhWlE+PLgWUFBdSUpzPoJ5dGDOwB8d3LaR3tyJ6lxTRp3sR/XoU079HF/r1KKZLoe7uFels2uMcwSBgU0J7BXA20AfY6+4NCd2PeK9xIzObCcwEKC0tpby8/KgDqd1dR6+8+GHdWvP8ysQyiTnPEhosbEsu29je0DVOYWEMC8uY/a1/XvhtGGaQZ3kflslL/AB5eZCHkZ8H+WH3fIP8PKPAoCCv8WMU5hF88o2iPCjKN4ryoSiPpMc3x8JPUq3coeBTtROqgHWtWFaJqqurj+l3ymaa59zQ2ea5xURgZi8AqR73+G13f7oV00i1r/Vmuqfk7vcD9wOUlZX5pEmTWjHpwx3DIBlTXl7OscSczTTPuUHznP1aTATuflGa06gAhiS0Dwa2ADuBnmZWEB4VNHYXEZF21B538SwCRprZCDMrAq4F5nhQ2b4AuDosNwNozRGGiIhkUFqJwMw+aWYVwLnAs2Y2L+x+gpnNBQj/7d8CzANWAU+4+4pwFN8EvmZmawnOGfw2nXhEROTopXvV0GxgdoruW4DLEtrnAnNTlFtPcFWRiIhERA94ERHJcUoEIiI5TolARCTHKRGIiOQ4a+oxCx2Zme0ANkYdx1HqS3DvRC7RPOcGzXP2GObu/ZI7ZmUiyEZmttjdy6KOoz1pnnOD5jn7qWpIRCTHKRGIiOQ4JYL2c3/UAURA85wbNM9ZTucIRERynI4IRERynBKBiEiOUyKIgJndZmZuZn2jjqWtmdkPzexdM1tqZrPNrGfUMbUVM5tmZqvNbK2Z3R51PG3NzIaY2QIzW2VmK8zsn6KOqT2YWb6ZvW1mz0QdS6YoEbQzMxsCTAU+iDqWdjIfONXdTwfeA+6IOJ42YWb5wH3ApcBY4DozGxttVG2uAbjV3ccA5wD/mAPzDPBPBI/U7zSUCNrfT4Bv0MxrOTsTd38+4b3UCwneRNcZTQTWuvt6d68DZgHTI46pTbn7Vnd/K2yuItg5Nvne8c7AzAYDlwMPRB1LJikRtCMzuxLY7O7vRB1LRP4P8Keog2gjg4BNCe0VdPKdYiIzGw6cCbwebSRt7qcEf+TiUQeSSWm9mEaOZGYvAANS9Po28C3g4vaNqO01N8/u/nRY5tsEVQmPtmds7chSdMuJoz4zKwH+CHzV3fdHHU9bMbMrgO3u/qaZTYo6nkxSIsgwd78oVXczOw0YAbxjZhBUkbxlZhPdfVs7hphxTc1zIzObAVwBXOid98aVCmBIQvtgYEtEsbQbMyskSAKPuvv/RB1PGzsfuNLMLgO6AMeZ2SPufkPEcaVNN5RFxMzeB8rcPRufYNhqZjYN+DFwgbvviDqetmJmBQQnwy8ENgOLgOsT3s/d6Vjwj+ZhYLe7fzXqeNpTeERwm7tfEXUsmaBzBNLWfgH0AOab2RIz+3XUAbWF8IT4LcA8gpOmT3TmJBA6H7gRmBL+tkvCf8uSZXREICKS43REICKS45QIRERynBKBiEiOUyIQEclxSgQiIjlOiUBEJMcpEYiI5Lj/H+bokbiVeK5GAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "def tanh(x):\n",
    "    return (2/(1+np.exp(-2*x))-1)\n",
    "x = np.linspace(-5,5,100)\n",
    "plt.title('Graph of the tanget hyperbolic activation function')\n",
    "plt.plot(x,tanh(x) )\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Función de activación Unidad de rectificación lineal (ReLu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*ReLu* es la función de activación más utilizada actualmente. Implementada principalmente en capas ocultas de la red neuronal. *ReLu* se define como\n",
    "\n",
    "$$\n",
    "f(x) = ReLu(x) = max(0,x),\n",
    "$$\n",
    "para $x\\in \\mathcal{R}$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ejercicio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verifique que\n",
    "\n",
    "1. $0\\le f(x)< \\infty$\n",
    "2. $f'(x)= 1$ for $x > 0$, and  $f'(x)= 0$ for $x < 0$.\n",
    "\n",
    "*ReLu* es menos costosa computacionalmente que *Tanh* y *sigmoide* porque involucra operaciones matemáticas más simples. A la vez, solamente se activan unas pocas neuronas, lo que hace que la red sea dispersa, lo que la hace eficiente y fácil de calcular. En palabras simples, *ReLu* permite a la red aprender mucho más rápido que la funciones *sigmoide* y *Tanh*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAEICAYAAAB25L6yAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5wU9f3H8ddHOhwICBzSURBFkSqopIAlUVFjmt3EWFDsUaPRWNKMiYk19hh/UQEp9hA1MZGT2PWOXgSk917u4ODK5/fHDGY977i943Zny/v5eOzj9nZmZz7fmd3Pfvc7s58xd0dERFLXflEHICIie6dELSKS4pSoRURSnBK1iEiKU6IWEUlxStQiIilOiTpNmNkvzWx0HS0r18ymmNl2M7s3zucsMbMT6mL9+8LMLjSzd1MgjjfM7McJWnahmR2UgOX2MrOp4X6/pq6Xv5f1dgnbVC9Z68w0StS1ZGZnm9lHZlZkZuvC+1eYmUUdWxxGAhuAFu5+Q8WJZvY3M/ttolZuZh5ut0IzW2lm96Xym7iyD0l3P9ndn6mDZeeZ2SUVlp3j7ov2ddmVuAnIc/fm7v5QApYPfPVD3d2XhW0qS9Q6M50SdS2Y2Q3Ag8AfgfZALnA5MBRoWMVzUikRdQXmeLS/durr7jnAN4GzgIsijCVbdAVmRx2E1IK761aDG7A/UAR8v5r5/gY8Brwezn8CMAKYCmwDlgO/jJm/G+AEvd1VwGrghpjpvwQmAM8C2wnecIP2sv5jgU+AreHfY2PiKgF2A4XACRWeN7LC9L+Hjy8BbgRmhMscDzSOed6pwDRgC/A+cOReYnOgR8z/E4BHKmzjv4bbYCXwW6BeOO1C4N0K26x+zHPzgEuqWO9g4IMwxtXAw0DDmOmHA28Bm4C1wK3ASeG2KAm3x/TY9QCNwuUdEbOctsBOoB3QCpgErAc2h/c7hfPdBZQBxeGyH664fcJt8Wz4/KXAbcB+sdsC+FO47MXAyVW0/e0K6zqk4raK3bYxcVwOLAiX/whgMdMvBeYSvB7nAAOA54DysP2FBL34L+0noAPwWridFwKX1vZ1ni23yANIt1v4xi2NTQ5VzPc3goQ2lOCbS2NgGNAn/P/IMBmcEc6/58X8PNAsnG89YSINX8DFwClAPeBu4MMq1t06fGNdANQHzgn/PyAmtt9WE/tvKzy2BPg4fJO1Dt+gl4fTBgDrgCFhbD8O529UxfJjE9GhBEnzpzHTXwGeCLdDu3C9l4XTvkgmFRNA+FgeVSfqgcDR4TbpFrbhunBa8zCOG8J91RwYErPtR1dY1hfrAZ4G7oqZdiXwZnj/AOD7QNNwmROBV/YWb4Xt8yzwavjcbsB84OKYbVFCkDDrAaMIPuStivZ/aV2V/P/Fto2JYxLQEuhC8Ho8KZz2Q4IP0aMAA3oAXWNeKyfELOdL+wl4B3g03M79wuUeX9PXeTbdNPRRc22ADe5euucBM3vfzLaY2U4z+0bMvK+6+3vuXu7uxe6e5+4zw/9nECTlb1ZY/q/cvcjdZwL/R5Bk93jX3V/3YKzvOaBvFTGOABa4+3PuXuruzwPzgNP2rek85O6r3H0T8HeCNxkEieIJd//I3cs8GLvdRZAUq1JgZkUEyTKP4I2LmeUCJxMk0CJ3XwfcD5y9j7Hj7vnu/mG4TZYQfBjs2f6nAmvc/d5wX21394/iXPRYvryfzg0fw903uvuL7r7D3bcT9KIr7vNKhcNlZwG3hPEsAe4l+ADeY6m7/yV8TTwDHEgwFFdXfu/uW9x9GTCZ/+3zS4B73P0TDyx096VxtKkz8DXg5nA7TwOeqtCmeF/nWUOJuuY2Am3MrP6eB9z9WHdvGU6L3abLY59oZkPMbLKZrTezrQRfK9tUWH7sc5YS9GD3WBNzfwfQODaOGB3C58ZaCnSsullxqbj+nPB+V+CG8MNqi5ltATrz5dgrGhA+/yyCnnizmGU1AFbHLOsJgp71PjGzQ8xskpmtMbNtwO/43/bvDHxey0W/DTQJ929XgmT2crjOpmb2hJktDdc5BWgZ5zGLNgTHPGL3ZcX9+MU+cfcd4d0c6k5V+7y226sDsCn80Nqjyjax99d51lCirrkPCHqL34lj3ooH68YSjM11dvf9gccJvjbG6hxzvwvBV9maWkWQ8GJ1IfiqGo+aHmRcTvDVv2XMrWnYk696JYEJBNv0jphl7QLaxCyrhbsfXskiisK/TWMea7+XVT5G8M2ip7u3IBiD3rP9lwMHVxVqNe0oJxhXPYegNz0pJhHdAPQiGEZpAez5xrVnvXtb9gaCoY3YfVmT/VidIuLfdhXVdnutAlqbWfOYx+qyTRlJibqG3H0L8CvgUTP7gZnlmNl+ZtaP//UKq9KcoDdRbGaDCd7UFd0e9sIOB35CcNCupl4HDjGzc82svpmdBfQmGG+Mx1qgJufx/gW4POxRmpk1M7MRFd6Me/N7YKSZtXf31cC/gHvNrEW4bQ82s68MF7j7eoI3+PlmVs/MLqLq5AHB9t8GFJrZoQRjuntMAtqb2XVm1sjMmpvZkHDaWqCbme3t/TKW4NvBeeH92HXuBLaYWWvgzgrPq3Jbh1/9JwB3hfF0Ba4H6uR8eoKDv98LX289gItr8NyngBvNbGC4z3uE8cHe27Sc4GDz3WbW2MyODNc7pvbNyHxK1LXg7vcQvGFuIjiItpbg6/nNBC/CqlwB/NrMthP0ICdUMs87BEfC/wP8yd3/VYv4NhKMud5AMBxzE3Cqu2+IcxF/BXqHQw+vxLG+TwnGqR8mOGi5kODAVLzxziRo98/Ch35E8JV/Tri8FwjGXitzafi8jQRnbext+99I8OG4neDD5YsPwbAHfCLBOP4agjMdhoeTJ4Z/N5pZQRVt+Iigh9oBeCNm0gNAE4Le8YfAmxWe+iDwAzPbbGaVndt8dbjcRQRneIwlOHhZF+4nOKNlLcH4dtzJ0t0nEoy3jyXYnq8QHGSG4ADgbeHr58ZKnn4OwQHGVQRDRHe6+1u1bENWMHddOCAVmFk3gtOrGsQeqBQRUY9aRCTFKVGLiKQ4DX2IiKQ49ahFRFJcQk4ib9OmjXfr1i0Ri06YoqIimjWr7uy6zKI2Zwe1OT3k5+dvcPe2lU1LSKLu1q0bn376aSIWnTB5eXkMGzYs6jCSSm3ODmpzejCzKn+Cr6EPEZEUp0QtIpLilKhFRFKcErWISIpTohYRSXFxnfVhZksICq+UAaXuPiiRQYmIyP/U5PS84TWoviYiInVEQx8iInXg48WbeOq/i0hEWY64an2Y2WKCusBOcG28JyuZZyTBFazJzc0dOG7cuDoONbEKCwvJyanLKxilPrU5O6jNibdlVzl3vl9M43rw62Ob0Kh+xQs3VW/48OH5VQ4rx3MFXKBD+LcdMB34xt7mHzhwoKebyZMnRx1C0qnN2UFtTqyS0jI/8/H3vddtr/vc1VtrvRzgU9+Xq5C7+6rw7zqCKzIMrvHHhYhIBrr3rfl8tHgTv/tuHw5t3yIh66g2UYfXv2u+5z7wLWBWQqIREUkjb81Zy2N5n3PukC58b0CnhK0nnrM+coGXzWzP/GPdveJ130REssrSjUVcP2EafTruzx2n9k7ouqpN1O6+COib0ChERNJIcUkZo0YXsJ8Zj543gMYN6iV0fQkpcyoiksnufHU2c1Zv4+kLB9G5ddOEr0/nUYuI1MCET5cz/tPlXDW8B8cdmpuUdSpRi4jEafaqrdz+yiyG9jiAn554SNLWq0QtIhKHrTtLuGJMAa2aNuTBs/tTb7+a/6iltjRGLSJSDXfnxonTWbl5J+MvO5o2OY2Sun71qEVEqvHElEW8NWctt55yGAO7tk76+pWoRUT24sNFG7nnzXmM6HMgPxnaLZIYlKhFRKqwblsxV42dSrc2zfjDD44k/OFf0mmMWkSkEqVl5Vz1/FSKdpUy9tIh5DSKLl0qUYuIVOKP//yMjxdv4oGz+nFIbvNIY9HQh4hIBf+cvYYnpizivCFdOKN/x6jDUaIWEYm1ZEMRN06YzpGd9ueO0xJbbCleStQiIqHikjJGjSmgXr2g2FKj+oktthQvjVGLiIRuf2UW89Zs4+kLj6JTq8QXW4qXetQiIsD4T5YxMX8FVw/vwfBe7aIO50uUqEUk681auZXbX53N13u24doTkldsKV5K1CKS1fYUWzqgWUMeOKtfUostxUtj1CKStcrLnRsmTGfVlp2Mv+wYDkhysaV4qUctIlnr8Smf8++5a/nFiMMY2LVV1OFUSYlaRLLS+59v4E///IwRRx7Ihcd2izqcvVKiFpGss3ZbMdc8P5XubZrxh+9HV2wpXhqjFpGsUlJWzlVjCyjaVcbYS4+OtNhSvFI/QhGROnTPm/P4ZMlmHjw7+mJL8dLQh4hkjTdnreYv/13MBUd35Tv9oi+2FC8lahHJCos3FPGziTPo27klt516WNTh1IgStYhkvJ27yxg1Op/6KVZsKV4aoxaRjObu3PbKLD5bu53/u/AoOrZsEnVINaYetYhktHGfLOfFghVcc1xPhqVYsaV4KVGLSMaatXIrd74WFFu65vieUYdTa0rUIpKRtu4o4fLR+bRp1pAHz+6fksWW4qUxahHJOOXuXD9hGmu3FTPhsmNo3axh1CHtk7h71GZWz8ymmtmkRAYkIrKv/rGohP/MW8dtI3rTv0vqFluKV02GPq4F5iYqEBGRuvDewg28tKCE0/p24EfHdI06nDoRV6I2s07ACOCpxIYjIlJ7a7YGxZYObGb8/nt9Ur7YUrzM3aufyewF4G6gOXCju59ayTwjgZEAubm5A8eNG1fHoSZWYWEhOTk5UYeRVGpzdsiWNpeWO3/4uJhl28u5sa/Ts116tXn48OH57j6osmnVHkw0s1OBde6eb2bDqprP3Z8EngQYNGiQDxtW5awpKS8vj3SLeV+pzdkhW9r8m0lzWLBlMQ+d058Wm+dnVJvjGfoYCpxuZkuAccBxZjY6oVGJiNTA6zNX89d3F/PjY7pyet8OUYdT56pN1O5+i7t3cvduwNnA2+5+fsIjExGJw6L1hdz0wgz6dW7JL0b0jjqchNAPXkQkbe3YXcqo0QU0CIstNayfmSmtRj94cfc8IC8hkYiI1IC7c9vLs5i/bjvPXjSYDmlYbClemfnxIyIZb+zHy3hp6kquO/4Qvt6zbdThJJQStYiknRkrtvCr1+bwzUPacvVxPaIOJ+GUqEUkrWzZsZtRowto27wRD5zVj/3SuNhSvFSUSUTSRnm5c934aazbXszEy4+lVZoXW4qXetQikjYembyQvM/Wc8epvenXuWXU4SSNErWIpIV3F2zgvn/P54x+HTj/6MwothQvJWoRSXmrtuzkmnFT6dE2h99lULGleClRi0hK211azlVjC9hVUsbjFwykacPsO7SWfS0WkbRy9xtzKVi2hUfOHcDBbdOrIl5dUY9aRFLWpBmr+L/3lvCTod0YceSBUYcTGSVqEUlJC9cVcvMLMxjQpSW3nHxY1OFESolaRFJO0a5SRo3Op1GDejySwcWW4qUxahFJKe7OrS/PZOH6Qp67aAgH7p+5xZbild0fUyKSckZ/uJRXp63i+hMO4Ws920QdTkpQohaRlDFt+RZ+PWkOw3u15crhmV9sKV5K1CKSEjYX7ebKMQW0a96Y+7Ok2FK8NEYtIpErK3euHT+N9dt38cKoY2jZNDuKLcVLPWoRidyf317AlPnrueO03hzZKXuKLcVLiVpEIvXO/PU8+J8FfLd/R84b0iXqcFKSErWIRGbllp1cN24qh7Rrzl3fPSLrii3FS4laRCKxu7ScK8cUUFLmPHr+gKwsthQvbRkRicRd/5jDtOVbePS87C22FC/1qEUk6V6bvopnPljKRUO7c0qf7C22FC8lahFJqgVrt/PzF2cwsGsrbjnl0KjDSQtK1CKSNEW7Shk1poAmDerxyLkDaFBPKSgeGqMWkaRwd37+0kwWrS9k9MVDaL9/46hDShv6OBORpHj2g6X8ffoqbvhWL47toWJLNaFELSIJV7BsM7/9xxyOP7Qdo755cNThpB0lahFJqE1Fu7lqTAG5LRpz35kqtlQbGqMWkYQpK3euHTeVDUW7eWnUsezftEHUIaWlanvUZtbYzD42s+lmNtvMfpWMwEQk/T30nwX8d8EGfnX64RzRcf+ow0lb8fSodwHHuXuhmTUA3jWzN9z9wwTHJiJpLO+zdTz09gK+P6ATZx/VOepw0lq1idrdHSgM/20Q3jyRQYlIelu5ZSfXjZ9Gr9zm/PYMFVvaVxbk4WpmMqsH5AM9gEfc/eZK5hkJjATIzc0dOG7cuDoONbEKCwvJycmuegNqc3ZIdptLyp27PypmdVE5dx7ThPbNkn/OQjru5+HDh+e7+6BKJ7p73DegJTAZOGJv8w0cONDTzeTJk6MOIenU5uyQ7Dbf/spM73rzJH9j5qqkrjdWOu5n4FOvIqfW6KPO3bcAecBJ+/TRISIZ6dVpK3n2g6Vc+vXunHSEii3VlXjO+mhrZi3D+02AE4B5iQ5MRNLL/LXb+fmLMzmqWytuOknFlupSPGd9HAg8E45T7wdMcPdJiQ1LRNJJ4a5SLh+dT7NG9XlYxZbqXDxnfcwA+ichFhFJQ+7OzS/OYMmGIsZccjS5LVRsqa7pY09E9snf3l/CP2as5sZv9+KYgw+IOpyMpEQtIrWWv3Qzd/1jLicclsvl31CxpURRohaRWtlYuIurxhbQoWUT7j2zr4otJZCKMolIjQXFlqaxcU+xpSYqtpRI6lGLSI09+O/5vLtwA7/5jootJYMStYjUyOR563jo7YX8cGAnzjqqS9ThZAUlahGJ2/JNO7hu/DQOO7AFvznjiKjDyRpK1CISl12lZVw5toDycuex8wbQuEG9qEPKGjqYKCJx+fXf5zBjxVaeuGAg3do0izqcrKIetYhU6+WpKxjz0TIu+8ZBfPvw9lGHk3WUqEVkrz5bs51bX5rF4O6t+dm3e0UdTlZSohaRKm0vLmHU6HxyGtfn4XP6U1/FliKhMWoRqdSeYktLN+1g7CVDaKdiS5HRx6OIVOrp95bw+sw13PTtXgw5SMWWoqRELSJfkb90E3e/Ppdv9c5l5DcOijqcrKdELSJfsqFwF1eMKaBjqyb88Yd9dQXxFKAxahH5QlBsaSpbdpTw8hWDVWwpRShRi8gX7n9rPu8t3Mg9PziS3h1aRB2OhDT0ISIAvD1vLQ9PXsjZR3XmzEGdow5HYihRi0hQbGncNA7v0IJfnn541OFIBUrUIlmuuKSMUWPyAXjsvIEqtpSCNEYtkuV+PWkOs1Zu4y8/GkSXA5pGHY5UQj1qkSz2UsEKxn60jMu/eTAn9s6NOhypghK1SJaat2Ybt748kyHdW3Pjtw6JOhzZCyVqkSy0rbiEUaMLaNG4AX8+V8WWUp3GqEWyjLtz08QZLNu0g+cvPZp2zVVsKdXpY1Qky/z13cW8OXsNPz/pUAZ3bx11OBIHJWqRLPLJkk3c/cY8Tjq8PZd8vXvU4UiclKhFssT67bu4ckwBnVs14Z4fHqliS2lEY9QiWaC0rJxrnp/KtuISnrloMC0aq9hSOlGiFskC9701nw8WbeRPP+zLYQeq2FK6qXbow8w6m9lkM5trZrPN7NpkBCYidWPqulIezfuccwZ35gcDO0UdjtRCPD3qUuAGdy8ws+ZAvpm95e5zEhybiOyjZRt38OSMXRzRsQV3nqZiS+mq2h61u69294Lw/nZgLtAx0YGJyL7ZU2zJULGldGfuHv/MZt2AKcAR7r6twrSRwEiA3NzcgePGjau7KJOgsLCQnJycqMNIKrU5sz09axdTVpRyWW/nmC7Z0eY90nE/Dx8+PN/dB1U2Le6DiWaWA7wIXFcxSQO4+5PAkwCDBg3yYcOG1S7aiOTl5ZFuMe8rtTlzTfx0OVNWzOCKYQczuPGarGhzrEzbz3GdR21mDQiS9Bh3fymxIYnIvpizahu3vTKLYw46gOtPVLGlTBDPWR8G/BWY6+73JT4kEamtbcUlXDEmn5ZNG/DQOSq2lCni2YtDgQuA48xsWng7JcFxiUgNuTs/mzidFZt38si5A2jbvFHUIUkdqXaM2t3fBfRbU5EU95f/LuKfs9dy24jDGNRNxZYyib4XiWSAjxZt5A9vfsYpfdpz8ddUbCnTKFGLpLl124u56vmpdG3dlD98X8WWMpFqfYiksdKycq4eO5XtxSU8d/FgmqvYUkZSohZJY3/613w+WryJ+87sy6HtVWwpU2noQyRN/Wv2Gh5/53POHdKF7w1QsaVMpkQtkoaWbizihonT6dNxf+44tXfU4UiCKVGLpJnikjJGjS5gPzMePW+Aii1lAY1Ri6SZO16dxZzV23j6wkF0bt006nAkCdSjFkkjEz5ZzoRPV3DV8B4cd2hu1OFIkihRi6SJ2au2cvursxja4wB+qmJLWUWJWiQNbN1ZwhVjCmjVtCEPnt2fevvpRy3ZRGPUIinO3blx4nRWbt7J+MuOpk2Oii1lG/WoRVLcE1MW8dactdx6ymEM7KpiS9lIiVokhX24aCP3vDmPEUceyE+Gdos6HImIErVIilq3rZirxk6lW5tmKraU5TRGLZKCSsvKuer5qRTtKmXMJUPIaaS3ajbT3hdJQX/852d8vHgTD5zVj17tm0cdjkRMQx8iKebNWWt4Ysoizj+6C2f07xh1OJIClKhFUsiSDUX8bOJ0+nban9tVbElCStQiKWLn7jIuH51PvXrGI+cNoFF9FVuSgMaoRVKAu3P7q7P4bO12nr7wKDq1UrEl+R/1qEVSwPhPlvNC/gquHt6D4b3aRR2OpBglapGIzVq5lTtem83Xe7bh2hNUbEm+SolaJEJbd5Rw+eh8DmjWkAfO6qdiS1IpjVGLRKS83Ll+wjTWbitm/GXHcICKLUkV1KMWichj73zOf+at4xenHMaALq2iDkdSmBK1SATe/3wD9/7rM07r24EfH9st6nAkxSlRiyTZmq3FXPP8VLq3acbd3+ujYktSLY1RiyRRSVk5V40tYMfuMp6/9GgVW5K46FUikkR/eGMeny7dzINn96NnrootSXyqHfows6fNbJ2ZzUpGQCKZ6o2Zq3nq3cX86JiufKefii1J/OIZo/4bcFKC4xDJaIvWF/KzF2bQt3NLfjHisKjDkTRTbaJ29ynApiTEIpKRdu4uY9ToAhrUMx5VsSWpBXP36mcy6wZMcvcj9jLPSGAkQG5u7sBx48bVUYjJUVhYSE5OTtRhJJXanHjuzlMzd/P+qlKuH9iIPm2Tf1hI+zk9DB8+PN/dB1U2rc5eNe7+JPAkwKBBg3zYsGF1teikyMvLI91i3ldqc+KN/WgZ762aybXH9+TqE6Op46H9nP50HrVIgsxYsYVfhsWWrjm+Z9ThSBpTohZJgC07djNqdAFtchry4Nn9VWxJ9kk8p+c9D3wA9DKzFWZ2ceLDEklf5eXOT8dPY932Yh49fyCtmzWMOiRJc9WOUbv7OckIRCRTPJq3kMmfrefX3zmcfp1bRh2OZAANfYjUofcWbuC+t+Zzet8OXHB016jDkQyhRC1SR/YUWzqobY6KLUmdUq0PkTpQUlbOlWML2FlSxvjzB9BMxZakDunVJFIH7n59HvlLN/Pnc/rTo52KLUnd0tCHyD6aNGMVT7+3mAuP7cZpfTtEHY5kICVqkX2wcF0hN78wg/5dWnLrKSq2JImhRC1SSzt2l3LFmHwaNajHI+cOoGF9vZ0kMTRGLVIL7s6tL81kwbpCnr1oMB1aNok6JMlg6gKI1MLoj5bxyrRVXHf8IXy9Z9uow5EMp0QtUkPTl2/hN3+fw7Bebbn6uB5RhyNZQIlapAY2F+3mijEFtG3eiPvP7Md+KrYkSaAxapE4lZc7P50wjfXbdzHx8mNopWJLkiTqUYvE6eHJC8n7bD13nNabviq2JEmkRC0Sh/8uWM/9/57Pd/t35LwhXaIOR7KMErVINVZt2cm146bRs10Od333CBVbkqRTohbZi92lQbGl3aXlPHb+QJo21GEdST696kT24nevz2Xqsi08cu4ADm6bXle1lsyhHrVIFV6bvoq/vb+Ei4Z2Z8SRB0YdjmQxJWqRSixct52fvziDgV1bccsph0YdjmQ5JWqRCop2lTJqdAFNwmJLDerpbSLR0hi1SAx355aXZvL5+kKeu3gI7fdvHHVIIupRi8R67sOlvDZ9FdefeAhDe7SJOhwRQIla5AtTl23mN5PmcNyh7bhimIotSepQohYBNhXt5soxBeS2aMx9Z/ZVsSVJKRqjlqxXVu5cN34aGwp38+KoY2nZVMWWJLUoUUvW+/PbC5gyfz2/+24f+nTaP+pwRL5CQx+S1d6Zv54H/7OA7w3oyDmDO0cdjkillKgla63aspPrxk2lV25z7jqjj4otScpSopastLu0nCvGFFBS5jx63gCaNKwXdUgiVdIYtWSlu/4xh2nLt/D4+QM4SMWWJMWpRy1Z59VpK3nmg6Vc8rXunHSEii1J6osrUZvZSWb2mZktNLOfJzookUT5dE0pt7w0k6O6teLmk1VsSdJDtYnazOoBjwAnA72Bc8ysd6IDE6lL67YXM2p0Pg9P20X3Ns14WMWWJI3EM0Y9GFjo7osAzGwc8B1gTl0Hc9qf36W4pKyuFxuXoh07aFbwTiTrjko2tXn11mJ2l5Xzg0MacPePhypJS1qJJ1F3BJbH/L8CGFJxJjMbCYwEyM3NJS8vr8bB5JQX0yiiM6RyGpVTz3ZGs/KIZFOb27cxTu7eiBbs4L3/Tok6nKQqLCys1fsxnWVam+NJ1JWlTv/KA+5PAk8CDBo0yIcNG1bjYGrxlDqTl5dHbWJOZ2pzdlCb01883/9WALE/2eoErEpMOCIiUlE8ifoToKeZdTezhsDZwGuJDUtERPaodujD3UvN7Crgn0A94Gl3n53wyEREBIjzl4nu/jrweoJjERGRSugcJRGRFKdELSKS4pSoRURSnBK1iEiKM/ev/HZl3xdqth5YWucLTqw2wIaog0gytTk7qM3poau7t61sQkISdToys0/dfVDUcSST2pwd1Ob0p6EPEZEUp0QtIpLilKj/58moA4iA2pwd1P8fuzMAAAI6SURBVOY0pzFqEZEUpx61iEiKU6IWEUlxStSVMLMbzczNrE3UsSSamf3RzOaZ2Qwze9nMWkYdUyJk2wWazayzmU02s7lmNtvMro06pmQxs3pmNtXMJkUdS11Roq7AzDoDJwLLoo4lSd4CjnD3I4H5wC0Rx1PnsvQCzaXADe5+GHA0cGUWtHmPa4G5UQdRl5Sov+p+4CYqudxYJnL3f7l7afjvhwRX8Mk0X1yg2d13A3su0Jyx3H21uxeE97cTJK6O0UaVeGbWCRgBPBV1LHVJiTqGmZ0OrHT36VHHEpGLgDeiDiIBKrtAc8YnrT3MrBvQH/go2kiS4gGCjlZ51IHUpbguHJBJzOzfQPtKJv0CuBX4VnIjSry9tdndXw3n+QXB1+UxyYwtSeK6QHMmMrMc4EXgOnffFnU8iWRmpwLr3D3fzIZFHU9dyrpE7e4nVPa4mfUBugPTzQyCIYACMxvs7muSGGKdq6rNe5jZj4FTgeM9M0+sz8oLNJtZA4IkPcbdX4o6niQYCpxuZqcAjYEWZjba3c+POK59ph+8VMHMlgCD3D3dKnDViJmdBNwHfNPd10cdTyKYWX2CA6XHAysJLth8biZf+9OC3sYzwCZ3vy7qeJIt7FHf6O6nRh1LXdAYtTwMNAfeMrNpZvZ41AHVtfBg6Z4LNM8FJmRykg4NBS4Ajgv367SwpylpSD1qEZEUpx61iEiKU6IWEUlxStQiIilOiVpEJMUpUYuIpDglahGRFKdELSKS4v4fbCq31ulSSzgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def Relu(x):\n",
    "    y= np.zeros(x.shape)\n",
    "    return np.ndarray.flatten(np.max([[y],[x]], axis =0))\n",
    "\n",
    "x = np.linspace(-5,5,100)\n",
    "plt.title('Graph of the Relu activation function')\n",
    "plt.plot(x,Relu(x))\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Función de activación Softmax "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La función softmax también es un tipo de función sigmoidea, pero es útil cuando intentamos manejar problemas de clasificación.\n",
    "\n",
    "Generalmente se usa cuando se intenta manejar múltiples clases. La función softmax comprime las salidas para cada clase a valores entre 0 y 1 al dividir cada valor por la suma de las salidas (modificadas mediante una función exponencial).\n",
    "\n",
    "La función softmax se usa idealmente en la capa de salida del clasificador donde realmente estamos tratando de alcanzar las probabilidades para definir la categoría que corresponde de cada entrada. Formalmente la función softmax es definada como sigue-"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Sea $x=(x_1,\\ldots,x_n)^t \\in \\mathbb{R}^n$. Softmax es una función $S: \\mathbb{R}^n \\to \\mathbb{R}^n$ definida por\n",
    "\n",
    "$$\n",
    "S(x) : \\begin{pmatrix} x_1 \\\\ \\vdots \\\\x_n\\\\ \\end{pmatrix}    \\to \\begin{pmatrix} S_1 \\\\ \\vdots \\\\S_n\\\\ \\end{pmatrix}\n",
    "$$\n",
    "\n",
    "en donde\n",
    "\n",
    "$$\n",
    "S_j = \\frac{e^{x_j}}{\\sum_{k=1}^N e^{x_k}}.\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "Dado que $ S $ es una función vectorial de $ \\mathbb {R} ^ n $ a $ \\mathbb {R} ^ n $, la derivada es la matriz jacobiana dada por\n",
    "\n",
    "$$\n",
    "DS = \\begin{pmatrix} D_1 S_1 &\\cdots& D_1S_n\\\\\n",
    "                     \\cdots & \\ddots & \\cdots\\\\\n",
    "                     D_n S_1 & \\cdots & D_n S_n\n",
    "     \\end{pmatrix} \n",
    "$$\n",
    "\n",
    "Luego de unos simples cálculos se llega a que  \n",
    "\n",
    "$$\n",
    "D_j S_i =  S_i(\\delta_{ij}-S_j),\n",
    "$$ \n",
    "\n",
    "en donde  $\\delta_{ij}$ es llmado el delta de Kroneker definido por\n",
    "\n",
    "$$\n",
    "\\delta_{ij} = \\begin{cases} 1, &\\text{si } i=j\\\\\n",
    "                            0, & \\text{isi } i\\ne j\n",
    "               \\end{cases}              \n",
    "$$\n",
    "\n",
    "\n",
    "El siguiente fragmento de código muestra cómo implementar $ DS $ en Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ejercicio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verifique la ecuación anterior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# derivative (grad) of the softmax function\n",
    "def softmax_grad(s):\n",
    "    # input s is softmax value of the original input x. Its shape is (1,n) \n",
    "    # i.e.  s = np.array([0.3,0.7]),  x = np.array([0,1])\n",
    "\n",
    "    # make the matrix whose size is n^2.\n",
    "    jacobian_m = np.diag(s)\n",
    "\n",
    "    for i in range(len(jacobian_m)):\n",
    "        for j in range(len(jacobian_m)):\n",
    "            if i == j:\n",
    "                jacobian_m[i][j] = s[i] * (1 - s[i])\n",
    "            else: \n",
    "                jacobian_m[i][j] = -s[i] * s[j]\n",
    "    return jacobian_m\n",
    "\n",
    "# testing the function\n",
    "import numpy as np\n",
    "import scipy.special\n",
    "\n",
    "\n",
    "x = np.array([1, 2])\n",
    "scipy.special.softmax(x)\n",
    "softmax_grad(scipy.special.softmax(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Seleccionado una función de activación"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La `regla básica de combate` es que si realmente no sabe qué función de activación usar, simplemente use RELU, ya que es una función de activación general y se usa en la mayoría de los casos en estos días.\n",
    "Si su salida es para clasificación binaria, la función sigmoidea es una elección muy natural para la capa de salida."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Estado del arte. Variaciones de la función de activación de ReLu</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En una red neuronal, la función de activación es responsable de transformar la entrada ponderada sumada del nodo en la activación del nodo o salida para esa entrada.\n",
    "\n",
    "La función de activación *ReLU* es una función lineal por partes que tiene como salida a la entrada directamente si es positiva; de lo contrario, dará salida a cero. Se ha convertido en la función de activación predeterminada para muchos tipos de redes neuronales porque un modelo que la usa es más fácil de entrenar y, a menudo, logra un mejor rendimiento.In a neural network, the activation function is responsible for transforming the summed weighted input from the node into the activation of the node or output for that input.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Limitaciones de las funciones de activación sigmoide y Tanh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se prefieren las funciones de activación no lineales, ya que permiten que los nodos aprendan estructuras más complejas en los datos. Tradicionalmente, dos funciones de activación no lineales ampliamente utilizadas son las funciones de activación tangente sigmoidea e hiperbólica.\n",
    "\n",
    "\n",
    "Un problema general con las funciones *sigmoide* y *Tanh* es que *saturan* en los extremos. Esto significa que los valores grandes se ajustan a 1 y los valores pequeños a -1 (Tanh) o 0 (sigmoide) respectivamente. Además, las funciones solo son realmente sensibles a los cambios alrededor del punto medio de su entrada, como 0.5 para sigmoide y 0.0 para Tanh.\n",
    "\n",
    "La sensibilidad y saturación limitadas de la función ocurren independientemente de si la activación sumada del nodo proporcionado como entrada contiene información útil o no. Una vez saturado, se vuelve un desafío para el algoritmo de aprendizaje seguir adaptando los pesos para mejorar el rendimiento del modelo.\n",
    "\n",
    "$ \\to$ Las unidades sigmoidales se saturan en la mayor parte de su dominio: se saturan a un valor alto cuando $ z $ es muy positivo, se saturan a un valor bajo cuando $ z $ es muy negativo y solo son muy sensibles a su entrada cuando z está cerca de 0.\n",
    "\n",
    "\n",
    "Finalmente, a medida que la capacidad del hardware aumentó con el uso de las GPU, las redes neuronales muy profundas que utilizan funciones de activación sigmoidea y tanh, no se pueden entrenar fácilmente.\n",
    "\n",
    "Las capas profundas en redes grandes que utilizan estas funciones de activación no lineales no reciben información de gradiente útil. El error que se propaga a través de la red y que se usa para actualizar los peso hace que el gradiente de la red sufra de unp de dos problemas.\n",
    "\n",
    "- Tiende a ser cero (el gradiente se devance).\n",
    "- Tiende a ser infinito (el gradiente explota).\n",
    "\n",
    "Los gradientes que desvanencen o que explotan hacen que sea difícil saber en qué dirección deben moverse los parámetros para mejorar la función de pérdida.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Evitando que los  gradientes desaparezcan mediante el uso de la función de activación ReLU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Para utilizar el descenso de gradiente estocástico con retropropagación de errores para entrenar redes neuronales profundas, se necesita `una función de activación que se vea y actúe como una función lineal`, pero que sea, de hecho, una función no lineal que permita aprender relaciones complejas en los datos.\n",
    "\n",
    "La función también debe proporcionar más sensibilidad a la entrada de suma de activación y evitar una fácil saturación.\n",
    "\n",
    "La solución había estado dando vueltas en el campo durante algún tiempo, aunque no se destacó hasta que algunos artículos de 2009 y 2011 arrojaron luz sobre ella.\n",
    "\n",
    "La solución es utilizar la **función de activación lineal rectificada**, o *ReLU* para abreviar.\n",
    "\n",
    "Un nodo o unidad que implementa esta función de activación se conoce como unidad de activación lineal rectificada, o ReLU. A menudo, las redes que utilizan la función rectificadora para las capas ocultas se denominan redes rectificadas.\n",
    "\n",
    "$\\leadsto$ La adopción de ReLU puede considerarse fácilmente **uno de los pocos hitos en la revolución del aprendizaje profundo**, p. ej. las técnicas que ahora permiten el desarrollo rutinario de redes neuronales muy profundas.\n",
    "\n",
    "Debido a que las unidades lineales rectificadas son casi lineales, conservan muchas de las propiedades que hacen que los modelos lineales sean fáciles de optimizar con métodos basados en gradientes. También conservan muchas de las propiedades que hacen que los modelos lineales generalicen bien."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ventaja de la función de activación ReLU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La función de activación lineal rectificada se ha convertido rápidamente en la función de activación predeterminada al desarrollar la mayoría de los tipos de redes neuronales.\n",
    "\n",
    "Como tal, es importante tomarse un momento para revisar algunos de los beneficios del enfoque, destacados por primera vez por Xavier Glorot, et al. en su documento histórico de 2012 sobre el uso de ReLU titulado [Deep Sparse Rectifier Neural Networks](https://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf).\n",
    "\n",
    "1. **Simplicidad computacional**. La función del rectificador es trivial de implementar y requiere una función max(). Esto es diferente a la función de activación sigmoidea y tanh que requieren el uso de un cálculo exponencial.\n",
    "2. **Escasa representación**. Un beneficio importante de la función de rectificador es que es capaz de generar un valor cero verdadero. Esto es diferente a las funciones de activación tanh y sigmoidea que aprenden a aproximarse a una salida muy cercana a cero, pero no un verdadero valor cero. Esto significa que las entradas negativas pueden generar valores cero verdaderos, lo que permite la activación de capas ocultas en redes neuronales para contener uno o más valores cero verdaderos. Esto se denomina representación dispersa y es una propiedad deseable en el aprendizaje representacional, ya que puede acelerar el aprendizaje y simplificar el modelo.\n",
    "3. **Comportamiento lineal**. La función rectificadora se ve y actúa principalmente como una función de activación lineal. En general, una red neuronal es más fácil de optimizar cuando su comportamiento es lineal o casi lineal.\n",
    "4. **Entrene redes profundas**. Es importante destacar que el (re) descubrimiento y la adopción de la función de activación lineal rectificada significó que se hizo posible explotar las mejoras en el hardware y entrenar con éxito redes profundas de múltiples capas con una función de activación no lineal mediante retropropagación."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extensiones y alternativas para ReLU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El ReLU tiene algunas limitaciones. Una de las principales limitaciones de ReLU es el caso en el que las actualizaciones de gran peso pueden significar que la entrada sumada a la función de activación es siempre negativa, independientemente de la entrada a la red.\n",
    "\n",
    "Esto significa que un nodo con este problema siempre generará un valor de activación de 0.0. Esto se conoce como un \"ReLU moribundo\".\n",
    "\n",
    "Algunas extensiones populares de ReLU relajan la salida no lineal de la función para permitir pequeños valores negativos de alguna manera."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  ReLU parametrizada (PReLu), 2015"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La función de activación PRelu es definida por\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "f(y_i) = \\begin{cases} y_i, & \\text{ si } y_i > 0\\\\\n",
    "a_i y_i, & \\text{ if } y_i \\le 0 \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Aquí $ y_i $ es la entrada de la activación no lineal $ f $ en el $ i $ -ésimo\n",
    "canal, y $ a_i $ es un coeficiente que controla la pendiente del\n",
    "parte negativa. El subíndice i en ai indica que permitimos\n",
    "la activación no lineal para variar en diferentes canales.\n",
    "\n",
    "Observe que\n",
    "\n",
    "1. $f(y_i) = max(0,y_i) + a_i min(0,y_i)$\n",
    "2.$\n",
    "\\begin{equation}\n",
    "f'(y_i) = \\begin{cases} 1, & \\text{ si } y_i > 0\\\\\n",
    "a_i , & \\text{ si } y_i \\le 0 \\end{cases}\n",
    "\\end{equation}\n",
    "$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEICAYAAABCnX+uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU5dn/8c/FvgQICIQt7O6sEoJLreDS4tbF2ipWQEDRrrbVam37a+1i26fWtlptK5Xdhepj7cKjtrYarVXZkV1lEcIie4AACSS5fn+cAx0iS0JO5mRmvu/XK69MZrnPdZ8zc+Wee+65jrk7IiKSuurFHYCIiNSMErmISIpTIhcRSXFK5CIiKU6JXEQkxSmRi4ikOCXyNGJm95rZ4xG1lWNmr5nZHjN7oIqPed/MLo1i+zVhZjeZ2et1II4XzGx0LbVdbGY9a6Hd081sQXjcvxp1+8fZbtewT/WTtc10okRei8zsejObZWZ7zWxLePmLZmZxx1YF44FtQEt3v6PyjWY2xcx+XFsbNzMP91uxmW0ws1/W5Rf50f6Juvvl7j41grYLzOzmSm1nufvqmrZ9FHcBBe7ewt0fqoX2gQ//03f3dWGfymtrm+lMibyWmNkdwIPA/UAHIAe4DbgAaHSMx9SlRNUNWObxfmOsv7tnARcB1wFjY4wlU3QDlsYdhFSTu+sn4h+gFbAX+MwJ7jcF+B3wfHj/S4ErgQXAbqAQuDfh/t0BJxgtbwQ2AXck3H4v8DQwDdhD8ILMO872zwfmALvC3+cnxHUQOAAUA5dWetz4Srf/Lbz+feBOYFHY5h+BJgmPuwpYCBQBbwD9jhObA70T/n4aeKTSPp4Y7oMNwI+B+uFtNwGvV9pnDRIeWwDcfIzt5gNvhjFuAh4GGiXcfjbwErAD2Ax8Gxge7ouD4f54O3E7QOOwvT4J7bQD9gPtgdbATGArsDO83CW8331AOVAStv1w5f0T7otp4ePXAt8F6iXuC+AXYdtrgMuP0feXK23rtMr7KnHfJsRxG/Be2P4jgCXcfguwnOD5uAw4B5gOVIT9LyZ4F3DEcQI6AX8N9/NK4JaTfZ5nwk/sAaTjT/jCLktMHse43xSChHcBwbujJsBQoG/4d78wWXwqvP+hJ/tTQPPwflsJE234BC8BrgDqAz8F3jrGttuEL7yRQANgRPj3KQmx/fgEsf+40nXvA7PDF2Gb8AV8W3jbOcAWYEgY2+jw/o2P0X5iojqDIKl+PeH2PwOPhvuhfbjdW8PbDiebygkivK6AYyfyQcC54T7pHvbha+FtLcI47giPVQtgSMK+f7xSW4e3A0wC7ku47UvAi+HlU4DPAM3CNp8B/ny8eCvtn2nAX8LHdgfeBcYl7IuDBAm1PvAFgkGAHaP/R2zrKH8f3rcJccwEsoGuBM/H4eFtnyX4JzsYMKA30C3huXJpQjtHHCfgVeC34X4eELZ7SXWf55nyo6mV2tEW2ObuZYeuMLM3zKzIzPab2UcT7vsXd/+Pu1e4e4m7F7j74vDvRQRJ+6JK7f/A3fe6+2JgMkESPuR1d3/eg7nG6UD/Y8R4JfCeu0939zJ3fwpYAVxds67zkLtvdPcdwN8IXoQQJJJH3X2Wu5d7MHdcSpA0j2W+me0lSKYFBC9szCwHuJwgwe519y3Ar4Draxg77j7P3d8K98n7BP8sDu3/q4AP3P2B8FjtcfdZVWz6SY48TjeE1+Hu2939WXff5+57CEbhlY/5UYXTcdcB94TxvA88QPAP+pC17v6H8DkxFehIMNUXlZ+5e5G7rwNe4b/H/Gbg5+4+xwMr3X1tFfqUC3wEuDvczwuBxyr1qarP84ygRF47tgNtzazBoSvc/Xx3zw5vS9zvhYkPNLMhZvaKmW01s10Eb1vbVmo/8TFrCUbAh3yQcHkf0CQxjgSdwscmWgt0Pna3qqTy9rPCy92AO8J/ZkVmVgTkcmTslZ0TPv46gpF884S2GgKbEtp6lGBkXiNmdpqZzTSzD8xsN/AT/rv/c4FVJ9n0y0DT8Ph2I0h2z4XbbGZmj5rZ2nCbrwHZVfzMpC3BZy6Jx7LycTx8TNx9X3gxi+gc65if7P7qBOwI/6kdcsw+cfzneUZQIq8dbxKMNj9ZhftW/jDxSYK5wVx3bwX8nuBtaaLchMtdCd4qV9dGgoSYqCvBW+GqqO6HoIUEUwvZCT/NwncCx95I4GmCffq9hLZKgbYJbbV097OP0sTe8HezhOs6HGeTvyN4Z3Kqu7ckmAM/tP8LgV7HCvUE/aggmNcdQTAan5mQqO4ATieYpmkJHHrHdmi7x2t7G8HUSeKxrM5xPJG9VH3fVXay+2sj0MbMWiRcF2Wf0o4SeS1w9yLgB8BvzexaM8sys3pmNoD/jiqPpQXBaKTEzPIJXvSV/b9wFHc2MIbgQ8Xqeh44zcxuMLMGZnYdcBbBfGdVbAaqs475D8Bt4YjUzKy5mV1Z6cV6PD8DxptZB3ffBPwDeMDMWob7tpeZfWg6wt23EiSAG82svpmN5djJBYL9vxsoNrMzCOaUD5kJdDCzr5lZYzNrYWZDwts2A93N7HivqScJ3l18PrycuM39QJGZtQG+X+lxx9zX4dTC08B9YTzdgG8AkXyfgODD6WvC51tvYFw1HvsYcKeZDQqPee8wPjh+nwoJPgz/qZk1MbN+4XafOPlupDcl8lri7j8neEHdRfAh32aCt/93EzxJj+WLwA/NbA/BCPTpo9znVYJP8v8F/MLd/3ES8W0nmPO9g2C65y7gKnffVsUmJgJnhVMbf67C9uYSzJM/TPCh6kqCD86qGu9ign5/M7xqFMGUwrKwvf8lmPs9mlvCx20nWHVyvP1/J8E/zz0E/3wO/5MMR9CXEXyO8AHBSo1h4c3PhL+3m9n8Y/RhFsEItxPwQsJNvwaaEoyu3wJerPTQB4FrzWynmR1tbfdXwnZXE6xQeZLgw9Uo/IpgRc5mgvn1KidTd3+GYL7/SYL9+WeCD8Eh+IDyu+Hz586jPHwEwQegGwmmoL7v7i+dZB/SnrnrxBKpwsy6Eywfa5j4QaqIZDaNyEVEUpwSuYhIitPUiohIitOIXEQkxcWygL5t27bevXv3ODZdI3v37qV58xOtHkwfmdZfUJ8zRar2ed68edvcvV3l62NJ5N27d2fu3LlxbLpGCgoKGDp0aNxhJE2m9RfU50yRqn02s6OWONDUiohIilMiFxFJcUrkIiIpTolcRCTFKZGLiKS4SFatmNn7BEVxyoEyd8+Lol0RETmxKJcfDqtG5TwREYmIplZERJJg/4Fy7v3rUnbtOxh525HUWjGzNQQ1oZ3gvIwTjnKf8QRnXycnJ2fQjBkzarzdZCsuLiYrK8ozZNVtmdZfUJ8zRbL7XFbhPDS/lMXbyvn6oMb0a3dykyHDhg2bd9Sp6yjO4Ax0Cn+3B94GPnq8+w8aNMhT0SuvvBJ3CEmVaf11V58zRTL7XF5e4V95cr53u3umPzlrbY3aAub6UXJqJFMr7r4x/L2F4Gwe+VG0KyKSytydH/xtKX99eyN3DT+dEflda2U7NU7k4bkXWxy6DHwMWFLTdkVEUt2v//keU99cyy0X9uALFx3vVLE1E8WqlRzgOTM71N6T7l75nIMiIhllyn/W8OC/3uPaQV349hVnEubIWlHjRO7uq4H+EcQiIpIW/rxgA/f+bRkfOyuHn13Tt1aTOGj5oYhIpF5esZk7n3mbc3u24aERA2lQv/bTrBK5iEhEZq/ZwRcen88ZHVvwh1F5NGlYPynbVSIXEYnAso27GTd1Dp2zmzJ1TD4tmjRM2raVyEVEauj9bXsZNWk2WY0bMP3mIZyS1Tip21ciFxGpgc27S7hx4izKKyqYPi6fztlNkx6DErmIyEkq2neAURNns3PvAaaMyad3+xaxxBHLyZdFRFLdvgNljJ0yhzXb9jJ5zGD652bHFotG5CIi1XSgrILbHp/PwsIiHhoxkAt6t401Ho3IRUSqobzC+cbTC3nt3a38/DP9GN6nQ9whaUQuIlJV7s73/rKEmYs2cc/lZ/C5wblxhwQokYuIVNkvX3qXJ2at47aLenFrLRbBqi4lchGRKpj4+hp+8/JKrsvL5e7hp8cdzhGUyEVETuDZeev50cxlDD+7A/d9uk+tF8GqLiVyEZHj+Oeyzdz17CIu6H0KD44YkJQiWNVV9yISEakj3lq9nS8+OZ8+nVry6Mg8GjdIThGs6lIiFxE5iiUbdnHL1Lnktm7K5DH5ZDWuu6u1lchFRCpZs20voyfNpmXThjx+8xDaNG8Ud0jHpUQuIpJg06793PjYLACmjcunY6vkF8Gqrrr7XkFEJMl27g2KYO3af5CnbjmXXu2y4g6pSpTIRUSAvaVljJkyh7U79jF1TD59u7SKO6Qq09SKiGS80rJybnt8HovWF/HwiIGc1+uUuEOqFo3IRSSjlVc43/jj2/z7vW3cf20/PnZ2/EWwqksjchHJWO7O//vLEv5v8Sa+e+WZfDavbhTBqi4lchHJWL/4xzs8OWsdXxzai5sv7Bl3OCctskRuZvXNbIGZzYyqTRGR2vLYv1fzyCurGJHflW9+vG4VwaquKOfIbweWAy0jbFNEJHKvbzjIY4uXc0XfDvz4U3WvCFZ1RTIiN7MuwJXAY1G0JyJSW/6x9AMmLTnAhae25VfXDaB+vdRO4gDm7jVvxOx/gZ8CLYA73f2qo9xnPDAeICcnZ9CMGTNqvN1kKy4uJisrNb4gEIVM6y+oz+lu+fZyHphXQudmzj3nNqdJg9RK4sOGDZvn7nmVr6/x1IqZXQVscfd5Zjb0WPdz9wnABIC8vDwfOvSYd62zCgoKSMW4T1am9RfU53S2eP0uHnnlLXq0zeL2PhUMv3RY3CFFJoqplQuAT5jZ+8AM4GIzezyCdkVEIrFqazGjJ8+mVdOGTB83hKxGqTUSP5EaJ3J3v8fdu7h7d+B64GV3v7HGkYmIRGDTrv2MmjgbA6aPy6dDqyZxhxQ5fbNTRNLWjr0HuPGxWezaf5AZ48+lZ4oUwaquSBO5uxcABVG2KSJyMopLyxgzeTaFO/czbWw+fTqnThGs6tI3O0Uk7ZSWlXPr9Lks2bibR244h3N7plYRrOpSIheRtFJe4XxtxkL+s3I7//OZflx2Vk7cIdU6JXIRSRvuzneeW8wLSz7gu1eeybWDusQdUlIokYtI2vj5399hxpxCvjysd0oXwaouJXIRSQsTXlvF7wpWccOQrtzxsdPiDieplMhFJOU9PaeQnzy/giv7deRHn0z9IljVpUQuIintxSUf8K0/LQqKYH0uPYpgVZcSuYikrDdWbuOrTy2gf242j44cRKMGmZnSMrPXIpLyFq0v4pZpc+nethmTbxpMs0aZ+0V1JXIRSTkrtxRz0+Q5tG7eiGljh5DdrFHcIcVKiVxEUsqGov2MnDiLegbTxw1JyyJY1aVELiIpY3txKSMnzqK4pIypY/Pp0bZ53CHVCZk7qSQiKWVPyUFumjyHDTv3M33cEM7ulL5FsKpLiVxE6rySg+WMnzaPZZt2M2HkIPJ7tIk7pDpFUysiUqeVlVfw1acW8Obq7fzis/245Mz0L4JVXUrkIlJnuTv3/Gkx/1i2me9ffRafHpgZRbCqS4lcROokd+enL6zgmXnr+eolpzLmgh5xh1RnKZGLSJ30+1dXM+G11Yw+rxtfv/TUuMOp05TIRaTOeWr2Ov7nxRV8on8nvn/12RlXBKu6lMhFpE55YfEmvvPcYoae3o4HPtefehlYBKu6lMhFpM54/b1t3D5jIed0bc3vPj+IhvWVoqpCe0lE6oSFhUWMnz6Xnu2aM3H0YJo2qh93SClDiVxEYrdyyx5umjybtlmNmTY2n1bNGsYdUkpRIheRWK3fuY8bH5tNw/r1eHzcENq3VBGs6qpxIjezJmY228zeNrOlZvaDKAITkfS3rbiUURNns+9AGdPG5tP1lGZxh5SSoqi1Ugpc7O7FZtYQeN3MXnD3tyJoW0TSVFAEazYbd+3n8XFDOLNjy7hDSlk1TuTu7kBx+GfD8Mdr2q6IpK+Sg+XcPHUuKzbt4Q+j88jrriJYNWFBHq5hI2b1gXlAb+ARd7/7KPcZD4wHyMnJGTRjxowabzfZiouLycrKijuMpMm0/oL6nAzlFc7DC0tZuKWcW/s15txOyS/CmqrHediwYfPcPe9DN7h7ZD9ANvAK0Od49xs0aJCnoldeeSXuEJIq0/rrrj7XtvLyCv/GHxd6t7tn+tQ31iRtu5Wl6nEG5vpRcmqkq1bcvQgoAIZH2a6IpD53577nl/Ps/PV8/dLTGHVe97hDShtRrFppZ2bZ4eWmwKXAipq2KyLp5bcFq5j4+hpuOr87X72kd9zhpJUoJqc6AlPDefJ6wNPuPjOCdkUkTTwxay33//0dPjWgE9+76iwVwYpYFKtWFgEDI4hFRNLQzEUb+e6fl3DxGe25/7MqglUb9M1OEak1r727la//cSF53VrzyA3nqAhWLdFeFZFaMX/dTm6dPo9e7bJ4TEWwapUSuYhE7t3NexgzeQ7tWzZm2rh8WjVVEazapEQuIpEq3LGPkRNn0bhBWASrhYpg1TYlchGJzNY9pYycOIv9B8qZPm4IuW1UBCsZkv/dWBFJS7tLDjJ60mw27y7l8ZuHcHqHFnGHlDE0IheRGis5WM7NU+by3pY9/H7kIAZ1ax13SBlFI3IRqZGD5RV86Yn5zFm7g4euH8hFp7WLO6SMoxG5iJy0igrnrv9dxL9WbOGHn+zD1f07xR1SRlIiF5GT4u786P+W8dyCDdxx2WmMPLdb3CFlLCVyETkpD7+8ksn/eZ+xF/TgyxerCFaclMhFpNqmv7WWB156l2sGdua7V56pIlgxUyIXkWr569sb+d5flnDpme35n2v7qQhWHaBELiJVVvDOFr7xx4UM7taGh1UEq87QURCRKpm3dgdfeHw+p+W04LGb8mjSUEWw6golchE5oRUf7GbM5DnktGzM1LH5tGyiIlh1iRK5iBzXuu37GDVxNk0b1Wf6uCG0a9E47pCkEn2zU0SOacueEkZOmkVpWQXP3HaeimDVURqRi8hR7dp/kNGT5rBldymTxwzmtBwVwaqrlMhF5EP2Hyjn5qlzWLllDxNGDeKcriqCVZdpakVEjnCwvIIvPTmfuWt38vCIc7jwVBXBqus0IheRwyoqnG8+8zYvr9jCjz/Vhyv7dYw7JKkCJXIRAYIiWD+cuYw/L9zINz9+Op8foiJYqUKJXEQAeOhfK5nyxvvccmEPvji0V9zhSDXUOJGbWa6ZvWJmy81sqZndHkVgIpI80958n1/9812uHdSFb1+hIlipJooPO8uAO9x9vpm1AOaZ2UvuviyCtkWklr25sYxHFy3lsrNy+Nk1fZXEU1CNR+Tuvsnd54eX9wDLgc41bVdEat8rK7bw2OJShvRow29GDKSBimClJHP36Boz6w68BvRx992VbhsPjAfIyckZNGPGjMi2myzFxcVkZWXFHUbSZFp/IbP6/N7Ocu6fU0JOU+fb5zWnaYPMGYmn6nEeNmzYPHfPq3x9ZInczLKAV4H73P1Px7tvXl6ez507N5LtJlNBQQFDhw6NO4ykybT+Qub0efmm3Vz36Ju0zWrM1/o6n/j4sLhDSqpUPc5mdtREHsn7KDNrCDwLPHGiJC4i8Vq7fS+jJs2meeMGTBuXT8vGmTMST1dRrFoxYCKw3N1/WfOQRKS2bNldwsiJsykrr2D6uHy6tFYRrHQQxYj8AmAkcLGZLQx/roigXRGJ0K59Bxk1aTbbikuZPCaf3u1VBCtd1Hj5obu/Dui9mUgdtv9AOWOnzmH11r1MumkwA3Kz4w5JIqS1RiJp7kBZBV94Yh4L1u3kwesH8JFT28YdkkRM1Q9F0lhFhXPnM29T8M5WfnZNXy7vqyJY6UgjcpE05e7c+7el/PXtjdw9/Ayuz+8ad0hSS5TIRdLUr//5HtPeXMutH+3JF1QEK60pkYukoSn/WcOD/3qPz+V14VuXnxF3OFLLlMhF0sxzC9Zz79+W8fGzc/jJp1UEKxMokYukkZdXbObOZxZxfq9TePB6FcHKFDrKImli9podfOHx+ZzdqSUTRuXRpGH9uEOSJFEiF0kDSzfuYtyUOXRu3ZTJNw0mq7FWFmcSJXKRFLdm215GT5pNiyYNeHzcEE7Jahx3SJJkSuQiKeyDXSWMnDiL8gpn2rghdMpuGndIEgMlcpEUVbTvAKMmzWLn3gNMHZtP7/apd6IEiYYm0kRS0L4DZYyZMof3t+1jypjB9OuiIliZTCNykRRzoKyCW6fP4+3CIh4aMZDze6sIVqbTiFwkhZRXON94eiH/fm8bP/9MP4b36RB3SFIHaEQukiLcne/9ZQkzF23insvP4HODc+MOSeoIJXKRFPHLl97liVnruO2iXtx6kYpgyX8pkYukgImvr+E3L6/k+sG53D389LjDkTpGiVykjnt23np+NHMZw8/uwH0qgiVHoUQuUoe9tGwzdz27iAt6n8KDIwZQv56SuHyYErlIHfXW6u186cn59OnUkkdH5tG4gYpgydEpkYvUQUs27OKWqXPp2qYZk8fkqwiWHJcSuUgds3pr8eEiWNPG5tOmeaO4Q5I6TolcpA7ZtGs/IyfOBmD6zSqCJVUTSSI3s0lmtsXMlkTRnkgm2rn3AKMmzmbX/oNMGZNPr3YqgiVVE9WIfAowPKK2RDLO3tKgCNbaHfuYMGoQfbu0ijskSSGRJHJ3fw3YEUVbIpmmtKycW6fPY9H6In4zYiDn91IRLKkec/doGjLrDsx09z7HuH08MB4gJydn0IwZMyLZbjIVFxeTlZU5b3czrb+Q/D5XuPPbhaXM3VzOuD6NuLBLw6Rt+xAd59QxbNiwee6eV/n6pK1pcvcJwASAvLw8Hzp0aLI2HZmCggJSMe6TlWn9heT22d359nOLmbu5kO9ccSa3fLRnUrZbmY5z6tOqFZGY3P/3d3hqdiFfHNortiQu6UGJXCQGf3htNb8tWMUNQ7ryzY+rCJbUTFTLD58C3gRON7P1ZjYuinZF0tEzcwu57/nlXNmvIz/6ZB8VwZIai2SO3N1HRNGOSLr7+9IPuPvZRVx4alt+9TkVwZJoaGpFJEneWLWNrzy1gP652Tw6chCNGujlJ9HQM0kkCRav38X4afPofkozJt80mGaNVARLoqNELlLLVm0tZvTk2bRq2pBpY4eQ3UxFsCRaSuQitWhj0X5GPjaLegaP3zyEDq2axB2SpCElcpFasmPvAUZOnMWekjKmjs2nR9vmcYckaUoTdSK1oLi0jJsmz2b9zv1MG5vP2Z1UBEtqjxK5SMRKy8oZP20uSzfuZsLIQQzpeUrcIUma09SKSITKyiu4/amFvLFqO/df249LzsyJOyTJAErkIhFxd77z3BJeXPoB37vqLK45p0vcIUmGUCIXicjPXlzBH+cW8pWLezP2Iz3iDkcyiBK5SAR+/+oqHn11NSPP7cY3Ljst7nAkwyiRi9TQH+es42cvrODq/p34wSfOVhEsSTolcpEaeHHJJu7502I+elo7Hvhsf+qpCJbEQIlc5CS9sXIbX31qIQNys/n9jeeoCJbERs88kZOwaH0Rt0ybS4+2zZmkIlgSMyVykWpauWUPoyfNpk1WI6aNy1cRLImdErlINWwo2s/IibOpX68e08cOIaelimBJ/JTIRapoe3EpIyfOori0jGlj8+muIlhSRyiRi1TBnpKDjJ48m41F+5l002DO6tQy7pBEDlMiFzmBkoPl3DJtLis27eG3nz+Hwd3bxB2SyBH0UbvIcZSVV/CVpxbw1uod/Pq6AVx8hopgSd2jEbnIMbg79/xpMS8t28y9V5/FpwZ2jjskkaNSIhc5CnfnJ88v55l567n9klO56QIVwZK6S4lc5Ch+9+oq/vDvNYw+rxtfu/TUuMMROa5IErmZDTezd8xspZl9K4o2ReLg7kx7831+/uI7fKJ/J75/tYpgSd1X4w87zaw+8AhwGbAemGNmf3X3ZTVtWySZNu8u4aEFpSzYspRhp7fjFyqCJSkiilUr+cBKd18NYGYzgE8CSuRS55WVV/Du5mL+s3IbD738HiUHyvn2FWcw9oIeNKivmUdJDVEk8s5AYcLf64Ehle9kZuOB8QA5OTkUFBREsOnkKi4uTsm4T1Y69ndnSQWrd1WwqqiCVUXlrNldwYHy4LYz2tTjurOdHhWFvP7vwuM3lEbS8TifSLr1OYpEfrT3nv6hK9wnABMA8vLyfOjQoRFsOrkKCgpIxbhPVqr3d/+BcpZs3MWCdTtZWFjEwnVFbNxVAkDD+sZZHVtyw5mtGZCbzYDcbLqd0oxXX301pft8MlL9OJ+MdOtzFIl8PZCb8HcXYGME7YpUWUWFs2b7XhauK2JBYZC4V2zaQ1lFMKbo0ropg7q3YVxuNgO7ZnNWx5Y0aVg/5qhFohFFIp8DnGpmPYANwPXADRG0K3JMO/ceYOH6IhasKwpH2zvZXVIGQFbjBvTr0opbL+rJgNxgxN2uReOYIxapPTVO5O5eZmZfBv4O1AcmufvSGkcmEjpQVsGKD3b/N2kXFrFm214A6hmcltOCK/t1ZEBuNgO7tqZXuyzqa7WJZJBIaq24+/PA81G0JZnN3dlQtP/wnPaCwiIWb9jFgbIKANq3aMyA3Gw+m9eFgbmt6dulFVmNVTJIMpteARKr4tIyFhUGCfvQaHvrnlIAGjeoR9/OrRh9XjcG5LZmYNdsOrZqoi/oiFSiRC5JU17hrNxSzMLCnYenSd7ZvAcP1zj1bNucC3u3ZWDXbAbktuaMji1oqLXcIiekRC61ZsueEhYmzGu/XVjE3nDRdqumDRmQm83Hz+7AgK7ZDMzN1rkvRU6SErlEouRgOUs37j68ZnvBuiI2FO0HoEE948yOLbnmnC7haDubHm2ba4pEJCJK5FJt7s7a7fvChL2TBYVFLN+0m4PlwRxJ5+ymDMjN5qbzuzOwazZ9OrfSmm2RWqRELie0a99BFq4vCqdJghH3zn0HAWjWqD59O7di3Ed6MjCcImmvM8uLJJUSuRyhrLyCFR/sYUFhES8uKuWH8wpYvTVYs20Gp7bP4uckn/UAAAbESURBVLKzcg6vIjktp4XWbIvETIk8w23atf/wCpIF63ayeMMuSg4Ga7ZbNoL8Xm34zDldGJCbTb8urWjRpGHMEYtIZUrkGWTfgTIWrd91+Ms2CwuL+GB3UESqUYN69OnUkhvyux1eRbLy7VkMGzY45qhF5ESUyNNURYWzamsxCwr/W4/k3c17KA+LSHU7pRlDerZhYPi19jM7tqRRgyPXbK/SqhKRlKBEnia2F5ceXq+9YF2wZntPaVBEqkWTBgzIzeayM3sxoGs2/btkc0qWikiJpAsl8hRUWlbOso27DyfthYVFrNuxD4D69YzTc1pw9YBO4Wg7m55ts3TKMpE0pkRex7k7hTv2syDha+3LNu7mQHnwgWSHlk0Y2DWbzw/pyoDcbPp2aUWzRjqsIplEr/g6ZnfJQRYV7jqiHsn2vQcAaNKwHv06ZzPmgu7BWW26ZtOxVdOYIxaRuCmRx+jQiX8XFO48vIpk5dbiw0WkerVrztDT2x9eRXJGhxY6IbCIfIgSeRJt3l1yxJrtRet3sf9gUESqdbOGDOzamqv7d2Jg12z6dcmmVVOt2RaRE1MiryUlB8tZvGHX4XNILlhXxKbEE/92asV1g3PDs9pk07VNMxWREpGTokQegYoKZ/W2veHyv6AWyfJN/12zndumKXndgzXbA3TiXxGJmBL5STjRiX/757bitvDEvwO7ZtNWa7ZFpBYpkZ9A4ol/X1xUwr1zXuH97cGa7UMn/r2ib0fO6dqaAV2zdeJfEUk6JfIEiSf+PTTaTjzxb6vGxpBeLfhcOLfdv0s2zXXiXxGJWUZnoeLSMhYlTJEsWFfEtuIjT/w76txuDOwaTJG8s+Athg3LizlqEZEjZUwiP3Ti30OnIjtURKoi4cS/Hz21bbhm++gn/n1Xq0pEpA5K20SeeOLfBeuKWLT+6Cf+PXQOSZ34V0RSVY0SuZl9FrgXOBPId/e5UQRVXcGJf3cdMUVS+cS/nxnUJVyz3Zrup2jNtoikj5qOyJcA1wCPRhBLlRw68e+hr7Uf68S/h+qR6MS/IpLuapTI3X05kLTR7UP/eo/J/1lzxIl/+3Vpxc0X9jz8Dcn2LXTiXxHJLOaHKjTVpBGzAuDO402tmNl4YDxATk7OoBkzZlR7O6+tP8h7OyvolV2PXtn16Zxl1EviFElxcTFZWVlJ217cMq2/oD5nilTt87Bhw+a5+4eWzp1wRG5m/wQ6HOWm77j7X6oagLtPACYA5OXl+dChQ6v60MOq/4hoFRQUcDJxp6pM6y+oz5ki3fp8wkTu7pcmIxARETk5Km4tIpLiapTIzezTZrYeOA/4PzP7ezRhiYhIVdV01cpzwHMRxSIiIidBUysiIilOiVxEJMUpkYuIpDglchGRFBfJNzurvVGzrcDapG+45toC2+IOIokyrb+gPmeKVO1zN3dvV/nKWBJ5qjKzuUf7emy6yrT+gvqcKdKtz5paERFJcUrkIiIpTom8eibEHUCSZVp/QX3OFGnVZ82Ri4ikOI3IRURSnBK5iEiKUyI/CWZ2p5m5mbWNO5baZmb3m9kKM1tkZs+ZWXbcMdUWMxtuZu+Y2Uoz+1bc8dQ2M8s1s1fMbLmZLTWz2+OOKRnMrL6ZLTCzmXHHEhUl8moys1zgMmBd3LEkyUtAH3fvB7wL3BNzPLXCzOoDjwCXA2cBI8zsrHijqnVlwB3ufiZwLvClDOgzwO3A8riDiJISefX9CrgLyIhPid39H+5eFv75FtAlznhqUT6w0t1Xu/sBYAbwyZhjqlXuvsnd54eX9xAkt87xRlW7zKwLcCXwWNyxREmJvBrM7BPABnd/O+5YYjIWeCHuIGpJZ6Aw4e/1pHlSS2Rm3YGBwKx4I6l1vyYYiFXEHUiUanRiiXR0vJNNA98GPpbciGpfVU6wbWbfIXgr/kQyY0siO8p1GfGuy8yygGeBr7n77rjjqS1mdhWwxd3nmdnQuOOJkhJ5Jcc62bSZ9QV6AG+bGQRTDPPNLN/dP0hiiJE70Qm2zWw0cBVwiafvFw/WA7kJf3cBNsYUS9KYWUOCJP6Eu/8p7nhq2QXAJ8zsCqAJ0NLMHnf3G2OOq8b0haCTZGbvA3nunooV1KrMzIYDvwQucvetccdTW8ysAcGHuZcAG4A5wA3uvjTWwGqRBSOSqcAOd/9a3PEkUzgiv9Pdr4o7lihojlxO5GGgBfCSmS00s9/HHVBtCD/Q/TLwd4IP/Z5O5yQeugAYCVwcHtuF4WhVUoxG5CIiKU4jchGRFKdELiKS4pTIRURSnBK5iEiKUyIXEUlxSuQiIilOiVxEJMX9f1XWfYDWng0mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def PRelu(a, x):\n",
    "    y= np.zeros(x.shape)\n",
    "    return np.ndarray.flatten(np.max([[y],[x]], axis =0)) + a*np.ndarray.flatten(np.min([[y],[x]], axis =0))\n",
    "\n",
    "x = np.linspace(-5,5,100)\n",
    "a = 0.2\n",
    "plt.title('Graph of the PRelu activation function, a=0.2')\n",
    "plt.plot(x,PRelu(a,x))\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Función de activación Elu, 2016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Elu es definida como\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "f(x) = \\begin{cases} x, & \\text{ si } x > 0\\\\\n",
    "\\alpha (\\exp(x)-1), & \\text{ if } x \\le 0 \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Observe que\n",
    "$$\n",
    "\\begin{equation}\n",
    "f'(x) = \\begin{cases} 1, & \\text{ si } x > 0\\\\\n",
    "f(x)+\\alpha, & \\text{ if } x \\le 0 \\end{cases}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEICAYAAABCnX+uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU5dn/8c/FvoR9CbIGBUE2WQIIqW1wqVpxqUvrRiGo1GqtbbW11tbaxz6tT23tYhfrTw0oQupa11qxGq1BloRVQBBJgLDvECD79ftjhueJEUJgJjmZme/79cormZkz97nuc2a+OXPPmXvM3RERkdjVKOgCREQkMgpyEZEYpyAXEYlxCnIRkRinIBcRiXEKchGRGKcgbwDM7H4zmxmltpLN7H0zO2Bmv63lfQrM7LxorP8464laPyOsY4WZpddBu73NrMjMGtdB22lm9km4/cuj3X4N6z3bzFbX1/rk5CjIj8LMrjGz+WZ20My2h/++1cws6NpqYRqwE2jr7ndWv9HMppvZL+pq5Wbm4e1WVOXnh3W1vlrU87n+uvtgd8+OQtuf+Qfo7hvcPcndKyJt+yj+C/hTuP1/1EH7wP/uv35HLrv7f9x9QF2tL9rMrKOZvRR+DK43s+tqWPYHZvZR+KAn38x+UJ+1RlOToAtoaMzsTuCHwG3Av4AiYDhwF/AEUHKU+zSuoyfvyegDrPRgP+l1pruvDXD98agPsCLoImLAn4FSIJnQ8/Z1M1vq7kfbdgZ8A1gGnAa8ZWYb3T2r3qqNFnfXT/gHaAccBK48znLTgb8Cb4SXPw+4GFgM7Ac2AvdXWT4FcEJHy5uBLcCdVW6/H3gWeAo4QOgJm1rD+scDC4F94d/jq9RVRuiBXAScV+1+06rd/mr4+gJC/6iWhdv8O9Ciyv0mAkuAvcBcYFgNtTnQ7xi33Q/MDP+dDhRWu72ges1Vbjvm9g3f/oVwbXvDt085Tn/PA7oDh4GOVdoZQegVTVNCT+53gF3h654B2oeXexqoDN+/iNA//yP7uUl4me7AK8BuYC1w88nsc+DTautqXn1bVdu2R+qYDGwI135vlWUbAz8Ot3sAyAN6Ae+H73cwvJ6vV99PwBlAdng7rwAurfa8+DPwerjd+cBptXzuHXNbn8Dzt3V4X59e5bqngQdref8/Ao8EkT2R/gReQEP6AS4Eyo88EWtYbjqhwEsjNDzVIvyAHxq+PAzYBlweXv7IE2t2+ME2FNhx5IkYfhIWA18JP8l+Bcw7xro7AnuASYReUV0bvtypSm2/OE7tv6h2XQGwgFDwdARWAbeEbxsJbAfGhmubHF6++THar6sgr2n79g4Hx7WEArgTMPw4/T2y7d/hswH7EPBo+O9+wPmEgrMLoaD7/bHq5fNB/h7wl/DjY3h4n597ovv8GOuqfrnqtj1Sx/8DWgJnEnoleUb49h8Ay4EBhI5Kz6zy+PnM/qu6n8Lbdi2hfwLNgHPC231AlW29GxhD6LH5DJBVy+fe8bb1a4T+eRzt57XwMiOAw9XavYvwP/DjrN8IHSjcEkT2RPqjMfLP6gzsdPfyI1eY2Vwz22tmh83si1WWfdndc9y90t2L3T3b3ZeHLy8jFNpfqtb+z939oLsvBzIJBc8RH7j7Gx4aonma0JPraC4GPnH3p9293N1nAx8Dl0TWdf7o7pvdfTfwKqHgAbgZ+Ju7z3f3CnefQSgUzqqhrUXhbXbk54IIa+M42/d64G13n+3uZe6+y92X1LLpWYT3Q/g9kGvC1+Hua919jruXuPsO4GE+v0+Pysx6EXqVcHf48bEEeJzQP+AjarvPT9bP3f2wuy8FllZp/ybgJ+6+2kOWuvuuWrR3FpBE6Ai31N3fIRSwVR/HL7r7gvBz6Bn+73FUo+Nta3ef6O7tj/EzMbxYEqEDrKr2AW1qUcL9hA4SMmtTb0OjMfLP2gV0NrMmR8Lc3ccDmFkhn31zeGPVO5rZWOBBYAiho5XmwHPV2q96n/WEjjCP2Frl70NAi6p1VNE9fN+q1gM9au7acVVff/fw332AyWZ2e5Xbm1W5/WhGepTHyI+zfXsRGiY4Gc8Dj5hZd6A/oSPS/4TX2ZXQy+2zCYVBI0KvfmqjO7Db3Q9UuW49kFrlcm33+cmq3n5S+O+T3V7dgY3uXlnluuqPvWOts0YRbusjioC21a5rS+hVQ03r/jahsfKz3f1z74HFAh2Rf9aHhI42L6vFstXfTJxFaDy0l7u3Ax4l9HKtql5V/u5NaLz8RG0mFK5V9QY21fL+J/om6Ebgv6sdAbUKvxKIxEGg1ZEL4VP2utSwfE3bdyOhMdajqbG/7r4XeAv4GnAdMNvDr7UJDXc4ofcE2gI38Nl9WlPbm4GOZlb1aPBE9tPxfGb7Ad1O4L41ba+abAZ6mVnV3IhWn2rc1mb2z2pnQlX9+Wd4sTVAEzPrX6XdM6nhTWIzmwr8iNCQV2EU+hEIBXkV4Sf1z4G/mNlVZpZkZo3MbDihse2atCF0BFZsZmMIhUJ1PzWzVmY2GMgg9KbiiXoDON3MrjOzJmb2dWAQoZe4tbENOPUE1vf/gFvMbKyFtDazi6sF1MlYQ+gI9GIzawr8hNBR9rHUtH2fAc4zs6+Ft0mn8D6D2vV3FqEjsivDf1ddZxGw18x6EBpbruqYbbv7RkJvvv7KzFqY2TDgxnCt0bAEuMbMmppZKnDVCdz3ceABM+sf3qfDzKxT+Laattd8Qv9AfhhebzqhIb1aneVhZtlmdv8xbq5xW7v7RR469fJoPxeFlzkIvAj8V/hxmkbooOzpY9RzPfBL4Hx3X1ebPjRUCvJq3P3XwPcJnYWwndAD+2/A3YSemMdyK6EH0AHgPkJnJFT3HqE3i/4N/Mbd3zqJ+nYROovkTkJDQT8EJrr7zlo28QQwKDx2fdzzkd09l9A4+Z8IvdRdS+iMkJosrXbE9PujtLuP0DZ7nNAR3UGgpiOiY25fd99A6E3DOwm92baE/xsPrk1/XyE0rLItPJ58xM8Jvdm7j9CZGC9Wu9+vgJ+E277rKO1eS+iNx83AS8DP3H1ODX08ET8ldFS9J1znrJoX/4yHCW2/twidBfQEoTdFITRWPCPcp69VvZO7lwKXAhcROrPkL8A33P3jWq63F5BzjNuOt61r61ZCfdlO6H2Ub3n41EMLfbipqMqyvyD0xvjCKo/VR09yvYGy/3sVKXXFzFKAfKBpFMc/RWKGmfUEnnP3cUHXEo8U5PVAQS4idUlDKyIiMU5H5CIiMU5H5CIiMS6QDwR17tzZU1JSglh1RA4ePEjr1sc7CzF+JFp/QX1OFLHa57y8vJ3u/rnPWwQS5CkpKeTm5gax6ohkZ2eTnp4edBn1JtH6C+pzoojVPptZ9U91AxpaERGJeQpyEZEYpyAXEYlxCnIRkRinIBcRiXG1DnIze9JCX0T8UZXrOprZHAt9u/ccM+tQN2WKiMixnMgR+XRCX4VW1Y+Af7t7f0Iz+v0oSnWJiEgt1TrI3f19QlOEVnUZMCP89wzg8ijVJSISV4pKyrn/lRXsLy6LetsnNNdKeBa/19x9SPjyXndvX+X2Pe5+1OEVM5tG6FvNSU5OHpWVVau56BuUoqIikpJq9c1VcSHR+gvqc6Ko7z7vKa7kd3klFBZV8p0RzRne9eQ+izlhwoQ8d0+tfn29fbLT3R8DHgNITU31WPxUVax+GuxkJVp/QX1OFPXZ55Wb9/Oj6Qs5UGI8OWU06QO6Rn0dkZ61ss3MTgEI/94eeUkiIvHhvTU7+NrfPgTguVvG10mIQ+RB/gowOfz3ZODlCNsTEYkLs+ZvYOr0hfTq2Ip/3JbGoO5t62xdtR5aMbPZQDrQ2cwKgZ8BDwLPmtmNwAbg6rooUkQkVlRWOg+9tZq/Zn9K+oAu/Om6kSQ1r9tR7Fq37u7XHuOmc6NUi4hITCsuq+DO55by+rItXD+2Nz+/dDBNGtf95y4DmcZWRCTe7D5YyrSncsldv4d7LhrItC+eipnVy7oV5CIiEcrfeZCMzAVs3lfMn68bycXDTqnX9SvIRUQikFuwm5ufysXMmH3zWEb16VjvNSjIRURO0qtLN3Pnc0vp0b4l0zNG06dTMF8fpyAXETlB7s6j763jf978mNEpHXhsUiodWjcLrB4FuYjICSirqOS+lz9i9oKNXHJmdx66ahgtmjYOtCYFuYhILR0oLuO2WYt5f80ObptwGneeP4BGjernzJSaKMhFRGphy77DZGQu5JPtRTx4xVCuGdM76JL+l4JcROQ4Vmzex9TpCzlYUkHmlNF88fQuQZf0GQpyEZEavLt6O99+ZhHtWjbl+W+NY2C3upsz5WQpyEVEjmHmvPX87JUVDOzWhienjCa5bYugSzoqBbmISDWVlc7/vPkxf3t/HecM7Moj146gdR1PfBWJhluZiEgAissq+P6zS3hj+VYmndWHn10yqF4mvoqEglxEJGxXUQk3P5XL4o17ufcrZ3DT2X3rbeKrSCjIRUSAdTuKmJK5kG37i/nr9SO5cEj9TnwVCQW5iCS8Bfm7mfZ0Lo3NyJp2FiN6H/U75BssBbmIJLSXl2ziB88to2fHlkyfMobenVoFXdIJi8oIvpl9z8xWmNlHZjbbzBrmOToiImHuzp/e+YQ7spYwond7XvzW+JgMcYhCkJtZD+A7QKq7DwEaA9dE2q6ISF0pr3TufmEZv3lrDZcP785TN46hfavgZi+MVLSGVpoALc2sDGgFbI5SuyIiUbW/uIzf5RWzYlch3zmnH987//SYODOlJhEHubtvMrPfABuAw8Bb7v5WxJWJiETZpr2HmZq5kLW7K3noqmFcndor6JKiwtw9sgbMOgAvAF8H9gLPAc+7+8xqy00DpgEkJyePysrKimi9QSgqKiIpKSnoMupNovUX1Od4VrCvgt8vKqGkwrlpoDOqZ+z1ecKECXnunlr9+mgMrZwH5Lv7DgAzexEYD3wmyN39MeAxgNTUVE9PT4/CqutXdnY2sVj3yUq0/oL6HK/+vWobv35nMR1ateDZjNFsXpUXV32OxlkrG4CzzKyVhQaazgVWRaFdEZGIPfVhATc/lUu/rkm8dNt4Tk9uE3RJUReNMfL5ZvY8sAgoBxYTPvIWEQlKZaXzyzdW8fgH+Zx3Rlf+eO0IWjWLz4/ORKVX7v4z4GfRaEtEJFKHSyv43t+X8OaKrUwZn8JPJw6icQP4Sra6Ep//nkQkYe0sKuHGGbksK9zLfRMHMfULfYMuqc4pyEUkbqzdXkTG9AXsOFDCozeM4oLB3YIuqV4oyEUkLsxbt4tvPp1H08bG36eN48xe7YMuqd4oyEUk5v1j8SZ+8PxS+nRqTeaU0fTqGJtzppwsBbmIxKzQxFdr+e2cNYw7tROP3jCKdq2aBl1WvVOQi0hMKi2v5McvLef5vEKuGNmDB68YRrMmDfsr2eqKglxEYs6+w2Xc+kweOWt38d3z+nPHuf1jfuKrSCjIRSSmFO45REbmQgp2HeS3V5/JlaN6Bl1S4BTkIhIzlhXu5cYZuRSXVTBj6hjGn9Y56JIaBAW5iMSEt1du4/bZi+mU1IxZN42lfxzOmXKyFOQi0uBNz8nnv15bydAe7Xh88mi6tGkedEkNioJcRBqsikrnv19fxZM5+Xx5UDJ/uGYELZs1DrqsBkdBLiIN0qHScr6btYS3Vm5jalpf7r34jLie+CoSCnIRaXC2Hyjmphm5fLRpH/dfMogpafE/8VUkFOQi0qB8su0AUzIXsvtgKX+blMr5g5KDLqnBU5CLSIMxd+1OvjkzjxZNG/PsN8cxtGe7oEuKCQpyEWkQXsgr5EcvLqNv59Y8OWU0PTsk1sRXkVCQi0ig3J3fvf0Jf/z3J6T168Rfrh9Fu5aJN/FVJKIS5GbWHngcGAI4MNXdP4xG2yISv0rLK/nRC8t4cfEmrhrVk19+dWjCTnwViWgdkf8BeNPdrzKzZoBeE4lIjfYdKuObM3OZt2433z//dG4/p19CT3wViYiD3MzaAl8EpgC4eylQGmm7IhK/Nu4+xJTMBWzcfZjff304l4/oEXRJMc3cPbIGzIYDjwErgTOBPOAOdz9YbblpwDSA5OTkUVlZWRGtNwhFRUUkJSUFXUa9SbT+gvpcH9btreD3i4qpcLh9RAsGdqz/T2rG6n6eMGFCnrunVr8+GkGeCswD0tx9vpn9Adjv7j891n1SU1M9Nzc3ovUGITs7m/T09KDLqDeJ1l9Qn+vav1Zs5Y6sxXRp05zMKWPo1zWYMI3V/WxmRw3yaLyrUAgUuvv88OXngZFRaFdE4siTH+Rzy8w8BnZry0u3pgUW4vEo4jFyd99qZhvNbIC7rwbOJTTMIiJCRaXzwGsrmT63gAsGJ/P7r2viq2iL1lkrtwPPhM9YWQdkRKldEYlhh0rL+c7sxby9ajs3n92Xey46g0aa+CrqohLk7r4E+Ny4jYgkru37i7lxRi4rNu/jgcsGM2lcStAlxS19slNEom711gNMnb6QPYdK+X/fSOXcMzTxVV1SkItIVH3wyU6+NTOPls1CE18N6aGJr+qaglxEoubZ3I38+MXlnNYlicyM0XRv3zLokhKCglxEIubuPDxnDY+8s5az+3fmz9ePpG0LTXxVXxTkIhKRkvIK7n5+Gf9Yspmvp/biF18dQtPGmviqPinIReSk7T1UyrSn81iQv5sfXDCAW9NP08RXAVCQi8hJ2bDrEFOmL6Bw92H+cM1wLhuuia+CoiAXkRO2aMMebp6RS4U7M28ay5i+HYMuKaEpyEXkhPxz+Ra++/clJLdtQWbGaE7rojlTgqYgF5FacXce/08+v/znKob3as/j30ilU1LzoMsSFOQiUgvlFZXc/+oKZs7bwFeGduPhrw2nRVNNfNVQKMhFpEYHS8q5ffZi3vl4O9/84qncfeFATXzVwCjIReSYtu0vZur0hXy89QC/uHwIN5zVJ+iS5CgU5CJyVB9v3U9G5kL2Hy7j8cmpTBjQNeiS5BgU5CLyOe+v2cGtzyyidfPGPHvLOAZ318RXDZmCXEQ+I2vBBu79x0f07xqa+OqUdpr4qqFTkIsIAJWVzm/eWs1fsj/l7P6d+cv1I2mjia9iQtSC3MwaA7nAJnefGK12RaTuFZdV8IPnl/Hq0s1cO6YX/3WZJr6KJdE8Ir8DWAW0jWKbIlLHikqdSU/MZ2HBHu6+cCC3fOlUTXwVY6LyL9fMegIXA49Hoz0RqR8FOw/ywLzDLC3cxyPXjuBbmr0wJpm7R96I2fPAr4A2wF1HG1oxs2nANIDk5ORRWVlZEa+3vhUVFZGUlDjzSiRafyGx+vzJngr+sKgYd+e7o1rSv0PifFIzVvfzhAkT8tz9c190H/HQiplNBLa7e56ZpR9rOXd/DHgMIDU11dPTj7log5WdnU0s1n2yEq2/kDh9fn3ZFh56ewnd27XilkHONRefE3RJ9Sre9nM0hlbSgEvNrADIAs4xs5lRaFdEoszdefS9T7lt1iKG9WjHS7em0a213tSMdRHvQXe/x917unsKcA3wjrvfEHFlIhJV5RWV/OQfH/HgPz/mkjO7M/OmsXRo3SzosiQKdB65SAIoKinn27MWkb16B7emn8ZdXx6gia/iSFSD3N2zgexotikikdmy7zBTp+eyZtsBHrxiKNeM6R10SRJlOiIXiWMrNu9j6vSFHCyp4Mkpo/nS6V2CLknqgIJcJE69u3o7335mEW1bNuW5W8Zxxin6rF68UpCLxKFn5q/nvpdXMCC5DZkZo0lu2yLokqQOKchF4khlpfM///qYv723jgkDuvDIdSNJaq6nebzTHhaJE8VlFdz57FJeX76F68f25ueXDqaJJr5KCApykTiw+2ApNz+VS976Pdxz0UCmfVETXyUSBblIjMvfeZCMzAVs2VfMX64fyVeGnhJ0SVLPFOQiMWxhwW5ufiqXRmbMuvksRvXpEHRJEgAFuUiMemXpZu56dik9O7QkM2M0fTq1DrokCYiCXCTGuDt/fe9Tfv3masakdOSxb4yifSvNmZLIFOQiMaSsopL7Xv6I2Qs2ctnw7vz6qmE0b5I484jL0SnIRWLEgeIybpu1mPfX7OD2c/rx/fNP15kpAijIRWLCln2HychcyNrtRfz6ymF8bXSvoEuSBkRBLtLAfbRpHzfOWMihkgoyM0Zzdn9NfCWfpSAXacDe/Xg7t81aRPuWTXnuW+MY2E0TX8nnKchFGqiZ89Zz38sfMah7W56YrImv5NgU5CINTGWl8+CbH/PY++s4Z2BXHrl2BK018ZXUQI8OkQakuKyC7z+7hDeWb+Ub4/pw38RBmvhKjiviIDezXsBTQDegEnjM3f8QabsiiWZXUQk3PZXLko17+cnFZ3DjF/rq9EKplWgckZcDd7r7IjNrA+SZ2Rx3XxmFtkUSwqc7isjIXMi2/cX89fqRXDhEE19J7UUc5O6+BdgS/vuAma0CegAKcpFamL9uF9OezqNJIyNr2lmM6K2Jr+TEmLtHrzGzFOB9YIi776922zRgGkBycvKorKysqK23vhQVFZGUlBR0GfUm0foL9d/nDzeX88TyErq0Mr43qgVdW9X/eLj2c+yYMGFCnrunVr8+akFuZknAe8B/u/uLNS2bmprqubm5UVlvfcrOziY9PT3oMupNovUX6q/P7s6f3lnLb+esYWzfjvxtUnATX2k/xw4zO2qQR+WsFTNrCrwAPHO8EBdJdGUVldz70nKezS3kqyN68OCVQzXxlUQkGmetGPAEsMrdH468JJH4tb+4jFtnLuKDtTv5zjn9+J4mvpIoiMYReRowCVhuZkvC1/3Y3d+IQtsicWPT3sNkZC5g3Y6DPHTVMK5O1cRXEh3ROGvlA0CHFCI1+GjTPjKmL6S4rIIZU8eQ1q9z0CVJHNEnO0Xq2Nsrt3H77MV0bN2MZ24ay+nJbYIuSeKMglykDj31YQH3v7KCIT3a8fjkVLq20cRXEn0KcpE6UFHp/PKNVTzxQT7nnZHMH68dTqtmerpJ3dAjSyTKDpdW8N2/L+ZfK7YxZXwKP504iMaN9DaS1B0FuUgU7Swq4cYZuSwr3Mt9Ewcx9Qt9gy5JEoCCXCRK1m4vImP6AnYcKOFvN4ziy4O7BV2SJAgFuUgUzFu3i2lP5dKsSSOypo1jeK/2QZckCURBLhKhlxYX8sPnl9GnU2syp4ymV8dWQZckCUZBLnKS3J1H3lnLw3PWMO7UTjx6wyjatWoadFmSgBTkIiehtLySH7+0nOfzCrliZA8evGIYzZroK9kkGApykRO073AZ35qZx9xPd/G9807nO+f208RXEigFucgJ2Lj7EFOnL6Rg10F+e/WZXDmqZ9AliSjIRWprWeFepk7PpaQ8NPHV+NM08ZU0DApykVp4a8VW7shaQsfWzZh981j6a+IraUAU5CLH8eQH+Tzw+kqG9WjH45NH06VN86BLEvkMBbnIMVRUOg+8tpLpcwu4YHAyv//6CFo201eyScOjIBc5ikOl5Xxn9hLeXrWNqWl9uffiMzTxlTRYUTnx1cwuNLPVZrbWzH4UjTZFgrL9QDHXPDaPdz7exs8vHcx9l2j2QmnYovHly42BPwPnA4XAQjN7xd1XRtq2SH1bvL2cux/5gP2Hy3lsUirnDUoOuiSR44rG0MoYYK27rwMwsyzgMkBBLjFjZ1EJ97+ygteWlTCwWxuemDyaIT3aBV2WSK2Yu0fWgNlVwIXuflP48iRgrLt/u9py04BpAMnJyaOysrIiWm8QioqKSEpKCrqMepMI/d1+qJI568v4T2E5ZZVwYS/nqwNb0ySBhlISYT9XF6t9njBhQp67p1a/PhpH5Ed7xH/uv4O7PwY8BpCamurp6elRWHX9ys7OJhbrPlnx2l93J2ftLjJz8nln9XYam3HxsO58e0I/Nq3Ki8s+1yRe93NN4q3P0QjyQqBXlcs9gc1RaFckqg6XVvDS4k1Mn5vPmm1FdGrdjNsn9OP6s/qQ3Db0pcibVgVcpMhJiEaQLwT6m1lfYBNwDXBdFNoViYrNew/z1IfryVq4gb2Hyhh0SlseumoYl5zZnRZNdV64xL6Ig9zdy83s28C/gMbAk+6+IuLKRCLg7uSt30NmTgFvrtiKu/PlQd3ISEthTN+Omq1Q4kpUPhDk7m8Ab0SjLZFIlJRX8PqyLWTmFLB80z7atmjCTV/oy6RxfejZQd/cI/FJn+yUuLD9QDGz5m9g5rwN7CwqoV/XJH5x+RCuGNmDVs30MJf4pke4xLTlhfvIzMnn1WWbKatwzhnYlYy0FL7Qr7OGTyRhKMgl5pRXVPLmiq1k5hSQt34PrZs15vqxfZg8PoW+nVsHXZ5IvVOQS8zYc7CU2Qs38PSH69myr5jeHVvx04mDuDq1J21b6EuPJXEpyKXBW731AJk5+by0eBMl5ZWk9evEA5cNYcLArprMSgQFuTRQFZXOOx9vJzMnn7mf7qJ5k0ZcMbIHU8b3ZUA3fTuPSFUKcmlQ9heX8VxuITPmFrBh9yFOadeCH144gGtH96ZD62ZBlyfSICnIpUFYt6OIGXMLeD6vkIOlFaT26cDdFw7kgsHJNGkclWnzReKWglwC4+7855OdZObk8+7qHTRtbFwyrDsZaX0Z2lNTyIrUloJc6t2h0nJeWLSJ6Tn5fLrjIJ2TmnHHuf25/qzedG3TIujyRGKOglzqTeGeQ6HJqxZsYH9xOUN7tOPhr53JxcNOoXkTTV4lcrIU5FKn3J0F+bvJzCngrZVbMTMuHNKNjPEpjOrTQZ++FIkCBbnUieKyCl5dupnMnAJWbtlP+1ZNmfbF0/jGuD50b98y6PJE4oqCXKJq2/5iZs5bz6z5G9h1sJTTk5P41RVDuXx4D1o20/CJSF1QkEtULN4Qmvv7jeVbqHDn3IFdyUjry/jTOmn4RKSOKcjlpJVVVPLG8tDc30s27iWpeRMmjevD5HEppGjyKpF6oyCXE7arqITZCzbw9Lz1bNtfQt/Orbn/kkFcldqLpOZ6SInUt4iedWb2EHAJUAp8CmS4+95oFCYNz6ot+8nMyecfSzZTWl7J2f078+AVw/jS6V1opMmrRAIT6eHTHOCe8Pd2/g9wD3B35GVJQ1FR6cxZuY3pc/OZtwxm33UAAAo8SURBVG43LZs25upRPZkyPoX+yZq8SqQhiCjI3f2tKhfnAVdFVo40FPsOl/HP/DJ+Mv9dCvccpkf7ltxz0UCuGd2bdq0097dIQ2LuHp2GzF4F/u7uM49x+zRgGkBycvKorKysqKy3PhUVFZGUlBR0GXVqc1Elb28oI2dTOSUVcHqHRpzfpykjuzZOiLm/E2EfV6c+x44JEybkuXtq9euPG+Rm9jbQ7Sg33evuL4eXuRdIBa7wWvxnSE1N9dzc3FoV3pBkZ2eTnp4edBlRV1npvPfJDjJzCnh/zQ6aNW7EpcO7M7T5TiZfem7Q5dWreN3HNVGfY4eZHTXIjzu04u7nHafhycBE4NzahLg0HAdLynlhUSHT5xawbsdBurZpzp3nn861Y3vTOak52dnZQZcoIrUQ6VkrFxJ6c/NL7n4oOiVJXdu4+xAz5hbw99yNHCgu58xe7fnDNcO5aMgpNGuiub9FYk2kZ638CWgOzAl/em+eu98ScVUSde7Oh+t2kZlTwNurttHYjIuGnkJGWgoje3cIujwRiUCkZ630i1YhUjeKyyp4eckmMnMK+HjrATq0asqt6acx6awUurXT3N8i8UAfw4tTW/Yd/t/Jq/YcKmNgtzb8+sphXDq8Oy2aavIqkXiiII8j7s6iDXvJzMnnzY+2UuHO+Wckk5HWl7NO7ajJq0TilII8DpSWH5m8Kp+lhfto06IJU8anMHl8Cr06tgq6PBGpYwryGLbjQAmz5m9g5vz17DhQwqldWvPAZYO5YmRPWmvyKpGEoWd7DPpo0z4ycwp4delmSisqSR/QhYy0vpzdr7MmrxJJQAryGFFeUcmcldvIzClgQcFuWjVrzNdH92Ly+BT6dY29jxqLSPQoyBu4fYfKyFq4gac+XM+mvYfp2aElP7n4DK5O7UW7lpq8SkQU5A3WJ9sOkDm3gJcWbeJwWQVnndqR+y4ZxHlnJCfE5FUiUnsK8gakstLJXrOdzJwC/vPJTpo3acTlw3swJS2FM05pG3R5ItJAKcgbgAPFZTyfV8iMuQUU7DpEt7Yt+MEFA7h2TG86tm4WdHki0sApyAO0ftdBps8t4LncQopKyhnZuz3f//IALhrSjaaNNXmViNSOgryeuTs5a3eRmZPPO6u306SRcfHQU8hI68uZvdoHXZ6IxCAFeT05XFrBS4s3MX1uPmu2FdE5qRm3n9OfG8b2pmtbTV4lIidPQV7HNu09zNMfridr4Qb2HipjcPe2/ObqM5k47BRNXiUiUaEgrwPuTu76PWTm5POvFdtwdy4Y3I2MtL6MTumgyatEJKoU5FFUUl7Bq0u3MH1uPh9t2k/bFk246Qt9mTSuDz07aPIqEakbCvIo2H6gmJnzNjBr/np2FpXSr2sS//3VIXx1RA9aNdMmFpG6pZSJwLLCvWTmFPDass2UVTjnDOxKRloKX+jXWcMnIlJvohLkZnYX8BDQxd13RqPNhqqsopJ/rdhKZk4Beev30LpZY64f24fJ41Po27l10OWJSAKKOMjNrBdwPrAh8nIart0HS3nt01Lu+fBdtuwrpk+nVtw3cRBXpfakbQtNXiUiwYnGEfnvgB8CL0ehrQbn4637mZ5TwEuLN1FSXklav7Y8cNkQJgzsqsmrRKRBMHc/+TubXQqc6+53mFkBkHqsoRUzmwZMA0hOTh6VlZV10uuta5XuLN1RwVsFZazaXUmzRjC+exPSupTRPzlx5v4uKioiKSlx+gvqc6KI1T5PmDAhz91Tq19/3CA3s7eBbke56V7gx8CX3X3f8YK8qtTUVM/Nza1V4fVpf3EZz+WGJq/asPsQ3du1YNK4FK4Z3YsOrZuRnZ1Nenp60GXWm0TrL6jPiSJW+2xmRw3y4w6tuPt5x2hwKNAXWBo+Q6MnsMjMxrj71gjrrVfrdhQxY24Bz+cVcrC0gtQ+Hbj7woFcMDiZJpq8SkQauJMeI3f35UDXI5dP5Ii8IXB3/vPJTjJz8nl39Q6aNjYuGdadjLS+DO3ZLujyRERqLeHOIz9UWs4LizYxY24Ba7cX0TmpOd89rz/Xje1N1zaavEpEYk/UgtzdU6LVVl3YuPsQT89bT9aCDewvLmdoj3Y8/LUzuXjYKTRvosmrRCR2xfURubszP383mTn5zFm5DTPjwiHdmJqWwsjemrxKROJDXAZ5cVkFryzdTGZOAau27Kd9q6Z880unMemsPnRv3zLo8kREoiqugnzb/mJmzlvPrPkb2HWwlNOTk/jVFUO5fHgPWjbT8ImIxKe4CPIlG/eSmZPP68u2UOHOuQOTyUhLYfxpnTR8IiJxL2aDvKyikjeWbyEzp4AlG/fSpnkTvjEuhcnj+9CnkyavEpHEEXNBvquohNkLNvD0vPVs219C386t+fmlg7lyVE+Smsdcd0REIhZTyffIvz/hkXfXUlpeydn9O/PgFcP40uldaKTJq0QkgcVUkHdv35KrR/UkIy2Ffl3bBF2OiEiDEFNBfuWonlw5qmfQZYiINCiaEUpEJMYpyEVEYpyCXEQkxinIRURinIJcRCTGKchFRGKcglxEJMYpyEVEYpy5e/2v1GwHsL7eVxy5zkBMfCdplCRaf0F9ThSx2uc+7t6l+pWBBHmsMrNcd08Nuo76kmj9BfU5UcRbnzW0IiIS4xTkIiIxTkF+Yh4LuoB6lmj9BfU5UcRVnzVGLiIS43RELiIS4xTkIiIxTkF+EszsLjNzM+scdC11zcweMrOPzWyZmb1kZu2DrqmumNmFZrbazNaa2Y+CrqeumVkvM3vXzFaZ2QozuyPomuqDmTU2s8Vm9lrQtUSLgvwEmVkv4HxgQ9C11JM5wBB3HwasAe4JuJ46YWaNgT8DFwGDgGvNbFCwVdW5cuBOdz8DOAu4LQH6DHAHsCroIqJJQX7ifgf8EEiId4nd/S13Lw9fnAfE63ftjQHWuvs6dy8FsoDLAq6pTrn7FndfFP77AKFw6xFsVXXLzHoCFwOPB11LNCnIT4CZXQpscvelQdcSkKnAP4Muoo70ADZWuVxInIdaVWaWAowA5gdbSZ37PaEDscqgC4mmmPry5fpgZm8D3Y5y073Aj4Ev129Fda+mPrv7y+Fl7iX0UvyZ+qytHtlRrkuIV11mlgS8AHzX3fcHXU9dMbOJwHZ3zzOz9KDriSYFeTXuft7RrjezoUBfYKmZQWiIYZGZjXH3rfVYYtQdq89HmNlkYCJwrsfvBw8KgV5VLvcENgdUS70xs6aEQvwZd38x6HrqWBpwqZl9BWgBtDWzme5+Q8B1RUwfCDpJZlYApLp7LM6gVmtmdiHwMPAld98RdD11xcyaEHoz91xgE7AQuM7dVwRaWB2y0BHJDGC3u3836HrqU/iI/C53nxh0LdGgMXI5nj8BbYA5ZrbEzB4NuqC6EH5D99vAvwi96fdsPId4WBowCTgnvG+XhI9WJcboiFxEJMbpiFxEJMYpyEVEYpyCXEQkxinIRURinIJcRCTGKchFRGKcglxEJMb9f6ye4v7ypAhSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def Elu(a, x):\n",
    "    y= np.zeros(x.shape)\n",
    "    return np.ndarray.flatten(np.max([[y],[x]], axis =0)) + a*np.ndarray.flatten(np.min([a*(np.exp([x]) -1),[x]], axis =0))\n",
    "\n",
    "x = np.linspace(-5,5,100)\n",
    "a = 1.0\n",
    "plt.title('Gráfica de la función de activación Elu, a=0.2')\n",
    "plt.plot(x,Elu(a,x))\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La siguiente tabla tomada de [Wikipedia](https://en.wikipedia.org/wiki/Activation_function) presenta las funciones de activación mas conocidas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table class=\"wikitable sortable bs-exportable exportable\">\n",
    "\n",
    "<tbody><tr>\n",
    "<th>Name\n",
    "</th>\n",
    "<th class=\"unsortable\">Plot\n",
    "</th>\n",
    "<th>Function, <span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle f(x)}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>f</mi>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle f(x)}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/202945cce41ecebb6f643f31d119c514bec7a074\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:4.418ex; height:2.843ex;\" alt=\"f(x)\"/></span>\n",
    "</th>\n",
    "<th><a href=\"/wiki/Derivative\" title=\"Derivative\">Derivative</a> of <span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle f}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>f</mi>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle f}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/132e57acb643253e7810ee9702d9581f159a1c61\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.671ex; width:1.279ex; height:2.509ex;\" alt=\"f\"/></span>, <span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle f'(x)}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>f</mi>\n",
    "          <mo>&#x2032;</mo>\n",
    "        </msup>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle f'(x)}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/d0cd7d7c75340e779d82658e19d1720ce84ab127\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:5.144ex; height:3.009ex;\" alt=\"f&#039;(x)\"/></span>\n",
    "</th>\n",
    "<th><a href=\"/wiki/Interval_(mathematics)#Notations_for_intervals\" title=\"Interval (mathematics)\">Range</a>\n",
    "</th>\n",
    "<th><a href=\"/wiki/Smoothness#Order_of_continuity\" title=\"Smoothness\">Order of continuity</a>\n",
    "</th></tr>\n",
    "<tr>\n",
    "<td><a href=\"/wiki/Identity_function\" title=\"Identity function\">Identity</a>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_identity.svg\" class=\"image\"><img alt=\"Activation identity.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Activation_identity.svg/120px-Activation_identity.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Activation_identity.svg/180px-Activation_identity.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Activation_identity.svg/240px-Activation_identity.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle x}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>x</mi>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle x}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/87f9e315fd7e2ba406057a97300593c4802b53e4\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:1.33ex; height:1.676ex;\" alt=\"x\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle 1}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mn>1</mn>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle 1}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/92d98b82a3778f043108d4e20960a9193df57cbf\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:1.162ex; height:2.176ex;\" alt=\"1\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (-\\infty ,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (-\\infty ,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/0c8c11c44279888c9e395eeb5f45d121348ae10a\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:9.299ex; height:2.843ex;\" alt=\"(-\\infty ,\\infty )\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{\\infty }}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{\\infty }}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/971ed05871d69309df32efdfd2020128c9cf69d8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:3.673ex; height:2.343ex;\" alt=\"C^{\\infty }\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td><a href=\"/wiki/Heaviside_step_function\" title=\"Heaviside step function\">Binary step</a>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_binary_step.svg\" class=\"image\"><img alt=\"Activation binary step.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/4/4b/Activation_binary_step.svg/120px-Activation_binary_step.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/4/4b/Activation_binary_step.svg/180px-Activation_binary_step.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/4/4b/Activation_binary_step.svg/240px-Activation_binary_step.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2265;<!-- ≥ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/5620555e698f283ad8d1db03be0bb1ec5e3bba25\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:13.51ex; height:6.176ex;\" alt=\"{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x\\neq 0\\\\{\\text{undefined}}&amp;{\\text{if }}x=0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2260;<!-- ≠ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>undefined</mtext>\n",
    "                  </mrow>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>=</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x\\neq 0\\\\{\\text{undefined}}&amp;{\\text{if }}x=0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/035155c3d067d90da5c17df00d70406d06e8c285\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:22.233ex; height:6.176ex;\" alt=\"{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x\\neq 0\\\\{\\text{undefined}}&amp;{\\text{if }}x=0\\end{cases}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\{0,1\\}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo fence=\"false\" stretchy=\"false\">{</mo>\n",
    "        <mn>0</mn>\n",
    "        <mo>,</mo>\n",
    "        <mn>1</mn>\n",
    "        <mo fence=\"false\" stretchy=\"false\">}</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\{0,1\\}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/28de5781698336d21c9c560fb1cbb3fb406923eb\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:5.684ex; height:2.843ex;\" alt=\"\\{0,1\\}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{-1}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mo>&#x2212;<!-- − --></mo>\n",
    "            <mn>1</mn>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{-1}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/fbeead216a363d09a6d0a05e192bdc3e7ed1067f\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:4.131ex; height:2.676ex;\" alt=\"{\\displaystyle C^{-1}}\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td><a href=\"/wiki/Logistic_function\" title=\"Logistic function\">Logistic</a>, sigmoid, or soft<span class=\"nowrap\">&#160;</span>step\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_logistic.svg\" class=\"image\"><img alt=\"Activation logistic.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Activation_logistic.svg/120px-Activation_logistic.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Activation_logistic.svg/180px-Activation_logistic.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Activation_logistic.svg/240px-Activation_logistic.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\sigma (x)={\\frac {1}{1+e^{-x}}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>&#x03C3;<!-- σ --></mi>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "        <mo>=</mo>\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mfrac>\n",
    "            <mn>1</mn>\n",
    "            <mrow>\n",
    "              <mn>1</mn>\n",
    "              <mo>+</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mo>&#x2212;<!-- − --></mo>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "            </mrow>\n",
    "          </mfrac>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\sigma (x)={\\frac {1}{1+e^{-x}}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/b2a78e39c42d7d51c4041d142740a7719e55b314\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.171ex; width:15.941ex; height:5.509ex;\" alt=\"{\\displaystyle \\sigma (x)={\\frac {1}{1+e^{-x}}}}\"/></span><sup class=\"reference plainlinks nourlexpansion\" id=\"ref_logistic1\"><a class=\"external autonumber\" href=\"https://en.wikipedia.org/wiki/Activation_function#endnote_logistic1\">[1]</a></sup>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle f(x)(1-f(x))}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>f</mi>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mn>1</mn>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mi>f</mi>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle f(x)(1-f(x))}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/4c31efde5f30f83edf924af9feb31f0eb59d1a98\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:14.647ex; height:2.843ex;\" alt=\"{\\displaystyle f(x)(1-f(x))}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (0,1)}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mn>0</mn>\n",
    "        <mo>,</mo>\n",
    "        <mn>1</mn>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (0,1)}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/c79c6838e423c1ed3c7ea532a56dc9f9dae8290b\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:5.168ex; height:2.843ex;\" alt=\"(0,1)\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{\\infty }}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{\\infty }}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/971ed05871d69309df32efdfd2020128c9cf69d8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:3.673ex; height:2.343ex;\" alt=\"C^{\\infty }\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Hyperbolic tangent (<a href=\"/wiki/Hyperbolic_function#Hyperbolic_tangent\" class=\"mw-redirect\" title=\"Hyperbolic function\">tanh</a>)\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_tanh.svg\" class=\"image\"><img alt=\"Activation tanh.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/c/cb/Activation_tanh.svg/120px-Activation_tanh.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/c/cb/Activation_tanh.svg/180px-Activation_tanh.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/c/cb/Activation_tanh.svg/240px-Activation_tanh.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\tanh(x)={\\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>tanh</mi>\n",
    "        <mo>&#x2061;<!-- ⁡ --></mo>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "        <mo>=</mo>\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mfrac>\n",
    "            <mrow>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <mo>&#x2212;<!-- − --></mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mo>&#x2212;<!-- − --></mo>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "            </mrow>\n",
    "            <mrow>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <mo>+</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mo>&#x2212;<!-- − --></mo>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "            </mrow>\n",
    "          </mfrac>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\tanh(x)={\\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/f8e81902c8d71b06c246769bad0fe17c9cf1efd9\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.171ex; width:20.357ex; height:5.843ex;\" alt=\"{\\displaystyle \\tanh(x)={\\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle 1-f(x)^{2}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mn>1</mn>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mi>f</mi>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <msup>\n",
    "          <mo stretchy=\"false\">)</mo>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mn>2</mn>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle 1-f(x)^{2}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/2bb42f038020ac20689657ce6f6f683296554e1f\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:9.475ex; height:3.176ex;\" alt=\"{\\displaystyle 1-f(x)^{2}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (-1,1)}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mn>1</mn>\n",
    "        <mo>,</mo>\n",
    "        <mn>1</mn>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (-1,1)}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/e120a3bd60fc89b495dd7ef6039465b7e6a703b1\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:6.976ex; height:2.843ex;\" alt=\"(-1,1)\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{\\infty }}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{\\infty }}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/971ed05871d69309df32efdfd2020128c9cf69d8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:3.673ex; height:2.343ex;\" alt=\"C^{\\infty }\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td><a href=\"/wiki/Rectifier_(neural_networks)\" title=\"Rectifier (neural networks)\">Rectified linear unit</a> (ReLU)<sup id=\"cite_ref-9\" class=\"reference\"><a href=\"#cite_note-9\">&#91;9&#93;</a></sup>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_rectified_linear.svg\" class=\"image\"><img alt=\"Activation rectified linear.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/f/fe/Activation_rectified_linear.svg/120px-Activation_rectified_linear.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/f/fe/Activation_rectified_linear.svg/180px-Activation_rectified_linear.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/f/fe/Activation_rectified_linear.svg/240px-Activation_rectified_linear.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{aligned}&amp;{\\begin{cases}0&amp;{\\text{if }}x\\leq 0\\\\x&amp;{\\text{if }}x&gt;0\\end{cases}}\\\\{}={}&amp;\\max\\{0,x\\}=x{\\textbf {1}}_{x&gt;0}\\end{aligned}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mtable columnalign=\"right left right left right left right left right left right left\" rowspacing=\"3pt\" columnspacing=\"0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em\" displaystyle=\"true\">\n",
    "            <mtr>\n",
    "              <mtd />\n",
    "              <mtd>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mrow>\n",
    "                    <mo>{</mo>\n",
    "                    <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "                      <mtr>\n",
    "                        <mtd>\n",
    "                          <mn>0</mn>\n",
    "                        </mtd>\n",
    "                        <mtd>\n",
    "                          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                            <mtext>if&#xA0;</mtext>\n",
    "                          </mrow>\n",
    "                          <mi>x</mi>\n",
    "                          <mo>&#x2264;<!-- ≤ --></mo>\n",
    "                          <mn>0</mn>\n",
    "                        </mtd>\n",
    "                      </mtr>\n",
    "                      <mtr>\n",
    "                        <mtd>\n",
    "                          <mi>x</mi>\n",
    "                        </mtd>\n",
    "                        <mtd>\n",
    "                          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                            <mtext>if&#xA0;</mtext>\n",
    "                          </mrow>\n",
    "                          <mi>x</mi>\n",
    "                          <mo>&gt;</mo>\n",
    "                          <mn>0</mn>\n",
    "                        </mtd>\n",
    "                      </mtr>\n",
    "                    </mtable>\n",
    "                    <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "                  </mrow>\n",
    "                </mrow>\n",
    "              </mtd>\n",
    "            </mtr>\n",
    "            <mtr>\n",
    "              <mtd>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "\n",
    "                </mrow>\n",
    "                <mo>=</mo>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "\n",
    "                </mrow>\n",
    "              </mtd>\n",
    "              <mtd>\n",
    "                <mi></mi>\n",
    "                <mo movablelimits=\"true\" form=\"prefix\">max</mo>\n",
    "                <mo fence=\"false\" stretchy=\"false\">{</mo>\n",
    "                <mn>0</mn>\n",
    "                <mo>,</mo>\n",
    "                <mi>x</mi>\n",
    "                <mo fence=\"false\" stretchy=\"false\">}</mo>\n",
    "                <mo>=</mo>\n",
    "                <mi>x</mi>\n",
    "                <msub>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                      <mtext mathvariant=\"bold\">1</mtext>\n",
    "                    </mrow>\n",
    "                  </mrow>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mi>x</mi>\n",
    "                    <mo>&gt;</mo>\n",
    "                    <mn>0</mn>\n",
    "                  </mrow>\n",
    "                </msub>\n",
    "              </mtd>\n",
    "            </mtr>\n",
    "          </mtable>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{aligned}&amp;{\\begin{cases}0&amp;{\\text{if }}x\\leq 0\\\\x&amp;{\\text{if }}x&gt;0\\end{cases}}\\\\{}={}&amp;\\max\\{0,x\\}=x{\\textbf {1}}_{x&gt;0}\\end{aligned}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/281532984565d3931de1cd1edc6e006d59fc67f4\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -4.005ex; width:23.452ex; height:9.176ex;\" alt=\"{\\displaystyle {\\begin{aligned}&amp;{\\begin{cases}0&amp;{\\text{if }}x\\leq 0\\\\x&amp;{\\text{if }}x&gt;0\\end{cases}}\\\\{}={}&amp;\\max\\{0,x\\}=x{\\textbf {1}}_{x&gt;0}\\end{aligned}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x&gt;0\\\\{\\text{undefined}}&amp;{\\text{if }}x=0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&gt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>undefined</mtext>\n",
    "                  </mrow>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>=</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x&gt;0\\\\{\\text{undefined}}&amp;{\\text{if }}x=0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7def1a8bcfe903e95398e0a77542b0b096dae3cf\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -3.671ex; width:22.556ex; height:8.509ex;\" alt=\"{\\displaystyle {\\begin{cases}0&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x&gt;0\\\\{\\text{undefined}}&amp;{\\text{if }}x=0\\end{cases}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle [0,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">[</mo>\n",
    "        <mn>0</mn>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle [0,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/8dc2d914c2df66bc0f7893bfb8da36766650fe47\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:6.072ex; height:2.843ex;\" alt=\"[0,\\infty )\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{0}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mn>0</mn>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{0}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7c14274cad45f7c22b662e7a4e56b1db052883d1\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:2.852ex; height:2.676ex;\" alt=\"C^0\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Gaussian Error Linear Unit (GELU)<sup id=\"cite_ref-ReferenceA_4-1\" class=\"reference\"><a href=\"#cite_note-ReferenceA-4\">&#91;4&#93;</a></sup>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_gelu.png\" class=\"image\" title=\"Visualization of the Gaussian Error Linear Unit (GELU)\"><img alt=\"Visualization of the Gaussian Error Linear Unit (GELU)\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/4/4e/Activation_gelu.png/120px-Activation_gelu.png\" decoding=\"async\" width=\"120\" height=\"80\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/4/4e/Activation_gelu.png/180px-Activation_gelu.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/4/4e/Activation_gelu.png/240px-Activation_gelu.png 2x\" data-file-width=\"1200\" data-file-height=\"800\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{aligned}&amp;{\\frac {1}{2}}x\\left(1+{\\text{erf}}\\left({\\frac {x}{\\sqrt {2}}}\\right)\\right)\\\\{}={}&amp;x\\Phi (x)\\end{aligned}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mtable columnalign=\"right left right left right left right left right left right left\" rowspacing=\"3pt\" columnspacing=\"0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em\" displaystyle=\"true\">\n",
    "            <mtr>\n",
    "              <mtd />\n",
    "              <mtd>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mfrac>\n",
    "                    <mn>1</mn>\n",
    "                    <mn>2</mn>\n",
    "                  </mfrac>\n",
    "                </mrow>\n",
    "                <mi>x</mi>\n",
    "                <mrow>\n",
    "                  <mo>(</mo>\n",
    "                  <mrow>\n",
    "                    <mn>1</mn>\n",
    "                    <mo>+</mo>\n",
    "                    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                      <mtext>erf</mtext>\n",
    "                    </mrow>\n",
    "                    <mrow>\n",
    "                      <mo>(</mo>\n",
    "                      <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                        <mfrac>\n",
    "                          <mi>x</mi>\n",
    "                          <msqrt>\n",
    "                            <mn>2</mn>\n",
    "                          </msqrt>\n",
    "                        </mfrac>\n",
    "                      </mrow>\n",
    "                      <mo>)</mo>\n",
    "                    </mrow>\n",
    "                  </mrow>\n",
    "                  <mo>)</mo>\n",
    "                </mrow>\n",
    "              </mtd>\n",
    "            </mtr>\n",
    "            <mtr>\n",
    "              <mtd>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "\n",
    "                </mrow>\n",
    "                <mo>=</mo>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "\n",
    "                </mrow>\n",
    "              </mtd>\n",
    "              <mtd>\n",
    "                <mi>x</mi>\n",
    "                <mi mathvariant=\"normal\">&#x03A6;<!-- Φ --></mi>\n",
    "                <mo stretchy=\"false\">(</mo>\n",
    "                <mi>x</mi>\n",
    "                <mo stretchy=\"false\">)</mo>\n",
    "              </mtd>\n",
    "            </mtr>\n",
    "          </mtable>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{aligned}&amp;{\\frac {1}{2}}x\\left(1+{\\text{erf}}\\left({\\frac {x}{\\sqrt {2}}}\\right)\\right)\\\\{}={}&amp;x\\Phi (x)\\end{aligned}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/36c937195a62f5db3de7bbadf3990d2356dd2470\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -4.171ex; width:25.388ex; height:9.509ex;\" alt=\"{\\displaystyle {\\begin{aligned}&amp;{\\frac {1}{2}}x\\left(1+{\\text{erf}}\\left({\\frac {x}{\\sqrt {2}}}\\right)\\right)\\\\{}={}&amp;x\\Phi (x)\\end{aligned}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\Phi (x)+x\\phi (x)}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi mathvariant=\"normal\">&#x03A6;<!-- Φ --></mi>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "        <mo>+</mo>\n",
    "        <mi>x</mi>\n",
    "        <mi>&#x03D5;<!-- ϕ --></mi>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>x</mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\Phi (x)+x\\phi (x)}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/bb43f09e198e0ef8e01fa00096ef3008100531f5\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:13.512ex; height:2.843ex;\" alt=\"{\\displaystyle \\Phi (x)+x\\phi (x)}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (-0.17\\ldots ,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mn>0.17</mn>\n",
    "        <mo>&#x2026;<!-- … --></mo>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (-0.17\\ldots ,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/6c15eea6a98f95d0a273d74ac6a4d491bf279b44\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:14.607ex; height:2.843ex;\" alt=\"{\\displaystyle (-0.17\\ldots ,\\infty )}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{\\infty }}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{\\infty }}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/971ed05871d69309df32efdfd2020128c9cf69d8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:3.673ex; height:2.343ex;\" alt=\"C^{\\infty }\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Softplus<sup id=\"cite_ref-10\" class=\"reference\"><a href=\"#cite_note-10\">&#91;10&#93;</a></sup>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_softplus.svg\" class=\"image\"><img alt=\"Activation softplus.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/7/72/Activation_softplus.svg/120px-Activation_softplus.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/7/72/Activation_softplus.svg/180px-Activation_softplus.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/7/72/Activation_softplus.svg/240px-Activation_softplus.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\ln \\left(1+e^{x}\\right)}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>ln</mi>\n",
    "        <mo>&#x2061;<!-- ⁡ --></mo>\n",
    "        <mrow>\n",
    "          <mo>(</mo>\n",
    "          <mrow>\n",
    "            <mn>1</mn>\n",
    "            <mo>+</mo>\n",
    "            <msup>\n",
    "              <mi>e</mi>\n",
    "              <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                <mi>x</mi>\n",
    "              </mrow>\n",
    "            </msup>\n",
    "          </mrow>\n",
    "          <mo>)</mo>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\ln \\left(1+e^{x}\\right)}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/a8cca29e3bdc9be84b90b9fcde7d23d9019cdc31\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:10.007ex; height:2.843ex;\" alt=\"{\\displaystyle \\ln \\left(1+e^{x}\\right)}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\frac {1}{1+e^{-x}}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mfrac>\n",
    "            <mn>1</mn>\n",
    "            <mrow>\n",
    "              <mn>1</mn>\n",
    "              <mo>+</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mo>&#x2212;<!-- − --></mo>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "            </mrow>\n",
    "          </mfrac>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\frac {1}{1+e^{-x}}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/a0730f7ecb26c15b341f9410687d23d0939977af\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.171ex; width:8.374ex; height:5.509ex;\" alt=\"{\\displaystyle {\\frac {1}{1+e^{-x}}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (0,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mn>0</mn>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (0,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/da17102e4ed0886686094ee531df040d2e86352a\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:6.329ex; height:2.843ex;\" alt=\"(0,\\infty )\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{\\infty }}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{\\infty }}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/971ed05871d69309df32efdfd2020128c9cf69d8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:3.673ex; height:2.343ex;\" alt=\"C^{\\infty }\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Exponential linear unit (ELU)<sup id=\"cite_ref-11\" class=\"reference\"><a href=\"#cite_note-11\">&#91;11&#93;</a></sup>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_elu.svg\" class=\"image\"><img alt=\"Activation elu.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/b/bc/Activation_elu.svg/120px-Activation_elu.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/b/bc/Activation_elu.svg/180px-Activation_elu.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/b/bc/Activation_elu.svg/240px-Activation_elu.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}\\alpha \\left(e^{x}-1\\right)&amp;{\\text{if }}x\\leq 0\\\\x&amp;{\\text{if }}x&gt;0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>&#x03B1;<!-- α --></mi>\n",
    "                  <mrow>\n",
    "                    <mo>(</mo>\n",
    "                    <mrow>\n",
    "                      <msup>\n",
    "                        <mi>e</mi>\n",
    "                        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                          <mi>x</mi>\n",
    "                        </mrow>\n",
    "                      </msup>\n",
    "                      <mo>&#x2212;<!-- − --></mo>\n",
    "                      <mn>1</mn>\n",
    "                    </mrow>\n",
    "                    <mo>)</mo>\n",
    "                  </mrow>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2264;<!-- ≤ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>x</mi>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&gt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}\\alpha \\left(e^{x}-1\\right)&amp;{\\text{if }}x\\leq 0\\\\x&amp;{\\text{if }}x&gt;0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/0f409bbcde02f828392ce0db27105ac46cc41477\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:22.29ex; height:6.176ex;\" alt=\"{\\displaystyle {\\begin{cases}\\alpha \\left(e^{x}-1\\right)&amp;{\\text{if }}x\\leq 0\\\\x&amp;{\\text{if }}x&gt;0\\end{cases}}}\"/></span>\n",
    "<dl><dd>with parameter <span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\alpha }\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>&#x03B1;<!-- α --></mi>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\alpha }</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/b79333175c8b3f0840bfb4ec41b8072c83ea88d3\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:1.488ex; height:1.676ex;\" alt=\"\\alpha \"/></span></dd></dl>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}\\alpha e^{x}&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x&gt;0\\\\1&amp;{\\text{if }}x=0{\\text{ and }}\\alpha =1\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>&#x03B1;<!-- α --></mi>\n",
    "                  <msup>\n",
    "                    <mi>e</mi>\n",
    "                    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                      <mi>x</mi>\n",
    "                    </mrow>\n",
    "                  </msup>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&gt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>=</mo>\n",
    "                  <mn>0</mn>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>&#xA0;and&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>&#x03B1;<!-- α --></mi>\n",
    "                  <mo>=</mo>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}\\alpha e^{x}&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x&gt;0\\\\1&amp;{\\text{if }}x=0{\\text{ and }}\\alpha =1\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/928d2cf6b7b334bad1f3177cfa75202da302ee96\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -3.671ex; width:27.071ex; height:8.509ex;\" alt=\"{\\displaystyle {\\begin{cases}\\alpha e^{x}&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x&gt;0\\\\1&amp;{\\text{if }}x=0{\\text{ and }}\\alpha =1\\end{cases}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (-\\alpha ,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mi>&#x03B1;<!-- α --></mi>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (-\\alpha ,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/9a966cc9f5c4412bfc563ac9790d9ed43177bfdd\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:8.463ex; height:2.843ex;\" alt=\"{\\displaystyle (-\\alpha ,\\infty )}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}C^{1}&amp;{\\text{if }}\\alpha =1\\\\C^{0}&amp;{\\text{otherwise}}\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <msup>\n",
    "                    <mi>C</mi>\n",
    "                    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                      <mn>1</mn>\n",
    "                    </mrow>\n",
    "                  </msup>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>&#x03B1;<!-- α --></mi>\n",
    "                  <mo>=</mo>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <msup>\n",
    "                    <mi>C</mi>\n",
    "                    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                      <mn>0</mn>\n",
    "                    </mrow>\n",
    "                  </msup>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>otherwise</mtext>\n",
    "                  </mrow>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}C^{1}&amp;{\\text{if }}\\alpha =1\\\\C^{0}&amp;{\\text{otherwise}}\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/98153cab376da330237e4a712fe848ae44dbced6\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:17.246ex; height:6.176ex;\" alt=\"{\\displaystyle {\\begin{cases}C^{1}&amp;{\\text{if }}\\alpha =1\\\\C^{0}&amp;{\\text{otherwise}}\\end{cases}}}\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Scaled exponential linear unit (SELU)<sup id=\"cite_ref-12\" class=\"reference\"><a href=\"#cite_note-12\">&#91;12&#93;</a></sup>\n",
    "</td>\n",
    "<td>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\lambda {\\begin{cases}\\alpha (e^{x}-1)&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>&#x03BB;<!-- λ --></mi>\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>&#x03B1;<!-- α --></mi>\n",
    "                  <mo stretchy=\"false\">(</mo>\n",
    "                  <msup>\n",
    "                    <mi>e</mi>\n",
    "                    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                      <mi>x</mi>\n",
    "                    </mrow>\n",
    "                  </msup>\n",
    "                  <mo>&#x2212;<!-- − --></mo>\n",
    "                  <mn>1</mn>\n",
    "                  <mo stretchy=\"false\">)</mo>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>x</mi>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2265;<!-- ≥ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\lambda {\\begin{cases}\\alpha (e^{x}-1)&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/b5f46708e4a264b8f1bb18b177b389c08e33c30f\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:23.258ex; height:6.176ex;\" alt=\"{\\displaystyle \\lambda {\\begin{cases}\\alpha (e^{x}-1)&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}\"/></span>\n",
    "<dl><dd>with parameters <span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\lambda =1.0507}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>&#x03BB;<!-- λ --></mi>\n",
    "        <mo>=</mo>\n",
    "        <mn>1.0507</mn>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\lambda =1.0507}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/740e4b3506d22e67bfc7bfd3e995d53bb81b3327\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:10.913ex; height:2.176ex;\" alt=\"{\\displaystyle \\lambda =1.0507}\"/></span> and <span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\alpha =1.67326}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>&#x03B1;<!-- α --></mi>\n",
    "        <mo>=</mo>\n",
    "        <mn>1.67326</mn>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\alpha =1.67326}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/169a7cbcf6cf284fa0e53c32238fa079f5147325\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:12.208ex; height:2.176ex;\" alt=\"{\\displaystyle \\alpha =1.67326}\"/></span></dd></dl>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\lambda {\\begin{cases}\\alpha e^{x}&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>&#x03BB;<!-- λ --></mi>\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>&#x03B1;<!-- α --></mi>\n",
    "                  <msup>\n",
    "                    <mi>e</mi>\n",
    "                    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                      <mi>x</mi>\n",
    "                    </mrow>\n",
    "                  </msup>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2265;<!-- ≥ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\lambda {\\begin{cases}\\alpha e^{x}&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/6e70f5d905524d91daaff606a6e7b409479ed6f9\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:17.446ex; height:6.176ex;\" alt=\"{\\displaystyle \\lambda {\\begin{cases}\\alpha e^{x}&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (-\\lambda \\alpha ,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mi>&#x03BB;<!-- λ --></mi>\n",
    "        <mi>&#x03B1;<!-- α --></mi>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (-\\lambda \\alpha ,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/e1357562409324a8c59d68435f87b4acb04d2045\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:9.818ex; height:2.843ex;\" alt=\"{\\displaystyle (-\\lambda \\alpha ,\\infty )}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{0}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mn>0</mn>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{0}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7c14274cad45f7c22b662e7a4e56b1db052883d1\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:2.852ex; height:2.676ex;\" alt=\"C^0\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Leaky rectified linear unit (Leaky ReLU)<sup id=\"cite_ref-13\" class=\"reference\"><a href=\"#cite_note-13\">&#91;13&#93;</a></sup>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_prelu.svg\" class=\"image\"><img alt=\"Activation prelu.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Activation_prelu.svg/120px-Activation_prelu.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Activation_prelu.svg/180px-Activation_prelu.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Activation_prelu.svg/240px-Activation_prelu.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}0.01x&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>0.01</mn>\n",
    "                  <mi>x</mi>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>x</mi>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2265;<!-- ≥ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}0.01x&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/ae7d94c8e173aa0a742c9d8cef64d6b64830926e\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:17.811ex; height:6.176ex;\" alt=\"{\\displaystyle {\\begin{cases}0.01x&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}0.01&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>0.01</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2265;<!-- ≥ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}0.01&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7d33d15f95972e19f5ab5c67cb0a596762fd8ada\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:16.481ex; height:6.176ex;\" alt=\"{\\displaystyle {\\begin{cases}0.01&amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (-\\infty ,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (-\\infty ,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/0c8c11c44279888c9e395eeb5f45d121348ae10a\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:9.299ex; height:2.843ex;\" alt=\"(-\\infty ,\\infty )\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{0}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mn>0</mn>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{0}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7c14274cad45f7c22b662e7a4e56b1db052883d1\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:2.852ex; height:2.676ex;\" alt=\"C^0\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Parameteric rectified linear unit (PReLU)<sup id=\"cite_ref-14\" class=\"reference\"><a href=\"#cite_note-14\">&#91;14&#93;</a></sup>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_prelu.svg\" class=\"image\"><img alt=\"Activation prelu.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Activation_prelu.svg/120px-Activation_prelu.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Activation_prelu.svg/180px-Activation_prelu.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Activation_prelu.svg/240px-Activation_prelu.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}\\alpha x&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>&#x03B1;<!-- α --></mi>\n",
    "                  <mi>x</mi>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>x</mi>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2265;<!-- ≥ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}\\alpha x&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/c3a5ba694377a9623afaec3b5356616242d3b98e\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:15.164ex; height:6.176ex;\" alt=\"{\\displaystyle {\\begin{cases}\\alpha x&amp;{\\text{if }}x&lt;0\\\\x&amp;{\\text{if }}x\\geq 0\\end{cases}}}\"/></span>\n",
    "<dl><dd>with parameter <span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle \\alpha }\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>&#x03B1;<!-- α --></mi>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle \\alpha }</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/b79333175c8b3f0840bfb4ec41b8072c83ea88d3\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:1.488ex; height:1.676ex;\" alt=\"\\alpha \"/></span></dd></dl>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\begin{cases}\\alpha &amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mrow>\n",
    "            <mo>{</mo>\n",
    "            <mtable columnalign=\"left left\" rowspacing=\".2em\" columnspacing=\"1em\" displaystyle=\"false\">\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mi>&#x03B1;<!-- α --></mi>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&lt;</mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "              <mtr>\n",
    "                <mtd>\n",
    "                  <mn>1</mn>\n",
    "                </mtd>\n",
    "                <mtd>\n",
    "                  <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                    <mtext>if&#xA0;</mtext>\n",
    "                  </mrow>\n",
    "                  <mi>x</mi>\n",
    "                  <mo>&#x2265;<!-- ≥ --></mo>\n",
    "                  <mn>0</mn>\n",
    "                </mtd>\n",
    "              </mtr>\n",
    "            </mtable>\n",
    "            <mo fence=\"true\" stretchy=\"true\" symmetric=\"true\"></mo>\n",
    "          </mrow>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\begin{cases}\\alpha &amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/e18e0768b83bced2a28b7bea9937939e66323dd7\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.505ex; width:13.835ex; height:6.176ex;\" alt=\"{\\displaystyle {\\begin{cases}\\alpha &amp;{\\text{if }}x&lt;0\\\\1&amp;{\\text{if }}x\\geq 0\\end{cases}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (-\\infty ,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (-\\infty ,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/0c8c11c44279888c9e395eeb5f45d121348ae10a\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:9.299ex; height:2.843ex;\" alt=\"(-\\infty ,\\infty )\"/></span><sup class=\"reference plainlinks nourlexpansion\" id=\"ref_alpha\"><a class=\"external autonumber\" href=\"https://en.wikipedia.org/wiki/Activation_function#endnote_alpha\">[2]</a></sup>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{0}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mn>0</mn>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{0}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7c14274cad45f7c22b662e7a4e56b1db052883d1\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:2.852ex; height:2.676ex;\" alt=\"C^0\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Sigmoid linear unit (SiLU,<sup id=\"cite_ref-ReferenceA_4-2\" class=\"reference\"><a href=\"#cite_note-ReferenceA-4\">&#91;4&#93;</a></sup> Sigmoid shrinkage,<sup id=\"cite_ref-refssbs1_15-0\" class=\"reference\"><a href=\"#cite_note-refssbs1-15\">&#91;15&#93;</a></sup> SiL,<sup id=\"cite_ref-16\" class=\"reference\"><a href=\"#cite_note-16\">&#91;16&#93;</a></sup> or Swish-&#8205;1<sup id=\"cite_ref-17\" class=\"reference\"><a href=\"#cite_note-17\">&#91;17&#93;</a></sup>)\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Swish.svg\" class=\"image\" title=\"Swish Activation Function\"><img alt=\"Swish Activation Function\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/1/12/Swish.svg/120px-Swish.svg.png\" decoding=\"async\" width=\"120\" height=\"72\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/1/12/Swish.svg/180px-Swish.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/1/12/Swish.svg/240px-Swish.svg.png 2x\" data-file-width=\"1000\" data-file-height=\"600\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\frac {x}{1+e^{-x}}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mfrac>\n",
    "            <mi>x</mi>\n",
    "            <mrow>\n",
    "              <mn>1</mn>\n",
    "              <mo>+</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mo>&#x2212;<!-- − --></mo>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "            </mrow>\n",
    "          </mfrac>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\frac {x}{1+e^{-x}}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/623852d52f9dc328d4168d79f470a3c3186730f6\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.171ex; width:8.374ex; height:5.009ex;\" alt=\"{\\displaystyle {\\frac {x}{1+e^{-x}}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\frac {1+e^{-x}+xe^{-x}}{\\left(1+e^{-x}\\right)^{2}}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mfrac>\n",
    "            <mrow>\n",
    "              <mn>1</mn>\n",
    "              <mo>+</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mo>&#x2212;<!-- − --></mo>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <mo>+</mo>\n",
    "              <mi>x</mi>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mo>&#x2212;<!-- − --></mo>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "            </mrow>\n",
    "            <msup>\n",
    "              <mrow>\n",
    "                <mo>(</mo>\n",
    "                <mrow>\n",
    "                  <mn>1</mn>\n",
    "                  <mo>+</mo>\n",
    "                  <msup>\n",
    "                    <mi>e</mi>\n",
    "                    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                      <mo>&#x2212;<!-- − --></mo>\n",
    "                      <mi>x</mi>\n",
    "                    </mrow>\n",
    "                  </msup>\n",
    "                </mrow>\n",
    "                <mo>)</mo>\n",
    "              </mrow>\n",
    "              <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                <mn>2</mn>\n",
    "              </mrow>\n",
    "            </msup>\n",
    "          </mfrac>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\frac {1+e^{-x}+xe^{-x}}{\\left(1+e^{-x}\\right)^{2}}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/01a6a4b826d59685162b5312f5172f985169a170\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -3.171ex; width:16.078ex; height:6.843ex;\" alt=\"{\\displaystyle {\\frac {1+e^{-x}+xe^{-x}}{\\left(1+e^{-x}\\right)^{2}}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle [-0.278\\ldots ,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">[</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mn>0.278</mn>\n",
    "        <mo>&#x2026;<!-- … --></mo>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle [-0.278\\ldots ,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/958ab90852b76ffe763b61fd8aae0a4175ea25f1\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:15.511ex; height:2.843ex;\" alt=\"{\\displaystyle [-0.278\\ldots ,\\infty )}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{\\infty }}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{\\infty }}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/971ed05871d69309df32efdfd2020128c9cf69d8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:3.673ex; height:2.343ex;\" alt=\"C^{\\infty }\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td>Mish \n",
    "<td>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle x\\tanh(\\ln(1+e^{x}))}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mi>x</mi>\n",
    "        <mi>tanh</mi>\n",
    "        <mo>&#x2061;<!-- ⁡ --></mo>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mi>ln</mi>\n",
    "        <mo>&#x2061;<!-- ⁡ --></mo>\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mn>1</mn>\n",
    "        <mo>+</mo>\n",
    "        <msup>\n",
    "          <mi>e</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi>x</mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle x\\tanh(\\ln(1+e^{x}))}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/e30109e9cceea5f6b732bdd7251be21201b10f4f\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:18.186ex; height:2.843ex;\" alt=\"{\\displaystyle x\\tanh(\\ln(1+e^{x}))}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle {\\frac {(e^{x}(4e^{2x}+e^{3x}+4(1+x)+e^{x}(6+4x)))}{(2+2e^{x}+e^{2x})^{2}}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "          <mfrac>\n",
    "            <mrow>\n",
    "              <mo stretchy=\"false\">(</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <mo stretchy=\"false\">(</mo>\n",
    "              <mn>4</mn>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mn>2</mn>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <mo>+</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mn>3</mn>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <mo>+</mo>\n",
    "              <mn>4</mn>\n",
    "              <mo stretchy=\"false\">(</mo>\n",
    "              <mn>1</mn>\n",
    "              <mo>+</mo>\n",
    "              <mi>x</mi>\n",
    "              <mo stretchy=\"false\">)</mo>\n",
    "              <mo>+</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <mo stretchy=\"false\">(</mo>\n",
    "              <mn>6</mn>\n",
    "              <mo>+</mo>\n",
    "              <mn>4</mn>\n",
    "              <mi>x</mi>\n",
    "              <mo stretchy=\"false\">)</mo>\n",
    "              <mo stretchy=\"false\">)</mo>\n",
    "              <mo stretchy=\"false\">)</mo>\n",
    "            </mrow>\n",
    "            <mrow>\n",
    "              <mo stretchy=\"false\">(</mo>\n",
    "              <mn>2</mn>\n",
    "              <mo>+</mo>\n",
    "              <mn>2</mn>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <mo>+</mo>\n",
    "              <msup>\n",
    "                <mi>e</mi>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mn>2</mn>\n",
    "                  <mi>x</mi>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "              <msup>\n",
    "                <mo stretchy=\"false\">)</mo>\n",
    "                <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                  <mn>2</mn>\n",
    "                </mrow>\n",
    "              </msup>\n",
    "            </mrow>\n",
    "          </mfrac>\n",
    "        </mrow>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle {\\frac {(e^{x}(4e^{2x}+e^{3x}+4(1+x)+e^{x}(6+4x)))}{(2+2e^{x}+e^{2x})^{2}}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/04af231bb225292d9229140240a7f7ae9e14dd53\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -2.671ex; width:41.415ex; height:6.676ex;\" alt=\"{\\displaystyle {\\frac {(e^{x}(4e^{2x}+e^{3x}+4(1+x)+e^{x}(6+4x)))}{(2+2e^{x}+e^{2x})^{2}}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle [-0.308\\ldots ,\\infty )}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">[</mo>\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mn>0.308</mn>\n",
    "        <mo>&#x2026;<!-- … --></mo>\n",
    "        <mo>,</mo>\n",
    "        <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "        <mo stretchy=\"false\">)</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle [-0.308\\ldots ,\\infty )}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/e82abe83a0dfe1b7eb7d9c57c59f814c8e440d83\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:15.511ex; height:2.843ex;\" alt=\"{\\displaystyle [-0.308\\ldots ,\\infty )}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{\\infty }}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{\\infty }}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/971ed05871d69309df32efdfd2020128c9cf69d8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:3.673ex; height:2.343ex;\" alt=\"C^{\\infty }\"/></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td><a href=\"/wiki/Gaussian_function\" title=\"Gaussian function\">Gaussian</a>\n",
    "</td>\n",
    "<td><a href=\"/wiki/File:Activation_gaussian.svg\" class=\"image\"><img alt=\"Activation gaussian.svg\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/f/fe/Activation_gaussian.svg/120px-Activation_gaussian.svg.png\" decoding=\"async\" width=\"120\" height=\"60\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/f/fe/Activation_gaussian.svg/180px-Activation_gaussian.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/f/fe/Activation_gaussian.svg/240px-Activation_gaussian.svg.png 2x\" data-file-width=\"120\" data-file-height=\"60\" /></a>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle e^{-x^{2}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>e</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mo>&#x2212;<!-- − --></mo>\n",
    "            <msup>\n",
    "              <mi>x</mi>\n",
    "              <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                <mn>2</mn>\n",
    "              </mrow>\n",
    "            </msup>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle e^{-x^{2}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/da5decb0035215bdd45d3d40b4b2c3a158d00fc8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:4.366ex; height:3.009ex;\" alt=\"e^{-x^{2}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle -2xe^{-x^{2}}}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo>&#x2212;<!-- − --></mo>\n",
    "        <mn>2</mn>\n",
    "        <mi>x</mi>\n",
    "        <msup>\n",
    "          <mi>e</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mo>&#x2212;<!-- − --></mo>\n",
    "            <msup>\n",
    "              <mi>x</mi>\n",
    "              <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "                <mn>2</mn>\n",
    "              </mrow>\n",
    "            </msup>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle -2xe^{-x^{2}}}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7da9606f8917a9a4d2120b139dd111ab6031157e\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.505ex; width:8.666ex; height:3.176ex;\" alt=\"{\\displaystyle -2xe^{-x^{2}}}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle (0,1]}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <mo stretchy=\"false\">(</mo>\n",
    "        <mn>0</mn>\n",
    "        <mo>,</mo>\n",
    "        <mn>1</mn>\n",
    "        <mo stretchy=\"false\">]</mo>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle (0,1]}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/7e70f9c241f9faa8e9fdda2e8b238e288807d7a4\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.838ex; width:4.91ex; height:2.843ex;\" alt=\"{\\displaystyle (0,1]}\"/></span>\n",
    "</td>\n",
    "<td><span class=\"mwe-math-element\"><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\"  alttext=\"{\\displaystyle C^{\\infty }}\">\n",
    "  <semantics>\n",
    "    <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "      <mstyle displaystyle=\"true\" scriptlevel=\"0\">\n",
    "        <msup>\n",
    "          <mi>C</mi>\n",
    "          <mrow class=\"MJX-TeXAtom-ORD\">\n",
    "            <mi mathvariant=\"normal\">&#x221E;<!-- ∞ --></mi>\n",
    "          </mrow>\n",
    "        </msup>\n",
    "      </mstyle>\n",
    "    </mrow>\n",
    "    <annotation encoding=\"application/x-tex\">{\\displaystyle C^{\\infty }}</annotation>\n",
    "  </semantics>\n",
    "</math></span><img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/971ed05871d69309df32efdfd2020128c9cf69d8\" class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"vertical-align: -0.338ex; width:3.673ex; height:2.343ex;\" alt=\"C^{\\infty }\"/></span>\n",
    "</td></tr></tbody></table>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}