Gaussian Kernal (RBF) Regression using Pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><center>Gaussian Kernal Regression</center></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Kernal Regression** tries to solve the problem \n",
    "<h1><center> $\\hat{Y} = \\Phi(X) \\theta \\\\\n",
    "                      = 1 \\theta_0 + k(x, \\mu, \\lambda) \\theta_1 + .....+ k(x, \\mu, \\lambda) \\theta_d$ </center></h1>\n",
    "where \n",
    "\n",
    "$k(x, \\mu, \\lambda) = e ^{- \\frac{1}{\\lambda} || x - \\mu ||^2}$, \n",
    "$\\Phi = \\begin{bmatrix}\n",
    "1 & k(x, \\mu, \\lambda) & k(x, \\mu, \\lambda)\\\\\n",
    "1 & -------& ------- \\\\\n",
    "---&-------&------- \\\\\n",
    "\\end{bmatrix}$\n",
    "\n",
    "The parameters $\\theta = (\\Phi^T\\Phi + \\delta^2)^{-1} \\Phi^T Y$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nData, kWidth = 10, 1\n",
    "xTrain = torch.linspace(-1, 1, nData)\n",
    "yTrain = torch.pow(xTrain, 2)\n",
    "yTrain = yTrain + torch.mul(torch.randn(nData), 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def phi(x, y):\n",
    "    \"\"\"\n",
    "    :param x: one number\n",
    "    :param y: one number\n",
    "    :return: Gaussian, phi = exp(-(1/variance) * (|| X - mu ||)^2)\n",
    "    \"\"\"\n",
    "    return torch.exp(-(1/kWidth) * torch.sum(torch.pow(x-y, 2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def kernal(xTrain, xTest):\n",
    "    \"\"\"\n",
    "    Finds the matrix of Gaussians or kernal, Phi =   [ phi_11 phi_12 ....phi_1n]\n",
    "                                                     [ phi_21 phi_22 ....phi_2n]\n",
    "                                                     [.........................]\n",
    "                                                     [.........................]\n",
    "                                                     [phi_m1 phi_m2 .....phi_mn]\n",
    "    :param xTrain: array of training data\n",
    "    :param xTest: array of training/testing data\n",
    "    :return: gaussian kernal\n",
    "    \"\"\"\n",
    "    Phi = torch.Tensor(len(xTrain), len(xTest))\n",
    "    for i in range(len(xTrain)):\n",
    "        for j in range(len(xTest)):\n",
    "            Phi[i, j] = phi(xTrain[i], xTest[j])\n",
    "    return Phi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regularizer = torch.mul(torch.eye(nData), 0.001)\n",
    "PhiTrain = kernal(xTrain, xTrain)\n",
    "theta = torch.inverse((PhiTrain.t() @ PhiTrain) + regularizer) @ PhiTrain.t() @ yTrain  # ((Phi^T Phi)^(-1) + d^2 I) Phi^T Y\n",
    "yPredTrain = PhiTrain.t() @ theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nTestData = 100\n",
    "xTest = torch.linspace(-1, 1, nTestData)\n",
    "PhiTest = torch.Tensor(nData, nTestData)\n",
    "PhiTest = kernal(xTrain, xTest)\n",
    "yPredTest = PhiTest.t() @ theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(xTrain, yTrain, 'b', label='yTrain')\n",
    "plt.plot(xTrain, yPredTrain, 'r', label='yPredicted_train')\n",
    "plt.plot(xTest, yPredTest, 'k', label='yPredicted_test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A good resource :\n",
    "https://krasserm.github.io/2018/03/19/gaussian-processes/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}