SVM/SVM_implicit_feature_space.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Support Vector Machines: what do features in the implicit space mean?\n",
    "===\n",
    "\n",
    "**Abstract** I think a lot of SVM tutorials give all the details about the maximum margin separating hyperplane, the quadratic programming and support vectors, and radial kernels, but don't give a good intuition on one important part of the big picture. When the kernel does its implicit mapping from the original feature space to a new feature space, what does that new feature space look like? What do the features in that new space mean? This short notebook tries to fill in the missing link."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Introduction\n",
    "---\n",
    "\n",
    "SVMs implicitly create a new feature space and then carry out linear classification in that. [Cover's theorem](https://en.wikipedia.org/wiki/Cover%27s_theorem) explains why the data are more likely to be linearly separable in the new feature space if it's higher-dimensional. A common illustration of the main idea is a dataset where one class is enclosed in the unit circle: that's not linearly separable. But if the implicit mapping does something like this: $$(x_0, x_1) \\rightarrow (x_0, x_1, \\sqrt{x_0^2 + x_1^2})$$ (ie add a new feature equal to the distance of the point from the origin), the data will be separable by that feature alone. This diagram (from [hackerearth.com](https://www.hackerearth.com)) illustrates the idea (note, here the data is in an ellipse not centered at the origin, so the new feature is not using the exact equation above, but the idea is the same): ![Cover and separation](https://www.hackerearth.com/blog/wp-content/uploads/2017/02/kernel.png)\n",
    "\n",
    "But while this illustrates the potential benefit of a higher-dimensional space, it is a bit misleading for two reasons. First, it suggests that the new space consists of the old space plus some additional features, and that's not the case. In reality the new space will have all-new features, none of the old ones. Second, the new feature in this example is not quite the type of feature that SVMs actually create! They don't create features by manipulation/combination of existing features. What features *do* they create?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at training data\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np, matplotlib.pyplot as plt\n",
    "from sklearn.svm import SVC\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose our training data $X$ is in two dimensions and consists of four points with labels -1 (blue) and +1 (red)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = np.array([[0.5, 0.1], [0.4, 0.4], [0.6, 0.6], [0.7, 0.8]])\n",
    "y = np.array([-1, -1, 1, 1])\n",
    "y_pos = (y == 1)\n",
    "y_neg = (y == -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARYAAAEMCAYAAAABAJmyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE85JREFUeJzt3X9s3PV9x/Hn6xKQWtrVVklzLLHVeCLQdC1d6tKqqTZa\nVJUw6cwtTIJWQ2JdI7S4yh/7A9RuZVqHuv7XdTagCCFUqRqrbEyuUls2baNsuKw4NeVHUVB2MF8o\nLlDs/qDbgnfv/fG9pK6J4x/9fL93Z78eUuS77/dj85Lxve77/dznvqeIwMwspVK7A5jZxuNiMbPk\nXCxmlpyLxcySc7GYWXIuFjNLLvdikXS3pBclPbnMfkn6sqQTkh6XtDfvTGaWryKOWO4BrjrH/v3A\nxa1/B4E7CshkZjnKvVgi4iHglXMMGQK+EplHgB5JF+Wdy8zy0wlzLDuAxqL7J1vbzKxLbW13gLWQ\ndJDsdIkLLrjgvZdeemmbE5ltXMeOHXs5Irat53s7oVieB/oW3d/Z2vY6EXEEOAIwODgYU1NT+acz\n26Qk/dd6v7cTToVqwA2tV4c+APwkIl5odygzW7/cj1gk/T1wBXChpJPArcB5ABFxJ/AN4GrgBPAL\n4Ma8M5lZvnIvloi4foX9ARzKO4eZFacTToXMbINxsZhZci4WM0vOxWJmyblYzCw5F4uZJediMbPk\nXCxmlpyLxcySc7GYWXIuFjNLzsViZsm5WMwsOReLmSXnYjGz5FwsZpaci8XMknOxmFlyLhYzS87F\nYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZci4WM0vOxWJmyblYzCw5F4uZJediMbPkXCxm\nltzWdgcw6yQzMzPUajXm5+fp6elhaGiIvr6+dsfqOi4WM2B2dpbh4WEmJiZoNptnth8+fJhqtcrI\nyAjlcrmNCbuLi8U2vdnZWfbt20e9Xuc8oApcAhwHjjabjI+PMz09zeTkJNu3b29v2C5RyByLpKsk\nHZd0QtItZ9n/Fklfl/R9SU9JurGIXGYAw8PD1Ot19gJ1YAy4rfX1Wci21+scOnSojSm7iyIi3/+A\ntAV4BvgocBJ4FLg+In6waMxngLdExM2StpE9WZQj4tRyP3dwcDCmpqZyzW4b38zMDLt27WJLs0kd\n2HmWMQ1gAGiWSjz33HObZs5F0rGIGFzP9xZxxHI5cCIi6q2iuBcYWjImgDdLEvAm4BVgoYBstsnV\najWazSYVzl4qAH1kf7DNZpNarVZcuC5WRLHsICv90062ti02ArwD+CHwBHA4IppLxiDpoKQpSVMv\nvfRSXnltE5mfnweyOZVz2d36Ojc3l2uejaJT1rF8DHgM+E3gPcCIpN9YOigijkTEYEQMbtu2reiM\ntgH19PQA2bn3uTzT+trb25trno2iiGJ5nuxo8rSdrW2L3QjcF5kTZHNmlxaQzTa5SqVCqVSiRnYo\nfTYN4ChQKpWoVCrFhetiRRTLo8DFknZJOh+4Dlh6ojoDXAkgaTvZkWm9gGy2yfX391OtVnmNbB6l\nsWR/A7iGbMKvWq1umonbX1fu61giYkHSMPAAsAW4OyKeknRTa/+dwOeBeyQ9AQi4OSJezjubGcDI\nyAjT09N8r15ngKxgdpOd/hwlK5WBgQFGR0fbGbOr5P5yc178crOltNzK21KpRLVaZXR0dNMtjvt1\nXm72ylszoFwuMzY2RqPRoFarMTc3R29vL5VKxac/6+BiMVukr6/PK2wT6JSXm81sA3GxmFlyLhYz\nS87FYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZci4WM0vOxWJmyblYzCw5F4uZJediMbPk\nXCxmlpyLxcySc7GYWXIuFjNLzsViZsm5WMwsOReLmSXnYjGz5FwsZpaci8XMknOxmFlyLhYzS87F\nYmbJuVjMLDkXi5klt7XdAczWY2Zmhlqtxvz8PD09PQwNDdHX19fuWNbiYrGuMjs7y/DwMBMTEzSb\nzTPbDx8+TLVaZWRkhHK53MaEBgWdCkm6StJxSSck3bLMmCskPSbpKUnfLiKXdZfZ2Vn27dvH+Pg4\nW5pNDgCfAQ4ApWaT8fFx9u3bx49+9KM2J7Xci0XSFmAU2A/sAa6XtGfJmB7gdqASEe8E/jDvXNZ9\nhoeHqdfr7AXqwBhwW+vrs5Btr9c5dOhQG1MaFHPEcjlwIiLqEXEKuBcYWjLm48B9ETEDEBEvFpDL\nusjMzAwTExOcBxwFdi7ZvxO4n+zcfmJigkajUXREW6SIYtkBLP6/fLK1bbHdQK+kByUdk3RDAbms\ni9RqNZrNJhVeXyqn9ZE9YzWbTWq1WnHh7HU6ZfJ2K/Be4ErgDcB3JD0SEc8sHiTpIHAQoL+/v/CQ\n1j7z8/MAXLLCuN2tr3Nzc7nmsXMr4ojlebInk9N2trYtdhJ4ICJejYiXgYeAy5b+oIg4EhGDETG4\nbdu23AJb5+np6QHg+ArjTj8T9fb25prHzq2IYnkUuFjSLknnA9cBS49TjwIfkrRV0huB9wNPF5DN\nukSlUqFUKlEjexY6mwbZH1KpVKJSqRQXzl4n92KJiAVgGHiArCy+FhFPSbpJ0k2tMU8D3wIeB74L\n3BURT+adzbpHf38/1WqV18jmUZZOzTaAa4AFoFqterFcmyki2p1hXQYHB2NqaqrdMaxAp9ex1Ot1\ntpIVzG6y05+jZKUyMDDA5OQk27dvb2fUDUHSsYgYXM/3+r1C1jXK5TIPP/wwBw4coFkqMQ58ARgH\nmqUSBw4ccKl0iE55VchsVcrlMmNjYzQaDWq1GnNzc/T29lKpVHz600FcLNaV+vr6vMK2g/lUyMyS\nc7GYWXIuFjNLzsViZsm5WMwsOReLmSXnYjGz5FwsZpaci8XMknOxmFly6y4WSTenDGJmG8eq3ysk\n6WuL7wLvAb6YPJGZdb21vAnxpxHxJ6fvSLojhzxmtgGs5VTotiX3P5syiJltHCsWi6S/laSIeHbx\n9oh4Jb9YZtbNVnPE8jOgJukCAEkfk/RwvrHMrJutOMcSEX8u6ePAg5JOAT8Hzvr5y2ZmsIpikXQl\n8CngVeAi4I8jYqWPdzGzTWw1p0KfBf4iIq4ArgX+QdJHck1lZl1tNadCH1l0+wlJ+8kujP7BPIOZ\nWfda88rbiHiB7DOWzczOal1L+iPiv1MHMbONw29CNLPkXCxmlpyLxcySc7GYWXIuFjNLzsViZsm5\nWMwsubVc6Mk2uJmZGWq1GvPz8/T09DA0NERfX1+7Y1kXcrEYs7OzDA8PMzExQbPZPLP98OHDVKtV\nRkZGKJfLbUxo3cbFssnNzs6yb98+6vU6cB5QBS4BjtNsHmV8fJzp6WkmJyfZvn17e8Na1yhkjkXS\nVZKOSzohadlruUh6n6QFSdcWkctgeHi4VSp7gTowRnYV0jHgWWAv9XqdQ4cOtTGldZvci0XSFmAU\n2A/sAa6XtGeZcV8E/jHvTJaZmZlhYmKC7EjlKLBzyYidwP3AViYmJmg0GkVHtC5VxBHL5cCJiKhH\nxCngXmDoLOM+TXY5hhcLyGRArVZrzalUeH2pnNYHDNFsNqnVasWFs65WRLHsABY/1Z1sbTtD0g6y\nk/tzfqSIpIOSpiRNvfTSS8mDbjbz8/OtW5esMHI3AHNzc7nmsY2jU9axfAm4OSKa5xoUEUciYjAi\nBrdt21ZQtI2rp6endWulK40+A0Bvb2+ueWzjKKJYnic7nj5tZ2vbYoPAvZKeI7v85e2Srikg26ZW\nqVQolUpAjexA8mwawFFKpRKVSqW4cNbViiiWR4GLJe2SdD5wHdlf8hkRsSsi3h4Rbyd7OeJPI+L+\nArJtav39/VSrVeA1smmvpZOzDeAaYIFqterFcrZqua9jiYgFScPAA8AW4O6IeErSTa39d+adwZY3\nMjLC9PQ09fr3gAGygtlNdvpzFFhgYGCA0dHRdsa0LqOIaHeGdRkcHIypqal2x9gQllt5WyqVqFar\njI6OenHcJiTpWEQMrud7vfLWKJfLjI2N0Wg0qNVqzM3N0dvbS6VS8emPrYuLxc7o6+vzCltLolNe\nbjazDcTFYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZci4WM0vOxWJmyblYzCw5F4uZJedi\nMbPkXCxmlpyLxcySc7GYWXIuFjNLzsViZsm5WMwsOReLmSXnYjGz5FwsZpaci8XMknOxmFlyLhYz\nS87FYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZcoUUi6SrJB2XdELSLWfZ/wlJj0t6QtKk\npMuKyGVm+ci9WCRtAUaB/cAe4HpJe5YMexb4vYh4F/B54EjeucwsP0UcsVwOnIiIekScAu4FhhYP\niIjJiJhr3X0E2FlALjPLSRHFsgNoLLp/srVtOZ8EvplrIjPL1dZ2B1hM0ofJiuVDy+w/CBwE6O/v\nLzCZma1FEUcszwN9i+7vbG37FZLeDdwFDEXEj8/2gyLiSEQMRsTgtm3bcglrZr++IorlUeBiSbsk\nnQ9cB9QWD5DUD9wH/FFEPFNAJjPLUe6nQhGxIGkYeADYAtwdEU9Juqm1/07gc8BbgdslASxExGDe\n2cwsH4qIdmdYl8HBwZiammp3DLMNS9Kx9T7Be+WtmSXnYjGz5FwsZpaci8XMknOxmFlyLhYzS87F\nYmbJuVjMLDkXi5kl52Ixs+Q66rIJtjHNzMxQq9WYn5+np6eHoaEh+vr6Vv5G61ouFsvN7Owsw8PD\nTExM0Gw2z2w/fPgw1WqVkZERyuVyGxNaXlwslovZ2Vn27dtHvV4HzgOqwCXAcZrNo4yPjzM9Pc3k\n5CTbt29vb1hLznMslovh4eFWqewF6sAYcFvr67PAXur1OocOHWpjSsuLi8WSm5mZYWJiguxI5Siv\nvzb6TuB+YCsTExM0Go2lP8K6nIvFkqvVaq05lQrLf+BCHzBEs9mkVqstM8a6lYvFkpufn2/dumSF\nkbsBmJubW2GcdRsXiyXX09PTunV8hZHZ5Y17e3tzzWPFc7FYcpVKhVKpRHbN9JPLjGoARymVSlQq\nleLCWSFcLJZcf38/1WoVeI3sQy+XTs42gGuABarVqhfLbUBex2K5GBkZYXp6mnr9e8AAWcHsJjv9\nOQosMDAwwOjoaDtjWk58xGK5KJfLPPzwwxw4cIBSqQmMA18AximVmhw4cMCL4zYwH7FYbsrlMmNj\nYzQaDWq1GnNzc/T29lKpVHz6s8G5WCx3fX19XmG7yfhUyMySc7GYWXIuFjNLzsViZsm5WMwsOReL\nmSXnYjGz5FwsZpaci8XMknOxmFlyLhYzS66QYpF0laTjkk5IuuUs+yXpy639j0vaW0QuM8tH7sUi\naQswCuwH9gDXS9qzZNh+4OLWv4PAHXnnMrP8FHHEcjlwIiLqEXEKuJfsqj+LDQFficwjQI+kiwrI\nZmY5KOKyCTv41WsTngTev4oxO4AXFg+SdJDsiAbgfyU9mTZqbi4EXm53iDXoprzdlBW6K+9KH7Ow\nrK66HktEHAGOAEiaiojBNkdalW7KCt2Vt5uyQnfllTS13u8t4lToebJPpzptZ2vbWseYWZcoolge\nBS6WtEvS+cB1ZJ8LsVgNuKH16tAHgJ9ExAtLf5CZdYfcT4UiYkHSMPAAsAW4OyKeknRTa/+dwDeA\nq4ETwC+AG1fxo4/kFDkP3ZQVuitvN2WF7sq77qyKiJRBzMy88tbM0nOxmFlyHV8s3fR2gFVk/UQr\n4xOSJiVd1o6ci/KcM++ice+TtCDp2iLzLcmwYlZJV0h6TNJTkr5ddMYlWVb6W3iLpK9L+n4r72rm\nFXMh6W5JLy63Lmxdj7GI6Nh/ZJO9/0n2GZ3nA98H9iwZczXwTUDAB4D/6OCsHwR6W7f3tyvravMu\nGvcvZBPs13ZqVqAH+AHQ37r/tk7+3QKfAb7Yur0NeAU4v015fxfYCzy5zP41P8Y6/Yilm94OsGLW\niJiMiLnW3UfI1uu0y2p+twCfJvt81BeLDLfEarJ+HLgvImYAIqLT8wbwZkkC3kRWLAvFxmwFiXio\n9d9fzpofY51eLMst9V/rmCKsNccnyZ4F2mXFvJJ2AFXa/6bQ1fxudwO9kh6UdEzSDYWle73V5B0B\n3gH8EHgCOBwRzWLirdmaH2NdtaR/o5D0YbJi+VC7s6zgS8DNEdHMnlg72lbgvcCVwBuA70h6JCKe\naW+sZX0MeAz4CPBbwD9J+reI+Gl7Y6XR6cXSTW8HWFUOSe8G7gL2R8SPC8p2NqvJOwjc2yqVC4Gr\nJS1ExP3FRDxjNVlPAj+OiFeBVyU9BFwGtKNYVpP3RuBvIpvEOCHpWeBS4LvFRFyTtT/G2jXBtcpJ\npa1AHdjFLyfB3rlkzO/zqxNL3+3grP1kq4s/2A2/2yXj76F9k7er+d2+A/jn1tg3Ak8Cv93Bee8A\n/rJ1e3vrgXphG/8e3s7yk7drfox19BFL5Pd2gHZl/RzwVuD21lHAQrTpna6rzNsRVpM1Ip6W9C3g\ncaAJ3BURbbmsxip/t58H7pH0BNkD9uaIaMvlFCT9PXAFcKGkk8CtwHmLsq75MeYl/WaWXKe/KmRm\nXcjFYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFciPpXyV9tHX7ryX9XbszWTE6eoGcdb1bgb+S9Dbg\nd4BKm/NYQbxAznLVuuDSm4ArIuJnki4AbgdOAQ9GxFfbGtBy4VMhy42kdwEXAaci4metzX8AjEXE\np/ARzIblYrFctC4E9FWyiwT9XNJVrV07+eW1Pf6vHdksfy4WS07SG4H7gD+LiKfJ3nB3a2v3SX55\n5Tz//W1QnmOxQrXmWEaA/wH+3XMsG5OLxcyS86GomSXnYjGz5FwsZpaci8XMknOxmFlyLhYzS87F\nYmbJuVjMLDkXi5kl9/9VTCeAOnAqoQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111b832e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(X[y_neg, 0], X[y_neg, 1], '.', c='b', markersize=20, mew=2, mec='k')\n",
    "plt.plot(X[y_pos, 0], X[y_pos, 1], '.', c='r', markersize=20, mew=2, mec='k')\n",
    "plt.xlabel(r\"$x_0$\")\n",
    "plt.ylabel(r\"$x_1$\")\n",
    "plt.xlim((0, 1))\n",
    "plt.ylim((0, 1))\n",
    "plt.axes().set_aspect('equal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of the SVM is to classify new unknown points, such as the point $q$ indicated by the green square here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARYAAAEMCAYAAAABAJmyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE+lJREFUeJzt3X9s3PV9x/Hn6xKQWtrVVklzLLHVeCLQdP2x1KVVU220\nqCph0plbmASthsS6RmxxlT/2B1W7lWodavtf29mAIoRQpWqssjG5Sm3ZtI2ykbLi1PxKUVB2MF8o\nLlDsltJtwbv3/vheqDFx/IPP93t39ushRb77fj82Lxnf677fz33ue4oIzMxSKrU7gJmtPy4WM0vO\nxWJmyblYzCw5F4uZJediMbPkci8WSbdLelbSY0vsl6RvSDoh6RFJu/POZGb5KuKI5Q7g8rPs3wtc\n2Pq3H7ilgExmlqPciyUi7gNeOMuQIeCbkXkA6JF0Qd65zCw/nTDHsg1oLLh/srXNzLrU5nYHWA1J\n+8lOlzjvvPPed/HFF7c5kdn6dfTo0ecjYstavrcTiuVpoG/B/e2tba8REYeAQwCDg4MxOTmZfzqz\nDUrSf631ezvhVKgGXNt6deiDwC8i4pl2hzKztcv9iEXS3wOXAudLOgncCJwDEBG3At8FrgBOAL8G\nrss7k5nlK/diiYhrltkfwIG8c5hZcTrhVMjM1hkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZci4WM0vO\nxWJmyblYzCw5F4uZJediMbPkXCxmlpyLxcySc7GYWXIuFjNLzsViZsm5WMwsOReLmSXnYjGz5Fws\nZpaci8XMknOxmFlyLhYzS87FYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZci4WM0vOxWJm\nyW1udwCzTjI9PU2tVmNubo6enh6Ghobo6+trd6yu42IxA2ZmZhgeHmZiYoJms/nK9oMHD1KtVhkZ\nGaFcLrcxYXdxsdiGNzMzw549e6jX65wDVIGLgOPA4WaT8fFxpqamOHLkCFu3bm1v2C5RyByLpMsl\nHZd0QtJnz7D/LZK+I+lhScckXVdELjOA4eFh6vU6u4E6MAbc1Pr6JGTb63UOHDjQxpTdRRGR739A\n2gQ8AXwMOAk8CFwTET9ZMOZzwFsi4gZJW8ieLMoRcWqpnzs4OBiTk5O5Zrf1b3p6mh07drCp2aQO\nbD/DmAYwADRLJZ566qkNM+ci6WhEDK7le4s4YrkEOBER9VZR3AkMLRoTwJslCXgT8AIwX0A22+Bq\ntRrNZpMKZy4VgD6yP9hms0mtVisuXBcroli2kZX+aSdb2xYaAd4B/BR4FDgYEc1FY5C0X9KkpMnn\nnnsur7y2gczNzQHZnMrZ7Gx9nZ2dzTXPetEp61g+DjwE/DbwXmBE0m8tHhQRhyJiMCIGt2zZUnRG\nW4d6enqA7Nz7bJ5ofe3t7c01z3pRRLE8TXY0edr21raFrgPuiswJsjmziwvIZhtcpVKhVCpRIzuU\nPpMGcBgolUpUKpXiwnWxIorlQeBCSTsknQtcDSw+UZ0GLgOQtJXsyLReQDbb4Pr7+6lWq7xMNo/S\nWLS/AVxJNuFXrVY3zMTt65X7OpaImJc0DNwDbAJuj4hjkq5v7b8V+BJwh6RHAQE3RMTzeWczAxgZ\nGWFqaoof1+sMkBXMTrLTn8NkpTIwMMDo6Gg7Y3aV3F9uzotfbraUllp5WyqVqFarjI6ObrjFca/n\n5WavvDUDyuUyY2NjNBoNarUas7Oz9Pb2UqlUfPqzBi4WswX6+vq8wjaBTnm52czWEReLmSXnYjGz\n5FwsZpaci8XMknOxmFlyLhYzS87FYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZci4WM0vO\nxWJmyblYzCw5F4uZJediMbPkXCxmlpyLxcySc7GYWXIuFjNLzsViZsm5WMwsOReLmSXnYjGz5Fws\nZpaci8XMknOxmFlym9sdwGwtpqenqdVqzM3N0dPTw9DQEH19fe2OZS0uFusqMzMzDA8PMzExQbPZ\nfGX7wYMHqVarjIyMUC6X25jQoKBTIUmXSzou6YSkzy4x5lJJD0k6JukHReSy7jIzM8OePXsYHx9n\nU7PJPuBzwD6g1GwyPj7Onj17+NnPftbmpJZ7sUjaBIwCe4FdwDWSdi0a0wPcDFQi4p3AH+edy7rP\n8PAw9Xqd3UAdGANuan19ErLt9ToHDhxoY0qDYo5YLgFOREQ9Ik4BdwJDi8Z8ArgrIqYBIuLZAnJZ\nF5menmZiYoJzgMPA9kX7twN3k53bT0xM0Gg0io5oCxRRLNuAhf+XT7a2LbQT6JV0r6Sjkq4tIJd1\nkVqtRrPZpMJrS+W0PrJnrGazSa1WKy6cvUanTN5uBt4HXAa8AfihpAci4omFgyTtB/YD9Pf3Fx7S\n2mdubg6Ai5YZt7P1dXZ2Ntc8dnZFHLE8TfZkctr21raFTgL3RMRLEfE8cB/wnsU/KCIORcRgRAxu\n2bIlt8DWeXp6egA4vsy4089Evb29ueaxsyuiWB4ELpS0Q9K5wNXA4uPUw8CHJW2W9EbgA8DjBWSz\nLlGpVCiVStTInoXOpEH2h1QqlahUKsWFs9fIvVgiYh4YBu4hK4tvR8QxSddLur415nHg+8AjwI+A\n2yLisbyzWffo7++nWq3yMtk8yuKp2QZwJTAPVKtVL5ZrM0VEuzOsyeDgYExOTrY7hhXo9DqWer3O\nZrKC2Ul2+nOYrFQGBgY4cuQIW7dubWfUdUHS0YgYXMv3+r1C1jXK5TL3338/+/bto1kqMQ58GRgH\nmqUS+/btc6l0iE55VchsRcrlMmNjYzQaDWq1GrOzs/T29lKpVHz600FcLNaV+vr6vMK2g/lUyMyS\nc7GYWXIuFjNLzsViZsm5WMwsOReLmSXnYjGz5FwsZpaci8XMknOxmFlyay4WSTekDGJm68eK3ysk\n6dsL7wLvBb6aPJGZdb3VvAnxlxHxZ6fvSLolhzxmtg6s5lTopkX3P58yiJmtH8sWi6SvS1JEPLlw\ne0S8kF8sM+tmKzlieRGoSToPQNLHJd2fbywz62bLzrFExF9J+gRwr6RTwK+AM37+spkZrKBYJF0G\nfBp4CbgA+NOIWO7jXcxsA1vJqdDngb+OiEuBq4B/kPTRXFOZWVdbyanQRxfcflTSXrILo38oz2Bm\n1r1WvfI2Ip4h+4xlM7MzWtOS/oj479RBzGz98JsQzSw5F4uZJediMbPkXCxmlpyLxcySc7GYWXIu\nFjNLbjUXerJ1bnp6mlqtxtzcHD09PQwNDdHX19fuWNaFXCzGzMwMw8PDTExM0Gw2X9l+8OBBqtUq\nIyMjlMvlNia0buNi2eBmZmbYs2cP9XodOAeoAhcBx2k2DzM+Ps7U1BRHjhxh69at7Q1rXaOQORZJ\nl0s6LumEpCWv5SLp/ZLmJV1VRC6D4eHhVqnsBurAGNlVSMeAJ4Hd1Ot1Dhw40MaU1m1yLxZJm4BR\nYC+wC7hG0q4lxn0V+Me8M1lmenqaiYkJsiOVw8D2RSO2A3cDm5mYmKDRaBQd0bpUEUcslwAnIqIe\nEaeAO4GhM4z7DNnlGJ4tIJMBtVqtNadS4bWlclofMESz2aRWqxUXzrpaEcWyDVj4VHeyte0VkraR\nndyf9SNFJO2XNClp8rnnnksedKOZm5tr3bpomZE7AZidnc01j60fnbKO5WvADRHRPNugiDgUEYMR\nMbhly5aCoq1fPT09rVvLXWn0CQB6e3tzzWPrRxHF8jTZ8fRp21vbFhoE7pT0FNnlL2+WdGUB2Ta0\nSqVCqVQCamQHkmfSAA5TKpWoVCrFhbOuVkSxPAhcKGmHpHOBq8n+kl8RETsi4u0R8XaylyP+IiLu\nLiDbhtbf30+1WgVeJpv2Wjw52wCuBOapVqteLGcrlvs6loiYlzQM3ANsAm6PiGOSrm/tvzXvDLa0\nkZERpqamqNd/DAyQFcxOstOfw8A8AwMDjI6OtjOmdRlFRLszrMng4GBMTk62O8a6sNTK21KpRLVa\nZXR01IvjNiBJRyNicC3f65W3RrlcZmxsjEajQa1WY3Z2lt7eXiqVik9/bE1cLPaKvr4+r7C1JDrl\n5WYzW0dcLGaWnIvFzJJzsZhZci4WM0vOxWJmyblYzCw5F4uZJediMbPkXCxmlpyLxcySc7GYWXIu\nFjNLzsViZsm5WMwsOReLmSXnYjGz5FwsZpaci8XMknOxmFlyLhYzS87FYmbJuVjMLDkXi5kl52Ix\ns+RcLGaWnIvFzJJzsZhZci4WM0vOxWJmyW1ud4D1RNKqxkdETknM2stHLGaWXCFHLJIuB74ObAJu\ni4ivLNr/SeAGQMCLwJ9HxMNFZMvFF1/nfrMul/sRi6RNwCiwF9gFXCNp16JhTwJ/EBHvAr4EHMo7\nl5nlp4hToUuAExFRj4hTwJ3A0MIBEXEkImZbdx8AtheQy8xyUkSxbAMaC+6fbG1byqeA7+WayMxy\n1VGvCkn6CFmxfHiJ/fuB/QD9/f0FJjOz1SjiiOVpoG/B/e2tba8i6d3AbcBQRPz8TD8oIg5FxGBE\nDG7ZsiWXsGb2+hVRLA8CF0raIelc4GqgtnCApH7gLuBPIuKJAjKZWY5yPxWKiHlJw8A9ZC833x4R\nxyRd39p/K/AF4K3Aza1FZvMRMZh3NjPLh7p19efg4GBMTk62O8areOWtrSeSjq71Cd4rb80suY56\nVajb+QjELOMjFjNLzsViZsm5WMwsOReLmSXnYjGz5FwsZpaci8XMknOxmFlyXiBnuZuenqZWqzE3\nN0dPTw9DQ0P09fUt/43WtVwslpuZmRmGh4eZmJig2Wy+sv3gwYNUq1VGRkYol8ttTGh5cbFYLmZm\nZtizZw/1eh04B6gCFwHHaTYPMz4+ztTUFEeOHGHr1q3tDWvJeY7FcjE8PNwqld1AHRgDbmp9fRLY\nTb1e58CBA21MaXlxsVhy09PTTExMkB2pHOa110bfDtwNbGZiYoJGo7H4R1iXc7FYcrVarTWnUmHp\nD1zoA4ZoNpvUarUlxli3crFYcnNzc61bFy0zcicAs7Ozy4yzbuNiseR6enpat44vMzK7vHFvb2+u\neax4LhZLrlKpUCqVyK6ZfnKJUQ3gMKVSiUqlUlw4K4SLxZLr7++nWq0CL5N96OXiydkGcCUwT7Va\n9WK5dcjrWCwXIyMjTE1NUa//GBggK5idZKc/h4F5BgYGGB0dbWdMy4mPWCwX5XKZ+++/n3379lEq\nNYFx4MvAOKVSk3379nlx3DrmIxbLTblcZmxsjEajQa1WY3Z2lt7eXiqVik9/1jkXi+Wur6/PK2w3\nGJ8KmVlyLhYzS87FYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZci4WM0vOxWJmyRVSLJIu\nl3Rc0glJnz3Dfkn6Rmv/I5J2F5HLzPKRe7FI2gSMAnuBXcA1knYtGrYXuLD1bz9wS965zCw/RRyx\nXAKciIh6RJwC7iS76s9CQ8A3I/MA0CPpggKymVkOirhswjZefW3Ck8AHVjBmG/DMwkGS9pMd0QD8\nr6TH0kbNzfnA8+0OsQrdlLebskJ35V3uYxaW1FXXY4mIQ8AhAEmTETHY5kgr0k1ZobvydlNW6K68\nkibX+r1FnAo9TfbpVKdtb21b7Rgz6xJFFMuDwIWSdkg6F7ia7HMhFqoB17ZeHfog8IuIeGbxDzKz\n7pD7qVBEzEsaBu4BNgG3R8QxSde39t8KfBe4AjgB/Bq4bgU/+lBOkfPQTVmhu/J2U1borrxrzqqI\nSBnEzMwrb80sPReLmSXX8cXSTW8HWEHWT7YyPirpiKT3tCPngjxnzbtg3PslzUu6qsh8izIsm1XS\npZIeknRM0g+Kzrgoy3J/C2+R9B1JD7fyrmReMReSbpf07FLrwtb0GIuIjv1HNtn7n2Sf0Xku8DCw\na9GYK4DvAQI+CPxHB2f9ENDbur23XVlXmnfBuH8hm2C/qlOzAj3AT4D+1v23dfLvFvgc8NXW7S3A\nC8C5bcr7+8Bu4LEl9q/6MdbpRyzd9HaAZbNGxJGImG3dfYBsvU67rOR3C/AZss9HfbbIcIusJOsn\ngLsiYhogIjo9bwBvliTgTWTFMl9szFaQiPta//2lrPox1unFstRS/9WOKcJqc3yK7FmgXZbNK2kb\nUKX9bwpdye92J9Ar6V5JRyVdW1i611pJ3hHgHcBPgUeBgxHRLCbeqq36MdZVS/rXC0kfISuWD7c7\nyzK+BtwQEc3sibWjbQbeB1wGvAH4oaQHIuKJ9sZa0seBh4CPAr8D/JOkf4uIX7Y3VhqdXizd9HaA\nFeWQ9G7gNmBvRPy8oGxnspK8g8CdrVI5H7hC0nxE3F1MxFesJOtJ4OcR8RLwkqT7gPcA7SiWleS9\nDvhKZJMYJyQ9CVwM/KiYiKuy+sdYuya4VjiptBmoAzv4zSTYOxeN+UNePbH0ow7O2k+2uvhD3fC7\nXTT+Dto3ebuS3+07gH9ujX0j8Bjwux2c9xbgi63bW1sP1PPb+PfwdpaevF31Y6yjj1giv7cDtCvr\nF4C3Aje3jgLmo03vdF1h3o6wkqwR8bik7wOPAE3gtohoy2U1Vvi7/RJwh6RHyR6wN0REWy6nIOnv\ngUuB8yWdBG4EzlmQddWPMS/pN7PkOv1VITPrQi4WM0vOxWJmyblYzCw5F4uZJediMbPkXCyWG0n/\nKuljrdt/K+nv2p3JitHRC+Ss690I/I2ktwG/B1TanMcK4gVylqvWBZfeBFwaES9KOg+4GTgF3BsR\n32prQMuFT4UsN5LeBVwAnIqIF1ub/wgYi4hP4yOYdcvFYrloXQjoW2QXCfqVpMtbu7bzm2t7/F87\nsln+XCyWnKQ3AncBfxkRj5O94e7G1u6T/ObKef77W6c8x2KFas2xjAD/A/y751jWJxeLmSXnQ1Ez\nS87FYmbJuVjMLDkXi5kl52Ixs+RcLGaWnIvFzJJzsZhZci4WM0vu/wHAhTb+9x55PgAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1141407f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q = np.array([0.2, 0.2])\n",
    "plt.plot(X[y_neg, 0], X[y_neg, 1], '.', c='b', markersize=20, mew=2, mec='k')\n",
    "plt.plot(X[y_pos, 0], X[y_pos, 1], '.', c='r', markersize=20, mew=2, mec='k')\n",
    "plt.plot(q[0], q[1], 's', c='g', markersize=10, mew=2, mec='k')\n",
    "plt.xlabel(r\"$x_0$\")\n",
    "plt.ylabel(r\"$x_1$\")\n",
    "plt.xlim((0, 1))\n",
    "plt.ylim((0, 1))\n",
    "plt.axes().set_aspect('equal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kernels\n",
    "---\n",
    "\n",
    "We'll use a Gaussian kernel $K_h$ with bandwidth $h$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def d(x, y):\n",
    "    return np.linalg.norm(x - y) # Euclidean distance\n",
    "\n",
    "h = 0.5\n",
    "def K(x, y):\n",
    "    return np.exp(- d(x, y)**2.0 / (2 * h**2.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That means that the value of $K_h(x, y)$ is the value of a 1-dimensional Gaussian centered at $0$ with $\\sigma^2=h$, evaluated at the point $|x-y|$, that is the point on the axis which is $|x-y|$ in distance from the centre of the Gaussian. \n",
    "\n",
    "A kernel is a *measure of similarity*, so $K$ will be large when $x$ and $y$ are close together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEOCAYAAAB8aOvdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VfW97vHPNxMhzJAwD2EIKKhQjMggFRUrWlt6W9ui\nR604HZxqT9t7aq8dzm17zj12uq1WaznWqkfr0Dpxna1iqSCVoMiMRGQIhnkeQqbv/WNvMMVs2CF7\nr7VX8rxfr7zYe+0fO89yYZ6svX5rLXN3REREGpMVdgAREclcKgkREUlIJSEiIgmpJEREJCGVhIiI\nJKSSEBGRhFQSIiKSkEpCREQSUkmIiEhCOWEHaK7CwkIvLi4OO4aISKQsXLhwm7sXHW9c5EuiuLiY\nsrKysGOIiESKma1LZpw+bhIRkYRUEiIikpBKQkREElJJiIhIQioJERFJKLCSMLP7zWyLmS1N8LqZ\n2Z1mVm5mi81sdFDZRESkcUHuSTwATDnG6xcCJfGv64HfBpBJRESOIbDzJNx9jpkVH2PIVOAhj91P\ndb6ZdTazXu5emY485Vv2Muu9Svp0zqdXp7b061pAcbcCzCwd305EJJIy6WS6PsCGBs8r4ss+URJm\ndj2xvQ369+9/Qt9sReVe7np9NQ1v8T2osB0Xj+zN50f2Zkj39if0viIiLUkmlUTS3H0mMBOgtLTU\njzO8UZ8b2Zspp/Rk0+4qKndXsWrzXl5YXMldr6/mztdWc9GpPfnuhSfTr2tBSrOLiERJJpXERqBf\ng+d948vSJjc7i35dC+jXtYAxA7tyxdgBbNlTxR/fXs/v/rqGv6zYwrVnDeTmc4dQkJdJ/6lERIKR\nSVNgZwFXxmc5jQV2p+t4xLF075jPNyYPZfa3J3Hxqb24540P+OI981i3fX/QUUREQhfkFNhHgbeA\nYWZWYWbXmNkMM5sRH/ICsAYoB/4LuDGobI3p2SmfX351FA9ePYbK3VV87q43mb1qS5iRREQCZ+4n\n9JF+xigtLfV0XwV2/fYD/PPDC1m5aQ//9rkRfG18cVq/n4hIupnZQncvPd64TPq4KWP171bAUzeM\nZ/LJPfjhrGU8OG9t2JFERAKhkkhS27xs7r5sNOcPjxXFQ2+tDTuSiEjaqSSaIC8n60hR/ODZZTy+\nYH3YkURE0kol0USHi+LTQ4u4/emlvPXB9rAjiYikjUriBOTlZPGbyz5FcWE7bnhkIeu3Hwg7kohI\nWqgkTlDH/FzuuzI2MeCaBxewt6om5EQiIqmnkmiG4sJ23PNPo/lw236+/af3iPp0YhGRo6kkmmn8\n4EK+M+UkXl62mUff3nD8vyAiEiEqiRS45qyBTCwp5EfPLaN8y96w44iIpIxKIgWysoxffHkkBXk5\nfP3RRRyqrQs7kohISqgkUqR7x3x++qXTWF65h1+88n7YcUREUkIlkUKTh/fg0jH9ue9va3hvw66w\n44iINJtKIsW+e9FJFLZvw21PLaGmrj7sOCIizaKSSLGO+bn8aOoprKjcw31/+zDsOCIizaKSSIMp\np/TkghE9+NVf3mftNt2sSESiSyWRJj+aegp52Vl875mlOslORCJLJZEmPTrm863PDOXN8m28snxz\n2HFERE6ISiKNLh87gKE92vPvz6+gqkbnTohI9Kgk0ignO4sfXDyC9TsOcP9cHcQWkehRSaTZWSWF\nfGZ4D37zejmb91SFHUdEpElUEgG4/bMnU1vn3PHSyrCjiIg0iUoiAAO6tWP6WcU8/e5GVlTuCTuO\niEjSVBIBufHsIXRok8NPtTchIhGikghIp4JcbjxnCLNXbWX+Gt0XW0SiQSURoKvGF9OzYz7/+eJK\nnWAnIpGgkghQfm4235hcwqINu3h5mU6wE5HMp5II2CWn92VwUTt+/soq6uq1NyEimU0lEbCc7Cy+\nef4wyrfs47nFH4UdR0TkmFQSIbjwlJ4M69GBX7+2WnsTIpLRVBIhyMoybp1cwpqt+/l/72lvQkQy\nV6AlYWZTzGyVmZWb2W2NvN7JzP6fmb1nZsvMbHqQ+YI0ZURPTurZgTu1NyEiGSywkjCzbOBu4EJg\nOHCpmQ0/athNwHJ3HwlMAn5hZnlBZQxSVpZx63klrNm2n1nvbQw7johIo4LckxgDlLv7GnevBh4D\nph41xoEOZmZAe2AHUBtgxkBdcGRvolx7EyKSkYIsiT7AhgbPK+LLGvoNcDLwEbAEuNXd64OJF7zD\nexMfbtvPC0sqw44jIvIJmXbg+gJgEdAbGAX8xsw6Hj3IzK43szIzK9u6dWvQGVPqghE9GVzUjnve\n+EBnYYtIxgmyJDYC/Ro87xtf1tB04CmPKQc+BE46+o3cfaa7l7p7aVFRUdoCByEry7hh0hBWVO7h\njVXRLjwRaXmCLIkFQImZDYwfjJ4GzDpqzHrgPAAz6wEMA9YEmDEUU0f1pk/ntvxmdrn2JkQkowRW\nEu5eC9wMvAysAJ5w92VmNsPMZsSH/RgYb2ZLgNeA77j7tqAyhiU3O4sZZw9i4bqdvP3hjrDjiIgc\nYVH/zbW0tNTLysrCjtFsVTV1nHXHbIb37shDV48JO46ItHBmttDdS483LtMOXLda+bnZXDtxIHPe\n38rSjbvDjiMiAqgkMsplZ/anfZscZs5p8YdhRCQiVBIZpGN+Lped2Z/nl1SyYceBsOOIiKgkMs30\nCcUY8Ps3Pww7ioiISiLT9OrUlqmj+vD4gg3s3F8ddhwRaeVUEhno+k8P4mBNHQ/PXxd2FBFp5VQS\nGWhYzw5MGlbEg2+tpaqmLuw4ItKKqSQy1PWfHsS2fdU8864uIy4i4VFJZKhxg7oxvFdHfv/mh7pU\nh4iERiWRocyMaycOZPWWfcxZ3eKvTCIiGUolkcEuPq033Tu04b6/6eQ6EQmHSiKD5eVk8bXxxfxt\n9TZWbdobdhwRaYVUEhnusjH9yc/N4n6dXCciIVBJZLgu7fK45PS+PL1oI9v2HQo7joi0MiqJCJg+\nYSDVtfU8Mn992FFEpJVRSUTA4KL2TBpWxMN/X8ehWp1cJyLBUUlExPQJA9m69xDPL64MO4qItCIq\niYj4dEkhQ7q35/65OrlORIKjkogIM+Oq8cUs3biHsnU7w44jIq2ESiJCvji6Dx3zc/jDXE2HFZFg\nqCQipCAvh0vP7M9LSzdRsVN3rhOR9FNJRMyV44oB+G/da0JEAqCSiJg+ndtywYiePPb2Bg5Wazqs\niKSXSiKCrhpfzO6DNTyzSPeaEJH0UklE0JiBXTm5V0cemLtW02FFJK1UEhFkZkwfX8yqzXt5a832\nsOOISAumkoioz4/qTZeCXB6YuzbsKCLSgqkkIio/N5tLx/TnLys2s2GHpsOKSHqoJCLsinEDMDNN\nhxWRtFFJRFivTm2ZMqInjy/QdFgRSQ+VRMR9TdNhRSSNAi0JM5tiZqvMrNzMbkswZpKZLTKzZWb2\n1yDzRdEZxV0YrumwIpImgZWEmWUDdwMXAsOBS81s+FFjOgP3AJ939xHAl4PKF1WHrw67avNe5q/Z\nEXYcEWlhgtyTGAOUu/sad68GHgOmHjXmMuApd18P4O5bAswXWUemw87T1WFFJLWCLIk+wIYGzyvi\nyxoaCnQxszfMbKGZXdnYG5nZ9WZWZmZlW7duTVPc6MjPzWbamP68unyzrg4rIimVaQeuc4DTgc8C\nFwDfN7OhRw9y95nuXurupUVFRUFnzEiXj9V0WBFJvSBLYiPQr8HzvvFlDVUAL7v7fnffBswBRgaU\nL9L6dG7LZ4b30HRYEUmpIEtiAVBiZgPNLA+YBsw6asyzwFlmlmNmBcCZwIoAM0baVeOL2XWghmc1\nHVZEUiSwknD3WuBm4GViP/ifcPdlZjbDzGbEx6wAXgIWA28D97n70qAyRt2Rq8PO03RYEUmNnCC/\nmbu/ALxw1LJ7j3r+M+BnQeZqKWLTYQfwnSeXMH/NDsYN7hZ2JBGJuCbvSZhZu/g5D5KBpo7qQ+eC\nXB6ctzbsKCLSAhy3JMwsy8wuM7PnzWwLsBKoNLPlZvYzMxuS/piSrMNXh31l+SZNhxWRZktmT2I2\nMBj4LtDT3fu5e3fgLGA+cIeZXZ7GjNJER6bDvqXpsCLSPMkck5js7jVHL3T3HcCTwJNmlpvyZHLC\nDk+HfWzBBm6dXEJBXqCHnkSkBTnunsThgjCzV82s0XMWGisRCdf0CQPZfbCGp9/VdFgROXFNOXD9\nHeBXZvYHM+uVrkCSGmcUd2FEb10dVkSaJ+mScPd33P0c4DngJTP7oZm1TV80aQ4zY/qEgazeso+5\n5dvDjiMiEdWkKbBmZsAq4LfALcBqM7siHcGk+T43sheF7fP4w1xdHVZETkzSJWFmc4lda+n/Ert6\n61XAJGCMmc1MRzhpnjY52Vw2pj+vr9rC2m37w44jIhHUlD2J64E+7n6+u3/f3Z9z93J3vwWYmKZ8\n0kyXjx1ATpbxgE6uE5ET0JRjEss88RHQz6Yoj6RY9475XHxab/5UtoE9VZqEJiJNk5IL/Ln7mlS8\nj6TH1RMGsr+6jicWbDj+YBGRBk64JMysl5m1SWUYSY9T+3ZiTHFXHpi3lrp6TYcVkeQ1Z0/iv4GV\nZvbzVIWR9Ln6rGIqdh7k1eWbwo4iIhFywiXh7pOBQcAfUhdH0uX84T3p26Utv39T02FFJHlNmQL7\n6/h5Ekd4zLLUx5JUy84yrhpfzIK1O1lcsSvsOCISEU3Zk9gLzDKzdgBmdkH83AmJiK+e0Y/2bXK4\nX3sTIpKkpkyB/R7wKPBGvBy+CdyWrmCSeh3yc/lKaT+eW1xJ5e6DYccRkQhoysdN5wHXAfuBQuDr\n7v63dAWT9Jg+oZh6dx6YuzbsKCISAU35uOl24PvuPgm4BHjczM5NSypJm35dC7jw1F788e317DtU\nG3YcEclwTfm46Vx3fzP+eAlwIfCTdAWT9Llu4iD2VtXyuE6uE5HjSOYe19bYcnevBM471hjJTKP6\ndeaM4i7c/+aH1NbVhx1HRDJYUve4NrNbzKx/w4VmlgeMM7MHga+lJZ2kzbUTB7Fx10FeWqaT60Qk\nsWRK4mKgDnjUzD4ys+VmtgZYDVwK/MrdH0hjRkmDySf3YGBhO2bOWaM714lIQsmUxF3ufo+7TwAG\nEPuIabS7D3D369z93fRGlHTIzjKunTiQxRW7mb9mR9hxRCRDJVMS2Wb2IwB3r3H3SnffZWYdzOxP\nac4nafSl0X0pbJ/H7+Z8EHYUEclQyZTE1cSOPVx7eIGZjQTKgBXpCibpl5+bzVXji3lj1VZWVO4J\nO46IZKDjloS71wJfBG4ws4vM7DrgWeAmd/9BugNKel0+dgAFednMnKNbgojIJyUzBfYe4J+AnxK7\n4utXgTPd/S9pziYB6FyQx7Qz+jPrvY+o2Hkg7DgikmGS+bhpETACuBHIBYYC/2Vm/25m09IZToJx\nzcSBGOgy4iLyCcl83DTT3W9x97PdvSswHvgtsIcm3tvazKaY2SozKzezhBcHNLMzzKzWzC5pyvvL\nienTuS2fH9mbx97ewI791WHHEZEM0uSbDrl7hbu/6O53uPsVyf49M8sG7iZ2OY/hwKVmNjzBuDuA\nV5qaTU7cDZMGc7Cmjgfmam9CRD7WnNuXNtUYoNzd17h7NfAYMLWRcbcATwJbAszW6pX06MAFI3rw\nwLy17K2qCTuOiGSIIEuiD9DwinIV8WVHmFkf4H8Q+zhLAnbTOUPYU1XLw/PXhx1FRDJEkCWRjF8B\n33H3Y151zsyuN7MyMyvbunVrQNFavtP6dmZiSSG/f3MNVTV1YccRkQwQZElsBPo1eN43vqyhUuAx\nM1tL7J4V95jZF45+o/jB9FJ3Ly0qKkpX3lbppnOGsG1ftS4jLiJAsCWxACgxs4HxK8hOA2Y1HODu\nA9292N2LgT8DN7r7MwFmbPXOHNiV0wd04Xd//YDqWl1GXKS1C6wk4mdu3wy8TOxyHk+4+zIzm2Fm\nM4LKIcdmZtxy7hA+2l3FnxdWhB1HREKWE+Q3c/cXgBeOWnZvgrFXBZFJPunsoUWM6teZu2eXc8np\nfcnLybRDVyISFP3fL59gZtw6uYSNuw7y5DvamxBpzVQS0qhJQ4sY2bcTd88up0a3OBVptVQS0qjD\nexMVOw/ylPYmRFotlYQkdM6w7pzWtxO/0d6ESKulkpCEzIx/mTyUDTsO8kSZzpsQaY1UEnJMk4YV\ncfqALtz1WrnOwhZphVQSckxmxrc/M4xNe6p4eP66sOOISMBUEnJc4wZ346whhfz2jQ/Yf6g27Dgi\nEiCVhCTl2xcMY/v+av6g+02ItCoqCUnKqH6dmXxyD343Zw07dfc6kVZDJSFJ+58XDGP/oVrunl0e\ndhQRCYhKQpI2rGcHLjm9Lw+9tY4NOw6EHUdEAqCSkCb5l/OHkpUFP39lVdhRRCQAKglpkl6d2nLN\nWQN5dtFHLN24O+w4IpJmKglpsn8+ezBdCnL5jxdW4O5hxxGRNFJJSJN1zM/l6+eVMO+D7by+ckvY\ncUQkjVQSckIuHzuAwUXt+MnzK3SbU5EWTCUhJyQ3O4vvXzycD7ft58F5a8OOIyJpopKQEzZpWHfO\nGVbEna+tZtu+Q2HHEZE0UElIs3zv4uEcrKnjF5oSK9IiqSSkWQYXteeq8cU8tmADSyo0JVakpVFJ\nSLN9fXIJhe3bcPszS6ir15RYkZZEJSHN1jE/l+999mQWV+zmj3/XPSdEWhKVhKTE50f2ZsKQbvz0\n5VVs2VsVdhwRSRGVhKSEmfHjqadwqKae/3h+RdhxRCRFVBKSMoOK2vPPZw/imUUf8ebqbWHHEZEU\nUElISt10zhAGFbbjtqcW61anIi2ASkJSKj83mzsuOY2Nuw7ys5d17oRI1KkkJOXOKO7KlWMH8OBb\naylbuyPsOCLSDCoJSYt/nXISvTu15V//vJiqmrqw44jICVJJSFq0a5PDHV86jTXb9utjJ5EIC7Qk\nzGyKma0ys3Izu62R1//JzBab2RIzm2dmI4PMJ6l1VkkhV44bwO/f/JC55ZrtJBJFgZWEmWUDdwMX\nAsOBS81s+FHDPgTOdvdTgR8DM4PKJ+nx3QtPZnBRO771xHvsOlAddhwRaaIg9yTGAOXuvsbdq4HH\ngKkNB7j7PHffGX86H+gbYD5Jg7Z52fx62qfYtu8Qtz+9VLc7FYmYIEuiD7ChwfOK+LJErgFebOwF\nM7vezMrMrGzr1q0pjCjpcEqfTvzL+UN5fkklf1pYEXYcEWmCjDxwbWbnECuJ7zT2urvPdPdSdy8t\nKioKNpyckBlnD2bsoK784NmlrNq0N+w4IpKkIEtiI9CvwfO+8WX/wMxOA+4Dprr79oCySZplZxl3\nTvsU7dvkcuMjC3U2tkhEBFkSC4ASMxtoZnnANGBWwwFm1h94CrjC3d8PMJsEoHvHfO6cNoo12/bz\nvWd0fEIkCgIrCXevBW4GXgZWAE+4+zIzm2FmM+LDfgB0A+4xs0VmVhZUPgnG+CGFfOO8oTz97kb+\n+Pb6sOOIyHFY1H+bKy0t9bIydUmU1NU7Vz+wgLnl23jk2jM5c1C3sCOJtDpmttDdS483LiMPXEvL\nlp1l3Hnpp+jftYAbHnmHip0Hwo4kIgmoJCQUndrm8l9fK6Wmrp7rHlrIgWodyBbJRCoJCc3govbc\neemnWLlpD7c+toi6+mh/9CnSEqkkJFTnDOvOv31uBK8u38wPZ2nGk0imyQk7gMjXxhdTubuKe//6\nAb06teWmc4aEHUlE4lptSZhZ2BGkETffEZsnDWivQiQD6OMmERFJqNXuSei31Mx0sLqOgjaxf5Yv\nLqnkwlN7hZxIpHXTnoRklLZ52Uce3/Lou7yybFOIaUREJSEZa0SfTtz4yDs8u+gT14EUkYCoJCRj\nPXzNGE4f0IVvPL6IR/6+Luw4Iq2SSkIyVof8XB68egznDuvO7U8v5e7Z5TqWJBIwlYRktPzcbO69\n4nS+MKo3P3t5Fbc9uYTq2vqwY4m0Gq12dpNER252Fr/8yij6dy3gztfLWb/jAPdefjqdCnLDjibS\n4mlPQiIhK8v45meG8cuvjKRs3Q6+cM9cVm7aE3YskRZPJSGR8sXRffnjdWPZd6iWL9w9lycXVoQd\nSaRFU0lI5JxR3JXnv34Wo/p15lt/eo/bnlyse2aLpIlKQiKpe4d8Hr7mTG6cNJjHyzZw0Z1/Y+G6\nnWHHEmlxVBISWTnZWfzrlJN49Lqx1NY5X753Hj99aSVVNXVhRxNpMVQSEnljB3XjpW9M5Euj+3LP\nGx9wwa/mMOf9rWHHEmkRVBLSInTIz+VnXx7JI9eeSZYZV97/Njf/UffPFmkulYS0KBOGFPLirRO5\n9bwSXl2+mXN/8Vf+z4sr2H2wJuxoIpFkUb/MQWlpqZeVlYUdQ1Lo8A2hmvtv86NdB/n5K6t4+t2N\ndGqby3UTB3HluAF0yNdJeCJmttDdS487TiUhmSZVJXHY0o27+eWr7/P6yi10zM9h+oSBXDluAN3a\nt0nJ+4tEkUpCIivVJXHYkord3PX6al5Zvpm8nCymjuzNVROKGdG7U0q/j0gUqCQkstJVEoeVb9nL\nA/PW8uTCjRysqWNkv858pbQvnxvZm476KEpaCZWERFa6S+Kw3Qdq+PM7FfypbAMrN+2lTU4W557U\nnc+e1otzT+pOQZ6ufyktl0pCIiuokjjM3VmycTdPLqzghaWb2Lr3EPm5WZw1pJBzTurOOcO607tz\n20CyiARFJSGRFXRJNFRX7yxYu4MXl1Ty2sotVOw8CMDgonaMG9yNcYMKOWNgF7p3yA88m0gqqSQk\nssIsiYbcnfIt+5i9agvzPtjOgg93sL86dsmPPp3bMnpAF07r04kRvTsyvHdHOhfkhZpXpCkysiTM\nbArwayAbuM/d//Oo1y3++kXAAeAqd3/nWO+pkmh5MqUkjlZTV8+Sjbt5Z91O3l2/i3fX7+Sj3VVH\nXu/ZMZ+SHu0Z0r09g4vaU9ytHQO6FdCrUz452TpvVTJLsiUR2JE5M8sG7gbOByqABWY2y92XNxh2\nIVAS/zoT+G38T5HQ5WZnMbp/F0b373Jk2fZ9h1heuYdlH+3h/c17Kd+yj8cXbOBA9ccXGczOMnp2\nzKdP57b07pxPj475FHVoQ/eO+RS2y6Nb+zZ0bZdH54JcclUmkmGCnL4xBih39zUAZvYYMBVoWBJT\ngYc89ivkfDPrbGa93L0ywJwiSevWvg0TS4qYWFJ0ZFl9vbNl7yHWbt/Puu37Wb/jAB/tqmLjzoOU\nrdvJlr2HEt6nu32bHDq1zaVj21w65OfQMT+Hdm1iX+3b5FCQl03b3GwK8rJpk5tNfm42+TlZ5OVk\n0SYnO/5nFrnZWeRmG7nZWeRkGzlZsefZWbHH2Vmxx1n28Z6bSGOCLIk+wIYGzyv45F5CY2P6ACoJ\niYysLKNnp3x6dspn7KBun3jd3dlzsJbNe6vYvq+a7fsPsX1fNbsP1rDrQA27DlSzp6qWPVU1bNxV\nxf5DtbGv6lqqahovl2bltdjejpmRbbHiyDLDLLYuWWYYsTIxI/4YjIbPY0VzuG8Ov/7x4481LKV/\nqCdr9OExpbvgMr0+v3pGP66dOCit3yOSE8HN7HrgeoD+/fuHnEakacyMTgW5dCrIhR5N+7v19c7B\nmjoOVNdRVVPHodo6qmrqOVRbx6Haeqpr66mpc2rqDj+up7beqY3/WVfvR57X1UOdO/X1Tr37kcfu\nUO9Q7467f/yYWMG5E/si/piPnwMQX8bh8Q3yNzzM9I/LvdHlx5TmQ1ae7m+QAoUBXFomyJLYCPRr\n8LxvfFlTx+DuM4GZEDtwndqYErZMO2CdSbKy7MjHTyJBCPIo2QKgxMwGmlkeMA2YddSYWcCVFjMW\n2K3jESIi4Qns1xF3rzWzm4GXiU2Bvd/dl5nZjPjr9wIvEJv+Wk5sCuz0oPKJiMgnBbrP6u4vECuC\nhsvubfDYgZuCzCQiIolpUraIiCSkkhARkYRUEiIikpBKQkREElJJiIhIQpG/VLiZbQXWneBfLwS2\npTBO2FrS+mhdMlNLWhdoWevT1HUZ4O5FxxsU+ZJoDjMrS+ZSuVHRktZH65KZWtK6QMtan3Stiz5u\nEhGRhFQSIiKSUGsviZlhB0ixlrQ+WpfM1JLWBVrW+qRlXVr1MQkRETm21r4nISIix9AqSsLMppjZ\nKjMrN7PbGnndzOzO+OuLzWx0GDmTkcS6TDKz3Wa2KP71gzByJsPM7jezLWa2NMHrUdoux1uXKG2X\nfmY228yWm9kyM7u1kTGR2DZJrkuUtk2+mb1tZu/F1+d/NzImtdvG43efaqlfxC5L/gEwCMgD3gOG\nHzXmIuBFYncrHAv8PezczViXScBzYWdNcn0+DYwGliZ4PRLbJcl1idJ26QWMjj/uALwf4f9nklmX\nKG0bA9rHH+cCfwfGpnPbtIY9iTFAubuvcfdq4DFg6lFjpgIPecx8oLOZ9Qo6aBKSWZfIcPc5wI5j\nDInKdklmXSLD3Svd/Z34473ACmL3mm8oEtsmyXWJjPh/733xp7nxr6MPLKd027SGkugDbGjwvIJP\n/iNJZkwmSDbn+Phu5otmNiKYaGkRle2SrMhtFzMrBj5F7DfWhiK3bY6xLhChbWNm2Wa2CNgCvOru\nad02ulFuy/MO0N/d95nZRcAzQEnImSSC28XM2gNPAt9w9z1h52mO46xLpLaNu9cBo8ysM/C0mZ3i\n7o0eC0uF1rAnsRHo1+B53/iypo7JBMfN6e57Du+OeuxOgLlmVhhcxJSKynY5rqhtFzPLJfZD9RF3\nf6qRIZHZNsdbl6htm8PcfRcwG5hy1Esp3TatoSQWACVmNtDM8oBpwKyjxswCrozPChgL7Hb3yqCD\nJuG462JmPc3M4o/HENvG2wNPmhpR2S7HFaXtEs/5e2CFu/8ywbBIbJtk1iVi26YovgeBmbUFzgdW\nHjUspdumxX/c5O61ZnYz8DKx2UH3u/syM5sRf/1eYvfdvggoBw4A08PKeyxJrsslwA1mVgscBKZ5\nfMpDpjGzR4nNLCk0swrgh8QOxEVqu0BS6xKZ7QJMAK4AlsQ/+wb4X0B/iNy2SWZdorRtegEPmlk2\nsTJ7wt2cUYcnAAABm0lEQVSfS+fPM51xLSIiCbWGj5tEROQEqSRERCQhlYSIiCSkkhARkYRUEiIi\nkpBKQkREElJJiIhIQioJkSSY2RvxC8S1qO8lcjwqCRERSUglIdJE8TudnR9//BMzu+sE3uMUM5vX\n4PloM3stlTlFUqHFX7tJJA1+CPzIzLoTuz/B50/gPZYDg8wsO37p518C30xhRpGUUEmINJG7z4lf\nNfSbwKT4D/kjzOwvQM9G/urt7v5s/D3qzWwZMMLMSoB1h++gJpJJVBIiTWRmpxK7Guf2+C0x/4G7\nT07yreYTu0rpjXzyngAiGUHHJESaIH6v4EeI3Ud4n5k154f7fOAnwNPunpE37BFRSYgkrwB4CviW\nu68Afkzs+MSJWgkcAu5IQTaRtNDHTSLJO+Du4w4/cfc5wLhjjD+eW4Hvuvv+ZicTSRPtSYgEzMwG\nm9lKoK27Pxh2HpFj0Z6ESHIeAHal4o3c/QPgpCC+l0hz6falIiKSkD5uEhGRhFQSIiKSkEpCREQS\nUkmIiEhCKgkREUlIJSEiIgmpJEREJKH/D3l1Y56AIpr3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1142fcf60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "grid = np.linspace(0, 1, 100)\n",
    "x0 = np.array([0.4, 0.4])\n",
    "y0 = np.array([2.5, 2.5])\n",
    "K_vals = [K(x0, x0+(y0-x0)*gi) for gi in grid]\n",
    "d_vals = [d(x0, x0+(y0-x0)*gi) for gi in grid]\n",
    "plt.plot(d_vals, K_vals)\n",
    "plt.plot((0, d(x0, (1.1, 1.1))), (K(x0, (1.1, 1.1)), K(x0, (1.1, 1.1))), color='k', linestyle='-', linewidth=2)\n",
    "plt.plot((1, 1)                , (0                , K(x0, (1.1, 1.1))), color='k', linestyle='-', linewidth=2)\n",
    "plt.plot(())\n",
    "plt.xlabel(r\"$|x-y|$\")\n",
    "plt.ylabel(r\"$K(x, y)$\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the diagram above, we have plotted the value of $K$ as distance increases. We have two points $x'$ and $y'$: think of $x'$ as being at 0.0 on the horizontal axis, and as the point $y$ moves from $x'$ to $y'$, we move along the horizontal axis, and $K$ drops. For example, when $y=(1.1, 1.1)$, we have: $$|x - y| = d(x, y) = 1.0$$\n",
    "so we look at 1.0 on the horizontal axis, and see that $K(x, y)$ has a value near 0.14."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and the implicit mapping to a new space\n",
    "---\n",
    "\n",
    "Let's train our SVM. We'll talk about the training procedure shortly, but for now go straight to looking at the result. With the right choice of parameters, we'll get exactly two support vectors, that is two of our training points become support vectors. They'll be the points which are closest to the boundary between classes: the two points $(0.4, 0.4), (0.6, 0.6)$ -- one labelled -1, the other +1. We can also print the weights and bias, and we'll come back to those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.4  0.4]\n",
      " [ 0.6  0.6]]\n",
      "[-6.76332653  6.76332653]\n",
      "[-0.]\n"
     ]
    }
   ],
   "source": [
    "svc = SVC(kernel='rbf', gamma=1 / (2 * h**2), C=10)\n",
    "svc.fit(X, y)\n",
    "SVs = svc.support_vectors_\n",
    "w = svc.dual_coef_[0]\n",
    "b = svc.intercept_\n",
    "print(SVs)\n",
    "print(w)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our original training data maps to the new space as follows. For each point in the training data, and each support vector, we calculate $K$ between the training point and the SV, and that gives one feature $z_i$ for that training point. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def z(q, SVs):\n",
    "    return np.array([K(q, xi) for xi in SVs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.81873075,  0.59452055],\n",
       "       [ 1.        ,  0.85214379],\n",
       "       [ 0.85214379,  1.        ],\n",
       "       [ 0.60653066,  0.90483742]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z = np.array([z(xi, SVs) for xi in X])\n",
    "Z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the value 1.0, which occurs when evaluating $K(x, x)$, ie the similarity between a point and itself. It's the largest possible similarity.\n",
    "\n",
    "We can also do the same for our query point $q$ whose label is unknown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.85214379,  0.52729242])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qz = z(q, SVs)\n",
    "qz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the value approximately 0.85 is a measure of similarity between the green query point and the first SV, that is the upper blue point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at the implicit new space. In our example, it is of two dimensions, the same as the original space, but that isn't usually the case! The number of dimensions in the new space is equal to the number of SVs. (The reason we made sure to have just two SVs is so that the new implicit feature space could be plotted easily.) The training data and query points are plotted in the new space below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARIAAAEMCAYAAAAI6znIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFLhJREFUeJzt3X9s3Hd9x/Hn65JWg8GwRULM8oM0UwpkjKLWdIigDYY2\nmoIczHWoha1axRZlNCh/puo0Fg0hQNofjCW0yqrSsU1UKK7bW1fopiHoltBRl5Y2LWtlzBanJa0p\nThE/uuDde398z+3VteOz776f7/fuXg8pqv29b+x33dwz3/ve9/OtIgIzs3ZUih7AzLqfQ2JmbXNI\nzKxtDomZtc0hMbO2OSRm1jaHxMza5pCYWdscEjNrm0NiZm1bW/QAq7Vu3brYunVr0WOY9Zwnn/05\nz/zkLGdPT/4wIta38nu6NiRbt25lYmKi6DHMekZE8Jd3PsoXjv031+/cysGRN/1Pq7/XL23M7EUR\nuWbnVj7+vh0r+v0OiVmfWywiklb0NRwSsz7WiYiAQ2LWtzoVEXBIzPpSJyMCDolZ3+l0RMAhMesr\neUQEHBKzvpFXRMAhMesLeUYEHBKznpd3RMAhMetpKSICXbzWxqyMTp48Sa1W48yZMwwMDLB79242\nb95cyCypIgIOiVlHnD59mn379jE+Pk69Xn9++/79+xkdHeXQoUMMDQ0lmydlRCDBSxtJN0t6WtKJ\nJR6XpM9JmpT0kKSL857JrJNOnz7Nzp07GRsbY029ThW4HqgClXqdsbExdu7cyVNPPZVkntQRgTTn\nSG4BLjvH47uA7Y1fe4AbEsxk1jH79u1jamqKi4Ep4CjwycY/vw/Z9qkprr322txnKSIikCAkEXEP\n8KNz7LIb+GJk7gUGJL0277nMOuHkyZOMj49zHnAHsGnB45uA28nOIYyPjzM9PZ3bLEVFBMrxrs1G\noPmne6qxzaz0arUa9XqdEV4akXmbyf62rNfr1Gq1XOYoMiJQjpC0TNIeSROSJmZmZooex4wzZ84A\n8Ppl9ruw8c/Z2dmOz1B0RKAcIXmCLNrzNjW2vUREHImI4YgYXr++pVtJmuVqYGAAgMeW2e/xxj8H\nBwc7+v3LEBEoR0hqwNWNd2/eBjwbET8oeiizVoyMjFCpVKiRvSZfzDTZ+ZNKpcLIyEjHvndZIgJp\n3v79EvBN4PWSTkn6iKS9kvY2drmL7GT3JPC3wEfznsmsU7Zs2cLo6Ci/IDsPsvBU6jTwfmAOGB0d\n7djFaWWKCIAiorBv3o7h4eHwXeStDOavI5mammItWVAuJHs5cwdZRLZt28bx48fZsGFD298vVUQk\n3R8Rw63sW4aXNmZdbWhoiGPHjlGtVqlXKowBnwLGgHqlQrVa7bqIrJQvkTfrgKGhIY4ePcr09DS1\nWo3Z2VkGBwcZGRnp2ZczzRwSsw7avHlzLlewljki4Jc2ZqVX9oiAQ2JWat0QEXBIzEqrWyICDolZ\nKXVTRMAhMSudbosI+F0bK5ky3aqwCN0YEXBIrCTKdqvCInRrRMAhsRJovsT8PGCUbFn+Y8AdjVsV\nPvDAAx27OrSMujki4HMkVgJlulVhEbo9IuCQWMHKdKvCIvRCRMAhsYKV5VaFReiViIBDYgUrw60K\ni9BLEQGHxApW9K0Ki9BrEQGHxApW5K0Ki9CLEQGHxApW1K0Ki9CrEQFfR2IlcOjQIR544AG+PTXF\nNpa+VeHhw4eLHLMtvRwRcEisBOZvVTh/ZetY05WtlUqF6ugohw8f7tqL0VJEpOilBb75s5VKnrcq\nLELeEVlqaUGlUml7acFKbv7skJjlJEVE5pcWwHnACE2LC4C5tu5e77vImxUsxcuZ+aUFnGNxQaql\nBQ6JWYelOicyPj4OLSwuSLG0wCEx66BU787MLy2ghcUFKZYWOCRmHZLyLd75pQWtLi7Ie2mBQ2LW\nAamvE5lfWtDq4oK8lxY4JGZtKuJis/mlBbSwuCDF0gKHxKwNRV2xOr+0gBYWF6RYWuArW81WqejL\n3ueXFkxNfRuWWFyQammBj0jMVqHoiMALSwuq1SqVSh0YAz4FjFGp1KlWq8nuc5vkiETSZcBfA2uA\nmyLi0wsefxXwD8CWxkx/FRFfSDGb2UqVISLzhoaGOHr0aOFLC3K/RF7SGrJjrd8lOyt0H3BVRDza\ntM/1wKsi4oCk9WSnooci4uxSX9eXyFsRyhSRvJXtEvlLgcmImGqE4VayF3PNAnilsv8irwB+RLZ6\n3Kw0+ikiK5UiJBt58SnlU41tzQ4BbwSeBB4G9kdEHbOScETOrSwnW98DPAj8KvAW4JCkX1m4k6Q9\nkiYkTczMzKSe0fqUI7K8FCF5guyi/3mbGtuaXQPcFplJsqWLb1j4hSLiSEQMR8Tw+vXrcxvYbJ4j\n0poUIbkP2C7pAknnA1eSXY7X7CTwbgBJG8gWEEwlmM1sSY5I63J/+zci5iTtA+4me/v35oh4RNLe\nxuM3Ap8AbpH0MCDgQET8MO/ZzJbiiKxMkutIIuIu4K4F225s+vhJ4PdSzGK2HEdk5cpystWsFByR\n1XFIzBockdVzSMxwRNrlkFjfc0Ta55BYX3NEOsMhsb7liHSOQ2J9yRHpLIfE+o4j0nkOifUVRyQf\nDon1DUckPw6J9QVHJF8OifU8RyR/Don1NEckDYfEepYjko5DYj3JEUnLIbGe44ik55BYT3FEiuGQ\nWM9wRIrjkFhPcESK5ZBY13NEiueQWFdzRMrBIbGu5YiUh0NiXckRKReHxLqOI1I+Dol1FUeknBwS\n6xqOSHk5JNYVHJFyc0is9ByR8nNIrNQcke6wtugBzJZS5oicPHmSWq3GmTNnGBgYYPfu3WzevLno\nsQrjkFgplTUip0+fZt++fYyPj1Ov15/fvn//fkZHRzl06BBDQ0MFTliMJC9tJF0m6TFJk5KuW2Kf\nd0p6UNIjkr6RYi4rpzJHZOfOnYyNjVGvrwGqwPVAlXq9wtjYGDt37uSpp54qeNICRESuv4A1wPeA\nbcD5wHeAHQv2GQAeBbY0Pn/Ncl/3kksuCes99Xo9DtZOxOsO3BkHayeiXq8XPdLzqtVqAAEXB0wH\nRNOv6cZ2olqtFj1qRwAT0eLzPMURyaXAZERMRcRZ4FZg94J9PgTcFhEnASLi6QRzWclESY9EIDsn\nMj4+DpwH3AFsWrDHJuB2YC3j4+NMT0+nHrFQKUKyEWj+qZ5qbGt2ITAo6euS7pd09WJfSNIeSROS\nJmZmZnIa14pQ5ogA1Gq1xjmREV4akXmbgd3U63VqtVq64UqgLG//rgUuAd4LvAf4c0kXLtwpIo5E\nxHBEDK9fvz71jJaTskcE4MyZM42PXr/Mntkf29nZ2VznKZsU79o8QZbqeZsa25qdAp6JiJ8CP5V0\nD3AR8HiC+axA3RARgIGBgcZHjy2zZ/ZHdnBwMNd5yibFEcl9wHZJF0g6H7gSWHjcdwfwDklrJb0c\n+E3guwlmswJ1S0QARkZGqFQqZH90Ty2x1zRwB5VKhZGRkXTDlUDuIYmIOWAfcDdZHL4cEY9I2itp\nb2Of7wJfBR4CvgXcFBEn8p7NitNNEQHYsmULo6OjwC/I3itYeDJ1Gng/MMfo6GjfXZym7F2e7jM8\nPBwTExNFj2Gr0G0RmTd/HcnU1BTZWYHdZOdEHic7qJ5j27ZtHD9+nA0bNhQ5akdIuj8ihlvZtywn\nW61PdGtEAIaGhjh27BjVapVKpQ6MAZ8CxqhU6lSr1Z6JyEqt+mSrpAMR8ZlODmO9rZsjMm9oaIij\nR48yPT1NrVZjdnaWwcFBRkZG+u7lTLOWX9pI+nLzp8BbImJ7LlO1wC9tuksvRKTfrOSlzUqOSH4c\nEX/c9E1uWPFk1pd6OSIr/ffo1nOSy1n2HImkK5T9tD654KE/y2ck6yW9HBF7QStHJH9PtszxD+Y3\nSLomIr6Q21TWE/oqIgfbfLzLtfKuzX8B3wDGJJ3X2Pax/EayXtBXEbGWQhIRcSNwG1CT9DKyk61m\ni3JE+k8rL21mASLii5J+Bvwz8PJcp7Ku5Yj0p2VDEhHvbvr4qKTngFvyHMq6kyPSv1Z8QVpE3Ams\ny2EW62KOSH/zJfLWNkfEHBJriyNi4P8dhbXBEWlysOgBiuUjElsVR8Sa+YjEVswReUGvrp1ZKR+R\n2Io4IrYYh8Ra5ojYUhwSa4kjYufikNiyHBFbjkNi5+SIWCscEluSI2KtckhsUY6IrYRDYi/hiNhK\nOST2Io6IrYZDYs9zRGy1HBIDHBFrj0Nijoi1zSHpc46IdYJD0sccEeuUJCGRdJmkxyRNSrruHPu9\nVdKcpCtSzNXPHBHrpNxDImkNcBjYBewArpK0Y4n9PgP8S94z9TtHxDotxRHJpcBkRExFxFngVmD3\nIvt9DBgDnk4wU99yRCwPKUKyEZhu+vxUY9vzJG0ERoEbEszTtxwRy0tZTrZ+FjgQEfVz7SRpj6QJ\nSRMzMzOJRusNjojlKcU9W58ANjd9vqmxrdkwcGvjD/Y64HJJcxFxe/NOEXEEOAIwPDzsm2W2yBGx\nvKUIyX3AdkkXkAXkSuBDzTtExAXzH0u6BbhzYURsdRwRSyH3kETEnKR9wN3AGuDmiHhE0t7G4zfm\nPUO/ckQslST/O4qIuAu4a8G2RQMSEX+UYqZe54hYSmU52Wod5IhYag5Jj3FErAgOSQ9xRKwoDkmP\ncESsSA5JD3BErGgOSZdzRKwMHJIu5ohYWTgkXcoRsTJxSLqQI2Jl45B0GUfEysgh6SKOiJWVQ9Il\nHBErM4ekCzgiVnYOSck5ItYNHJISc0SsWzgkJeWIWDdxSErIEbFu45CUjCNi3cghKRFHxLqVQ1IS\njoh1M4ekBBwR63YOScEcEesFDkmBHBHrFQ5JQRwR6yUOSQEcEes1Dklijoj1IockIUfEepVDkogj\nYr3MIUnAEbFe55DkzBGxfuCQ5MgRsX6RJCSSLpP0mKRJSdct8viHJT0k6WFJxyVdlGKuPDki1k9y\nD4mkNcBhYBewA7hK0o4Fu30f+O2I+A3gE8CRvOfKkyNi/SbFEcmlwGRETEXEWeBWYHfzDhFxPCJm\nG5/eC2xKMFcuHBHrRylCshGYbvr8VGPbUj4CfGWxByTtkTQhaWJmZqaDI3aGI2L9qlQnWyW9iywk\nBxZ7PCKORMRwRAyvX78+7XDLcESsn61N8D2eADY3fb6pse1FJL0ZuAnYFRHPJJirYxwR63cpjkju\nA7ZLukDS+cCVQK15B0lbgNuAP4yIxxPM1DGOiFmCI5KImJO0D7gbWAPcHBGPSNrbePxG4OPAq4HP\nN56EcxExnPds7XJEzDKKiKJnWJXh4eGYmJgo7Ps7ItbrJN3f6l/opTrZ2i0cEbMXc0hWyBExeymH\nZAUcEbPFOSQtckTMluaQtMARMTs3h2QZjojZ8hySc3BEzFrjkCzBETFrnUOyCEfEbGUckgUcEbOV\nc0iaOCJmq+OQNDgiZqvnkOCImLWr70PiiJi1r69D4oiYdUbfhsQRMeucvgyJI2LWWX0XEkfErPP6\nKiSOiFk++iYkjohZfvoiJI6IWb56PiSOiFn+ejokjohZGj0bEkfELJ2eDIkjYpZWz4XEETFLr6dC\n4oiYFaNnQuKImBWnJ0LiiJgVq+tD4oiYFS9JSCRdJukxSZOSrlvkcUn6XOPxhyRd3MrXdUTMyiH3\nkEhaAxwGdgE7gKsk7Viw2y5ge+PXHuCGVr62I2JWDimOSC4FJiNiKiLOArcCuxfssxv4YmTuBQYk\nvfZcX/TJZ3/uiJiVRIqQbASmmz4/1di20n1e5JmfnHVEzEpibdEDrISkPWQvfQD+9+DIm04cLHCe\nZawDflj0EEso82xQ7vnKPBt0dr7XtbpjipA8AWxu+nxTY9tK9yEijgBHACRNRMRwZ0ftnDLPV+bZ\noNzzlXk2KG6+FC9t7gO2S7pA0vnAlUBtwT414OrGuzdvA56NiB8kmM3MOiD3I5KImJO0D7gbWAPc\nHBGPSNrbePxG4C7gcmAS+BlwTd5zmVnnJDlHEhF3kcWieduNTR8HcO0Kv+yRDoyWpzLPV+bZoNzz\nlXk2KGg+Zc9hM7PV6/pL5M2seKUPSV6X1yea7cONmR6WdFzSRalma2W+pv3eKmlO0hVlmk3SOyU9\nKOkRSd9INVsr80l6laR/kvSdxnzJzutJulnS05JOLPF4+udERJT2F9nJ2e8B24Dzge8AOxbscznw\nFUDA24D/LNFsbwcGGx/vSjVbq/M17fc1snNYV5RlNmAAeBTY0vj8NWX62QHXA59pfLwe+BFwfqL5\nfgu4GDixxOPJnxNlPyLJ5fL6VLNFxPGImG18ei/Z9TGptPKzA/gYMAY8XbLZPgTcFhEnASKibPMF\n8Epll1W/giwkcymGi4h7Gt9vKcmfE2UPSS6X13fISr/vR8j+lkhl2fkkbQRGaXGRZAe18rO7EBiU\n9HVJ90u6Otl0rc13CHgj8CTwMLA/IuppxltW8udEV10i360kvYssJO8oepYFPgsciIh6CdcrrQUu\nAd4NvAz4pqR7I+LxYsd63nuAB4HfAX4N+FdJ/x4RPy52rGKUPSQdu7w+By19X0lvBm4CdkXEMwnm\nmtfKfMPArY2IrAMulzQXEbeXYLZTwDMR8VPgp5LuAS4CUoSklfmuAT4d2UmJSUnfB94AfCvBfMtJ\n/5xIdQJrlSeV1gJTwAW8cNLr1xfs815efGLpWyWabQvZ1bpvL+PPbsH+t5DuZGsrP7s3Av/W2Pfl\nwAngTSWa7wbgYOPjDWRP1HUJ//tuZemTrcmfE6U+IokSX17f4mwfB14NfL7xt/5cJFpQ1eJ8hWhl\ntoj4rqSvAg8BdeCmiFj07c4i5gM+Adwi6WGyJ+yBiEiyKljSl4B3AusknQL+Ajivabbkzwlf2Wpm\nbSv7uzZm1gUcEjNrm0NiZm1zSMysbQ6JmbXNITGztjkkZtY2h8RyI+lrjfuJPCjpOUkfLHomy4cv\nSLPcSfpT4F3AVRHxf0XPY51X6kvkrfs1lv/vAqrAL0n6PHAW+HpE/GOhw1nH+KWN5UbS7wMfBj4Y\nEb8APgAcjYg/AUYKHc46yiGxXEh6H/BR4AMR8Vxj8yZeuOGOX+L0EL+0sbz8HdntAI81Vj7/Ddk9\nRjaR3RDIf4n1EJ9stWQk/TLZLQqfA/7D50h6h0NiZm3z4aWZtc0hMbO2OSRm1jaHxMza5pCYWdsc\nEjNrm0NiZm1zSMysbQ6JmbXt/wHsHYctye1hiQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1144ca828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(Z[y_neg, 0], Z[y_neg, 1], '.', c='b', markersize=20, mew=2, mec='k')\n",
    "plt.plot(Z[y_pos, 0], Z[y_pos, 1], '.', c='r', markersize=20, mew=2, mec='k')\n",
    "plt.plot(qz[0], qz[1], 's', c='g', markersize=10, mew=2, mec='k')\n",
    "plt.plot((0, 1.1), (0, 1.1))\n",
    "plt.xlabel(r\"$z_0$\")\n",
    "plt.ylabel(r\"$z_1$\")\n",
    "plt.xlim((0, 1.1))\n",
    "plt.ylim((0, 1.1))\n",
    "plt.axes().set_aspect('equal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prediction and the maximum-margin separating hyperplane\n",
    "---\n",
    "\n",
    "We have plotted the maximum-margin separating hyperplane also. This is the line which gives the greatest possible margin between support vectors in the implicit space. It's easy to see that in this case it should be a 45-degree line bisecting the two SVs. We see that the query point ends up on the \"blue\" side of it as it should. The hyperplane is indeed a hyperplane here, ie a line and not a curve, because the SVM is a linear classifier in the new implicit space. It may be non-linear when we view the decision function in the original space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can *see* the decision for the query point by seeing which side of the line it falls on. But how do we actually calculate the decision? We have a weight $w_i$ for each SV, and a bias $b$, given by the training. The *decision function* is: if $$b + \\sum_i w_i K(q, x_i) > 0$$ the output is +1, else -1 (where $x_i$ are the SVs).\n",
    "\n",
    "The line is determined by $w$ and $b$. In fact, the line is the set of points where the decision function is 0: $$b + \\sum_i w_i K(q, x_i) = 0$$ (where $x_i$ are the SVs). \n",
    "\n",
    "It's interesting to look at the left-hand side of the decision function, so we'll bring that out as its own function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decision_function(q, SVs):\n",
    "    wx = b + np.sum(w[i] * K(q, SVs[i]) for i in range(len(SVs)))\n",
    "    return wx\n",
    "\n",
    "def predict(q, SVs):\n",
    "    if decision_function(q, SVs) > 0: return +1\n",
    "    else                            : return -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the decision function for $q$ is negative:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-2.19707585])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decision_function(q, SVs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict(q, SVs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we might have noticed an interesting property of the weights $w$ when we printed them out. The weight $w_i$ was negative for an SV labelled -1, and positive for an SV labelled +1. That means that similarity to the first SV should give a *negative* contribution to the decision function, leading to a -1 outcome; similarity to the second SV should give a *positive* contribution, leading to a +1 outcome. (Things could be more complicated when some SVs are on the wrong side of the hyperplane because the data is not linearly separable in the implicit space, ie when we need a soft margin: but we'll ignore that.)\n",
    "\n",
    "We can rewrite the decision function as: if $$b + \\sum_{i: y_i=+1} w_i K(q, x_i) > \\sum_{i: y_i=-1} -w_i K(q, x_i)$$ (where $x_i$ are the SVs) then +1, else -1. Because of the negative/positive property of the weights mentioned above, the $w_i$ on the right-hand side are negative, so $-w_i$ on the right-hand side are positive. So (ignoring the bias): if the query point is more similar (in weighted average) to the SVs with label +1 than to the SVs with label -1, then the output is +1 (and vice versa).\n",
    "\n",
    "Now we can talk about what the SVM training procedure is trying to do. It's trying to maximise the margin of the separating hyperplane (an objective function), *subject to* putting each SV on the correct side of that hyperplane (a set of constraints). It uses quadratic programming to achieve this. The separating hyperplane (decision boundary) is the set of points where the two expressions above are *equal*: $$b + \\sum_{i: y_i=+1} w_i K(q, x_i) = \\sum_{i: y_i=-1} -w_i K(q, x_i)$$ $$b + \\sum_{i: y_i=+1} w_i z_i = \\sum_{i: y_i=-1} -w_i z_i$$ The SVM chooses the two closest-together opposite-labelled points as the SVs and gives them equal weights $w=(-6.76, 6.76)$. It also chose $b=0$. The equation of the hyperplane in the new space is then just $6.76z_1 = -(-6.76)z_0$, ie $z_1=z_0$, hence the 45-degree straight line. Of all possible lines, it is the one which gives the largest possible margin between each SV and the line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The effect of the decision boundary in the original space\n",
    "---\n",
    "\n",
    "Now, we can plot the decision function over the original space, and superimpose the training data and the query point. We'll see a decision boundary (white line), and as we move away from the decision boundary, the decision becomes more and more \"certain\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "g = 101\n",
    "grid = np.linspace(0, 1, g)\n",
    "im = np.zeros((g, g))\n",
    "for i in range(g):\n",
    "    for j in range(g):\n",
    "        im[i, j] = decision_function((grid[i], grid[j]), SVs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUkAAAEMCAYAAABJH0R3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXv0bUV15/uda/9+vwOHhwc9vASMkABREzVI+4ojIb4C\ntBHJNbbaN+k2D4Z2zOi+NxmtHftGO+mM1pEefROb+DjDEK897Bivr0NHIkE7XiRKhGaIggSCgHIA\nxQPK68D5/fZe8/6xataqmqtq7bV/+7nOmZ8x9th7rapVq/bjV79vzTlrFjEzDMMwjDTFsjtgGIax\nytggaRiG0YINkoZhGC3YIGkYhtGCDZKGYRgt2CBpGIbRwtwHSSK6jIjuJ6KbMuVERO8lotuJ6OtE\ndM68+2QYRv8gotOI6G+J6JtEdDMR/etEnZmPJ4tQkh8GcH5L+QUAznSPSwC8fwF9MgyjfwwB/DYz\nPxPACwH8JhE9U9WZ+Xgy90GSma8G8GBLlYsAfIQrrgWwi4hOnne/DMPoF8x8HzPf4F4/AuAWAKeo\najMfT1bBJnkKgLuD431ovnHDMAwPET0dwE8B+HtVNPPxZG2aixcNEV2CSkLjqKOOet5ZZ50NQm5Z\npTufXHaZKetQlxp1U22xO6XK9DMAcJksq68tE9e4OmX6Pv645OCSTN0ycxy9RX0/XSHovm5PtxG+\nDcTdZPncVJslmvCYa5rfBrK/FCPNfmzuZ+bjp2njNDqSn0h+g8n73QzgieDUHmbeo+sR0dEAPgng\n3zDzw9P0rwurMEjeA+C04PhUd66B+8D2AMA55zyPr/m7v0PhB6nqiyAZVMpRdD4uG1bPo2HyGpLy\n8Hp3jtw1jbqjrfqSYfWatzarE3LsnzfrugefiMrgynxdX15fU25V9xw9UZ0bbW5F54fufLlZvw9f\n19Upt6prho9vqmuDazZH7rmMyurzo7rulqvzuKuzNXJ9iNvYKuvvY9ONjs3ndDkAjDg+t+WORxxf\nMwpG4/ocjAn4IL797WnbOIgSr+s4230ff/sJZj63rQ4RraMaID/KzJ9KVOk8nnRlFQbJywG8lYg+\nBuAFAB5i5vvGX8YomQGqtF1BhTtbQWJIKMMrXJkUDqq3zzJYUlWZi/pj8YOgXFO4Z/kj9OcH9Y0K\nN2APqnMsA0MxStQduGZG7t6DqA25HwXX0EDaL1wTVZkMdIU7z4MiuMa1U7pzW4jqllvydqhxjXyI\nNKDoOUXhyoL/GTPFBrp+QQAGlP+9RIz5bomIAPwZgFuY+b9kqm1zPMkz90GSiP4CwHkAdhPRPgDv\nBLAOAMz8AQBXALgQwO0ADgB407z7ZBjGYiAAG0XHQXL8rPynAfwygG8Q0dfcud8F8DRgfuPJ3AdJ\nZn7DmHIG8Jvbahuo1CQwXlEC/ksQpehVYiHXqPMA2DUg6rM+LpPl1T1dXa8K3dRcq0QA8GozLhNl\nCa1GAYgMlPt4lSjPo6Y/zqtCrxjT1xaDYGrrptA5dVgESlXq1mXVNTxISPo5EE6vq+O53s7oCgEt\nE4+JYOZrELgFMnW2PZ7kWIXptmEYhygTTbdXFBskDcOYG9UguexeTEdvB0kGMCoZ5P9LtU+7gYQz\nxztd5GNw02wOpqvkQkxkih7frXEcnvO/DecJbzhwAGBtvXoeyZRcO3AGjWsKdw2P3NS8FOdOegoN\n1FPwQkwBbhoP5+yRa4HgPu7XLR9XUcRT6DKYQtfTa/eu1dRcnD0DDv9ibE586EOmJA3DMHIQVmPF\nyjT0dpCsnDbwcZIjremUogxKGqE+OsyHB0EIkLzQ6jMXEhSVObW2tlE9j5QDB6idOOtOHUp8pzhw\nRGmWtcITRaodNYVy8oSOFdbOnVI7bJw6DOZGNIpDfmrnTxwSVJ1DRK1M8w4bURh1M+OVpdSdxDFj\nTpzlQTSBd3tF6e0gaRhGP7Dp9rLgWCGMU5RAYKdUClLbG5EMASqTdbVtEgBYVv9QbCvE+kZ0TXWv\nOMCcXB1vx9S2SSAIKWoPAarVXCLUx9UpW+yYYp/MBZWHSBB6rq4PIxo2LvXUypLdM6JnwFRh3zDH\njWEYRgsWArRkmNkrRxEYA6UoOdRt2vOtFaVXfs2PxStGXVcFmVfXD+JrpKCMlysCqO2VpSjVUfK8\nt00Ctfp0dUhskOuyzLI6LgNVWJQ55TiI2gjVp1/e6NY+ew+2slWGdXUAOpfKQx78wYhiNA5tTEka\nhmFkMMfNEmFU9ilKWgUBEm9z9AXFdsrIyxy0ECQOaibD0HVTd9fpzUR1SnxjWFc83muxd1uSWEAv\ncQQasZPZJYbr9dfrYyoz3u3aQ14rSfF057zcsfdcqU0VL6ltlSHa9rgd5SFTOr08MWzP7JmLx2yS\nhmEYLZAFky+XKk6yYuQT3VbHA6cgR2XoAo/138DpP9I2yuAe2hZZx0VKvKRKmZZox7fBTbuf93gr\n2yOXadtkWKerbRKo7ZODjbhMr8CJVumopBfN4zCtmqhzpei9ym1eI6tvcnGSzTjKptqUd5hSkMZq\nYErSMAwjA5F5tw3DMLKYTXKJMFdTrDqdpPsmvH+A4/PBOZ9jQepmQoOA8Qkt2hJc1FP1dCB61Sc3\nzV5fBxLn6+NwWaKaijvnTyPxRRgCpKbgfurspt9l2bxmoK7RTpkimFqzcuboqfnIZ0IPrhkp84ea\nXm8nRMgC0FeLiZLurii9HSQNw+gHNt1eEpIqjUV9SBC5C14WtTgINN5IhQn5bfoyihLoniKtbVmi\nJMyo/RNNJ4zvZS4kSJYrArWqlGft/GkJARrrwAkUrVadg3W3l84oduQAAJWxY6ahOhuZygFxu4wL\n/Wlz3IhKtDCf1YQoXkDQR/qexcgwjJWGQINuj7EtEV1GRPcT0U2Z8vOI6CEi+pp7/N4s3kFvlSQg\nqiHWcqRCgcJgch147kVHRlFWl6fDgxqKMgzRySxL9PcLwlXIX5NRhTokCGiGBSm7ZiHJMYIQILE9\nFqO4LGebDMvqMJ7YNpmyL/rkulp1qlCgqK4KBWomuAiXMqafy0x5+NrChJYAxb+TKfkwgEsBfKSl\nzpeY+VWzuiHQ80HSMIzVhqAzS20fZr6aiJ4+k8YmoNeDJHO92YAksiCtIINg8toDnl7K2FCU1UWu\nuUzAudgXg0S9Wbulrzto1pW9tH1fAuUIxN5uea2Cymuvd6wogdBD7VSuU44D8bwr2yRQ2ycHrsyr\nW/fZFhv1+xDFWO/5HS9THGwUURvVOdeXkSjH2Xm5Q3J2TGMBFOS/+wXxYiL6OoB7APwOM988bYO9\nHiQNw1h9ohVm7ewmouuD4z3MvGeCW90A4GnM/CgRXQjgMwDOnOD6JL0dJJkZI+ZarbEsMXQnfExk\nYM9SHnBRlA37ZbSU0T1rz3fLUkbfR7nUn4iTZIRlzKqdNae4lLcbCDzeckJ7u3UcJYBiI1aDpcRW\nroutMi4HmrGTOm6Sg8/JK8Z1l5JtFKtObd+szsUe8do2mU+6m7NJai+3xUmuBjSZTXI/M5+73Xsx\n88PB6yuI6H1EtJuZ92+3TaDHg6RhGP2gi+d6JvchOgnA95iZiej5qCTOA9O229tBsoqTrJUjO/UR\n+rKrek3d5r3cKqaytkk2V+mIohSdtZbxdsd9UGWs+xjEVGqPuChJsTeG7TdskGlvt6hQIFiNs+5i\nE5Vy9LbEILbS11HPOm6y6iZHz4WPx5TNyWIbZXVOxVCORElqm2T93nNKMuflTtUxL/cCIZqZ44aI\n/gLAeaim5fsAvBPAOgAw8wcAvBbAW4hoCOBxAK9nnv7L7u0gaRjG6kNUm2CmhZnfMKb8UlQhQjPF\nBknDMOYIRcmZ+0ivB8lWx413ygTzrkKHCcm1cpxy3FTnZBo5UIv19bQbyGQrB3y4UGsyDF/Xtbum\nEl8AzWm1P59x4CCYVivni55Sl0EAetNhM0peAyRCfUZpB04ybGgjDjzfcN+ZTIvDqbNEk+QcNfp8\nqo6FAi0QWpxNcl70epA0DGO1IRsklwczsDViiJIXwViHxYhTJpAL5ZgwoYTjpizjAOekcwe1ogz7\n0Agq1wHjYR2tMsXJk7gmpRRdZ/Plqq4OCSqUSgRSDpuBuzYOG6quix03pB037nwYNqQDzMc5cMLX\nXR04bXXMgbMYbLptGIaRg7olr1hlejtIMhhbZYmSJci7Ou//aXl7Y6AKJUhZlQ3UMsUyVDu5pYwZ\nRQnkw4PawoUaweQDUZYb8XmgGWgubYhalNCfxFJGb7KVMKJMSFDVhVhJ1rZIpzo3gqWYpa5TxNcq\nRQk0k1+Ms00CtQrsaptMlbXVfQRDfBsHsIkSGyjwdOzE0f39M1k6lXd7ML7iCmPfvmEAOIARrsGD\nuAsHon9iX8YP8HTsxEvwZOxEv//Yl8UMswAthd4OkmKTLJ3Ck8SeJcdfSHiYtVeqZYpRPoUCyTpT\nKcog6W4+nVqs/ERpAkGguW6jxbvdCEqXZYkqrVrKJqmXLBZlU3WKGtTB46IcS5WUF6jj57zqLNOq\nM0z/r5XjOEUJ5NOoyfMBHmIvvouHMcQ6gFcDOBvArQD2ArgTB/AANnERTrKBclIOgen2QiyqRHQ+\nEd1KRLcT0dsT5U8iov9BRDcS0c1E9KZF9MswAOBqPIiHMcQ5AO4A8AkAf+ie7wRwDoCHMcQ1eHCJ\nvewnlXe76PRYVeauJIloAOBPAbwCwD4A1xHR5cz8zaDabwL4JjP/AhEdD+BWIvooM2/m2mUgaZMU\nZTly59eDhApZe2UhasfVS2we1llRhp1xNBVlnXgipzLDOgBARag+lZ2yVDZIHT+ZQl2jvd1A0+Od\ni62MzokaVEl3xfsd2iR9ijSlHP01TiaWB+trauUYx1DqzaZCO2ZOZQ4IeJiHuBMHsI5KNZ6KmFNR\npZI5A8BdOIBHMTQb5YT0fbq9iOH7+QBuZ+Y73KD3MQAXqToM4BiqEj4eDeBBAMMF9M04zLmDKxvk\nq9EcIIXTUP1gGcBdeHxRXTs0oGohQZfHqrKIf4mnALg7ON4H4AWqzqUALgdwL4BjAPwzZm7IISK6\nBMAlAHDCU3M/acPozkFnsTx7TL2z3PMmRq31jBgi8jOFvrIq84afB/A1AC8F8KMAriKiL4X54QDA\nJeDcAwBn/cRzuXLcVGWF87Y0pt9c/6jXuaoss0SJTGhMs4tg6izZhNSe3TKbYxXMHtFpCWM8dRb/\nDA03oxOB36ZRRipDeZJcoHmpHDjrLVPoTEhQdS4zNdfOmVE9eSnV0kUJAdJT88FaEAK0Vd0nN73O\nTb+rG7nnYP50FA8Arpw0bdzmnjfMcTMZNLvtG5bFInp/D6oZi3CqOxfyJgCf4orbUdnLf3wBfTMO\nc36s2AlCNY3Zl6lzNyp7JQF4Oo5cVNcOEQhUFJ0eq8oilOR1AM4kotNRDY6vB/BGVec7AF4G4EtE\ndCKq2c8dbY2WYBwclVj3ytGFAGWUpVwV1pX/EQ2Hzqi+hgu1LLFLCJCoTjnutIRRKcoitjZE6tMF\nmvuli2vNOlU/wgQXmUBzlXsytS+OVoepLOY6LChcslidbzpuysS56lhCgpp/OOu+btW+HOZCgcLX\nqZyTx9E6fox24h/5AC5C5aQJ/6PfDeA1qAzkp1tg+cRUmclXdwDswty/cWYeEtFbAVwJYADgMma+\nmYje7Mo/AOAPAHyYiL6B6m/9bdOmXDeMrryseAruH23iBgxxBionzVmopth7UQ2Qx2INL8GTl9nN\nnjK7pLvLYiH/Fpn5CgBXqHMfCF7fC+CVk7UJbI1Kv1+NpDArSgn9iZUlUNutJCxI7JU5W2VVSSlI\ndyxLHNvSq0XnwvMBjfAgb2d0x0Nf4K9pBJ67OpIMQy9bBIJA8wnQ1reyJVXaQClH1bVGAozwHGsb\nZCYkqCqLw4J0SFC9MV/zs5Y6Wm0eN1jHrxRPxd8MH8Bt/Bg+GVxDqBSkrbjZJkQ+PV9f6XfvDWNG\nHE1r+MX1E7F/tIVv8QEc4BF2oMCpfKRNsaeB6uxRfaW3334VTM6BaEsvT1wPbJJ1WRmVjZwdbd0b\nsgIPrASaS/ILsW/54HJE54NT0ylK9+wVZeOKIPBcfYudIs5ySxijpYxxoPlauaNLy1WftD1TJcCo\n6ugEvfG0TAeih8i5jaHYJvPB5FmPt+9KXfe4Yg3n4lhslrHq3NTfodEJOgSm2/3uvWEYqw0BRVF0\neoxtiugyIrqfiG7KlBMRvdctf/46EZ0zi7fQXyXJXCXdFS+22CSVshwFEm99UCTLtIJMxVZ6xi1P\nrBqILimpYUWs34e/dfVinKJMteLvLOnVvF0zTJWW8Wp3SNQ7k7RqCTtmLgmGbzMRW+mvVXbMZtxk\n45I8oTlV3WrTJ13eRrsGgJnGSX4Y1cKTj2TKLwBwpnu8AMD70Vy4MjG9HSQNw+gBM9xSlpmvJqKn\nt1S5CMBH3Day1xLRLiI6mZnvm+a+vR0kfYILp9JE0OWUJRB4s92XplVn6ezLoR1T7JU6YYZ4wCVJ\nBgerdGQ7CL/nt1KfZagFtd1yjKIEai92TtQkr9lI1ZwMMb9Pk1atbcsHveVZassHX1YqlekWIY2c\nnXaj5e9ykrRquh1tqzTaocm827uJ6PrgeI9bZdeV1BLoUwAcnoOkYRg9YLJlifuZ+dx5dmc72CBp\nGMZcWeCKmy5LoCemt4MkM3BwWPqwHpkeFW66OlBB5UA9BZcpmVyzruJ5YmdP2qlTchGVl6HjRjt3\nECfHSO6X2HHaHVwR7NEdO2xa7pKddjeWLQLZpYuDI+aTCWfLPeeCzENSgeYAsHEwPw+uA85bgqRU\nEoxNd5ybdlft5ps73CGiRa7LvhzAW4noY6gcNg9Na48EejxIGobRD2bluCGivwBwHirb5T4A74Qz\nZbsVfFcAuBDA7QAOoEqcMzW9HSSZGZuj0i9H1A4cWTYYBhlrdamV5aiQkKAgmDzr1Enk3VKIkKyD\ny9WliFYbujrtihJIO2aS54OUnD6zeSa9WlEeobuWRwWZV7cSx8wU2cwbOy3KB9b8mWpnj1ahEmQe\nUv8WEmFbjRu45zGKsiozZ06W2Xq33zCmnFHtcjBTejtIGoax+hARBus6bqFf9HaQLBnYHNZKcqDs\nTBsS5hP8d9fqsqEsWZ7ri0RVarUpbYjYCRWrDhPSGi9cwuiVo5IhjQD04P21hQe1nU/V0Voq/J/v\ntZgONN9xRKPuOCZJ1Nuk3slDli4OVIB5Y+ni4/U1uaWLs1aUtoQxTd+XJfZ2kDQMowdMFgK0kvR2\nkGRmPL45aihJeR7JcrJAgclrWWqolaU/n/Bc1l7yeGmj2CrD5L7lQK6Ps5+kAs/9NVrMDHRBMwC9\nq6JMlek6vo1UejWnHNsYt3RxrZwkmn3TXZtP1FszdHXiP8SUR3yjYQD2tYPXGVWZUZRArSpFQW74\n2YbZKiUzeZ/p7SBpGMbqQ6Ykl0fOJpl7Dl+Pilhl1goyXrYI1LbIUuIhJXGvzpWWtNClYyrjupk4\nSSlNJpHtvoRRX90oky0g3FJH2jHBHi5hMgynNifJHDhu6WKXH2dTMTplmdj6wWvZrbY4zzF2ysTX\nrbtQ2ybjNg9LRWlJdw3DMFogAhWWdHcpMDhSkkIXJbmxVkTHoiw3lCcbCFbUiOfb25vgro2P43MZ\nD3jwqZdKsbDyaqf9vWnP9ySK0q/SWSvT5W3o9GoJcskwojjJI9rtlHrrh+qW+US8QDoZRugdB5qb\niaXpqCgRiEq1GZngfyvhb+RwslfaIGkYhpGDAHPcGIZhZCDb42ZplCVwIBECtNbFcaPCg2T6nQob\n8k4XNYVel+WPyqFTnZMwIVfmfyPNdYkSeC7fRB0KNElI+PikGH567ffsLsfepTHRlGn2Rts01bU3\n52QYzft1mbfKtNt92D7gfIppN4Ips3LUbChTTBxkHn/yh+60m2y6bRiGkYOI6jwBPaW3gyTDJbhQ\nSwxdguqkktQqM6cg5XxYNnLndLiQKM0yuI926mjaAs/XG/90yb/fPO2Ksmq+XTl2UpQd9u6WMB7a\nEauzNi3BY5JhREsZZWfLsT1pI6cogbyqzCvKhgp0TbTti6OdOfXzIaYsCaYkDcMw8th0e2mwCyYX\nGqE/SmECtbLraptMnRutpcOFRsFGKdpe6e+fUJg66YIoU6zFHsFOKcxkb3CnRjjs02AjUXOypYzU\nwRZJo3a1Gf651CnSJt+AZ+jmDLXK3M5PWSlKALTp+p8NOHe/Gao/qc3MxyLftw4RCsvGPVev42+i\nbyrTliUahmHkIFOSS6NkxuObw2yqtEFRNM431CaN927rcw1F6RVfKslY7M2uk2KEddXWEfqfrmuf\nuU3jxTQyggHeQNZVUSbLxCPeIvxonN0y3Kv7iG76uBuSdncaRVkjb3EwlK0+OHpuS4cmv6tGMt4y\nrIOorF1Jxu1r+2UbOdU5SP1GNDNRrDZIGoZh5CECWdLd5VBytRGYVpC1BzuVKi1Wl14dcmxnSnm3\nG7ZJ/bwWxEl6jzer4+YSxvoaHVMZsx7965f+qWQY7nCtaNGHUyhKH2sp5xOKUlKmZTVioDTnY6ma\nRlECKVUJpLeDyJFTfqlz8hnklGXqXM5rnrpPJ8WYqzuLENcZe7eJ6HwAf4LKvP0hZn63Kj8PwF4A\nd7pTn2Lm35/mnr0dJA3D6AOzS3BBRAMAfwrgFQD2AbiOiC5n5m+qql9i5lfN5KaY1z9zwzAMoSi6\nPcbzfAC3M/MdzLwJ4GMALppr39FjJcnMyRAgHcAdB5OX0bn6ufqC/JQ6MIjLFHzoptU7xky7gXrq\nPfLtufMy1U1Nv1qm4tX54H35b02cOuzen76qxdkzZtqduKJ1KaOfesvui3Ko20g4drbzn/qehx7F\nVbd9Bz987HEcu2MDLz31BJx8zM6gxlbwevKfuewuSTIHdQHnqWm3VJHn2qkTf6dx2FDs1MlNv1N1\n9Pnc8UowWaq03UR0fXC8h5n3BMenALg7ON6Han9tzYuJ6OsA7gHwO8x88yRd1vR2kDQOT+5/5ADe\n8VfX4HO33Bmtl/99Al5xxil4x/N/AsfvHL/dhLEgiIDuyxL3M/O5U97xBgBPY+ZHiehCAJ8BcOY0\nDS5kkBxnbHV1zgPwx6hWnO1n5p9ta5MZOLg5Ak2QT1IHk2snz0iCmwPpr9VlNiQoVJIZp4526ISk\nEmWEbRzBCb0lSTE6ZYHMuGYyirLlikZ5WIfU+JRVlAnGKcrvPfwYXvOhz+DbDz6MdQAXAzgbwK0A\n9jJw5bfuwTe//0N8/H97KXYdGf5hdnfm1L+ntAOncMHmRRBBvlWm312tLKvnyZYnBvdU7eWuTTlp\nuoQJ1f2dwMvTEQLNMgvQPQBOC45Pdec8zPxw8PoKInofEe1m5v3bvencbZKBsfUCAM8E8AYieqaq\nswvA+wC8mpmfBeCX5t0vo3+8/f/9PL794MM4B8AdAD4B4A/d850AzgFw98OP4V1X37DEXhoR4t3u\n8hjPdQDOJKLTiWgDwOsBXB7djugkomq0J6LnoxrjHpjmLSxCSXpjKwAQkRhbQ4/UG1G56r8DAMx8\n/7hGmRmjUenDFNznAnZKQHQAtQWTZ553rIWqMFaXjWWJXZSkUpSjpvEwmSgD0IHn+hpnJ5X/1J2+\nzdkrylQd2ohtj0lFmQk8T73jux94CJ+98Xaso4rvOFWVn4pqXnUGgKvuuBff/+mhslECOUVJLXEy\nsvdP4epspSq5rCq5gPOmrTIMOK+OuwSTayWaU5hV3ebyyeUwu2ByZh4S0VsBXIlqVnoZM99MRG92\n5R8A8FoAbyGiIYDHAbyeeQI5nWARg2QXY+tZANaJ6IsAjgHwJ8z8kQX0zegJf33jbSiZcTGaA6Rw\nGqr/vp9kxhfuuhf/+0/+2OI6aGSZ5dptZr4CwBXq3AeC15cCuHRmN8TqOG7WADwPwMsAHAngK0R0\nLTPfFlYioksAXAIAO3adiFHgaSycAhNxIgqSwgQXqi4pBZlKcKHVpSjL0dogKg+vGeaUZEJ1yvo3\n7QH3fW75H6iTY/j0ETNQlECHgPPEP+iGolTlqcWbOcK6j2xWc4Ozx1xzltQflRgcsSPd7mB8lHSt\nLtX+OO45/J5EbY6cvXLDL2WsylMJMLQdMacoc+fazgMrFNtna7c7MdbYikpdPsDMjwF4jIiuBvAc\nANEg6cIB9gDAMaeevYoBD8aceJLzWN86pp78YI7dMXlmIWMeEFCsihbbHovovTe2ohocX4/KBhmy\nF8ClRLSGSlu9AMD/3dYoMzDcKgMFWY2ZpHYbpOBfqtSVZBGiMlmpwS5bPtTLFQfRMdD0hLcqSblG\necCFZExlm8e76mX1NI2iBLzMWVurFFkx/opsqynLmLdTjkmKUQB41QufjX/73/4alzNjH9JT7rtR\n/ZAKIpz/rNOxltmNUdKspQJTiox9shC16MpDO6aOqWx6wEVZ5uMk9VLDUH1uR0mOi5lsK59kKWMX\nmFZG126LufeemYcAxNh6C4CPi7E1MLjeAuBzAL4O4KuowoRumnffjP5w2u7j8OrnPwtbqOyOd6vy\nuwG8BtXk+PxnnI6nHnv0ortopCBUSqXLY0VZiA4eZ2x1x38E4I86twlGOSy9XUvHS8pxEW6VoNSm\nfC/aRsmBR3nUUI5pr3acXs2py7W4jrZVhq+P3Iiv8X1uianMIXloQ1XKa7I5mTtmqSPtp5NmhEy0\nOkfXUat1QrrGUv6Xt/wz3HjXvbjhew/iDFSD5Vmopth7UQ2QP/LkY/Ge174Ug/XxP+1R0YyF1DZh\n+W2MBl0iPWPI/Y4kkW9oQx9ni2xbcdPmCdfX5M7PWi3mocou2WNWd/g2DMVJxx2Lz//Bv8JrXvAT\nKInwSQD/CcAnAZRE+IXnnom/euvrcEIj9MdYKrNbu70U+m1RNQ47Ttp1DD7627+M79z3PVxxwz/g\nwYcew66dR+CVP/4jOOW4YzF6YnN8I8bCYCKwOW6WBKMKJneQCocpnFMmnCSRmlbLVNzNBEFuasvB\ndFjqjmicAyexw2IpAehuKs356XbzWjUND6rlHDZS54i2b7URyC6fm86mDsw64Dysp1+Hd+4ysX3a\nySfizf+9tL+AAAAgAElEQVT0RJSbTwDAtgZHSgTq68DyZijQWqOe/12pwPOR2icndAqJU0cHnncJ\nAap3VMyHi42bbi+UFbY3dqG/g6RhGD2AbJBcFswcGcIFHxLkjiOHjgo0LyhWlv58cI04eQZrEj7k\n1GbLskRRlc2g8qZzZpyS9OfX8hKgLCV8aAKZMFZR6n11AmagKMO60yjKaf78RgklqfdPbDgEB3G4\nT/g6FyYkIUKhshQVKwHoR4zi73AzCilrf67rhScoUwcLp+8hQL0dJA3D6Ak2SC4HHUwu+DAf+V4C\ns5CoQh9ULgrSKaZCBZdX7agAdEmkscZxecLOmEunJrZKoLZXCqkwodQxECS4SCTM6ExWUQJj9eA2\n9vVOsSxFmbJJCnVikdjWqe2Ore1nbJVArSpFbZZOUZL7/R4RSL7S2yBjW2R9jOg5pF1tLgDqfwhQ\nbwdJwzD6gXm3lwWjUzB5iIRiabWpFSYllKTYLwu1D7aog1BJio1wkAk8HyZUoZDygKeOgXY7ZdVW\n/bpU7Wp1snM9lYRgAj24jV0Yc3ebVVKMcaRskr4vKm6PBpW1UpJjhCnTZHmjDjwXe6PvW8KOOdqM\nVab8NsvgWlGXhXseZBVkoFRVneCdROXzh1Y6BrIL/R0kDcNYfWRZYo/p7SDJ4ChOUnuu5bgI7CFa\nddaK0f2n9ufrZgdObYidUhSYVpahbbRwaq2LJ9x3v8UDPu4ajSjZHdsQC0dENspJUlo45qgowx6V\nY9Jvtf1ZFi5JMT1xMFunbChJ9x22LGX03mzlAa/tjkFcr7JX8kjUZ9N+KeqS3bMoVFGWomTD6IbV\n2TTMQoAMwzDasUFySTBa4ySF5IobTqtNTtgk2SXZ1fbLgU8Y4exEgX2rFC+5u7l4wgtXJ7U2pM0D\nHp5vO6dX6cyO2SvKLq10UpT+RP49+97LFhxFfnWOfEdDVWck6rNllU7Om51KkuHVoChU7+12v6vQ\nJikrwZRCZa8w3flAHhZKKg4aNsjF2Sb7Hie57d4T0dtm2RHDMA5BJDP5bDYCAxGdT0S3EtHtRPT2\nRDkR0Xtd+deJ6Jxp30JnJUlEHw8PATwXwHum7YBhGIcys7NJBjuvvgLVbgbXEdHlzBxuKngBqn22\nz0SVvPv9aO6pNRGTTLcfZuZfDzr8/mluPC3MjHLUnG7rPW5Cwv1ugGZQecP5g5SDRu7vpkcquBwI\nnD3egeOuVQ6dsM64MKEw6FymSEeW8X/fxpR8xjsYMM9u2j1JK5R5DbQEnIfKRF4frGq3aZZcWJB3\n3CTKJfBchwnlHDrhOZky1yFBMg1POW5i545Mv8uR+40GfdJTcXnPpZqG62n5PJjhdLvLzqsXAfiI\n2yHxWiLaRUQnM/N9273pJL3/Q3X8ju3e1DCMw4jumcl3E9H1weMS1VJq59VTtlFnIsYqSSL6EwD/\nhpnvDM8z84PT3HgWjIbNcAyB2gz5uZ0VVVA5kA/98Tst+uDy2tAu6lAvWfSB6IHsEQVaZhSlf6/R\nssf0e8uFBAFo7Moo72eypBjyQitKoOsSxmmTYsjUbZoljL5LqeaVUhRnz9ClYks6booiKpMwoZxD\nBwjSqW3GyxO1wgzrsK/jfkejMqobqkT29yqiMmpRjpyYmU0LE4G7L0vcz8znzrwTU9JFST4C4HIi\nOgoAiOjniejv5tstwzAOCZgxKrs9OtBl59UudSZirJJk5n9PRG8E8EUi2gTwKICGV2nRMJcoh+MT\nraYUJatzUkcrzKrMKUYVVE6cVonVuTgExNeR4PLgB5GzV3KrkkyX5RRmWGdceFCnH2tDUQKd9eCU\nSTF0nVkoyrXg9zA6GAeYj1RQedImKaFFKkxIz3OKcO8kpRx9ELmEC23l7Zc+qNyFDdUqMXzXRVRX\nwogk8lx+o9H7GM0nEcUMrZ5ddl69HMBbnb3yBQAemsYeCXSbbr8MwG8AeAzAyQB+lZnHbX9sGIYB\nRr353NRtMQ+JSHZeHQC4THZedeUfQLXh4IUAbgdwAMCbpr1vF+/2OwD8X8x8DRH9JIC/JKL/k5n/\n57Q3nwpmlFvbU5K+TP7zuzr1ssX6Gnmt7Yy5oHKg6RGXhL1idww94Tl7JbcsS2wqycx2DkG9aQLM\ns3bL6Nez2CWMDUWpvucuSTHElshuCwigaads2Ci72CRVndIHjNfaMmev9IHhgf3Se7W3xJudDioP\noz3kdyUqU3pU2x1jpQkAA6cytQd8WniGAevjdl51Xu3fnNkN0W26/dLg9TeI6AJUG9S9eJYdMQzj\n0GOWSnJZTLwskZnvc1Pw5cLcySYZotUGlWnbpCjM8JzYMbX9Uv5LUuDB0+qSyyI6H22RkLVXSh+c\n3alFSeZSr3U0hs+GrOd7wYryCOlG/R0W6jtk9/3ywbzfUq7WyrEcNBV52RJDCTSVJgCMNofROR9b\nqZYnAkFCXuXV9jOTlJIcpct4ENsdU6px1osIez5Gbm/tNjM/PuuOGIZxCMIL/mc9B3qb4IJnoSRz\nx8PmOa0yJVVXsVapnyi2UiW98CtvEp5wsVfm0N7vEPlPlfsR6q0hFsIcFWVbKw1FOe1qI/l+Dz4R\nndbJeIFaQTYT9boZhfJ+h2Xe3qhiK6M4yUxCXol55KK5OZlWkKQ811Iudsiw7ixhTBZpsIr0dpA0\nDKMfLHpbnVljg6RhGHOl57PtHg+SXGI05XRbKNSUOnWNfy7jY3aR52GmbJmC++m1mnYXUYILV+a+\niUKFCaWm6P7aMf+il2oLmsO0O9mKWvKWustEU2/5HjfjaXadk7L5m6uXI6YDz5OOG583cisqawsX\n0skvdOKLcIquy7STp64XLJVMhAVNC/NsQ4CWQX8HScMweoHZJJcEo1JxPBqNrStQInwDyC9TBJoq\nc5yyBGp1yU5Rlq7OIKUGM04dvTwx/KKG/tp0W8JKeBWXlBQjdZesogwVYG5nP5VuLbWUMRd4ngoR\najh1nHJMqc46wHwYHeeWJwL1EkV2Dhuf2EKWJY6azp76fjNUkliR3+EU9HaQNAyjH/R7iOzzIKmW\nJXKZVpQphddWB4gVZz4ZRhwCFC1ldMpRVG6xLjZKd36t/tiLTAByW6qGVKKMqt3plOTD378Pd17/\nRRx89BHsOPoY/NTPvhK7Tjh5ojayuLdMUQLWyZNirK3tANDB0hnex29QVMR9UOqwFW2vDn4jOvDc\n76Xj0qulQoRy9kofEhS0X27FdksJRM/teQMEweNqeSIrhRm+q1KpzVnRcyHZ40HSmBmP/WA/rvrz\nd+Mfv/IFcDCHv+qD/wnPeskr8Lrf/g849inHL7GHRp/pud+mx4Nkx1RpOfUY0rZcMacyRWGWrv0i\n4d3WHnBRlCFir9Q2yEItaeTgm/KB5aq5enuFJrklUgd+uB+ffeev4pH796GSEBcDOBvAreByL266\n+krc+4/fxJv/61/ihJNOzLavKTolWt2G59sxyRLGXFmnxGBebT6hzjc/69p+Hf8uU97vnL2y3IqX\nK4ZltR1RJ92VBBiBR3yktodwAedYjyMtQjtmMwnG9DAYZc8n3LNepplk3A5nQb1/QkRDInrtIvpl\nANf++XvcAHkOgDsAfALVTh2fAHAngHPw4H134/L3/ocl9tLoLQyMym6PVWXuSrLjDmdS7z0A/qZL\nu5V3u+ykFMf2sSjVcbjncVpl6mWKoe1S1OUgYa8EEHnka3WpYisTadVqXFJXSeufrBNTJgxDBx74\nLu667m9RKci9qJI4h5wK4DMAzsDN11yF+/btm52NMiKtKOukIYn3t42kGNtRlDqNniflBZfv2ZU1\nbJVh0gr3u9G2Sa0aw+tKlRTDx0Aqm2VU5p6LMladbcsSdRKMaWD0f7q9CCXpdzhj5k0AssOZ5rdQ\npWC7fwF9MgB898YvuTiiV6M5QAqnAbgIXJa45ctfWFznjEOG0k25xz1WlUUMkmN3LyOiU1AZw1q3\nqSWiS2QnNd48MPOOHm5sHXjUvTp7TM2zAACPP/LwXPtjHJpUq27GP1aVVXHc/DGAtzFzSS0Gf2be\nA2APAAyOOYlTjptygul3oRwrQpyZPD0Vbwsm91NwNe32Dpy1eoroA89VAHrbV9O2VDEsD5HWNsOP\nbOMo92Lcbhy3VW0ceTQe35zMvDFI7H+uaTp55DjOol2h3ts2Mgd1mZLrrOWNgKzUwgQ/3Y7LBkVz\nOtzcYTE+LqNlifGUXKbOuel39XoU1dVOnjpEKJjWz2GPG8aEO3KuIItQkl12LzsXwMeI6C4ArwXw\nPiJ6zQL6dlhz/LN/2sUOXo5K4Ke4G8BeEBU460UvzdQxjAwLctwQ0ZOJ6Coi+kf3fFym3l1E9A0i\n+hoRXd+l7UUoybE7nDHz6fKaiD4M4K+Y+TOtrTKjnHBZomakrvVB5IGyLDLB5KIwtbIE0uoyPA6V\na6gqQ7TqbEsS4Jcp6jqJCKlQUQ6OPh4nPOdncP/XvojKTPwZxP/P7gbwGgBDnPGCl2PjSSdMrCQn\nYed6Y4cZ99zcBbCrogxbmUhR5vb3lu9QJcCoCtNKUlRnEc1Q4n2865Cg2KEDAIWoPxVMnlOWYV2v\nIAvZu1vySzZHJa0yZwGDsZXYmXEOvB3AF5j53S6C5u0A3pap+3PMvL9rw3MfJDvucGYsiWe87v/A\nI/tuw+P7bwBwBqrB8ixUU+y9AIY49sRT8TO//rvL7KbRUxY43b4IwHnu9f8D4IvID5ITsRCb5Lgd\nztT5f9mxzeRuidOEBOk9b4DEskSlCrSyrF7H6lIHk+s2wzopu2VFUyE1U6/F++JEqI9KvvjiyF14\n7m9dim99+r24/8arAf5kXYkKnPa8l+Ilv/bvQEftwoEpVGSX4PKB1GnYWutrC2J1bnaKEgBYliwO\nKSrzdVp23/RhQbpO6ryECSl7ZXo/nHQdsSdqZQm02CIb5+vvVFTmTG1wPNFUereaAu9xfogunBjs\nr/1dALmVDwzg80Q0AvDBLu2viuPGWCI7jn0Knvsb/xFP/OB7+ME3v4ytxx7B+lHH4PRzz8NRTzkR\nR27Yz8TYHhMqyf3MfG6ukIg+D+CkRNE7onsyMxHlbvoSZr6HiE4AcBUR/QMzX93WqR7/+rlKlTZG\nObaVN4K8E3V1ajSv+FQweWxvStstc/fNncuTtmPWFqnJFeXmJlActRsn//TFdZ82BlOpx5AuSrJR\nJ+G9bzbTriiBDgHnYTKMUVpH1V7vWSlJ97oxM4ltleHrUinGUqVXCz3VPglGIQHisS1SzpeD5vud\n5bJEABjNaLrNzC/PlRHR94joZLeb68nIxFsz8z3u+X4i+jSqOO7WQXIhyxINwzg8qZRkt8eUXA7g\nX7jX/wKVQT2CiI4iomPkNYBXArhpXMM9VpLopCTHXd+1Ts5T7dOiJeIkc9tCpNK3pZJfhOVhX+tz\n7pqNOLdVU1ECDVXZoihnicRJHhyOqYiWmMpAUebDLqdYwhgoybGpNsTrnfq6cgl7JWoiZYtW9/WX\nRL+nTJyk7Nzp7IpRbKVSl+UobaMsAvXJSmXOAmbG1mIWZr8bwMeJ6NcAfBvA6wCAiJ4K4EPMfCEq\nO+WnXSz2GoD/zsyfG9dwrwdJwzBWnxkmOs/CzA8AeFni/L0ALnSv7wDwnEnb7u8gyTyT5Badb6fu\npRVmMumuslv68oQnPHe/QSaOMsTHSbpkvmvqfEXGTplRlAAwdB7eXJq1NkQV+rjKDptxdbFbxkl7\nkzXccwdFKW0l7JBZT3g5bJz3qjIXF6sSX0ToMrkmiNzQ9kptiyR3XISJepW6JGWb1MoSQGNrh1lw\nKKy46e8gaRjG6sO2x41hGEYWU5JG0rGS21FRSIULaSYLCUqT2mGx67Q7xaabQg/8c92GP+em6INc\n6FBi2i3ROgc75FfwU/Kxv9x8igvZE2atWBtTMx8ulNyNUZsCWhw2nsx0O7pGErmoAPRyGAeZRzko\n1RRcpt/iyCkS023t5JkFDGDLlKRhGEYG5mTC5z5hg+QcyDmUZBllGC40cKE/pQqRmYWSDGk6c9KK\nkoazT5eVY7CjutcTY5RLGPZz0HfX9b/DL1jUp6zBGMofbRErSmCbyTCkbKRUZi4kKCxTQeVJJSl1\nnXJsJMxwvzdRlkBTXYqy9MsTVcKLsKyY4W6JjMV4t+eJDZKGYcwVs0kaWbS9MhUuJLSFCc0CUlHY\nWlES5e8niZClT5sTqM1Bw45ZX/t4JvxJEAXYbqtMK8rw7W413prShUFlbafMKspUKBLF04FGurWE\nOsyGCaWUpNQVW6QKFwpt3ezsmPXe3LGy1AkwwtflFOkHNZWStEHSMAwjDac3oesTNkgukJStUrag\naPOAzwKvcdQSxtKptdSqQa0+2xh4+54oxkqVjFOLybZcn4oJvKyyjK9wum0rmwQmJKEXlZ0ya4sc\nNT8xX1cZgMkdRzMIUYHq2iQ5e6X3eqtjoFaqohiVskwtQayTYMwupYN5tw3DMFpgsE23jclJJauQ\nJLs+kbAkvEhsdjYNo8bSuVjDlMHxUC0plLpiegs3beu6dHEtaD9nr5wkrdrAKcYnxKbqf9Etm4c1\nSEQ9jlGUrbGVzhbpz8sGW4lEGp6GWgz20BYbpJzQdcSrHXrT5Zz81pSyTHnE/f1mmZDCVtwYhmHk\nYdggacwIb5vUtsgZK8q6/XTGiThO0iV3HTqvZ4v9UsitypHjAzS5jTJUlrJ9w8Fxaif6ZWc2D0uS\n9ny32SgbylG3JF7u4Dtk9/n7VTqNFTfh9rPKAy7tKPXJYZykiqHUylJiOiPbt08oPUPvtilJwzCM\nPAzG5nAh+STnhg2ShmHMD1OSxqxhb1BX0+swc/kUU+9mILtM+/LOEh03LT+a0MlTulASyUE5btoN\nBMkwMg6cSbKaC9qhU52rXlPpHEPyvlqn392m3UENH+pTJ8OId1zk4K/N56VUSxlT34JvX03F66Dy\nRAC6hPzoPeAz0++oL4ldQ7eL2SQNwzBaMJukMTeyihLYljNHlESyPQBlsSNxjYT8iL4pM+dDJTr7\nbPGSCAOol9MN3G22XJFfwtjm0Mnu5x2oTinxb7ldUaZa8YpSK+DEUsb62tiRkwo81983JRw2HnHq\nyBLDccoSAFR40KxYxCBJRL8E4F0AngHg+cx8fabe+QD+BNV61g8x87vHtW27JRqGMTcYjFHZ7TEl\nNwH4RbRsD0tEAwB/CuACAM8E8AYieua4hk1JrjhdFCX5RBpFdE0YtD7KKEhJrDEaiuypfxKlU4ql\nl1VO5VCsKGPiPcc3ld0xDiYvo7KcjXI7SxtbySpKoBHII2ZAdnVUmrWquUzAuSjKMh8C5NWlROin\nAs8zYUKS2ELbKoGEvVKUpSxW8L+VcG+mUVQ2C5iBgwvwbjPzLUC7bR3VHtu3uw3BQEQfA3ARgG+2\nXWSDpGEYc2PFbJKnALg7ON4H4AXjLrJBsiekljKWW9MHmLftCT4apRWjeLXLhELILV3U3u7qdezx\nlsQWaxkvd/hae7EL94coSTEGQZflcskvWzi1JqKjCP6GSStGTZEozwWcq1RpkXd7FO+62GXZY8MD\nnrFVAk17ZZ2uzynILac0I5tkU13OggnWbu8motCWuIeZ98gBEX0ewEmJ697BzHun6GIrNkgahjE3\nxCbZkf3MfG62LeaXT9mdewCcFhyf6s61YoNkDxH7pCTF4FG8nExvBRHiFUbG1hkry3QMpXi1U3ln\nxX7p1WYmfhJIbyg2jkYspeRvkPRqTlkOgj/Mwu8foJVvyxYQoljlUr0HQaiuM9tBNDzXweUNBSmq\nM7HE0au/UVxH7wEefncNe6V8z6IS/fnaMy6/I5qhd3vFptvXATiTiE5HNTi+HsAbx11k3m3DMObK\nIrzbRHQxEe0D8CIAnyWiK935pxLRFQDAzEMAbwVwJYBbAHycmW8e17YpyR6TU4OyuVhoWxpnx/Rx\nlEG5P+dseIVTSt5WGcVJcrJsO1s/5LzdAPC43qo2nacjg9uSVeyYzks/CLy5RS7NrthavRht2iR1\nnt+coozPtR8DADvFmKtD667/oyBespGYV6VVK2NPeXVuFD/PAGbG5nD2sbOJ+3wawKcT5+8FcGFw\nfAWAKyZp2wbJGbL1tT+fqP76c980p54YxmpgyxINwzBaYA628O0pCxkkxy0FIqJ/DuBtqGYQjwB4\nCzPfuIi+zYV3TVneEb1njp9Sl00DfC7Q3NdN5LNsOnVUKFDg0KmTXailiy3ZzKX13P44etodndPP\n4rhRyxaB5tLFgQ8XSjh0VGZzMSMUjarBUsnM1Nw3mUyKES9DzE27k+fEYyaB5/Idhte4z7B26ozi\n4zK1LNG1awkuIuY+SAZLgV6BKnjzOiK6nJnDKPc7AfwsM/+AiC4AsAcdgjwNw1hxVsu7vS0WoSTH\nLgVi5i8H9a9FFb9kdKShDicIMh8XEhTWGQ1lj+7YgQOEYUHpbOY6JAioHUGSlDWXVm0QqM9coHku\nyBxoDzR3Nfyr2qkTp1erE1y49xv+3es6vtm0ogRaHDdeJQZxXPL+c6E/+tqgTsOpoxw6PAqvSajL\nKbGku92YdCnQrwH467n2yDCMhbBicZLbYqUcN0T0c6gGyZdkyi8BcAkAYP2oxXWsJzRslKPAJik2\nqVIpx618MLmukwsJAlKhP+0hQVV7sf1ynKIEuifqDW2S9R45aUUTCj69dLHGvWd/HIYAuafcksZE\nIpAuuzDqO41TnxSsIvA94DKqK595IyQIaOx/MyvYBsmxdFoKRETPBvAhABcw8wOphtw6zj0AUOzc\n3e9P3jAOA5iB0gbJsYxdCkRETwPwKQC/zMy3LaBPhyQ5bzfQ9HjnvN1hSrWcvdLbKEObpEp6ITbI\nwrmFRU3Edkz3LDZOCVZ3dWtlWV8zLq1ayo4pHm+RfHrpYhEsOaxtmfLZiR3WteC93IEiztkiXfuk\nzwfMRFFy7MmOrnf91HWhkm8AYULeWQZ/M7h7gouVZO6DJDMPiUiWAg0AXMbMNxPRm135BwD8HoCn\nAHif+4MZti10NwyjP9h0uwOppUBucJTXvw7g1xfRl4XwruXePpV0VyfB0LZJXQ40laPU8Qk2gi0f\nRsPYvlgq5ajjJgGgdApu3CZi4bK2rmnVQjumtknmtn4A4sQYVVlz6WJVUL/U6dV8USqtmi5ztO7r\nrS71qpDi2UCkClUy3zpJhnvzylZZ1ZHtG2abdHc0tEHSMAwji023DY+txTYMhTlujFUmmc28owMn\nrFuosrqt+sdPykEjDpzCB6I3s5znpuZt++J4p44Eho9x4MSvVRYgFWRelSGqK06dgUrxUwQTVaLY\npVI7d+I8k3EAenOvnOq9tuzr7W8o31k6O5ArjMv09Fo5dKr+xU6d2cBmkzQMw8jBMMeNscKklGRX\nB05YZ5wDB6idOFox1seifuo/mHHhQjq4vHodB5iLsjzYCERvKsnBevW85ZSjzmYOhI4bqSPn/Ttt\nXOOXLoqE9BFHKhQoEo3KqdNlCaM0oBw1ybAhTof+1Ekymo4bZqcgucTMYKA0m6RhGEae1IZxfcIG\nycOEbIq0jG0yVTdnm6zqOLWk1CCTHFf1RsEfTMMmOZQ2lBoN1GetGFUSjEwgeqpOQZUi3mpNq+bq\nNGySTmEGyxbrCCNRxO69cqzxKBwrCqX/MooSCFSltjPGLcQ2yVLbJFXweCoAXb6kstmH7cLMvXfc\n2B43hmHMFWbu9JgGIvolIrqZiEoiyi5EIaK7iOgbRPQ1tX1tFlOShynjbJPpOup5lLB5itdUvNxO\nMdVKM+hDwxMeXyuqM7XDYnPpYnq5YrKOD0ivOrPVklbNe8TlvJONW4Es9IHmPtGFK1C2ymR6tYyi\nLIJ8bqV4zcWr3WUpI6mku5JeLaNKgdoDHkve6ZmlibOFmwD8IoAPdqj7c8y8v2vDNkgahjE3FpXg\ngplvAdDY/ngW2CB5mKBVYLy/9vbaSNsv4+QXVBC2Hvk+fnjXdRgdfAzrO4/BsWe+EBvHHp9Vm7Vt\n0h0Hf2Q5VThyak7K17qoT/FuF3X7YoP0STBcWe0Rl5rNRL11iVaQTduqv1wpSlGqybrUTVECQdq0\nwVpUVtsdEbURlc3au71ajhsG8HkiGgH4oMss1ooNksbcGB74Ab53zZ/h0Tv+PvrDu/cLe/CkM1+E\np134r7B+9JOX2ENj/vAkIUC7lZ1wTziIEdHnAZyUuO4dzLy34z1ewsz3ENEJAK4ion9g5qvbLrBB\n8jCjq5e7W92UTbJ63nr0h7h377/H8OHvAVgHcDGAswHcCvBePHTb3+HW+7+FH/uV/4wjd+121zpP\nuPaQB6pKFJZ+1gpz2EF9bqp4SaBWhUNXNt7bXXvJpY6PoXQqUa/IqcrEm+2etNos8nXHKUogoRzl\nvLZFMgVlzeS90zJhMPn+tuxfzPzyqfvDfI97vp+IPo1qe5nWQdK828ZceOCaP3MD5DkA7gDwCQB/\n6J7vBHAONn/4Xdxz5fuX2Etj7nA1SHZ5zBsiOoqIjpHXAF6JyuHTig2SxswZProfj931VVQKci+a\n+7qdCuAzANbw0G1fweZD3190F40FUpbc6TENRHQxEe0D8CIAnyWiK935pxKRpGk8EcA1RHQjgK8C\n+Cwzf25c2zbdNgCE0+XmHjepXJON64M6B+66ztkgL0Z+48vTAFwE8Cfxw1u/guPP/YXakaOeUxnQ\nc8sRtQMHqJ04ual5aDOrHTQScB6HBPmwoeCPuhkW5Kat7q9r1Aj3qa+RWzem2ZHjRgecy4eRnnYD\ngTPH56xwZRIKNJDj4LPVYUIzYhGp0pj50wA+nTh/L4AL3es7ADxn0rZtkDRmTrn5mHt19piaZwEA\nho8/Otf+GMuDmaNVVn3EBsnDlEZA+AQhQW0hQABQbMhOlreOaanazqjYcVQ15XKKg7zDRtqulUgu\n+UVOJQK1Eyd7TaBUG0sWVcC5ToBRXeNfuWvEqRSdrndTjMo6LE/MqcyMogQS4UGZZYlAuMOiu2bG\n0VrkXk4AAAozSURBVN9676W+YTZJY+Yc+bRz3NKay1Fts57ibgB7ASrwpLNeuLjOGYuFGVyOOj1W\nFVOSRkSsDtMJeX3Ci8z1g53HYeePnIsDd30VwEWonDThrsJ3A3gNgCGOPfPFWD9md7V+VyfFYLFN\n1ld6O2X2uXTPde90gHmb6swpyKIRVF4rbx0W1AwJgrtv8BH4tGouTEgCz6mZoFcilEib9jKKsmov\nnhloe6MP8wntj7KUcTTLECBe6QGwCzZIGnPhyS/6VWw+cBeGj9wA4AxUg+VZqKbYewEMsf6kk/DU\nl79lmd005g3Ha/z7iA2Shxm5ZYmiDgfbsE2Gr+UPYrBzF0561e/jB9d+uAoH4k/WF1KBY370RXjq\ny9+CwZG7GrbHRlIMCgOe4wBzUYVidxyo8wAwKuI6Odtk+Hrk7YjK005Nj7hc7j3hY7zdQL1UUQLN\nR5yxNwLZZBj1VgzNawZSRwecZxJdhGWylHE2mJI0jCyDnbtw4it/B8NH9+PAd/4XyoOPYv2oY3D0\nGc/H+jHHYzAwk/ghD3OUwb6P2CBpzIUw4UWx8ynY9ZMXAAAGa/U2DqyMbF45quWIHMQXerVJsWLM\nPXepk0rQ29U2CQS2R9eeKMiBy74rqjO2ScqbjhP1iqAMQwsZcn0mvZocRR+nO8gtYVSJLpJlM8Bs\nkoZhGG1w/0OAbJA0OpNLwqtfJ69VtkQAgDN/phLyNuoiriumU60Kh/64VkOiwJp1OqjOMbZJoFaV\nojpLlg3H4pjK0CMuHm+p4bd6SKzOEcU4gHj9ZZMv1xcdYxneIKcopeXI9hn3aTawt3f3FRskDcOY\nH2zTbcMwjCzMjHLLHDdGj5gmI3mXa1NLFnOIQ0P7uP00W03HgTocKBcK1DqF5gnqioNIBbRrB05V\nx5X5ECB5rs4XZWJfHOepqQPF45AgCpYwElQolFqm6KfhgQlAJ8ygMdNuIJh6z3iptSlJwzCMHDbd\nNox29C6KSIjROgN5XEcnvAjLtNocKSdPrA7jpYqjTPhQ6tymWoaoHThVe06d+RAgF9iuUrGFXayd\nOo7MnjcVaQVJsTiMbiD3FsU90jvhpJJiyN1mGr5qg6RhGEYWBsDljOfvC8YGycMUbV8stmGr3A6R\nfVGpQV0nmUDDR1lTVDcXCgQAG+55pO6nlzKm2vF9UrbJUdC5QinFUtk+tW2yOsfROR0SFAeTu2cV\naN4MMq+v0ckwBjpcCFqGpu2UU2PTbcMwjBaYMbJliUuCCFQMev9fqg9sxyM+CblNoHTqtJA6AD1v\ni9Tncl7u1ro+qNwpP04kuBB1qWyTPlFH6hqlOksl7CLvtvJ4U209BNAMMgeAkaojJUVOUQY3L2Zo\nlGT0PwvQQjIMENH5RHQrEd1ORG9PlBMRvdeVf52IzllEvwzDmDMLSrpLRH9ERP/gxo9PE9GuTL3W\nsSjF3AdJIhoA+FMAFwB4JoA3ENEzVbULAJzpHpcA6LTPKBUD/zDaCT+r8POiwQA0GCTrNdpQdWdN\nl13zcnX0tqSjkhuPtrKuj5Krxyh6ACOGL5PHVlk9/PXhNWWV8EIfM49/NO6HKrSx5PpR1+dqnxl/\njZRX5xnwj7o9QqmSZ2yfhWUmvwrATzDzs1ElLf13ukLHsajBIpTk8wHczsx3MPMmgI+hysAachGA\nj3DFtQB2EdHJC+ibYRhzZhGDJDP/DTNLSvVrkd6ms8tY1GARNslTUOXrF/YBeEGHOqcAuC+sRESX\noFKaAHDw4PUfHLux+IqwG8D+ZXdiAvrU3z71FehXf8dtdzkWPrD/ys0bPrS7Y/UjiOj64HgPM+/Z\nxm1/FcBfJs53GYsa9Mpx4z6wPQBARNcz87lL7lIn+tRXoF/97VNfgX71Vw1Y24KZz59FXwCAiD4P\n4KRE0TuYea+r8w5UW0B+dFb3XcQgeQ/iXaBOdecmrWMYxmEMM7+8rZyI/iWAVwF4GdcBtSHbGmcW\nYZO8DsCZRHQ6EW0AeD2qvUZDLgfwK87L/UIADzHzfbohwzCMFER0PoB/C+DVzHwgU63LWNRg7kqS\nmYdE9FYAV6Ja63AZM99MRG925R8AcAWACwHcDuAAgDd1aHo7topl0ae+Av3qb5/6CvSrv33q66UA\ndgC4ymVLupaZ30xETwXwIWa+MDcWjWuY0qrUMAzDABYUTG4YhtFXbJA0DMNoYeUHyT4taezQ13/u\n+vgNIvoyET1nGf0M+tNpiRYR/RMiGhLRaxfZP9WHsX0lovOI6GtEdDMR/X+L7qPqy7jfwpOI6H8Q\n0Y2uv13s8HOBiC4jovuJKBl3vEp/Y0vBL09awQcq4+q3AJyBKuPVjQCeqepcCOCvUa3hfyGAv1/h\nvr4YwHHu9QXL6mvX/gb1/icq59prV7WvAHYB+CaAp7njE1b5swXwuwDe414fD+BBABtL6u/PADgH\nwE2Z8pX4G1vWY9WVZJ+WNI7tKzN/mZl/4A5zS6cWRdclWr8F4JMA7l9k5xRd+vpGAJ9i5u8AADOv\nen8ZwDFUuWKPRjVIDrEEmPlqd/8cq/I3thRWfZDMLVectM4imLQfv4bqv/OyGNtfIjoFwMXomHBk\njnT5bM8CcBwRfZGI/hcR/crCetekS38vBfAMAPcC+AaAf83Mq5rCe1X+xpZCr5YlHioQ0c+hGiRf\nsuy+jOGPAbyNmUuiWWWFmRtrAJ4H4GUAjgTwFSK6lplvW263svw8gK8BeCmAH0UV3/clZn54ud0y\nNKs+SPZpSWOnfhDRswF8CMAFzPzAgvqWokt/zwXwMTdA7gZwIRENmfkzi+mip0tf9wF4gJkfA/AY\nEV0N4Dmo0mYtmi79fROAd3Nl9LudiO4E8OMAvrqYLk7EqvyNLYdlG0XbHqgG8TsAnI7aAP4sVeef\nIjYqf3WF+/o0VKuKXtyHz1bV/zCW57jp8tk+A8AXXN2dAG5ClV9wVfv7fgDvcq9PRDXo7F7i7+Hp\nyDtuVuJvbFmPlVaSPL8ljcvq6+8BeAqA9zl1NuQlZYTp2N+VoEtfmfkWIvocgK+jykH7IWZeSiq9\njp/tHwD4MBF9A9Xg8zZmXkoKNSL6CwDnAdhNRPsAvBPAetDXlfgbWxa2LNEwDKOFVfduG4ZhLBUb\nJA3DMFqwQdIwDKMFGyQNwzBasEHSMAyjBRskDcMwWrBB0pgbRPS3RPQK9/o/EtF/XXafDGNSVjqY\n3Og97wTw+0R0AoCfAvDqJffHMCbGgsmNueKS3x4N4DxmfoSIjgLwPgCbAL7IzDPbH9kw5oFNt425\nQUQ/CeBkAJvM/Ig7/YsAPsHMvwFTlkYPsEHSmAsuKetHUSVsfdTtiwxUGWQkN+FoGX0zjEmwQdKY\nOUS0E8CnAPw2M9+CKpnDO13xPtQZ2e33Z6w8ZpM0FoqzSV4K4AkA15hN0lh1bJA0DMNowaY7hmEY\nLdggaRiG0YINkoZhGC3YIGkYhtGCDZKGYRgt2CBpGIbRgg2ShmEYLdggaRiG0YINkoZhGC38/zJ7\nv7pZkGhZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114254780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.pcolor(grid, grid, im.T, cmap='RdBu_r', vmin=-2, vmax=+2)\n",
    "plt.colorbar()\n",
    "plt.plot(X[y_neg, 0], X[y_neg, 1], '.', c='b', markersize=20, mew=2, mec='k')\n",
    "plt.plot(X[y_pos, 0], X[y_pos, 1], '.', c='r', markersize=20, mew=2, mec='k')\n",
    "plt.plot(q[0], q[1], 's', c='g', markersize=10, mew=2, mec='k')\n",
    "plt.xlabel(r\"$x_0$\")\n",
    "plt.ylabel(r\"$x_1$\")\n",
    "plt.axes().set_aspect('equal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hey, why isn't the decision boundary non-linear as promised? We can see non-linear behaviour of the decision *function* plotted above, but the boundary looks to be linear. That happens because we only have two SVs. \n",
    "\n",
    "So let's add in another training point in such a position that it will become a third SV, and see what effect that will have on the decision value and boundary in the original space. (We won't be able to plot it in the new space as well, since it will now have 3 dimensions.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUkAAAEMCAYAAABJH0R3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXuUZUd15vntc29mSaUnUumFSjKSW8K8DAgZYZrB4mVL\najcCD2YAtx/Ybg0e06vdg3vATbdxm8aNl2e1bZqHXAvTDGth0x4EkjwWyMiGFjIWD9NCTyRkCVEl\nsNSF1CqV6pF579nzx4kdJ2KfiHNPZt7XKe3fWrnuPXEi4kRm3oz8Yu8dO4iZYRiGYaQpFj0AwzCM\nZcYmScMwjBZskjQMw2jBJknDMIwWbJI0DMNowSZJwzCMFmY+SRLRR4joYSK6PXOfiOh9RHQvEd1K\nRBfMekyGYfQPIjqLiD5PRHcS0R1E9C8TdaY+n8xDSX4UwCUt9y8FcJ77ugLAh+YwJsMw+scIwNuY\n+ZkAXgTgV4nomarO1OeTmU+SzHwjgEdaqlwO4GNccTOAE4nojFmPyzCMfsHM32Pmr7v3jwO4C8CZ\nqtrU55NlsEmeCWB3cL0HzW/cMAzDQ0RPA/B8AF9Wt6Y+nwy30njeENEVqCQ0jjnmmBc8/fzzAbht\nlY3tlbnysCyuQ437iX7cK3OZLK/ep+812oT3Sl1XyqVN2H1cpvuVvqItp1LX34uv6+eGz8ncS9V1\nr76JK1FNUCJoo341uTaJ3wZUUwPB5zd4T6qgUDWiNhS32VMe3svMp2xlTGfR0Xwo+q3n2Yu1OwAc\nCop2MfMuXY+IjgVwFYBfY+Z9WxlfF5ZhknwQwFnB9U5X1sD9wHYBwAsuuID/5qYvgspRdbMcAwBI\nJgwpH498+0Zdf1290lhe1+uHuns8cmWjter68KGonF156p5vs+bK1+v+S1dWrlXPGR+q6o5dHX+9\nVn8fdR1pczjqY+TbjOs269XPZXTQtVkfx9eurtRL1S2lzlpVZ72s66652bH5mr4fvh+72XCd5Tp9\nPy6DoRgEM95qQa6Monu6fIUo0aa6ftv+bz2w1TEdRonXd1ztfpAfOMTMF7bVIaIVVBPkx5n5U4kq\nneeTrizDJHktgLcS0ScAXATgMWb+XqeWVFRfAEBeGtX3AKAILAqs6lLa2sBBObn35Prxf5vFwL2O\n0cDdI3ev0Qb1JElFPAYaFFEVfb9LHX8fibHNGJm8NjKJaZ0xVhLTJsRuhD+n+r28oWR5+Mmp74X6\ncmsQ6gl5IhN+z0REAP4YwF3M/J8y1TY/n2SY+SRJRH8K4GIAO4hoD4B3AVgBAGa+EsB1AC4DcC+A\nAwDePOsxGYYxHwi1Qp3I5FX5PwbwswBuI6JbXNm/AXA2MLv5ZOaTJDO/ccJ9BvCrG+6XCEzk1SBR\n6crdNXezg1SVleoMlYxWmzkF6VUiQAM3lpG+tx4/JxxCZ3UYdlOVlbqO3A/ayDK6cGspsSjQIP8B\n1nU3w5ibS+Z83XTbtjpGmvpnFy+hpVzUXRm1kXdT/CFTbAbYCsx8EybI3M3OJ20sw3LbMIwjlA0t\nt5cUmyQNw5gZ1SS56FFsjf5Oklytisk7bjJOmWC5zLmluXvV9yP8Mru6R4NBsrwa2np8T137tlEd\n51kftDtyqm9J13F9rI+wWWRpHnq3jf6il87aKZNaWsunarrKj0xJGoZh5CAsx46VrdDjSZJRMoPE\nczZBJQIIHDMTVGeg1LiM++kUCiTti1htNkKCwntOydG4cMVVeenUYeiEyTlqfB9+/PWYxAkjbQv3\nc+sSJKTbtiFLq2kusbo4fYx2csoyVHllo+7WIdqAd3tJ6fEkaRhGH7Dl9gJh1NE6E22Twfuutslk\nv4W2RbrdO0EIEHt16P43i/rUbaP2aVukVphhmVaQOvSnDOVcJuTH10UcIgQALO/1czpuMwOaOz5M\nFc4XrSDl95EK+p/F78gcN4ZhGC1YCNACYVTJFMj9h5xkm6zKdND4ZI+47Ppv2DiVp5qDfcyiKlns\nlNoDHipJ1cYrx1LbF4O6SmUWTmWWLQHo8vPRbbEu3vpYNcZjixWk1B1w+OE3hdgHalukXHPjXjHl\nSc2UpGEYRgZz3CwYZq62JqKDbTIq62abBAAeDF03sdqkhp2xu3c72paolGjWyx2owqLMKcdYjYaq\nUMdBiu2R9GvwgdY2yLYtjNr2uBllWaubdN/VPVOsmyGnIFNL4WlGyppN0jAMowWyYPLF4W2SIvDg\npSQAoCicAlRtdFl1Q9Kr5T3iDdvkYKUqL9UOHKBpg/RqMeHdHq5Wr+PYy93wbof2xXGmTqmuA6Up\nnnZRkOK51vGSXZJi+D7GeZtkrSxFsSB6Dd97Tytiak+sqcdp0aYgtdqcFqYkDcMwMhCZd9swDCOL\n2SQXTBVM7rZZ+UV0OiQIaAke986ZfNhQZwcOAAxlKa4cNYlgcu/MUQkzZHmcXjqnl9s6FChcOvvt\nlcrT6NsO5PsMEh7opbm/lufWdSVuXi+rN/IH0iXwPOfcMTZHKph8mmwo6e6S0utJ0jCM5ceW24uC\ngXHJgBxeJCft5UKCgPyWRVGLCWdP48wcccKwKL3YgVOVKcUozhmpI0oTAJcq4NzdozJ25AxWwzbi\nUHHjXqnGzePYiZRy9hRafTo1qEOBqmerAPSiqSCFSSFAzftNtVmyboNGG1OQs2eq2xJp+sHp86bv\nWYwMw1hqCDTo9jWxJ6KPENHDRHR75v7FRPQYEd3ivn5zGt9Bb5Ukowp6rUOAHHJkqUz/wX8xHRbk\nQ4I4TkThtysiUJfKXukTWoiiTKlDbZuU63EYeD6I2ytlWaw6lRhte9TKLmOjXKl/vaIyJenFYKWI\nyiUUiMOgdVGq7mfqbZJlbKOs+tEhP7Fy1KFA8T33Y2kEPMs1NdpYWNDWmFW4TwOKPydb5KMA3g/g\nYy11vsjMPzmtBwI9niQNw1h+CLHZZysw841E9LSpdLYBej1JMgPjTIh4I8gcTY+393ZrhRmqtkwd\nUZDMsdIEEqpQ2yQHTU94V9skUNsnZZzyKl7tgbJRAk11yWOOygerbvtjGfy8xE45VlsXVTB79d4p\nRW4PIk+pwoZtUt0PVWNOfRpLSkEYrM7VqvdiIroVwIMAfp2Z79hqh72eJA3DWH5Smawy7CCirwXX\nu5h51wYe9XUAZzPzfiK6DMDVAM7bQPskvZ0k2Xm3B2JLU97tsbdNBgrTlQ1UqjId+4hBYMvL1PHK\nsoyVZlVXvNhqy6FWjYl7Xn1mbJNVe2cHLWJ1WIxjZRkqvUKpy0m2SaBWl6I6m9eB7Vae7W2T2iYp\nr3lVmLNNxiatybGUxmTmFeBNG7NJ7mXmCzf7LGbeF7y/jog+SEQ7mHnvZvsEejxJGobRD7p4rqfy\nHKLTATzEzExEL0Tlvv3+Vvvt9SRZAoE32wfbAUCtMAOlkY2hlCS5rl4yTpJVHbFJ6p04ACAqU3bT\nrKy6rpqecFGMTZukUqGB+hRVOFDKUcdNDgKbZB1D2c02CdT2yUm2ybC92CZXOU5OkU5wkVOb7vlo\ntqmPIYjvmW1ySSGamuOGiP4UwMWoluV7ALwLwAoAMPOVAF4H4FeIaATgIIA3MG99qdHrSdIwjOWG\nqDbpbBVmfuOE++9HFSI0VWySNAxjhlCUQ6CP9HaSZMTbEkmL6hbHjQ40bwSZh+E87l5jma1DggbB\n0lbqSpkss1eUcwZohgdpx82Kuo/aWeSXyK5f75QZNx03dVjQOK6bceBUZbGjJufAqcri9uLA0ckN\n4oQKuTChuG6b40Zvg7Rldzt6H3XKXDjVvdY0P5vkrOjtJGkYxvJDNkkuFq8mgWAXulOW8nspQ4dK\n7NyRNo0g8/AMbek1pyA5EQKkwoJoWEZ9RdsSc1sWy4wKBVCIs2dFVJtz3GiHTsJxU6hAdB82pFRi\nWCYf8sKpxdL3FTh5XN2wDADKw7FaTMUV50J+fN3o0BXdX/VmrYzvm6KM0fNUSi3Oai6z5bZhGEYO\n6pa8Ypnp9SQ5LgH5J9VQlC4JQ5jnSCfDkOtxJsg8rAuOf1ReqLgIg/BjwJIwV7Ul6SK0SZaRTGra\nJEU1hnWc2vSKUilHdoHng7KpJP2rU6Y6jKgM1KeowoGSZd5GWTZtkrrOYOhCgkZVv6kthqtqFSCI\nOgz/xmR00matZFVHpy/ZmKp8HCM8gANYQ4lVFHgatuPYfv+ZRHRRlNPUfZV3u/l31SeOnN++YWyB\nAxjjJjyCb+NA9A/pS3gUT8N2vAQnYTv6/ce+KKaYBWgh9HaSZOZKlZQZG6RWlABQqHDxbCB64o8h\noRircmVvRGC/1G0TadUayjHn7Q4pY7tlsRqrQf/YUBWOc3WqvlJ2zLpO7LlOebfruvGWxUne7hAd\neJ6sq4an1ehaqX7H1dNd/+nnHsAY1+AfsA8jrAB4NYCnA7gbwDUA7scBfB9ruByn926iTCUWCcvj\nshlMZkfAcnsuFlUiuoSI7iaie4noHYn7JxDRnxPRN4joDiJ68zzGZRgAcBMewT6McAGA+wB8EsB7\n3Ov9AC4AsA8j3IRHFjjKflJ5t4tOX8vKzJUkEQ0AfADAqwDsAfBVIrqWme8Mqv0qgDuZ+Z8S0SkA\n7iaijzPzWq7fyrMNrw4HTjl43dXweiOwU3ZTlAAwnLB1sWF/RJA+TQoGZbocAIbN9kCwTVG2JYY3\nlfrU3m6hCLzoWilO8nbHY4njJOthND/YZaOOkm/Bb3TslHXO8920NyZsj15Bxn2slZG2l+/EPbe+\n8zhG+DYOYAWVatypvp+dqFLJnAvg2ziA/Rj1ykbZ3Po5f0XZ9+X2PKbvFwK4l5nvc5PeJwBcruow\ngOOIiAAcC+ARAKM5jM14kvOAs0G+Gs0JUjgL1QeWAXwbB+c1tCMDqkLsunwtK/P4l3gmgN3B9R4A\nF6k67wdwLYDvAjgOwP/G4bYXBxFdAeAKADjjzLNmMljjycWaM3I+fUK98339hI3YyEJEUextH1mW\ndcNPALgFwMsB/CCAzxHRF8P8cADgEnDuAoBnPff5XIIbS+jsshtIhAdNWHajPult0vk47Y4bcdi4\ny0Aj5wLNqS0ESAWaSxv9UYyyALm6Ax1yJPfFKZNYbksIUGMpXTY9IQO1BNfhQyGrhzcf8d1w0Ego\nWKJLKUsFnK+6hndPeN497nW1J46bXKiPXnaHS+vC153iQAhLbW/swjxG/yCqFYuw05WFvBnAp7ji\nXlT28h+aw9iMJzk/gO0gVMuYPZk6u1HZKwnA03D0vIZ2hECgouj0tazMQ0l+FcB5RHQOqsnxDQDe\npOp8B8ArAHyRiE5Dtfq5r61T2ZLIPvQnVof1P8OWBBeTFGXASpFLhtF0xnCRDib3TynCZBircV0V\n3pNEhw35vty3o4LMgbRCDOvUjp3mctKrUNVHSiVq5w4nnDu6vQSabwb/ONVF6AQS1SkhRdJmzMDJ\nWME55XbchwO4HJWTJvyPvhvAa1AZyM9Z8sDybrk64/LwtzOLEKAqM/nyToBdmPlvnJlHRPRWANej\nWhF+hJnvIKK3uPtXAng3gI8S0W2o5ou3bzXlumF05aU4CXuxhq9jhHNROWnOR7XEvgbVBHk8hngJ\nTlrkMHvK9JLuLoq5/Ftk5usAXKfKrgzefxfAj2+sT2B9zLX5UExFOri8iKyFrk5HRRnU9c+ZYKME\n0rbHqG6QVq25dXEFSUJlmVOZmSDzFD4ESAWZhx8IGX5Tha67Ppof/oaSTKjyXN3NKEqtEkVRRiNT\n9kr9ehyGeB2fjv/Gj+A+HMBVQVNCpSCXccdNLqwnvCc/ly42ybb+Ng1RdP57H+n36A1jShxDQ1xG\np+JxHuF+HMBBHmMbBjgbRy/1EnvpIYAGy/XPZaP09rfPYKyXpfc+e7ngfh+1ja/LtkRVN6U+fT4v\nKVaKkkM7o6siQeT+RiIZhr431OPvEHKSsVHGOu9Q9RgfNF7VbfsAFBM84TGiOzfykUrX3YiiFNWz\nphRrJFJVooza261UKIATeYjn4/iGl1y2TC4yBVtXj3WqTF+nPNlyr23r6EahI2C53e/RG4ax3BBQ\nFEWnr4ldEX2EiB4motsz94mI3ue2P99KRBdM41vor5LUNknv3nRHGhTRpasyKT5S1UvVzSnKwapv\nQm7vnfZcp0RI454TZA1FGdJFXap6uY9g3u7YrNO0W6Z2jW5eUfpECAerPtoUpVRNJ7Roj5f0P4tE\nbKW2V+bK28773orabItR1KqwrXySgtS2yrjuJgbewhSV5EdRbTz5WOb+pQDOc18XAfgQmhtXNkxv\nJ0nDMHrAFI+UZeYbiehpLVUuB/Axd4zszUR0IhGdwczf28pzeztJMoD1klH4E8Dc0QJcqacVTvxi\nXJGk7WKnEv153JDr4DnSvbc9x4qydH2uhLt0BqIgN6AoZZeO/EZEUUoCjdVEI+mvTCTByKCTYbR9\nAMT2mKsTl2tVuRlFGVOsuQQe9VYZDKh6v9YQmZIMgxr3td3SLwbcL3o9UIXdlWRzvOmDyzZHl8QT\nObtjWFYgXSfVJqUutwptzLu9g4i+FlzvcrvsupLaAn0mgCfnJGkYRg/Y2LbEvcx84SyHsxlskjQM\nY6bMccdNly3QG6a3kyQzY30chgC5M1o4vg5dFuKbWFGB59450yEEyK/iE7knBb+FccKyuyqL8Uvy\nYbocaC69G4cKpnJQKnTkWtsHocvCOe/Mmdy6TpPVITue715tkcyEAgHNZbAsxetld1hXti5y9JTm\nMpuCNi3hR1Ogi6MmLAe6L7PDcJ9cv1uBiOa5L/taAG8lok+gctg8tlV7JNDjSdIwjH4wLccNEf0p\ngItR2S73AHgXgBXA7+C7DsBlAO4FcABV4pwt09tJsnbcxOWl+n2Mg4DoFf8vMg48zyrKsCyT/IJb\n/utOUpRBr50VZXhPZzXXijKVm1uzSEWZO/tEnFWS0Xoc1POOLOfUGYzEkRNnMQ+7rh02sWNCO3LC\n9/q1LY1YzskzLSadcKhVY9hmkpJMtZlmMPmUvdtvnHCfUZ1yMFV6O0kahrH8EBEGK5l8BD2ht5Mk\nM3BoVHq1JrbIFbEpOTW4Ev0bjsOESmdglDql01uhkBRVJmWl/ifbYpsUZAhDUZRU2950KPSmFKWy\n+TRslInnNMaYKMt9OMYtykDbn2gwUtf5AHFRloWrM24xjvm6KkxoXdK6RQqp+u5r5RiHC42D5YBW\nl9oGmdqeqH8a0w7GbvbfHgqUGoPecphS3LOwSQL9T7rb20nSMIwesLEQoKWkt5MkM2NUHZcIACi9\nh7G6Th7Z7AK2V1o84I3LCVsZvS4KHyinC6buARgWgV1Ovh+5dqczNhQl1YPKqc2c1zsc50YU5WY+\n3GVDSSqVO2huqRTbY+F+TustXm5RkGMVTS7PEVvleBQGoMevohy1sqzqxOpSK8d0MHlcV5iWbXKS\nTbJLyrRcEt5Ze7clM3mf6e0kaRjG8kOmJBeHeLfl7GaxK/prZasEQgVZuLpOJfhEB86OGfznk5hK\ndvf0VsbUJsOSxMaZ8YSH/70HK8GIAHaKsdF7oBKy9kvVNsT3L+eHr03O8VcU6TqFyw9Ihw437mnV\nMFa5BMdFUyWK7VF7u8fiyQ6924XUVR7w9XFUV2yV1XuxU8a2Sa0sq/ftylErzPBeTlE2602mS6IL\nXbddScZ1Um20vXIqWNJdwzCMFohAmX+2faG3k2SVKq30diyfIs2pBe3tDutotaltk+L9rt6nPeA6\nrRoHuq5+ZKz5yoT6lHdDrSh1UgwdANroHYBL3EHqfsSkD2x4vzgUjck/t8XGVA7abZKppZeoP31A\nPaWUZMYDLuVyJEQqttIfWbEmu7OaNkm9GyefIi1Un/HrJEW5WXIKMnXdNRlGyiO+Mm0bok2ShmEY\nOagRotY3bJI0DGN2kJ1xszD8tkTZSejyJK645Vxd3jSwa6eOf3W/y3iJngkTkrsU1wrL/Fh1Tsqg\nQqmWZLLsDkN+9HVjESf3KN4KSFEV93C3hM7SsjSSFPuU6EOW0SPlzNGB5+G1LNvHh9ZcHyM3hKp8\n3V/X34mYV7RTR5bqcjpj7OyJT4TUy+/w/HDZ5lg7cHQoUH4pPWl74mZDgnKOlLbl92aW25IsZrA6\nTeVHttw2DMPIQUT5Y5J7Qm8nyZIZB9fHWB06Z4sPAo5DgkpqqsKmUyd26IwDB4JXDpkwIR0iBKDO\nveXkZUNZBv+odYKM0qvdquNCnaUDJALQx6Po2hPagkZOBeoM51JHv0Z1BvGroy3wnLzqXIvu06Fa\n7eYCzyVMyAeOrwTq0ztqYiU5WK1GM3ahP6ETSCtHrSzLQOJJ6JCoS3EEaSfPOJHZpLllcTEOnLBO\nLjxIVGOouOVnmEs8sikIpiQNwzDy2HJ7YTADa6MSYwkMdsrBK0uv+JqqUM6/KVUyg1QAurZX1tei\nMOIQoQq3tTATeB7bL+N7WnMMnL1xOEgccqNskb4nKR/Xqq2hPreJcmz5AMu9w7ENUlQiBwZ5sVN6\nxSi2YRV4HmapHimVOToUX5e+rzoAXdqPV932wzV3vaZslJFCcr9nUYnqmgMlKaFDpVKS2n4ZtilV\nWFBze+JkZabbdDlnZtJ2RaCpGAv/8ymi8ujelL3Rti3RMAwjB5mSXBjMjLVR6RWkvIqyFEUZ/hcf\nOONg6RSeeMLFFlkOJgegi71S2ypT6SR06jVRkKEZWweed9rKONxWFY3Xqzq+CzeGUeo8bF0n3sKY\nUofsFKSMW0I5+HDeblm4Nt5zXcRjSXm3GwpybeiGeDgqr/qL7ZW5bYqDlfr70FsWRQVqjzgAFM4u\nVzqV6RWlsl9ywlWd8pbn6gpahbZRZNRlyoaoFaOum1KNRUKFbx2bJA3DMPIQgSzp7mIoGTi4Nm7Y\nIrWiHAQKTKtLb4sUO5TYGyM7pqszIbYyZLIHPLBnicp0ikJUrAiMYdFso7cy+hjKsUqOEdqCxDY1\nVqnYpDyVHEPUoU+K4WyT3hMefPP+ntgxnS21EI+ps1Gu1x85rRS9ohTPuCj99cAmuVKp59KVFSvV\n68Bdiyd8vF5bfrW6FGWpPeJA4M1W/YhK9PdTSjLhLQ/bxGVOdTbuTCaXVadVURZFdJ1qU9uRl9e7\nTUSXAPhDVD+6DzPze9X9iwFcA+B+V/QpZv7trTyzt5OkYRh9YHoJLohoAOADAF4FYA+ArxLRtcx8\np6r6RWb+yak8FLltJIZhGNOiKLp9TeaFAO5l5vuYeQ3AJwBcPtOxo8dKsmTG4VHpl6O5UKBwud1w\n6hRxGI8YxsMMMisSF66cOnKGTsomPzlMKNhi2CFMCAi2NgIYypgmBJ7TOHiOvPFhQ2orY6HKgaYC\nkDprLQHna7HjRpboA7/srsOJmtsSnePGLaFlmZ0KQP/OE4fwV3+/B48dOIjjtq3iFWedhjOO3e7D\nhQar9U9QL8Fl252EDQ2CYHWpU8pyeFUFlcuyO3D2lCpMSC+vU0tzgUv9m+6ODq1JLZP1ElwvqcOg\n+yIRFrRlNpYqbQcRfS243sXMu4LrMwHsDq73oDpfW/NiIroVwIMAfp2Z79jIkDW9nSSNJycP7z+A\n3/zcl3H9Pd+J9r3/BwJedc6ZeOdFz8Yp249a4AiNCCKg+7bEvcx84Raf+HUAZzPzfiK6DMDVAM7b\nSodzmSQnGVtdnYsB/AGqCJm9zPxjbX36YHIVApRTlkCgILWiVNdRMLlSm+LUGYrq8Scv1n+w23xo\nURwm1AwbAlaH7WFCTQdOHSZU12FXRwWeh0kytHLUinIcB6QDQZZ0qePVoHPkBAqBhy4cSYcJqcQa\nRREq1TgAXZw62qEjCufhx5/A6z7+WTzw6D6sAHgtgKcDuBvANQxcf9+DuHPv/8T/+/pX4qSjgrPN\nlbrMBZWnymTDQUNJBupQ3/Pl5WRFyeMtKMmWYxEaDhp/Bnw6JCh8r+tsBQJNMwvQgwDOCq53ujIP\nM+8L3l9HRB8koh3MvHezD525TTIwtl4K4JkA3khEz1R1TgTwQQCvZuZnAfjpWY/L6B+/cfWNeODR\nfbgAwH0APgngPe71fgAXANi97wm86/N/t8BRGhHi3e7yNZmvAjiPiM4holUAbwBwbfQ4otOJKnsT\nEb0Q1Rz3/a18C/NQkt7YCgBEJMbW0CP1JlSu+u8AADM/PKlTSXChg8n9+cJDCUFp2iRzYUIp9Tn2\nYUPu2tsm4Z6H6H7Yz4rLJn7UUMYc2yqrsqrONvccf5aOV4f19ytMChNir3brZQ4p5UhOUfowIbGJ\nhupTzqNRYUJ++RR+sA/r8CD3ur4e1Q0zwojqJKdCxV4ptjYJ8xmvrWPPI/vwmTvuwwqq+I6diNmJ\nal11LoDP3fcgHh6XeOrxx1SPdmMo1yRsqIz6TwWeS1B5LvQnVIBd1GbqOkSr0I2QUn5NW2R8nbI/\nFj4wf5raaXrB5Mw8IqK3Arge1ar0I8x8BxG9xd2/EsDrAPwKVYfbHwTwBuatZRiZxyTZxdh6PoAV\nIvoCgOMA/CEzf2wOYzN6wmdvuxclM16L5gQpnIXqv+9VzLjh7/fg557/9PkN0Mgyzb3bzHwdgOtU\n2ZXB+/cDeP/UHojlcdwMAbwAwCsAHA3gb4noZma+J6xERFcAuAIAjjn5dKyNxhgUsRfbq0L3zyM6\nw8Pd29bRNhmWjYexbXKdZGtjrCyB2hMu6k9eta0SCO2VVZ2jxC4q2xRVMDtQn6cjvzyp45Wl8n5X\n7Z0HfBgrRx9crm2U4eBIqUPZ0rgtVJ1OFY4qpSj2ytLbHRNbGkVhOCUp9srCXZcjV74yxH6ntCZN\ne+e71/1liZVjKgeObHMcO6956ZQlr7rytSBYXSlInwSjzCvJNpUJxImfdRvNRhRlm+2woRwb5wfl\nA8en7d22bYmTmWhsRaUuv8/MTwB4gohuBPBcANEk6cIBdgHAyec+c0sS2ugXJziP9d0T6skH5oSj\nt810PEZXCCiWRYttjnmM3htbUU2Ob0Blgwy5BsD7iWgIYBXVcvz32zotS8aBtTGGLoFqwzbpVEmo\nCgc+xrGbbTJVJopy4Lcpite7aWcsvapF1MdKcPJhw17pVWf137fe0lh/71pBSpif1Ekm0vA2Tvfz\nkSQZaktANWv5AAAgAElEQVRj0iOu7/nyIFWaVpt6S+NIxWMC3l5Jo9g+yqIg12Wb4iH85EXPwtv/\n5Hpcy4w9SC+5d6P6IBVEuOQ5P4iB83DLuc+Fs0GKsixdjKJsdQSCrYXKfinlqVhIHTuZU5Qp9ZiL\noezi9d6Id7tuk7c7pmInpwFTfpx9YOajZ+YRADG23gXgz8TYGhhc7wLwWQC3AvgKqjCh22c9NqM/\nnLXjRPzTH3kG1lHZHXer+7sBvAbACMBlzz4XZ5543LyHaKQgVP9Uu3wtKXPRwZOMre769wD8Xvc+\n2+MkB07ZjYMdDdp+qW2RnZSkrjsUhRnYMTOecG2rDN83YivdsFdcAOVRQf9ir9QK0nvCEZcDtRfe\nq0///bnzvuVDWjbVofeAi3JMHTymduzQIPaAa1slEMRWjpS33ClJ8YSLF/z33/J63PrA+/D1hx7B\nuagmy/NRLbGvQTVBPm3Hifi/f+ZSrBxdx0mKKvQ2SG+bdOUr9Z/B2CfOcM8ei00yVpih7bBQKtN/\nf5n4yaq/7rtyNkNWSWbiJYF2lbl5COiQQHiZWd7p2zAUpz/leNzw7v8Dr7no2SiJcBWA/wjgKlQZ\n3l/9gh/CZ3/9n+FUF/pjLAnT27u9EPptUTWedJx+4nH4+Nt+Frsf+h/4i7/7Jh7d9wROPOYoXPKc\nf4SdJx3vlZ6xHDAR2Bw3i4FRZSYXmo6b+BWAd/LkltmbWm6rcqAZLpTb2gjknTo5hw5QO3VYImjU\nMltvaazquH7UWToDymxpBJqOGh9cnnDylOqeW5q3BqCLY8Ytt0u/NHfL8PVEG7dEP3vnCn5l51PB\nEs7jnD+SHCPMWyllpVs6F2q5HS2dZWmultC5ZXiqrryWKmdkKplFF+fOZsiF8fizbRLKrQ4wn7Kq\nW2J7Yxd6O0kahtEHyCbJRVGWwNra2AfJagdOSklKnVEp6ikOH1odukzVLUqybtuiJN37NaVcV0Ut\nBmPalnHq+EzlEgAfHNadc+oM1HbFUExIf7k6PiA9PGUw49RpOHSAplNHVKd3+gyjtkDgmJGQHx0u\nlFCfPlxInD2ypdFdyxbHYiVIr+aUo1d4wXZHIFZ4vm4pweSiIIdReRxM7vptKMrY6RPSVJD5kJ+u\nAebtweVF6zUw3Z0xIX0PAertJGkYRk+wSXJRMMpxWZ8b40xEYxdu4Dfrh6cMZlRmzkZZ3YvVpd7u\nOEooSbGV6n7bwoVEFcq5OKIw69MZ6+98xdk4jxIbpwpAF1tlpCTlZ6C3MGZslVX72F7ZCEBPJsNQ\noUQNe2YwKGfQz4YLib0xTIohZ+noQPSMsgRqdYnSpT8LtjsCaSWp7ZdeHZaxvTG6p+o0bJSdFOW4\nUWcam/p0urJkMPkslCT1PwSox5OkYRh9wLzbC4IZGI8Yon3qXXFic4sVJtBUmcMJXu7UvbFTSDqR\nRuhpbyjIzCvQ5gl3NlD3nPWgjU+CIXbLUp4Hd909AD1nqwSa9kpRoStiqww//GPZdjiMruUPhMbr\n8f2wjdgzpe4wrwp5WHnfRVF6BZlTlgBo1dkG1XZHSaTBZf0hydkva491086YU5KlLk8GlU+2Tc4i\nMe/8bJK01DGQXejtJGkYRg+QbYk9pr+TJDNG6+Pa5ujEgLdFSiqw0GymVKaksNL2y3Yl2a4sgVpd\nSt3DTmVKirYw6caqKtOecGkTpWJreL7jsYqSjBL1SvII8YSLSnT3ta2yvU7Tdjt09koSVSbbE50t\nsrG1Majj7ZUq1pJEdYZnpIi6XFF2S68gRVEGSSuccmxsd3QqTtRndS9tv8ypxdQ9rTZTSlIryMbR\nD1tQj220J8WYkZK0SdIwDKMFmyQXAzMwHpe1gnTKkQN1A8Te7ZzK1EpyTHklqT3XWiVWbSq1pOMk\n22IqG6/iCfeqMVCFYnP0STFcuRtj7fWufw7Svu7PKVZnqxTVGClJtZNH1xlE6lbquJ07TllmvdwA\nMHAfP1FypbtWdk2Jsay6UepS2S21sgzr6p08onrb7JcDqbMeK8zQjtlIr5axM0ZHPigv9pFsm+x7\nnOSmR09Eb5/mQAzDOAKRzOTTOQgMRHQJEd1NRPcS0TsS94mI3ufu30pEF2z1W+isJInoz8JLAM8D\n8LtbHYBhGEcy07NJBievvgrVaQZfJaJrmTk8VPBSVOdsn4cqefeH0DxTa0NsZLm9j5l/ORjwh7by\n4GlQBmE3rBw4Pm9eSc06UHXUMnwwDNqIE0aFD7XloMw5d2QJvRYsm1bdeTHaqaNfj16t/9P6cCHv\nqInDherQoNBxE4csraikG3JWDwfbH2U5XS+lXcC5c3yll9vuZ1BIyJEKFxoHy2C9vPbLbjnX24UE\nFWEb93OQbY4Tlt9AcwlOq0dF1/5ER8Avp8XJ46+VQ4fC5bYEnrtzvaOlPtq3Ms56mT2J5HJ7yg6c\nKS63u5y8ejmAj7kTEm8mohOJ6Axm/t5mH7qR0b9HXb9zsw81DONJRPfM5DuI6GvB1xWqp9TJq2du\nos6GmKgkiegPAfwaM98fljPzI1t58FZhZozWy1rJi4JUW6DCU+JIOWhEZbIq5+A0w5yTR4cPddmW\nuDaKE2kAgYNGJdBoC0CfFC5Ub3UM06tpJ49KpOF+FtsC09CKT7pRXYvA8H0ESlLGNxzEKlOHCw0G\n9QFdReF+aWUcTF47bCRJRpDNXJRcEavOrLJE0xnDKkUbrwaqUIUQiUqsnTxKYYb3JKSoPCqq47ca\nhqox4QAK+4jKMkpyMwqzi0qcppJkInD3bYl7mfnCqT18SnT5aTwO4FoiOgYAiOgniOhvZjsswzCO\nCJgxLrt9daDLyatd6myIiUqSmf8tEb0JwBeIaA3AfgANr9K8qbYl1v9J61CfWOFxGNOi1KYO/fEK\nk1rUp1eh8XPCs5W1uswpy+p9VbYmITnuZMK1UawWtyUC0MVOmQsXWgnssTI+2d640rBfxlsbw/Y+\nXGi48XAhrTrFrgk0E2d4O58kuihk22AQNqTUpVeWg1FUl8ZBG19H0qnFKdoi+6IO9RnFWxe1woz6\nV4oyW56455+fUp0a1WZqzOh87E7TXze6nLx6LYC3OnvlRQAe24o9Eui23H4FgH8O4AkAZwD4RWae\ndPyxYRgGGHG87pb6Yh4RkZy8OgDwETl51d2/EtWBg5cBuBfAAQBv3upzu3i33wng3zHzTUT0HAD/\nlYj+T2b+660+fEswV8HkggoqlzuUsElKrKycUOczfymFGbbxWxYH4llOK8uwTMYnbbSyBJpKUZTl\nWFK0qS2O4Xt5zXrEVwKPeEdP+HrwM/VbIjfgCZfmwyKdOCNM3yYqc6gC0b2y1MkxEGxzFHUpilFs\nk95jHipJqSs55KokGTRU2yGBhhfbHzGh1CGH2x6lrleHcdLdpEdc9ee/v5Tq9N/HBAW5EYU5I9WY\nIlrNbb2v1pNXnVf7V6f2QHRbbr88eH8bEV2K6oC6F09zIIZhHHlMU0kuig1vS2Tm77kl+EJhjuMk\nfbmOheRQulQvBcXqMGfPrO65V9n2OOSoje4LAAZOgek2WlkCTVVYb3NURz8EHvE1Z7fMxVh6W2XK\nI65sktoTHh4tMckTHm6V3KYSZ2iVOE70P1D3/NnpWlmuHOXbkFdlzvao7ZZaWQIAu8PNBko5Ks94\nVWc17t/HVKYVZlVXqUytKJXSjO6Nm/fC7zMkqS4TbbfMlFVmz+fIze3dZuaD0x6IYRhHIIyunuul\npccJLlSqNCETLwkEnmh3nfNqc4sdk3kc9Z86JoLLWG2K7U4rS6BWl6ORJIaI7ZejFjumqEtRlkeX\nsYJsS8nm1aZXkrEnO7qnEv8epWyVVd04ccakGEsg9Hi7V6Us5WiJcGdP1m4p6nAgyqzFJil2S9+m\nbNZlF2cpcZISf8kuFnK1VrdaZXq74jhWlIhskhm16cfRVIfU2J3TQUFOUpkztk0y6r+3vtLbSdIw\njH4wRb/NQrBJ0jCMmdLz1Xa/J8nxqF5SUWbZEDph6qTZbolcUrJuMmwo4+yp29T96OV2IxRoHDgv\nBpIxPHbuyPJ7LZHNvLl1MV5m60B0oOncyYUPhUvo3Dk7OjN6eG/bQDt30oHoQLhEj5fV2vkzCP7K\n9BK8sfz2TpLg86DDglicJs45Ey3N42W1tGG3/PZB6lEkfezs8UvyTHB5dE8vyf04mkvz+p7rPy5t\n1lsCmKcbArQIej1JGoax/JhNckEwlyhVSiqgqSjLlnty7R06SmkC3bcwRo6bofQT96HDh4BadYq6\n1MpSgsBHgeNG1GWtMvUZOtX3JVscgVpdbl9NO3f0KwCU7gBvUXorLUpSB6PrQPT1xAmOEmguAeiD\nol1ZAk116RNn+GunLMNEGj7nm4T+iDNGBaZXF+5ByoEjHw5x8qTaKGePV5T+ftlokw0eT6hO31Q7\nbDLKMkmb2pyBE4dh3m3DMIxW+j1F9nmSZEa53lSSQs5GCQA0UGrTJ1SIX0MaSnIDIUB1KBBHbYGm\numwqS6eQgkTAZcMmqRNpVK/bg0S9eptjKhD9sYe/i/u++gUc2v84jjr2ODzvx34cJ556RiMQvVaL\ndf86GF0HoouNcj2IAfKhRD69Wqws5YzwMHOXVpf56/o59XnqcQJgb2csm8kwtP2yVpZKNYZt5J5W\nm1IeqkWx03GsINtUp6/TIb1aY2wLpOdCsseTpDE19j+6F5/f9Tu4+0s3gIM/yM986Hfw7Je8Cj/9\ntn+P408+ZYEjNPpMz/02PZ4kmZM2SU1KFVKZtk3660Bpyj2eYM/soiRLl3S3CO1+TlkVEnztTliU\nOlpZArW6lLOaxyPx8Mae69AWlNvm+D/3PoKr/90vYN9DewCsAHgtgKcDuBtcXoPbbrwee751J/73\n930CJ51yatRHFBiuvNm1t9upxbLpPV93CqgOTo+V5ViOiQiiECbZLVN2TCkjKDum8owDgUqbZIsM\n1V1DZcpr5fVmb39M2TGVCmxTnbqOjFn3keo3R9d6m4TBKHu+4J7uYRYZJp1wFtT7ESIaEdHr5jEu\nA7jpj9/rJsgLANwH4JOoTur4JID7AVyAR7+3G3/+vt9e4CiN3sKV+aTL17IycyXZ8YQzqfe7AP6y\nS7+8BSWZu+evR80y/zqIrzlhx2woSWW/LAJlIN5xUY469ZpWlmH/frvjKK0s14I2Ot5y27DEE9//\nB9z/lb9GpSCvQZXEOWQngKsBnIs7b/ocHty9GyecUtso14b1J1u2OeZskNuGKY94EdU5ahi31coS\nyHvEB3KIm/J6V2VSV/pwr65KmObOe8f97zs+o9vHX0ZxkhmVqVVhaMf0dTjdpsUmqb3mfiRttskF\nrXl5cY+eGvNQkv6EM2ZeAyAnnGn+BaoUbA/PYUwGgO/+9y+6peCr0ZwghbMAXA7mEvf87WJTiBr9\npHRL7klfy8o8JsmJp5cR0ZmojGGtx9QS0RVyklp5eP/UB/pkY/3A4+7d0yfUPB8AcHD/vpmOxzgy\nqXbdTP5aVpbFcfMHAN7OzGUqe4/AzLsA7AKAlRN38rjDcjskv3Ux78jJLclzy3AgH1LUlsWc1NZF\nnwldluFBG1mCj8dF1EaW336pHhh69BJ8bVigXN3u7k46jeOeakxHHYvHD40aWxyBydscvZMmiOfR\n5+3oa738ru6lw4a840Ytqat7nKwrK/KUk0fa+LPYnZOncEHqRah8dFhQmQ4m59QSurEkTy230+FC\nE69zZRu5v0UY9eaDvjIPJdnl9LILAXyCiL4N4HUAPkhEr5nD2J7UnPrDL3Hbgq5FJfBT7AZwDYgK\n/OCPvGx+gzOODObkuCGik4joc0T0Lff6lEy9bxPRbUR0CxF9rUvf81CSE084Y+Zz5D0RfRTA/8fM\nV7d1yhOCydtoBJNLeSpcyKvAbooyec89TxRmMVz1dXUIUSMXpboGmuoypywHgdIrvLOHfb/Fsafg\n1Oe+FA/f8gVUZuKrEf8/2w3gNQBGOPvCV4CO34H9h0fJBBqTtjnKa3iqpD5vZ5KyBCY7d7RaBGp1\nWeScPalwIaUya9XZ3BDgVab/rKgkGR3UYUNtplSndszkwojC/hvl83UjM9ivImbMOwD8FTO/10XQ\nvAPA2zN1X8bMe7t2PPNJsuMJZ8aCeMbr/xUe33MPDu79OoBzUU2W56NaYl8DYIRjTz0TF70593kz\njDxzXG5fDuBi9/7/AfAF5CfJDTEXm+SkE85U+S907LRTCFASF6mRs1FGqlBUoLqn2xZtSlK9himt\ntP2Shy7Ttmzvc1sNB4kQoJzdUitLIDjl0YXbjF3YyuCoE/C8t/5n3Pvp9+F/3PpFgK8KfxA4/fkX\n44J/9q8x2nY89h+qMnCnEmiIktT2Sv0ahiXJaY46gUZOWabKVtz3oRVlmGHdq8zcVsbA6EQ6cUbG\nfhmazus6GTum/I6DNjQp9CecWHSiDCluVZI6YUZazSUD0acJb2gpvUMtgXc5P0QXTgvO1/4HAKfl\nR4QbiGgM4I+69L8sjhtjgawefzKe84vvxqFHH8Ijd34JowOPY7j9OJz5/Jfi6JNOi5bVhrERNqgk\n9zLzhbmbRHQDgNMTt94ZPZOZiSj30Jcw84NEdCqAzxHRN5n5xrZB9XaSZDDK4OzjzUBFevN/pBIz\nqlPbG/W2xbBOoWyRoZLUKrMcxXW9wgw+aFpdTlKWALInNobKsth+Mk57UR3CyoMCBw6NJqZmA4Ik\nGxNOcgwnXEnIIYHo+iRHrSyBhJJU2x1TJzlm7ZeiCgOlo73jog61/TJUhTmVmVOYYRtCWm1SQknm\ntzJmvN8IlKL+eOpA9MS9aTGe0nKbmV+Zu0dEDxHRGe401zOQibdm5gfd68NE9GlUcdytk6RJBMMw\nZkalJLt9bZFrAfy8e//zqAzqEUR0DBEdJ+8B/DiA2yd13FslCWZwOZ5KyvqmbXI9cW/dXcf/V1q9\n20pllm6sXeyXPq2/9FGuNtqIupykLIFmIg1RllJXtj2WQdYKsVuKbVOOlBgnzvVuerfjkxy1sgzr\nNhRl5oxwYLLdMpVIQ6tLOVCxbdujjNIry4ZKbG4t1XXa7Ji1kpTrWG2Gh4B2UptAN9tkToXOCGb2\nqfVmzHsB/BkR/RKABwC8HgCI6KkAPszMl6GyU37a/YyHAP6EmT87qeP+TpKGYfSC8WznYQAAM38f\nwCsS5d8FcJl7fx+A52607/5OkswId9x0OoN4Ajp+MiQfJyn/JUP1Wag6yu6YSMWm7Zf+XGZpE8SE\nFiuVqtT2y5yyrPqL1WXDy61iLKt+RZm6PtSREpMSaITXWlkCTXWZVZSh+pxgt0x5xEVdDpzxcWUQ\nq83Ujh5Rhw37ZcIjrlWnpGTLKczqXlwmfWgVWvUnZXm1WdULfndebjY960BChYZM0SZ5JOy46e8k\naRjG8sN2xo1hGEYWU5ILJLctcSuOHJ2xPOq3JXs5kHbG1M4eWVIX6n7CYSNLaeXkifNVxktxuc4t\nv4G8c0c7bGT5DdRL8NKfFy5L8m4JNID88rt6n3bu5LY4pspyy+/DwfqyDjiP62hnDwAUsiQXp04p\nS2a1/OYgnMe96i2NuWV49RxK1k0utyfUUStpd68ZdqT7zbWdZtALozaB9JXeTpKGYfQA5mi/fh/p\n8STZLQRoI8qyLYt5o44KMk8Gkw9iZ0zT2VMHtGtVqJ084dgGGSWZU5ZAi3NHOWyiDOiZulpZVu1j\ndTlJWQJ5545WkqmwodzrYBwrQSARTJ5Rn0DgsFFOHa8kVV9hGx8uJCnx5GejFGZ4r1O4kK8rhRy1\n0Uqzra6/jya6zjRgzMe7PUt6PEkahtEHzCa5KLaS4CLXZQfVqeu0pVcTG2cjOUZgz2yqzLT9Mkyv\nppXjJGUZluXslj5EKLAzlqNYXeq64Vk9+tydVGo2IB2APsqcIy4JNCQ1G9BUlw0l6eRQyo7Z3LoY\nK0sgVJKxvVIUnlwXwV7GFX8uTnytw4jalKS0Havyqh9XpkKLdFhPrCTjtro81aauMz1JWSlJmyQN\nwzDSMMwm+WQnpT5zajN1GmPjqAdlx+ySFGOSskzV0cpy7J6TCkAXxSjnho+Hsr0vUJ0ND3jaax7a\nMUdZu2WcQENslUBTXWpl2aYkc0dLhMHkYnvUwem1kmwGq687T7IoRrkulB1zECg0rTLFbqcVZlU3\nLuvm5e6mOtOqcXqTmnm3DcMwWmCwLbeNyYhqS6nOhi2y3HhSjJxKjOIxnf12oJSpbhMm0hiLMtUe\ncG9nLIK6sbqcpCyBfAKNnLIEmuoypyxli2NYNukVCLzYyvNdq8KmkqzvKful+57XJVtZFLspSjVW\nmdpTHpZNVJKRzVPaxBNUU3U27bFTxXbcGIZh5GHYJGlskZzKbCTFSCjJwUq7Kkzt7JF7OY94KpGG\nqEt/tIRSi0CYDMO9uk+W93KrpL/AZLulVpbh+5zdUpTlajC2wyNdt0VJKpum3tGznoiTFOWYs1/q\nRBfVPfGASx+xkozStzVUZtreWERWSY7vJdTmpDbTgE1JGoZh5GFwZProIzZJGoYxO0xJGrNCL8Pb\nls46fCgVAlQMV1r7bQsbklydjUD0Yf3xmXQGuM6MXrVPZ0fX+SvDJfqmwoaGceKM3PI7fC/qJxes\nHmY+z4UHrWunT+KsHn1v4Nv6qlmnTr2Mj++n6uSW2anzsqa5PdFskoZhGC2YTdKYG6FjRzt5/MmK\nEj6kzscBuifFSCXSkO2fXkGOm+FCXpE6dakdNlotAvns6LnwobC/Um1hFOXXKWxI1KJTqKvBtket\nLnOKcq1ohhjprOiFV53OERWG6BSxglynWEmGqlCrzEKFFCXDhspYDuacPyE51blV5jFJEtFPA/gt\nAM8A8EJm/lqm3iUA/hDV+ZEfZub3Tuo7tXXTMAxjKjAY47Lb1xa5HcBPoeV4WCIaAPgAgEsBPBPA\nG4nomZM6NiV5BKATfSSTYih1uZGkGNpuyUphhs8SdSnXPnXaQFKZBaEzKj1bLtlGGGqUS/wr/esk\nGUCoBtN2y1AV5k53zCnK1L01CSOSYHJ/drdvUgeaq/Cgofycgro5lVmf6x3fr8rg6mTsmWVTLqZs\nm1uFubYBzxJmvguYmJzjhQDudQeCgYg+AeByAHe2NbJJ0jCMmbFkNskzAewOrvcAuGhSI5skj0BE\n6Y3XDvoyrS4nKcuwH602i8RWSe0192pTPzeRXs0rRjm+QRJoDCmqB4RbIsXz7RRSJulv1U+sMnPK\nsqoTq8s1FZyuPeNVP2m1KX1ozziQt18OlNcbqAPLi7JdQRZRoo5cHUSkkm4MpmyE28De7R1EFNoS\ndzHzLrkgohsAnJ5o905mvmYLQ2zFJknDMGaG2CQ7speZL8z2xfzKLQ7nQQBnBdc7XVkrNkk+SdDq\nMqcsZatj2KYRYylxk2HyYLXdseERb0ugoQ8nUwk0OPiUajtlodrqpL9AU13mlCXQVJdaWbYl0MjZ\nL+U6VJI5+2VqC6NOgqHVp1aYUT8JlRnWjWMr02pzKyzZcvurAM4jonNQTY5vAPCmSY3Mu20YxkyZ\nh3ebiF5LRHsA/CiAvyCi6135U4noOgBg5hGAtwK4HsBdAP6Mme+Y1LcpyScpOWWZSu7byRM+Hifv\naY+41ANqe6hOoJHbvQPUHvFJCTTC2EpRjr7fjLIEJtst10aSAKP+3sXmllOQKZtkzn6ZSho8KGOv\n87qoQBWHGSnJFpUZ9hV5xDN1twIzY220+WOeN/CcTwP4dKL8uwAuC66vA3DdRvq2SXKKrN/yXzZU\nf+V5b57RSIxl47GHv4t7v/IFHNq/D0cdezye+eJX4IRTz1j0sGaObUs0DKOV/Y/uxed3/Q7u/tIN\nYK6Nfddf+Tt4xotfidf82rtw3EmnLHCEs4W5Puitr8xlkpy0FYiIfgbA21ElTH4cwK8w8zfmMbaZ\n8FtbvL8AUmFD2ulSqPyVqRMcc3X1shsIlu9qm6P0Wy8Zg7Ahtc2RXB2dQEPqperq5XcYluTPGM+G\nDcXl1b14CS7bHdf3PYJP/dufx76H9gBYAfBaAE8HcDe4vAZ33vSX+N7f34U3/6c/wQknVxOlLH9T\njqGB2p4oy/C2YHUdGD5QS/W2sKFpcCQoyZk7bjpuBbofwI8x83MAvBvALhhGz/niH/9HN0FeAOA+\nAJ8E8B73ej+AC/Do93bjMx949wJHOWN4Po6bWTIPJTlxKxAzfymofzOq+CVjwUgYT+McHNlyGKjC\n+uycMqqbC0QH6mD0roHoQGLrYhEHnOstjkBTHeq64bk7Xom69pOUJdBUl4OC8MTef8B9X/5rVAry\nGjQ/0jsBXA3gXHzzb27AQw/uwfGnnNFQi7rfLq9APlh93d3XziCgqTKnwZGQdHceIUCprUBnttT/\nJQCfmemIDGPG7PnvNzob5KuR/59/FoDLwVziW1/+/PwGN0fYlOR0IaKXoZokX5K5fwWAKwAAK8fM\nb2BPcnJKLxUCpLc55gLRAYBXYkWq7ZapsKTGeTtaSSZOcvR1lAqVuqGa0ol/yXWTU7BA2m75xL7H\n3N2no53zAQCPP/YYDqyNvRocblFJenXLsf1SvtcylWxX2TqnBS/xBNiFeUySnbYCEdEPA/gwgEuZ\n+fupjtw+zl0AUGzf0e+fvHFEs7L9OPfu7gk17wEAbDv2+JmOZ1Ew17k/+8o8JsmJW4GI6GwAnwLw\ns8x8zxzGZGyB1AmP2uPttznq8kSb3EmOyTbKE+7bZpL9VnVEbcaB6JQIPNcJNOq21f1xEdsoU3XH\nJeMpz3oxQL8P8LWoLEypJfduANeAqMBTn/e/4OB6rSTDxBMSQjOcoCBXE0HxE9VnqKLdjyylMjcP\nR/bhPjJzm2RuKxARvYWI3uKq/SaAkwF8kIhuUZlADKN3HH3S6Tj9+S9F5Sq5HLFZHu76NQBGOPvC\nl+HYHankNkcGXHKnr2VlLjbJ1FYgZr4yeP/LAH55HmOZC7+16AHMHx1n2YixFJUYxVaWrm76kLIu\nKQ/wnrsAAA3FSURBVNlysZuhepFErLkYyzCBhlai2uaplWVb3X/0v/4aHvvOPTi49+sAzkU1WZ6P\naol9DYARjjnlTDz3Z96Gg2uj6udTNLcwyrbHAaXV4LZE7GZX+2X4nFTZVmEGxqPlnQC7sFSOG8M4\nkth2/Ml4wb/6IL71yT/AQ7fcCPBV9U0q8NQLXoYLf/Zf46gTTl7cIOdA35fbNklOEduLbWi2HX8y\nLrjiPTj4yEPYe9vfYP3APqxsPx47X/BSbD/ptMiLfURijhvDSKMD0YUwBGjQWCqvJOtykOOya7hQ\nmLdSTnBshACp4PKozNXVS3WfhSjcKulyWOq64fK7OHYHTvvHr/FtaFjgYBDyUztnmrknJadlw1Ez\naAaxC5OcPPNabgPLbW/sgk2ShmHMDIbFSRpGKzmHTnhPO19yDp2wbqOPoSoPg9ZV4gyvJEWBhWeB\nq2D0XCB6GEzecOpIdvQBRfeLINTocCJsCOgWziOJNHJhPkDt5JF+JinLXNmW4fpn2ldskjQMY6aU\nPd+7bZOkMVfCs7ob9kp9nbBV6u2Nua2NbWnc6gQdzo4ZKB1tc6yv43ChYtBUktrGKdeplG++TE5h\nFJtkSzhPKpFGfH/QaDMpfGg4YyXJzL133NgZN4ZhzBRm7vS1FYjop4noDiIqiSh74iIRfZuIbtvI\nphVTksbCmBiAvtLcltiwPaq6ujzVr6jQuv8gWF22Nw7SqlBfA/Dng5fKxim2OFGjZZjg1peN4zbS\nf+ipzihIbW9Mq0/lGVcJL8YJJTntsCSez2r7dgA/BeCPOtR9GTPv7dqxTZKGYcyMeSW4YOa7gDoM\na5rYJGnMHF7bj/Kx7wDjNWCwiuKEs0GrxzbqZWMr1TWwydjKDipUe8u1wtMqsWoTn7rI7J6XsW9G\nZV6hlu57R1SeevZAKcfUaYxaXdb2xiJuw4GSTKjLLcNL57hhADcQ0RjAH7nMYq3YJGnMDF4/gPGe\nm8GPfQfVZ7OifPAroBPOxmDni0Ar2xc3QGMO8EZCgHYoO+GucBIjohsApDKBvJOZr+n4jJcw84NE\ndCqAzxHRN5n5xrYGNkkaM4HXD2D0reuAtcdRHWHwasghWMA14McewOjgIxie909AK0fHbeX4hoyy\nBOrkvnWbOLayvW46tjL1LLFXyq4dIXXEQdN+6cak4iXjOsoTzhS1rZ6tYjVlp01GLQJNdVnHSbpj\nKUrZ2VM02/AUvdvYUDD5XmbOOl2Y+ZVbHg/zg+71YSL6NKrjZWySNObPeM/NboK8AM0zXvYAuBxY\n+zrGe/4Ww3NevpAxGnOAl2fHDREdA6Bg5sfd+x8H8NuT2lkIkDF1eG2/W2JPOgRrCH7sO+C1/fMe\nojFHypI7fW0FInotEe0B8KMA/oKIrnflTyUiSdN4GoCbiOgbAL4C4C+Y+bOT+jYlaUyd0tsgJx+C\nBVyF8rHdGJzyjEaNVAZ0gcp4WVxkAtGBRNhQh7q50KLSL7/zjpVc6I8sm7u0oWg5L2UqQYc4hoqm\nw0U7dcaZIHJZfldt5IyepnljK8wjVRozfxrApxPl3wVwmXt/H4DnbrRvmySN6TOWXTXdDsHC+PAs\nR2MsEGb2x/L2FZskjekzEPXV7RAsDLZ17rprmFDK2ZNz2CQdQ5kUb7WyrPvQTpzGtQ8iD87S8W/L\nCW1S4UKupQpEp7CN6k9vOdTKEkgHpU+D1EqgT5hN0pg6xQlnAyAAcghWiuoQLIBQnHBWpo7Re5jB\n5bjT17JiStKYOrR6LOiEs8GPPYDK7ng14lOF60Ow6IQfSAaWT2JSmJAO+6nqpJd9qbraxtkMDQpS\nsXllKsl9YxWnVV1cFif51bZKACDVn9grC1/eVKMyhlwgeiqZhajLbcPpaScGL/UE2AWbJI2ZMNj5\nIowOPgKs5Q/BwupxGOz80UUO05g1HOf27CM2SRozgVa2Y3jeZcGOm6vCu6ATfgCDnT/aCCTfKFql\niLIsApvhGFWZbEssR2qsZcJ+OUFJptrk6zaVZFYdKltlVZa2VzbUaCCUxROu22pPeCoA/fBUHS2m\nJA0jC61sx/Ccl7u927srL/ZgG4oTztrUEtvoIcxRDtE+YpOkMXNo9dhkHOQ02UpMJRXNNuLFHo+0\nCnVKNfCQl+vxJFArSXdsRMIm2VUdAs04Q6mTU6NAaK/0o3R9ubaibgP749qoeQjZVjGbpGEYRhvc\n/xAgmySNI5JwiSeqzys+iXVMqcJRThWmr7vUqe2b9Z9bzgM+cAoyDMCuFaJLp+Z38Li64t0ObZ/K\nQy32SrFV+rbBc1I7d7YO+3jOvmKTpGEYs4NtuW0YhpGFmRs2275hk6RxxKOdLblld1Qn00Y7dIC8\nU6fZBkGb6k9vrJbQQuiUqZfgboy5LY3hEtsto72TJ7OcTwWgD6a8Ec+UpGEYRg5bbhtGf5ioKIGG\nM2eSOgQmhwlJm/RWyTj1mii8cFtiwzGT2zaYcMJoVej7SmQfL1Sd6WCTpGEYRpbq+AZLlWYYvSKr\nKIFseFBbm40EnvsxIL4n9sZUaI5PzCtST+7pMJ8g6NzbHsXmqeyNqWlLhwlNBVtuG4ZhtMAcObn6\niE2SxpOWlMKRjDVeaU1QlEDe8z0NRVmVZbzYSlEWCNKeue8ga5NM2De91zyRkGOzMPqfBWguSXeJ\n6BIiupuI7iWidyTuExG9z92/lYgumMe4DMOYMXNKuktEv0dE33Tzx6eJ6MRMvda5KMXMlSQRDQB8\nAMCrUKWp/ioRXcvMdwbVLgVwnvu6CMCH3KthzIzUH6ZOtSaKklTdZNuOihJIHU4W9xfHVMqBYjIW\nl6C3xUYpqjKnKBttg/bFVKXT3GySnwPwG8w8IqLfBfAbAN4eVug4FzWYh5J8IYB7mfk+Zl4D8AlU\nGVhDLgfwMa64GcCJRHTGHMZmGMaMmYeSZOa/ZGb513Iz0sd0dpmLGszDJnkmqnz9wh40VWKqzpkA\nvhdWIqIrAFzhLg+v3/Jfbp/uUGfGDgB7Fz2IDdCn8fZprEC/xjvpuMuJ8IG91699/cM7OlY/ioi+\nFlzvYuZdm3jsLwL4r4nyLnNRg145btwPbBcAENHXmPnCBQ+pE30aK9Cv8fZprEC/xqsmrE3BzJdM\nYywAQEQ3ADg9ceudzHyNq/NOACMAH5/Wc+cxST6I+BSona5so3UMw3gSw8yvbLtPRL8A4CcBvIJ1\npuKKTc0z87BJfhXAeUR0DhGtAngDqrNGQ64F8HPOy/0iAI8x8/d0R4ZhGCmI6BIA/xeAVzPzgUy1\nLnNRg5krSedteiuA6wEMAHyEme8gore4+1cCuA7AZQDuBXAAwJs7dL0ZW8Wi6NNYgX6Nt09jBfo1\n3j6N9f0AtgH4HBEBwM3M/BYieiqADzPzZbm5aFLHlFalhmEYBjCnYHLDMIy+YpOkYRhGC0s/SfZp\nS2OHsf6MG+NtRPQlInruIsYZjKfTFi0i+hEiGhHR6+Y5PjWGiWMloouJ6BYiuoOI/tu8x6jGMumz\ncAIR/TkRfcONt4sdfiYQ0UeI6GEiSsYdL9Pf2EJg5qX9QmVc/XsA56I6xPgbAJ6p6lwG4DMACMCL\nAHx5icf6YgBPce8vXdRYu443qPfXqJxrr1vWsQI4EcCdAM5216cu888WwL8B8Lvu/SkAHgGwuqDx\nvhTABQBuz9xfir+xRX0tu5Ls05bGiWNl5i8x86PuMrd1al503aL1LwBcBeDheQ5O0WWsbwLwKWb+\nDgAw87KPlwEcR5Ur9lhUk+QIC4CZb3TPz7Esf2MLYdknydx2xY3WmQcbHccvofrvvCgmjpeIzgTw\nWlQJRxZJl5/t+QCeQkRfIKK/I6Kfm9vomnQZ7/sBPAPAdwHcBuBfMvOypvBelr+xhdCrbYlHCkT0\nMlST5EsWPZYJ/AGAtzNz6WLPlpkhgBcAeAWAowH8LRHdzMz3LHZYWX4CwC0AXg7gB1HF932Rmfct\ndliGZtknyT5taew0DiL6YQAfBnApM39/TmNL0WW8FwL4hJsgdwC4jIhGzHz1fIbo6TLWPQC+z8xP\nAHiCiG4E8FwAi5gku4z3zQDey5XR714iuh/ADwH4ynyGuCGW5W9sMSzaKNr2hWoSvw/AOagN4M9S\ndf4JYqPyV5Z4rGej2lX04j78bFX9j2JxjpsuP9tnAPgrV3c7gNsBPHuJx/shAL/l3p+GatLZscDP\nw9OQd9wsxd/Yor6WWkny7LY0LmqsvwngZAAfdOpsxAvKCNNxvEtBl7Ey811E9FkAt6LKlfthZl5I\nKr2OP9t3A/goEd2GavJ5OzMvJIUaEf0pgIsB7CCiPQDeBWAlGOtS/I0tCtuWaBiG0cKye7cNwzAW\nik2ShmEYLdgkaRiG0YJNkoZhGC3YJGkYhtGCTZKGYRgt2CRpzAwi+jwRvcq9/w9E9J8XPSbD2ChL\nHUxu9J53AfhtIjoVwPMBvHrB4zGMDWPB5MZMcclvjwVwMTM/TkTHAPgggDUAX2DmqZ2PbBizwJbb\nxswgoucAOAPAGjM/7op/CsAnmfmfw5Sl0QNskjRmgkvK+nFUCVv3u3ORgSqDjOQmHC9ibIaxEWyS\nNKYOEW0H8CkAb2Pmu1Alc3iXu70HdUZ2+/wZS4/ZJI254myS7wdwCMBNZpM0lh2bJA3DMFqw5Y5h\nGEYLNkkahmG0YJOkYRhGCzZJGoZhtGCTpGEYRgs2SRqGYbRgk6RhGEYLNkkahmG0YJOkYRhGC/8/\nbDifvUSzBIQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1144bfac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X = np.array([[0.5, 0.1], [0.8, 0.2], [0.4, 0.4], [0.6, 0.6], [0.7, 0.8]])\n",
    "y = np.array([-1, -1, -1, 1, 1])\n",
    "y_pos = (y == 1)\n",
    "y_neg = (y == -1)\n",
    "\n",
    "svc = SVC(kernel='rbf', gamma=1 / (2 * h**2), C=10)\n",
    "svc.fit(X, y)\n",
    "SVs = svc.support_vectors_\n",
    "w = svc.dual_coef_[0]\n",
    "b = svc.intercept_\n",
    "\n",
    "for i in range(g):\n",
    "    for j in range(g):\n",
    "        im[i, j] = decision_function((grid[i], grid[j]), SVs)\n",
    "\n",
    "plt.pcolor(grid, grid, im.T, cmap='RdBu_r', vmin=-2, vmax=+2)\n",
    "plt.colorbar()\n",
    "\n",
    "plt.plot(X[y_neg, 0], X[y_neg, 1], '.', c='b', markersize=20, mew=2, mec='k')\n",
    "plt.plot(X[y_pos, 0], X[y_pos, 1], '.', c='r', markersize=20, mew=2, mec='k')\n",
    "plt.plot(q[0], q[1], 's', c='g', markersize=10, mew=2, mec='k')\n",
    "plt.xlabel(r\"$x_0$\")\n",
    "plt.ylabel(r\"$x_1$\")\n",
    "plt.axes().set_aspect('equal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the three SVs, we now have a non-linear boundary.\n",
    "\n",
    "It's interesting to see that the weights still sum to 0: two smaller negative weights for the two SVs labelled -1, and one larger positive weight for the SV labelled +1. The bias is now non-zero also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1.70835027 -5.90998055  7.61833082]\n",
      "[-0.43687287]\n"
     ]
    }
   ],
   "source": [
    "print(w)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the numerical values of the prediction for point $q$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svc.predict([q])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-2.28748131])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svc.decision_function([q])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict(q, SVs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-2.28748131])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decision_function(q, SVs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that our values are matching those from the `sklearn` SVM (phew), and the prediction for $q$ is negative as it should be."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is the example misleading?\n",
    "---\n",
    "\n",
    "Now that we understand the nature of the features created implicitly by the SVM, let's revisit the example at the top of the notebook, where one class of data is enclosed by a hypersphere in the original space. Can the SVM create a feature which directly represents \"distance from the centre\"? Well, nearly. Consider the unit circle in 2 dimensions. Let's suppose there is an SV *at the origin*. Then with a Gaussian kernel, one of the new features will be the *similarity to the origin*, that is a transformation of distance from the origin, which is just $\\sqrt{x_0^2 + x_1^2}$, just as in the example. More realistically, if the centre of the circle is $c=(c_0, c_1)$ (and one SV is at $c$) rather than at the origin, the formula becomes $\\sqrt{(x_0 - c_0)^2 + (x_1 - c_1)^2}$. It's important to think of this new feature as a distance from an SV, not as a transformation of the existing features. But still, it would nicely separate the data inside the circle from that outside. \n",
    "\n",
    "But the problem is that *there would never be an SV at the centre*! SVs occur near the decision boundary. So the common example is definitely slightly misleading."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conclusion\n",
    "---\n",
    "\n",
    "So, an SVM can be seen as an algorithm which decides the label of a query point by testing whether it is more similar in weighted average to a subset (the SVs) of the +1 examples, or to a subset (the SVs) of the -1 examples. Similarity is measured by the kernel, based ultimately on an inverse mapping of Euclidean distance. In this sense, an SVM is closely related to techniques such as $k$ nearest neighbours, kernel density classification, and dissimilarity representations (eg Duin & Pekalska, Pattern Recognition Letters 33 (2012) 826–832). \n",
    "\n",
    "In particular, we could make a crude proto-SVM by just calculating the mean of distances (in the original space) from $q$ to all the +1 points, and to all the -1 points, and finding out which is larger! This would leave out the kernel and the choice of SVs, but otherwise would be quite similar to an SVM. We could call it a mean-distance classifier. The computational complexity at query time would scale linearly with the size of the training data, which is a disadvantage relative to an SVM. If we used a kernel instead of just raw distance, it would become a kernel density classifier, with the same computational complexity disadvantage. Since similarity measured by a Gaussian kernel goes exponentially towards 0, this would give less emphasis to distant points than the mean-distance clasifier, hence would be a bit more robust to outlier points. If we go on to chose a subset of points to be the \"prototypes\" (the term used in the dissimilarity representation literature), instead of using all training data, the complexity would improve, and with a careful choice of these prototypes the generalisation would actually get better. The final refinement would be to define soft constraints that each labelled point must be on the correct side of the hyperplane, and an objective function which maximises the margin, and optimise using quadratic programming. This would make a careful choice of prototypes (now called SVs) and weights $w$ -- and then it is an SVM again.\n",
    "\n",
    "**Acknowledgements** Thanks to Stefano Mauceri and Loi Van Cao for useful discussions on these topics and to Sean McGarraghy for excellent notes in the UCD MIS40530 module."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}