线性回归/12. 利用梯度下降法解线性回归.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一元线性回归的梯度下降\n",
    "\n",
    "下面使用梯度下降算法来求解前面的线性回归问题，我们知道，对一元线性回归，损失函数为：\n",
    "\n",
    "$$\n",
    "loss = J(a, b) = \\frac{1}{2m} \\sum_{i=1}^{m}(y_i - (ax_i + b))^2\n",
    "$$\n",
    "\n",
    "这是一个二元二次方程，我们对该方程求梯度，也就是分别对参数 a 和 b 求偏导，得到参数的更新公式：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "a &:= a - \\eta \\frac{\\partial J(a, b)}{\\partial a} \\\\\n",
    "b &:= b - \\eta \\frac{\\partial J(a, b)}{\\partial b} \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "将损失函数带入上式求导，得到：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "a &:= a - \\eta \\frac{1}{m} \\sum_{i=1}^{m}(y_i - (ax_i + b))x_i \\\\\n",
    "b &:= b - \\eta \\frac{1}{m} \\sum_{i=1}^{m}(y_i - (ax_i + b))\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "我们使用代码来实现这个过程，先来看一个最简单的例子，手工造几个散点，然后使用梯度下降法来求解线性回归。如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADetJREFUeJzt3X+M5PVdx/HXy70bsJRY8LZ64bguJI2mGCu4IQ40OrCNpVctGv+hsaYo5oK/QqPR2JD465/1L22MJuakxDZWaKXFVCKm58CEVOaO7CHcHT1o4UqVHPGW0hYuJoysb/+Y77Zfxtmd79zN97v75p6PZLPf+X4/3/m+9rNfXvvd78wejggBAPL4nq0OAACYDsUNAMlQ3ACQDMUNAMlQ3ACQDMUNAMlQ3ACQDMUNAMlQ3ACQzI46nnTXrl2xsLBQx1MDwJvSkSNHXoqI+SpjaynuhYUFrays1PHUAPCmZPvrVcdyqwQAkqG4ASAZihsAkqG4ASAZihsAkqG4ASAZihsAZqDf72t5eVn9fr/2Y9XyPm4AOJ/0+30tLS1pMBio1Wqp2+2q3W7XdjyuuAHgHPV6PQ0GA62trWkwGKjX69V6vEpX3Lafl/SqpDVJr0fEYp2hACCTTqejVqv1nSvuTqdT6/GmuVVyQ0S8VFsSAEiq3W6r2+2q1+up0+nUeptE4h43AMxEu92uvbDXVb3HHZK+aPuI7f11BgIAbK7qFff1EXHK9tslHbT9dEQ8Uh5QFPp+Sdq7d++MYwIA1lW64o6IU8Xn05Lul3TtmDEHImIxIhbn5yv9k7IAgLMwsbhtX2T74vVlST8t6XjdwQAA41W5VfIDku63vT7+7yPiX2pNBQDY0MTijoiTkt7dQBYAQAX85SQAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AylYvb9pztf7f9QJ2BAACbm+aK+w5JJ+oKAgCoplJx294j6QOS7qo3DgBgkqpX3B+X9HuS/rfGLACACiYWt+2fkXQ6Io5MGLff9ortldXV1ZkFBAC8UZUr7uslfdD285LulXSj7b8bHRQRByJiMSIW5+fnZxwTALBuYnFHxMciYk9ELEi6RdJDEfHh2pMBAMbifdwAkMyOaQZHRE9Sr5YkAIBKuOIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobpw3+v2+lpeX1e/3tzoKcE52bHUAoAn9fl9LS0saDAZqtVrqdrtqt9tbHQs4K1xx47zQ6/U0GAy0tramwWCgXq+31ZGAszaxuG1faPsx20/afsr2HzcRDJilTqejVqulubk5tVotdTqdrY4EnLUqt0pek3RjRJyxvVPSl2w/GBGHas4GzEy73Va321Wv11On0+E2CVKbWNwREZLOFA93Fh9RZyigDu12m8LGm0Kle9y252w/Iem0pIMRcbjeWACAjVQq7ohYi4gfk7RH0rW2f2R0jO39tldsr6yurs46JwCgMNW7SiLiW5J6km4as+1ARCxGxOL8/PyM4gEARlV5V8m87bcVy98r6b2Snq47GABgvCrvKtkt6ZO25zQs+s9GxAP1xgIAbKTKu0qOSrq6gSwAgAr4y0kASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkJha37cttP2z7hO2nbN/RRDAAwHg7Kox5XdLvRMTjti+WdMT2wYj4cs3ZAABjTLzijogXI+LxYvlVSSckXVZ3MADAeFPd47a9IOlqSYfrCAMAmKxycdt+q6TPSfpoRLwyZvt+2yu2V1ZXV2eZEQBQUqm4be/UsLQ/HRGfHzcmIg5ExGJELM7Pz88yIwCgpMq7SizpE5JORMSf1R8JALCZKlfc10v6JUk32n6i+NhXcy4AwAYmvh0wIr4kyQ1kAQBUwF9OAkAyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyE4vb9t22T9s+3kQgAMDmqlxx/62km2rOAQCoaGJxR8Qjkl5uIAsAoIKZ3eO2vd/2iu2V1dXVWT0tAGDEzIo7Ig5ExGJELM7Pz8/qaQEAI3hXCQAks62Ku9/va3l5Wf1+f6ujAMC2tWPSANv3SOpI2mX7BUl/GBGfmHWQfr+vpaUlDQYDtVotdbtdtdvtWR8GANKr8q6SD0XE7ojYGRF76ihtSer1ehoMBlpbW9NgMFCv16vjMACQ3ra5VdLpdNRqtTQ3N6dWq6VOp7PVkQBgW5p4q6Qp7XZb3W5XvV5PnU6H2yQAsIFtU9zSsLwpbADY3La5VQIAqIbiBoBkKG4ASIbiBoBkKG4ASIbiBoBkHBGzf1J7VdLXz3L3XZJemmGcWSHXdMg1HXJN582Y6x0RUemfVq2luM+F7ZWIWNzqHKPINR1yTYdc0znfc3GrBACSobgBIJntWNwHtjrABsg1HXJNh1zTOa9zbbt73ACAzW3HK24AwCYaK27bN9l+xvaztn9/zPYLbH+m2H7Y9kJp28eK9c/Yfl/DuX7b9pdtH7Xdtf2O0rY1208UH19oONettldLx//V0raP2P5q8fGRhnP9eSnTV2x/q7Stzvm62/Zp28c32G7bf1HkPmr7mtK2OudrUq5fLPIctf2o7XeXtj1v+1gxXysN5+rY/nbp+/UHpW2bngM15/rdUqbjxTl1abGtzvm63PbDtk/Yfsr2HWPGNHeORUTtH5LmJD0n6UpJLUlPSnrXyJhfl/TXxfItkj5TLL+rGH+BpCuK55lrMNcNkt5SLP/aeq7i8ZktnK9bJf3lmH0vlXSy+HxJsXxJU7lGxv+WpLvrnq/iuX9S0jWSjm+wfZ+kByVZ0k9IOlz3fFXMdd368SS9fz1X8fh5Sbu2aL46kh4413Ng1rlGxv6spIcamq/dkq4pli+W9JUx/002do41dcV9raRnI+JkRAwk3Svp5pExN0v6ZLF8n6Ql2y7W3xsRr0XE1yQ9WzxfI7ki4uGI+O/i4SFJe2Z07HPKtYn3SToYES9HxDclHZR00xbl+pCke2Z07E1FxCOSXt5kyM2SPhVDhyS9zfZu1TtfE3NFxKPFcaXmzq8q87WRczk3Z52ryfPrxYh4vFh+VdIJSZeNDGvsHGuquC+T9J+lxy/o/3/R3xkTEa9L+rak76+4b525ym7T8Cfqugttr9g+ZPvnZpRpmly/UPxKdp/ty6fct85cKm4pXSHpodLquuario2y1zlf0xo9v0LSF20fsb1/C/K0bT9p+0HbVxXrtsV82X6LhuX3udLqRubLw9u4V0s6PLKpsXOsqf8DjsesG307y0Zjqux7tio/t+0PS1qU9FOl1Xsj4pTtKyU9ZPtYRDzXUK5/knRPRLxm+3YNf1u5seK+deZad4uk+yJirbSurvmqYivOr8ps36Bhcb+ntPr6Yr7eLumg7aeLK9ImPK7hn2Cfsb1P0j9Keqe2yXxpeJvk3yKifHVe+3zZfquGPyw+GhGvjG4es0st51hTV9wvSLq89HiPpFMbjbG9Q9L3afgrU5V968wl2++VdKekD0bEa+vrI+JU8fmkpJ6GP4UbyRUR3yhl+RtJP1513zpzldyikV9ja5yvKjbKXud8VWL7RyXdJenmiPjG+vrSfJ2WdL9md4twooh4JSLOFMv/LGmn7V3aBvNV2Oz8qmW+bO/UsLQ/HRGfHzOkuXOsjhv5Y27s79DwhvwV+u4LGleNjPkNvfHFyc8Wy1fpjS9OntTsXpyskutqDV+MeefI+kskXVAs75L0Vc3oRZqKuXaXln9e0qH47gshXyvyXVIsX9pUrmLcD2n4QpGbmK/SMRa08YttH9AbXzh6rO75qphrr4av21w3sv4iSReXlh+VdFODuX5w/funYQH+RzF3lc6BunIV29cv6i5qar6Kr/1Tkj6+yZjGzrGZTXaFL3yfhq/EPifpzmLdn2h4FStJF0r6h+IkfkzSlaV97yz2e0bS+xvO9a+S/kvSE8XHF4r110k6Vpy4xyTd1nCuZUlPFcd/WNIPl/b9lWIen5X0y03mKh7/kaQ/Hdmv7vm6R9KLkv5Hwyuc2yTdLun2Yrsl/VWR+5ikxYbma1KuuyR9s3R+rRTrryzm6sni+3xnw7l+s3R+HVLpB8u4c6CpXMWYWzV8w0J5v7rn6z0a3t44Wvpe7duqc4y/nASAZPjLSQBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGQobgBIhuIGgGT+D5OQ3FaX6WqLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "X = np.array([0,1,2])\n",
    "Y = np.array([0.9, 3.1, 5.1])\n",
    "\n",
    "plt.plot(X, Y, 'k.')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很显然这是一个一元线性回归，根据上面的介绍，我们可以定义出损失函数和参数 a, b 的更新公式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 损失函数\n",
    "def loss(X, Y, a, b):\n",
    "    s = 0\n",
    "    for i in range(X.size):\n",
    "        s += ((a*X[i] + b)-Y[i])**2\n",
    "    return s * (1./2*Y.size)\n",
    "\n",
    "# 参数 a 的更新公式\n",
    "def update_a(X, Y, a, b, eta):\n",
    "    s = 0\n",
    "    for i in range(X.size):\n",
    "        s += ((a*X[i] + b)-Y[i])*X[i]\n",
    "    return a - eta * s * (1./Y.size)\n",
    "\n",
    "# 参数 b 的更新公式\n",
    "def update_b(X, Y, a, b, eta):\n",
    "    s = 0\n",
    "    for i in range(X.size):\n",
    "        s += ((a*X[i] + b)-Y[i])\n",
    "    return b - eta * s * (1./Y.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们选取参数的初始值 $(a, b) = (0, 0)$，并令学习率 $\\eta = 0.01$，然后进行第一次迭代，并计算损失函数的值和上次损失函数的值之间的差值（这个差值可以用于判断梯度下降过程是否已经收敛，如果两次损失函数的差值足够小，就认为收敛）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a = 0.04433333333333332, b = 0.03033333333333333, loss = 52.07898433333334, diff = 2.566015666666658\n"
     ]
    }
   ],
   "source": [
    "a = 0\n",
    "b = 0\n",
    "eta = 0.01\n",
    "\n",
    "a_new = update_a(X, Y, a, b, eta)\n",
    "b_new = update_b(X, Y, a, b, eta)\n",
    "loss_new = loss(X, Y, a_new, b_new)\n",
    "diff = abs(loss(X, Y, a, b) - loss(X, Y, a_new, b_new))\n",
    "print(\"a = {0}, b = {1}, loss = {2}, diff = {3}\".format(a_new, b_new, loss_new, diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们不妨画出迭代一次后的图像，很显然，这个差值还比较大，我们需要继续这个过程："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADy5JREFUeJzt3X9s3PV9x/HXK7bPzg/bpI2BLD8wiGgMKnUgi/WGNN1IN7G1Ktsfk6jUqpom5Z/9oBNS1e6fan+lf0xV+0c1NQJW1DKqilINoa4tcntClW6sDrAVCNMqBm1GtphWJCEsudh+7w+fg+PYvq+Jv3f3vjwfUuS7y9f+vs8JT775fL/nc0QIAJDHlm4PAADYGMINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASCZwTK+6K5du2JycrKMLw0Afeno0aNvRsREkW1LCffk5KRmZmbK+NIA0Jdsv150W5ZKACAZwg0AyRBuAEiGcANAMoQbAJIh3ACQDOHuskajocOHD6vRaHR7FABJlHIdN4ppNBo6ePCgms2mKpWKpqenVa1Wuz0WgB5X6Ijb9mu2f2r7Bdu8smaT1Ot1NZtNzc/Pq9lsql6vd3skAAls5Ij7dyPizdImuQrVajVVKpWLR9y1Wq3bIwFIgKWSLqpWq5qenla9XletVmOZBEAhRcMdkn5gOyR9NSKOlDjTVaVarRJsABtSNNx3RcQbtq+V9LTtVyLimeUb2D4k6ZAk7d+/f5PHBAAsKXRyMiLeaH08Kek7ku5cZZsjETEVEVMTE4V+MiEA4D1oG27b222PLt2W9PuSXix7MADA6ooslVwn6Tu2l7b/x4j4XqlTAQDW1DbcEfGqpA92YBYAQAG85B0AkiHcAJAM4QaAZAg3ACRDuAEgGcINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiGcANAMoQbAJIh3ACQDOEGgGQINwAkQ7gBIBnCDQDJEG4ASIZwA0AyhBsAkiHcAJAM4QaAZAg3ACRDuAEgGcINAMkUDrftAdvP236qzIEAAOvbyBH3/ZKOlTUIAKCYQuG2vVfSRyQ9WO44AIB2ih5xf0nSZyQtlDgLAKCAtuG2/VFJJyPiaJvtDtmesT0zOzu7aQMCAC5V5Ij7Lkkfs/2apG9Kutv2N1ZuFBFHImIqIqYmJiY2eUwAwJK24Y6Iz0XE3oiYlHSfpB9GxCdKnwwAsCqu4waAZAY3snFE1CXVS5kEAFAIR9wAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiGcANAMoQbAJIh3ACQDOEGgGQINwAkQ7gBIBnCDQDJEG4ASIZwA0AyhBsAkiHcAJAM4QaAZAg3ACRDuAEgGcINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwi2p0Wjo8OHDajQa3R4FANoa7PYA3dZoNHTw4EE1m01VKhVNT0+rWq12eywAWFPbI27bI7b/1fa/2X7J9t92YrBOqdfrajabmp+fV7PZVL1e7/ZIALCuIkfc5yXdHRFv2x6S9GPb/xwR/1LybB1Rq9VUqVQuHnHXarVujwQA62ob7ogISW+37g61fkWZQ3VStVrV9PS06vW6arUayyQAel6hNW7bA5KOSrpZ0lci4tlSp+qwarVKsAGkUeiqkoiYj4jflLRX0p22P7ByG9uHbM/Ynpmdnd3sOQEALRu6HDAi3pJUl3TPKr93JCKmImJqYmJik8YDAKxU5KqSCdvXtG5vlfRhSa+UPRgAYHVF1rh3S3qktc69RdK3IuKpcscCAKylyFUl/y7p9g7MAgAogJe8A0AyhBsAkiHcAJAM4QaAZAg3ACRDuAEgGcINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiGcANAMoQbAJIh3ACQDOEGgGQINwAkQ7gBIBnCDQDJEG4ASIZwA0AyhBsAkiHcAJAM4QaAZAg3ACRDuAEgmbbhtr3P9o9sH7P9ku37OzEYAGB1gwW2mZP0QEQ8Z3tU0lHbT0fEyyXPBgBYRdsj7og4ERHPtW6fkXRM0p6yBwMArG5Da9y2JyXdLunZMoYBALRXONy2d0j6tqRPR8TpVX7/kO0Z2zOzs7ObOSMAYJlC4bY9pMVoPxoRT6y2TUQciYipiJiamJjYzBkBAMsUuarEkh6SdCwivlj+SACA9RQ54r5L0icl3W37hdavPyx5LgDAGtpeDhgRP5bkDswCACiAV04CQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiGcANAMoQbAJIh3ACQDOEGgGQINwAkQ7gBIBnCDQDJEG4ASIZwA0AyhBsAkiHcAJAM4QaAZAg3ACRDuAEgGcINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASCZtuG2/bDtk7Zf7MRAAID1FTni/pqke0qeAwBQUNtwR8Qzkn7VgVkAAAWwxg0AyWxauG0fsj1je2Z2dnazviwAYIVNC3dEHImIqYiYmpiY2KwvCwBYoaeWShqNhg4fPqxGo9HtUQCgZw2228D2Y5JqknbZPi7p8xHx0GYP0mg0dPDgQTWbTVUqFU1PT6tarW72bgAgvSJXlXw8InZHxFBE7C0j2pJUr9fVbDY1Pz+vZrOper1exm4AIL2eWSqp1WqqVCoaGBhQpVJRrVbr9kgA0JPaLpV0SrVa1fT0tOr1umq1GsskALCGngm3tBhvgg0A6+uZpRIAQDGEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiGcANAMoQbAJIh3ACQDOEGgCt0fu68Tp49qdffer0j++upHzIFAJ00vzCv0+dP6/T50zp1/tTix3On1r+/yuPN+aYk6fod1+vEAydKn5twA0gnIvTOhXeuKLanzp3S2Qtn2+5rwAMaGx7T+Mj44sfhce3esVu37LpFY5XFx8eHF39v17ZdHXj2hBtAh12Yv7BmSC+LbHPtKM/HfNt9bR/afklwx4bHtG9838XbFz+OXH5/6fa2oW2y3YHvTHGEG0AhC7Ggt5tvX4znuke7zbW3OTd3ru2+hrYMXRLP8ZFxTV4zuWZwlx5b/jmjw6Ma3NKfievPZwXgEufmzrVfTlge2lW2OXP+jEKx7n4svxvSVkQntk3o5vfdfHFZod2R7tjwmEYGRzr0ncmJcAM9rN3JsyJruafPn7548mw9Wwe3XhbS63dd/+7RbJvYjg+Pa3tlu7aYi9XKRriBEkSEzl44W+zE2TpruUVPni2P5/jIuPaM7dGtw7cWXscdGx7T0MBQB74z2AyEG1ihOd8sFtt1jnSLnjzbUdlxaUhHxrV/fP+awV1tTXfr4NaeO3mGchFu9I2FWNCZ82eueFmhyMmzykDlsrDeeM2NlwV2vaPd0cqoBrYMdOA7g35DuNF1EaFzc+fWPlFW8JrcM80zbfe18uTZ2PCYrt1+rQ68/0Dhk2fjw+MaHhzuwHcGWB3hxhWZW5jTmfNn1r4Od0Vw19rmwsKFtvsaGRy57JKv3aO7Lw3rKuu3nDxDvyHcV6mNnjxb62h3IyfPlod079jeQi+CWH6fk2fAIsKdUHO+2T6wm3TybLQyeklEd47s1A3jNxS6UmHp9zh5Bmwuwt1B7U6eFV1eKHLybHhg+LKQ3rTzpjWXFVY7qbajsoOTZ0APItwFLJ08K7SUsM41uUVOnm3xlosnz5YCet2O63Tg/QcKr+OODY9x8gzoY4XCbfseSV+WNCDpwYj4QqlTbaK5hbm2R7ZFrlqYW5hru69tQ9suC+vu0d2FX3U2NjymHZUdLCsAWFfbcNsekPQVSb8n6bikn9h+MiJeLnOwpZNn73Udd+n2Oxfeabuv5SfPlkK6b2yfbpu4rVBwl67J5eQZgE4ocsR9p6SfRcSrkmT7m5LulbTp4b7jq3fol//3y4snzxZioe3njFZGL4nozpGdmrxmckM/0IaTZwAyKRLuPZJ+sez+cUm/VcYwt11727s/tLzAVQujw6NckwvgqlMk3Ksdil72sx1tH5J0SJL279//nob5+h9//T19HgBcTYocrh6XtG/Z/b2S3li5UUQciYipiJiamJjYrPkAACsUCfdPJB2wfaPtiqT7JD1Z7lgAgLW0XSqJiDnbfyHp+1q8HPDhiHip9MkAAKsqdB13RHxX0ndLngUAUACXZABAMoQbAJIh3ACQDOEGgGQccdlraa78i9qzkl5/j5++S9KbmzhOBjzn/ne1PV+J57xRN0REoRfBlBLuK2F7JiKmuj1HJ/Gc+9/V9nwlnnOZWCoBgGQINwAk04vhPtLtAbqA59z/rrbnK/GcS9Nza9wAgPX14hE3AGAdPRNu2/fY/g/bP7P92W7PUzbb+2z/yPYx2y/Zvr/bM3WK7QHbz9t+qtuzdILta2w/bvuV1p93tdszlc32X7f+Xr9o+zHbI92eabPZftj2SdsvLnvsfbaftv2frY87y9h3T4R72fta/oGkWyV93Pat3Z2qdHOSHoiI35D0IUl/fhU85yX3SzrW7SE66MuSvhcRt0j6oPr8udveI+mvJE1FxAe0+FNF7+vuVKX4mqR7Vjz2WUnTEXFA0nTr/qbriXBr2ftaRkRT0tL7WvatiDgREc+1bp/R4n/Me7o7Vfls75X0EUkPdnuWTrA9Jul3JD0kSRHRjIi3ujtVRwxK2mp7UNI2rfLmK9lFxDOSfrXi4XslPdK6/YikPypj370S7tXe17LvI7bE9qSk2yU9291JOuJLkj4jqf07QfeHmyTNSvqH1vLQg7a3d3uoMkXEf0v6O0k/l3RC0qmI+EF3p+qY6yLihLR4cCbp2jJ20ivhLvS+lv3I9g5J35b06Yg43e15ymT7o5JORsTRbs/SQYOS7pD09xFxu6SzKumfz72ita57r6QbJf2apO22P9HdqfpLr4S70Pta9hvbQ1qM9qMR8US35+mAuyR9zPZrWlwOu9v2N7o7UumOSzoeEUv/mnpciyHvZx+W9F8RMRsRFyQ9Iem3uzxTp/yv7d2S1Pp4soyd9Eq4r7r3tbRtLa57HouIL3Z7nk6IiM9FxN6ImNTin/EPI6Kvj8Qi4n8k/cL2r7ceOijp5S6O1Ak/l/Qh29taf88Pqs9PyC7zpKRPtW5/StI/lbGTQm9dVrar9H0t75L0SUk/tf1C67G/ab1NHPrLX0p6tHVQ8qqkP+3yPKWKiGdtPy7pOS1ePfW8+vBVlLYfk1STtMv2cUmfl/QFSd+y/Wda/B/Yn5Syb145CQC59MpSCQCgIMINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJPP/RhLBCG6lxZUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(X, Y, 'k.')\n",
    "\n",
    "x = np.linspace(0, 10, 10)\n",
    "y = a_new*x+b_new\n",
    "\n",
    "plt.plot(x, y, 'g-')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们把这个过程改为迭代 10 次："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a = 0.04433333333333332, b = 0.03033333333333333, loss = 52.07898433333334, diff = 2.566015666666658\n",
      "a = 0.08762444444444442, b = 0.059919999999999994, loss = 49.63396493034815, diff = 2.4450194029851886\n",
      "a = 0.12989817037037035, b = 0.08877788888888888, loss = 47.30423372457149, diff = 2.3297312057766604\n",
      "a = 0.17117875530864196, b = 0.11692446162962962, loss = 45.08435194574915, diff = 2.219881778822341\n",
      "a = 0.21148986477053494, b = 0.14437676279358025, loss = 42.969137414855254, diff = 2.115214530893894\n",
      "a = 0.25085459939642357, b = 0.17115142985127244, loss = 40.95365243850431, diff = 2.0154849763509475\n",
      "a = 0.2892955084413038, b = 0.1972647028921288, loss = 39.03319227448755, diff = 1.9204601640167596\n",
      "a = 0.32683460293836075, b = 0.22273243411212781, loss = 37.20327414148798, diff = 1.8299181329995662\n",
      "a = 0.3634933685482668, b = 0.24757009707495625, loss = 35.45962674730114, diff = 1.7436473941868442\n",
      "a = 0.3992927781017128, b = 0.27179279575205734, loss = 33.79818031109894, diff = 1.6614464362021977\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    a_new = update_a(X, Y, a, b, eta)\n",
    "    b_new = update_b(X, Y, a, b, eta)\n",
    "    loss_new = loss(X, Y, a_new, b_new)\n",
    "    diff = abs(loss(X, Y, a, b) - loss(X, Y, a_new, b_new))\n",
    "    print(\"a = {0}, b = {1}, loss = {2}, diff = {3}\".format(a_new, b_new, loss_new, diff))\n",
    "    a = a_new\n",
    "    b = b_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到 loss 和 diff 都变得越来越小，说明损失函数在不断收敛，画出现在的图像："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHlxJREFUeJzt3Xd0VHXCh/Hnl0bovUoHaYoUoxIQCAQVhRUUKyrusorrui6uuiYZIKEkJCCLAhYEUYpgQUWxuxkYmgGkCIKIIiK99xKGJL/3D3CP8qoEyOTOzXw/53hIwjjznUN4uLm5SYy1FhERcY8wpweIiMj5UbhFRFxG4RYRcRmFW0TEZRRuERGXUbhFRFxG4RYRcRmFW0TEZRRuERGXiQjEnVaqVMnWrVs3EHctIlIkLV++fK+1tnJ+bhuQcNetW5dly5YF4q5FRIokY8xP+b2tTpWIiLiMwi0i4jIKt4iIyyjcIiIuo3CLiLiMwi0i4jIKt8OysrJIT08nKyvL6Ski4hIBuY5b8icrK4v4+Hj8fj9RUVF4vV5iY2OdniUiQU5H3A7y+Xz4/X5yc3Px+/34fD6nJ4mIC+TriNsYswk4AuQCOdbamECOChVxcXFERUX974g7Li7O6Uki4gLnc6qkk7V2b8CWhKDY2Fi8Xi8+n4+4uDidJhGRfNE5bofFxsYq2CJyXvJ7jtsCnxtjlhtj+v3WDYwx/Ywxy4wxy/bs2VNwC0VE5FfyG+521trWwI3AI8aYDmffwFo7wVobY62NqVw5X9+ZUERELkC+wm2t3X7m193ALODqQI4SEZHfd85wG2NKGmNK//wycD2wJtDDRETkt+Xnk5NVgVnGmJ9vP8Na+2lAV4mIyO86Z7ittRuBFoWwRURE8kFfOSki4jIKt4iIyyjcIiIuo3CLiLiMwi0i4jIKt4iIyyjcIiIuo3CLiLiMwi0i4jIKt4iIyyjcIiIuo3CLiLiMwi0i4jIKt4iIyyjcIiIuo3CLiLiMwi0i4jIKt4iIyyjcIiIuo3CLiLiMwi0i4jIKt4iIyyjcIiIuo3CLiBSA7Ue28+F3HxbKYyncIiIX4WD2QTxeDw3HNqTPrD4cP3U84I8ZEfBHEBEpgrJzsnl+6fOkLUjjQPYB7ml+D0M7DaVEZImAP7bCLSJyHnLzcpm2ehrJc5PZcngLXRt2JT0+nZbVWhbaBoVbRCQfrLV8+N2HJHmTWLtnLVfVuIopPafQqV6nQt+icIuInMOizYtIyExg0ZZFNKrYiJm3z6RX014YYxzZo3CLiPyOtbvX4pnjYfb62VQrVY3x3cbTt1VfIsMjHd2lcIuInGXLoS2k+FKYsmoKpaJKkdY5jf7X9KdkVEmnpwEKt4jI/+w/sZ/0BemMWzoOi+Wxax7D095DxRIVnZ72K/kOtzEmHFgGbLPWdg/cJBGRwnX81HHGLhlLxsIMDp88TJ8WfRgSN4Q65eo4Pe03nc8Rd39gHVAmQFtERApVTl4Or658lcHzBrP9yHa6N+rO8M7DaV61udPT/lC+wm2MqQl0A9KAxwO6SEQkwKy1zPp2Fh6vh/X71hNbM5Y3er1B+zrtnZ6WL/k94n4WeAoo/Xs3MMb0A/oB1K5d++KXiYgEwLxN80jITGDJtiU0rdSUWXfOokfjHo5d2nchzvm9Sowx3YHd1trlf3Q7a+0Ea22MtTamcuXKBTZQRKQgrN61mm4zuhE3JY6th7fy8p9eZvXDq+nZpKerog35O+JuB9xsjLkJiAbKGGNes9beG9hpIiIXb9PBTSTPTea11a9RNrosI7qM4NGrH6V4ZHGnp12wc4bbWpsEJAEYY+KAJxVtEQl2e4/vJW1+Gi8se4EwE8a/2/6bxGsTKV+8vNPTLpqu4xaRIuWY/xjPLH6GkYtGcuzUMf7S8i8MjhtMzTI1nZ5WYM4r3NZaH+ALyBIRkYtwKvcUL694maHzh7Lz6E56NunJ8M7DaVq5qdPTCpyOuEXE1ay1zPxmJgPmDGDD/g20r92ed+94l9hasU5PCxiFW0Rcy7vRS6I3kWXbl3F5lcv58O4PuenSm1x3lcj5UrhFxHVW7lhJojeRz3/4nNplazO5x2TuveJewsPCnZ5WKBRuEXGNH/b/wMC5A3ljzRtUKF6B0deP5uGrHiY6ItrpaYVK4RaRoLfr6C5S56cyfvl4IsMiGdB+AP9u+2/KRpd1epojFG4RCVpHTh5h1Bej+E/Wf8jOyebB1g+S3DGZ6qWrOz3NUQq3iAQdf66fl5a9xLD5w9hzfA+3N7ud1M6pNKrYyOlpQUHhFpGgkWfzeGPNGwycM5AfD/5Ip7qdGNFlBFddcpXT04KKwi0ijrPW8tkPn5HkTeKrnV/RslpLPr3nU65vcH2Rv7TvQijcIuKoL7d9SUJmAnM3zaVeuXpMv3U6d11+F2HmnN+8NGQp3CLiiO/2fceAOQN4+5u3qVyiMmO7juWhmIeICo9yelrQU7hFpFDtOLKDIfOG8PKKl4mOiCalYwpPxD5B6WK/+3Na5CwKt4gUikPZhxi5aCTPLH6GnLwcHo55mIEdBlK1VFWnp7mOwi0iAZWdk80LX75A2oI09p/Yz92X382wTsNoUKGB09NcS+EWkYDIzcvltdWvkexLZvOhzVzf4Hoy4jNoVb2V09NcT+EWkQJlreWj7z8iyZvEmt1riKkRwys3v0J8/XinpxUZCreIFJgvtnxBQmYCCzcvpGGFhrx121vc1uw2XYtdwBRuEblo3+z5Bo/Xw/vr36dqyaq82O1F/trqr0SGRzo9rUhSuEXkgm09vJWUuSlMXjWZkpElSe2UymNtHqNkVEmnpxVpCreInLf9J/aTsTCDcUvHkWfz6H9NfzztPVQqUcnpaSFB4RaRfDtx6gRjl4wlY1EGh7IPce8V9zKs0zDqlKvj9LSQonCLyDnl5OUw+avJDPYNZtuRbXS7tBvD44dzRdUrnJ4WkhRuEfld1lre+/Y9PHM8fLv3W9rUbMOMXjPoUKeD09NCmsItIr9p/k/zSchMYPHWxTSp1IRZd86iR+MeurQvCCjcIvIrX+/6miRvEh99/xGXlL6EiX+ayJ9b/pmIMOUiWOhPQkQA+OngTyT7kpm2ahplo8uSEZ/Bo9c8SonIEk5Pk7Mo3CIhbu/xvQxfMJznv3weg+HJtk+SeG0iFYpXcHqa/A6FWyREHfMf49nFzzLyi5Ec9R/lzy3+zOC4wdQqW8vpaXIOCrdIiDmVe4pJKycxZN4Qdh7dSY/GPRgeP5xmlZs5PU3ySeEGsrKy8Pl8xMXFERsb6/QckYCw1vL2N28zYM4Avt//PdfWvpZ37niHtrXaOj1NzlPIhzsrK4v4+Hj8fj9RUVF4vV7FW4qcOT/OISEzgWXbl3FZ5cv44O4P6HZpN13a51Ih/2OUfT4ffr+f3Nxc/H4/Pp/P6UkiBWbljpXc8NoNxE+NZ9fRXUzuMZlVf1tF90bdFW0XO+cRtzEmGpgPFDtz+7ettSmBHlZY4uLiiIqK+t8Rd1xcnNOTRC7axgMbGThnIK+veZ0KxSvwn+v/w9+v+jvREdFOT5MCkJ9TJSeBztbao8aYSGChMeYTa+3iAG8rFLGxsXi9Xp3jliJh97HdpM5PZfyy8USEReC51sNT7Z6ibHRZp6dJATpnuK21Fjh65tXIM//ZQI4qbLGxsQq2uNqRk0cYnTWaUVmjOHHqBA+0foDkjsnUKF3D6WkSAPn65KQxJhxYDjQEnrfWLvmN2/QD+gHUrl27IDeKyO/w5/p5adlLDJs/jD3H93Bbs9tI7ZRK40qNnZ4mAZSvcFtrc4GWxphywCxjzOXW2jVn3WYCMAEgJiamSB2RiwSbPJvHm2veZODcgWw8sJG4unGM6DKCqy+52ulpUgjO63JAa+1BY4wP6AqsOcfNRaSAWWv5/IfPSfImsXLnSlpUbcEn93zCDQ1u0FUiISQ/V5VUBk6diXZxoAswIuDLRORXvtz2JYneROb8OIe65eoy7ZZp9G7emzAT8lf1hpz8HHFXB6acOc8dBrxlrf0wsLNE5Gff7/ueAXMGMPObmVQqUYkxXcfw0JUPUSyimNPTxCH5uapkNdCqELaIyC/sOLKDofOGMnHFRKIjoknukMwTbZ+gTLEyTk8Th4X8l7yLBJtD2Yd4+ouneWbxM/hz/fwt5m8M6jCIqqWqOj1NgoTCLRIkTuac5IUvXyBtQRr7TuzjrsvvYlinYTSs0NDpaRJkFG4Rh+Xm5TL96+kMmjuIzYc2c13968jokkHr6q2dniZBSuEWcYi1lo+//5hEbyJrdq/hyupXMunmSXSp38XpaRLkFG4RByzeupiEzATm/zSfhhUa8uZtb3Jbs9t0aZ/ki8ItUojW7VmHZ46H9759j6olq/LCTS/wQOsHiAyPdHqauIjCLVIIth7eymDfYF796lVKRpZkWKdhPNbmMUpFlXJ6mriQwi0SQAdOHCBjYQZjl44lz+bxz6v/iae9h8olKzs9TVxM4RYJgBOnTjBu6TjSF6ZzKPsQ915xL0M7DaVuubpOT5MiQOEWKUA5eTlM+WoKKb4Uth3Zxk2X3kR6fDpXVL3C6WlShCjcIgXAWsv769/H4/Wwbu86rrnkGqbfOp2OdTs6PU2KIIVb5CIt+GkBCZkJZG3NonHFxrx7x7v0bNJT32ZVAkbhFrlAa3avIcmbxIfffUiN0jWY0H0Cf2n1FyLC9NdKAkvvYSLn6aeDP5HiS2HqqqmUKVaGjPgMHr3mUUpElnB6moQIhVskn/Yd38fwBcN57svnMBiebPskidcmUqF4BaenSYhRuEXO4Zj/GGOWjGHEohEc9R/l/hb3MyRuCLXK1nJ6moQohVvkd5zKPcUrK19h8LzB7Dy6kx6Ne5DWOY3Lqlzm9DQJcQq3yFmstbyz7h08Xg/f7/+edrXa8fbtb9Oudjunp4kACrfIr8z9cS4JmQl8uf1LLqt8GbPvmk33Rt11aZ8EFYVbBPhq51ckZiby2Q+fUatMLV7t8Sr3XXEf4WHhTk8T+X8UbglpGw9sZNDcQcz4egblo8sz6rpRPHL1I0RHRDs9TeR3KdwSknYf203q/FTGLxtPRFgESdcm8VS7pygXXc7paSLnpHBLSDly8gijs0YzKmsUJ06d4K+t/kpKXAo1StdweppIvincEhL8uX4mLJ/AsPnD2H1sN72a9iKtcxqNKzV2eprIeVO4pUjLs3m8ueZNBs4dyMYDG+lYpyOz75rNNTWvcXqayAVTuKXI+u8P/yUhM4GVO1dyRdUr+Lj3x3Rt2FWX9onrKdxS5CzbvozEzES8P3qpW64u026ZRu/mvfUT1KXIULilyPh+3/cMnDuQt9a+RaUSlRjTdQwPXfkQxSKKOT1NpEAp3OJ6O4/uZOi8oUxcMZFi4cUY1GEQT7Z9kjLFyjg9TSQgFG5xrcMnD/P0oqcZvXg0/lw//Vr3Y1DHQVQrVc3paSIBpXCL65zMOcmLy14kdX4q+07s487L7iS1cyoNKzR0eppIoVC4xTVy83KZ8fUMBs0dxE+HfqJL/S5kxGdwZY0rnZ4mUqjOGW5jTC1gKlANyAMmWGvHBHqYyM+stXyy4ROSvEms3rWa1tVbM/FPE7muwXVOTxNxRH6OuHOAJ6y1K4wxpYHlxpj/Wmu/CfA2ERZvXUxiZiLzfppHg/INeKPXG9x+2e26tE9C2jnDba3dAew48/IRY8w64BJA4ZaA+Xbvt3i8HmZ9O4sqJavw/E3P80DrB4gKj3J6mojjzusctzGmLtAKWPIbv9cP6AdQu3btApgmoWjb4W0MmTeESSsnUSKyBEPjhvKv2H9RKqqU09NEgka+w22MKQW8AzxmrT189u9baycAEwBiYmJsgS2UkHDgxAFGLBrBmCVjyM3L5dGrH2VA+wFULlnZ6WkiQSdf4TbGRHI62tOtte8GdpKEkuycbJ5b+hzDFwznYPZB7rniHobGDaVe+XpOTxMJWvm5qsQAk4B11trRgZ8koSA3L5epq6aS7Etm6+Gt3NjwRtLj02lRrYXT00SCXn6OuNsB9wFfG2O+OvM2j7X248DNkqLKWsvs9bPxzPHwzZ5vuPqSq5l2yzTi6sY5PU3ENfJzVclCQN8HUy7aws0LSchM4IstX9CoYiPeueMdbmlyi77Nqsh50ldOSsCt2b0Gj9fDB999QPVS1Xmp+0v0bdWXiDC9+4lcCP3NkYDZfGgzKb4Upq6aSumo0gzvPJz+bfpTIrKE09NEXE3hlgK37/g+0hem89zS5wB4vM3jJF6bSMUSFR1eJlI0KNxSYI6fOs6YxWMYsWgER/xH6NOiD0PihlC7rL4gS6QgKdxy0XLycnhl5SsM9g1mx9Ed3Nz4ZoZ3Hs5lVS5zeppIkaRwywWz1vLuuncZMGcA6/etp22ttrx1+1tcW/tap6eJFGkKt1wQ3yYfCZkJLN22lGaVm/H+Xe/zp0Z/0qV9IoVA4ZbzsmrnKpK8SXyy4RNqlqnJKze/Qp8WfQgPC3d6mkjIULglX3488CPJvmSmr55OuehyPH3d0zxy1SMUjyzu9DSRkKNwyx/ac2wPaQvSeOHLF4gIiyChXQIJ1yZQLrqc09NEQpbCLb/pqP8oz2Q9w9NfPM3xU8fp26ovKR1TuKTMJU5PEwl5Crf8ij/Xz8TlExk2fxi7ju3i1qa3ktY5jSaVmjg9TUTOULgFgDybx8y1MxkwZwA/HPiBjnU68t5d79GmZhunp4nIWRRuIXNjJgmZCazYsYLmVZrzce+P6dqwqy7tEwlSCncIW759OYneRDI3ZlKnbB2m9pxK7+a9dWmfSJBTuEPQhv0bGDhnIG+ufZOKxSvy7A3P8reYv1EsopjT00QkHxTuELLz6E6GzRvGhBUTiAqPYlCHQTzZ9knKFCvj9DQROQ8Kdwg4fPIwo74Yxeis0ZzMPcmDrR8kuWMy1UpVc3qaiFwAhbsIO5lzkvHLxpO6IJW9x/dyx2V3kNoplUsrXur0NBG5CAp3EZRn85jx9QwGzR3EpoObiK8XT0aXDGJqxDg9TUQKgMJdhFhr+XTDpyR5k1i1axWtq7dmQvcJXNfgOqeniUgBUriLiCVbl5DoTcS3yUf98vV5vdfr3HHZHYSZMKeniUgBU7hdbv3e9XjmeHh33btUKVmF5258jgevfJCo8Cinp4lIgCjcLrX9yHaG+IYwaeUkikcWZ0jcEB6PfZxSUaWcniYiAaZwu8zB7IOMXDSSZxc/S05eDo9c9QgDOgygSskqTk8TkUKicLtEdk42zy19juELhnMw+yC9m/dmaKeh1C9f3+lpIlLIFO4gl5uXy7TV00iem8yWw1vo2rAr6fHptKzW0ulpIuIQhTtIWWv58LsPSfImsXbPWq6qcRVTek6hU71OTk8TEYcp3EFo0eZFJGQmsGjLIhpVbMTM22fSq2kvfZtVEQEU7qCydvdaPHM8zF4/m+qlqjO+23j6tupLZHik09NEJIgo3EFgy6EtpPhSmLJqCqWiSpHWOY3+1/SnZFRJp6eJSBA6Z7iNMa8A3YHd1trLAz8pdOw/sZ/0BemMWzoOi+Wxax7D095DxRIVnZ4mIkEsP0fck4HngKmBnRI6jp86ztglY8lYmMHhk4fp06IPQ+KGUKdcHaeniYgLnDPc1tr5xpi6gZ9S9OXk5fDqylcZPG8w249sp3uj7gzvPJzmVZs7PU1EXETnuAuBtZZZ387C4/Wwft96YmvG8kavN2hfp73T00TEhQos3MaYfkA/gNq1axfU3bqeb5OPxMxElmxbQtNKTXnvzve4ufHNurRPRC5YgYXbWjsBmAAQExNjC+p+3WrVzlUkeZP4ZMMn1CxTk0k3T6JPiz5EhOmDHBG5OEFVkaysLHw+H3FxccTGxjo954JsOriJQXMHMX31dMpGl2Vkl5H84+p/UDyyuNPTRKSIyM/lgK8DcUAlY8xWIMVaO6mgh2RlZREfH4/f7ycqKgqv1+uqeO85toe0BWm8uOxFwkwYT7V7ioR2CZQvXt7paSJSxOTnqpK7C2OIz+fD7/eTm5uL3+/H5/O5ItxH/Ud5JusZnv7iaY6dOkbfln1JiUuhZpmaTk8TkSIqaE6VxMXFERUV9b8j7ri4OKcn/aFTuaeYuGIiQ+cNZdexXdzS5BbSOqfRtHJTp6eJSBEXNOGOjY3F6/UG/TnuPJvHzLUzGTBnAD8c+IEOdTow685ZxNYKzr0iUvQETbjhdLyDNdgAmRszSchMYMWOFTSv0pyPen/EjQ1v1KV9IlKogircwWr59uUkehPJ3JhJ7bK1mdpzKr2b9yY8LNzpaSISghTuP7Bh/wYGzhnIm2vfpGLxioy+fjQPX/Uw0RHRTk8TkRCmcP+GnUd3MmzeMCasmEBUeBQD2w/kybZPUja6rNPTREQU7l86fPIwo74Yxeis0WTnZPNg6wdJ7phM9dLVnZ4mIvI/CjdwMuck45eNJ3VBKnuP7+X2ZreT2jmVRhUbOT1NROT/Celw59k8Znw9g0FzB7Hp4CY61+vMiC4jiKkR4/Q0EZHfFZLhttby6YZPSfImsWrXKlpWa8ln937GdfWv06V9IhL0Qi7cS7YuIdGbiG+Tj3rl6jHj1hncefmdhJkwp6eJiORLyIR7/d71DJgzgHfWvUPlEpUZd+M4+l3Zj6jwKKeniYiclyIf7u1HtjPEN4RJKydRPLI4gzsO5vHYxyldrLTT00RELkiRDffB7IOMXDSSZxc/S05eDn+/6u8M7DCQKiWrOD1NROSiFLlwZ+dk8/zS50lbkMaB7AP0bt6bYZ2GUb98faeniYgUiCIT7ty8XKatnkby3GS2HN7CDQ1uID0+nVbVWzk9TUSkQLk+3NZaPvjuAzxeD2v3rCWmRgyTe06mc73OTk8TEQkIV4d70eZFJGQmsGjLIi6tcClv3fYWtzW7Tddii0iR5spwr929Fs8cD7PXz6ZaqWqM7zaevq36Ehke6fQ0EZGAc1W4txzaQoovhSmrplAqqhSpnVJ5rM1jlIwq6fQ0EZFC44pw7z+xn/QF6YxbOg6Lpf81/fG091CpRCWnp4mIFLqgDvfxU8cZu2QsGQszOHzyMPe1uI+hcUOpU66O09NERBwTlOHOycvh1ZWvMnjeYLYf2U63S7uRHp9O86rNnZ4mIuK4oAq3tZZZ387C4/Wwft962tRsw+u9XqdDnQ5OTxMRCRpBE+6D2Qfp+lpXlmxbQpNKTZh15yx6NO6hS/tERM4SNOEuW6wsDSo04MHWD3J/y/uJCAuaaSIiQSVo6miMYfqt052eISIS9PTTA0REXEbhFhFxGYVbRMRlFG4REZdRuEVEXEbhFhFxGYVbRMRlFG4REZcx1tqCv1Nj9gA/XeD/XgnYW4Bz3EDPuegLtecLes7nq461tnJ+bhiQcF8MY8wya22M0zsKk55z0Rdqzxf0nANJp0pERFxG4RYRcZlgDPcEpwc4QM+56Au15wt6zgETdOe4RUTkjwXjEbeIiPyBoAm3MaarMWa9MWaDMSbR6T2BZoypZYyZa4xZZ4xZa4zp7/SmwmKMCTfGrDTGfOj0lsJgjClnjHnbGPPtmT/vWKc3BZox5l9n3q/XGGNeN8ZEO72poBljXjHG7DbGrPnF2yoYY/5rjPn+zK/lA/HYQRFuY0w48DxwI9AMuNsY08zZVQGXAzxhrW0KtAEeCYHn/LP+wDqnRxSiMcCn1tomQAuK+HM3xlwC/BOIsdZeDoQDdzm7KiAmA13Pelsi4LXWXgp4z7xe4IIi3MDVwAZr7UZrrR94A+jh8KaAstbusNauOPPyEU7/Zb7E2VWBZ4ypCXQDXnZ6S2EwxpQBOgCTAKy1fmvtQWdXFYoIoLgxJgIoAWx3eE+Bs9bOB/af9eYewJQzL08BegbisYMl3JcAW37x+lZCIGI/M8bUBVoBS5xdUiieBZ4C8pweUkjqA3uAV8+cHnrZGFPS6VGBZK3dBowCNgM7gEPW2s+dXVVoqlprd8DpgzOgSiAeJFjC/Vs/yj0kLncxxpQC3gEes9YednpPIBljugO7rbXLnd5SiCKA1sCL1tpWwDEC9OFzsDhzXrcHUA+oAZQ0xtzr7KqiJVjCvRWo9YvXa1IEP7Q6mzEmktPRnm6tfdfpPYWgHXCzMWYTp0+HdTbGvObspIDbCmy11v780dTbnA55UdYF+NFau8daewp4F2jr8KbCsssYUx3gzK+7A/EgwRLuL4FLjTH1jDFRnP5ExmyHNwWUMcZw+rznOmvtaKf3FAZrbZK1tqa1ti6n/4znWGuL9JGYtXYnsMUY0/jMm+KBbxycVBg2A22MMSXOvJ/HU8Q/IfsLs4H7z7x8P/B+IB4kIhB3er6stTnGmH8An3H6M9CvWGvXOjwr0NoB9wFfG2O+OvM2j7X2Ywc3SWA8Ckw/c1CyEfiLw3sCylq7xBjzNrCC01dPraQIfhWlMeZ1IA6oZIzZCqQAGcBbxpi/cvofsNsD8tj6ykkREXcJllMlIiKSTwq3iIjLKNwiIi6jcIuIuIzCLSLiMgq3iIjLKNwiIi6jcIuIuMz/ASgkdhTmp9IuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(X, Y, 'k.')\n",
    "\n",
    "x = np.linspace(0, 10, 10)\n",
    "y = a_new*x+b_new\n",
    "\n",
    "plt.plot(x, y, 'g-')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "比迭代一次靠谱多了，我们再继续这个过程，迭代 1000 次："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a = 2.083663961231524, b = 0.9559983763104101, loss = 0.010980851812161636, diff = 5.500906925493335e-06\n"
     ]
    }
   ],
   "source": [
    "for i in range(1000):\n",
    "    a_new = update_a(X, Y, a, b, eta)\n",
    "    b_new = update_b(X, Y, a, b, eta)\n",
    "    loss_new = loss(X, Y, a_new, b_new)\n",
    "    diff = abs(loss(X, Y, a, b) - loss(X, Y, a_new, b_new))\n",
    "    a = a_new\n",
    "    b = b_new\n",
    "    \n",
    "print(\"a = {0}, b = {1}, loss = {2}, diff = {3}\".format(a_new, b_new, loss_new, diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "diff 的值已经趋近于 0，说明已经达到收敛，这个时候的图形，就是我们根据梯度下降法求得的结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAH8NJREFUeJzt3Xt8z/X///Hb05g55xg5NCFRJJ+VltJEksP66pO+Sumwg1gsh5QQPoSQ0Ic0Uzrx0UVczJiNZc75hCSHnCWMkTOx0/P3h30/vz4VZnu/99r7/b5fLxeXvd+vvbbX/WXb3ctrr8frbay1iIiI5yvidAAREXENFbqIiJdQoYuIeAkVuoiIl1Chi4h4CRW6iIiXUKGLiHgJFbqIiJdQoYuIeImiBbmxSpUq2cDAwILcpIiIx9u4ceMJa23l661XoIUeGBjIhg0bCnKTIiIezxjzc27W0ykXEREvoUIXEfESKnQRES+hQhcR8RIqdBERL6FCFxHxEip0EREvoUIXEXGjXy/+SnRCNGcunXH7tlToIiJuYK3lq21f0WBKA6ZumMrKn1e6fZsFOikqIuILjpw7Qs9FPVmwcwFBtwSxLHQZjW9u7PbtqtBFRFzEWkvsplheX/o66VnpjH90PNH3R1O0SMFUrQpdRMQF9pzcQ+TCSJYfWE5IYAjTO06nboW6BZpBhS4ikg+Z2ZlM+nYSQ5YPoZhfMWI6xBDeNBxjTIFnUaGLiOTRj8d+JCwujO+OfEdo/VCmtptK9bLVHcujQhcRuUGXMy/zzqp3GL16NOUDyjPnqTl0btjZkaPy31Ohi4jcgHW/rCMsLowdJ3bwfOPnef+x96lYsqLTsQAVuohIrpxPP8/gbwYzef1kaparSULXBNrWbet0rP+iQhcRuY6kvUlELozk5zM/8+q9rzKq1SjKFC/jdKw/UaGLiFzFyd9O0i+pHzM3z6R+xfqsemkVD9Z60OlYV6VCFxH5C19v/5qoxVGcuHiCQQ8NYnCLwQQUDXA61jWp0EVEfif1XCpRi6OY/9N8mlZrypLnltCkahOnY+WKCl1EhCtj+59s/oR+Sf24lHmJd1u/S9/gvgU2tu8K101qjKkJfAZUBbKBGGvtJGNMBWAOEAgcAJ621p5yX1QREffYd2of3eO7s2zfMlrc2oLpHadze8XbnY51w3Jz+9xMoJ+1tgFwPxBljGkIvAkkW2vrAck5z0VEPEZWdhbvr3ufRh82Yv2h9UxrP43lLyz3yDKHXByhW2tTgdScx+eMMTuA6sATQEjOap8CKcAbbkkpIuJiW9O2Eh4XzvrD62lfrz3TOkyjRtkaTsfKlxs6OWSMCQTuAdYDN+eUPdbaVGNMlat8TCQQCVCrVq38ZBURybf0rHRGrxrNO6veoVxAOWY9OYsud3VxfGzfFXJd6MaY0sDXwGvW2rO53XlrbQwQAxAUFGTzElJExBXWH1pP+MJwtqZt5dlGzzLxsYlULlXZ6Vguk6uXoDPGFONKmX9prZ2Xs/iYMaZazvurAWnuiSgikj8X0i/QN7EvwTOCOX3pNPHPxPPlk196VZlD7q5yMcAMYIe1dsLv3hUHvACMyXm7wC0JRUTyIXlfMhELI9h/ej89gnowpvUYyhYv63Qst8jNKZfmwPPAj8aYzTnL3uJKkX9ljAkDDgKd3RNRROTGnb50mv5J/Znx/QzqVajHihdX0OLWFk7HcqvcXOWyGrjaCfNWro0jIpJ/83fMJ2pxFGkX0hjwwACGhQyjRLESTsdyO88ZgRIRuY6j54/SK6EXc7fPpUnVJsQ/G0/Tak2djlVgVOgi4vGstXz2w2f0SezDxYyLjHpkFP0f6E8xv2JORytQKnQR8WgHTh+ge3x3kvYm0bxmc2JDY7mj0h1Ox3KECl1EPFJWdhZTvpvCW8lvYYxhSrspvBL0CkVMrq7G9koqdBHxONuPbyc8Lpx1h9bRtm5bPurwEbXKaRJdhS4iHiM9K52xa8YyYuUISvuX5vNOn9O1UVevGNt3BRW6iHiEDUc2EBYXxpZjW+hyVxcmtZ1ElVJ/eQspn6VCF5FC7WLGRYYuH8qEbydQtXRVFnRZQGj9UKdjFUoqdBEptJbvX07Ewgj2ntpLZNNIxj46lnIB5ZyOVWip0EWk0Dlz6QwDlg4gZlMMdcrX4Ztu39CydkunYxV6KnQRKVTidsbRY1EPjp4/Sv/g/gxvOZySxUo6HcsjqNBFpFBIu5BG74TezNk2h0ZVGrGgywKCbglyOpZHUaGLiKOstXyx5QteS3yN8+nnGdFyBAOaD8Dfz9/paB5HhS4ijjl45iDd47uzZM8SgmsEExsaS8PKDZ2O5bFU6CJS4LJtNh9+9yFvJr9Jts1mUttJRN0bhV8RP6ejeTQVuogUqJ9O/ETEwghWH1zNo7c9SkzHGAJvCnQ6lldQoYtIgcjIymDc2nEMXzGcUsVKMfOJmXS7u5vG9l1IhS4ibrcpdRMvL3iZH479wFMNn+KDxz+gaumqTsfyOip0EXGb3zJ+Y/iK4YxfO57KpSoz7+l5dGrQyelYXkuFLiJusfLnlYTHhbP75G7C7glj3KPjKF+ivNOxvJoKXURc6uzls7yx9A2mbZxG7Ztqs/T5pbS+rbXTsXyCCl1EXGbRrkW8sugVDp89zGvNXmPkIyMp5V/K6Vg+Q4UuIvl2/MJxXkt8jVk/zuLOyncyN2wuzWo0czqWz1Ghi0ieWWuZvXU20UuiOXPpDMMeHsbAhwZqbN8hKnQRyZNfzvxCj0U9WLR7EfdVv48ZoTO4q8pdTsfyaSp0Ebkh2TabmI0xDFg6gCybxYQ2E+jdrLfG9gsBFbqI5NquX3cRsTCClT+vpFXtVsR0jOG28rc5HUtyqNBF5LoyszN5b+17DE0ZSkDRAGaEzuClJi9pbL+QUaGLyDVtPrqZsLgwNqVuotMdnfhnu39yS5lbnI4lf0GFLiJ/6VLmJUasGMG7a96lUslKzO08l783/LvTseQaVOgi8ierD64mPC6cnb/u5MUmL/Jem/eoUKKC07HkOlToIvIf5y6fY2DyQKZ8N4XAmwJJfC6RNnXaOB1LckmFLiIAJOxOoHt8dw6dPUR0s2hGPjKS0v6lnY4lN0CFLuLjTlw8QZ/EPnyx5QsaVGrAmpfXEFwz2OlYkgcqdBEfZa3lq21f0SuhF6cunWJIiyEMemgQxYsWdzqa5JEKXcQHHT57mJ6LexK3M46gW4JYFrqMxjc3djqW5JMKXcSHZNtsYjfF8vrS18nIymD8o+OJvj+aokVUBd5AX0URH7Hn5B4iFkaQciCFkMAQpnecTt0KdZ2OJS5U5HorGGM+NsakGWO2/m7ZMGPMYWPM5pw/7dwbU0TyKjM7k/Frx9Pow0ZsSt1ETIcYvun2jcrcC+XmCH0m8E/gsz8sf99aO97liUTEZbYc20JYXBgbjmwgtH4oU9tNpXrZ6k7HEje5bqFba1caYwLdH0VEXOVy5mVGrhzJmDVjKB9QnjlPzaFzw866mZaXy8859FeNMd2ADUA/a+0pF2USkXxY+8tawuPC2XFiB93u7saENhOoWLKi07GkAFz3HPpVfAjUAZoAqcB7V1vRGBNpjNlgjNlw/PjxPG5ORK7nfPp5ohOiefDjB7mQcYGErgl8+j+fqsx9SJ6O0K21x/7vsTFmOhB/jXVjgBiAoKAgm5ftici1Je1NInJhJAfPHKTnvT0Z3Wo0ZYqXcTqWFLA8Fboxppq1NjXnaSdg67XWFxH3OPnbSfol9WPm5pnUr1ifVS+tonmt5k7HEodct9CNMbOBEKCSMeYQMBQIMcY0ASxwAOjuxowi8gfWWr7e8TWvLn6VExdPMOihQQxuMZiAogFORxMH5eYql2f+YvEMN2QRkVxIPZdK1OIo5v80n6bVmrLkuSU0qdrE6VhSCGhSVMRDWGv5ZPMn9E3sy+Wsy4xtPZY+wX00ti//oe8EEQ+w79Q+IhdGkrw/mRa3tiC2Yyz1KtZzOpYUMip0kUIsKzuLyesnM3j5YPyMH9PaTyPibxEUMXm94li8mQpdpJDamraV8Lhw1h9eT/t67ZnWYRo1ytZwOpYUYip0kUImPSud0atG886qdygXUI5ZT86iy11dNLYv16VCFylE1h9aT1hcGNuOb6Nro65MbDuRSiUrOR1LPIQKXaQQuJB+gSHLhzDx24lUL1ud+GfiaX97e6djiYdRoYs4LHlfMhELI9h/ej89gnowpvUYyhYv63Qs8UAqdBGHnL50mv5J/Znx/QzqVajHihdX0OLWFk7HEg+mQhdxwPwd84laHEXahTTeaP4GQx8eSoliJZyOJR5OhS5SgI6eP0qvhF7M3T6XJlWbEP9sPE2rNXU6lngJFbpIAbDW8tkPn9EnsQ8XMy4y6pFR9H+gP8X8ijkdTbyICl3EzQ6cPkD3+O4k7U2iec3mxIbGckelO5yOJV5IhS7iJlnZWUz9bioDkwdijGFKuym8EvSKxvbFbVToIm6w/fh2wuPCWXdoHW3rtuWjDh9Rq1wtp2OJl1Ohi7hQelY6Y9eMZcTKEZT2L83nnT6na6OuGtuXAqFCF3GRDUc2EBYXxpZjW+hyVxcmtZ1ElVJVnI4lPkSFLpJPFzMuMixlGO+te4+qpauyoMsCQuuHOh1LfJAKXSQfUg6kEB4Xzt5Te4loGsG4R8dRLqCc07HER6nQRfLgzKUzDFg6gJhNMdQpX4dvun1Dy9otnY4lPk6FLnKD4nbG0WNRD46eP0r/4P4MbzmcksVKOh1LRIUukltpF9LondCbOdvm0KhKIxZ0WUDQLUFOxxL5DxW6yHVYa/liyxe8lvga59PPM6LlCAY0H4C/n7/T0UT+iwpd5BoOnjlI9/juLNmzhOAawcSGxtKwckOnY4n8JRW6yF/Ittl8+N2HvJn8JtZaJrWdRNS9UfgV8XM6mshVqdBF/uCnEz8RsTCC1QdX06ZOGz7q8BGBNwU6HUvkulToIjkysjIYt3Ycw1cMp1SxUsx8Yibd7u6msX3xGCp0EWBT6iZeXvAyPxz7gacaPsUHj39A1dJVnY4lckNU6OLTfsv4jeErhjN+7Xgql6rMvKfn0alBJ6djieSJCl181sqfVxIeF87uk7sJuyeMcY+Oo3yJ8k7HEskzFbr4nLOXz/LG0jeYtnEatW+qzbLnl9HqtlZOxxLJNxW6+JRFuxbxyqJXOHLuCH3v78s/Wv6DUv6lnI4l4hIqdPEJxy8cJ3pJNLO3zubOyncyt/NcmtVo5nQsEZdSoYtXs9Yye+tseif05uzlswx7eBgDHxqosX3xSip08Vq/nPmFHot6sGj3IppVb8aM0BncWeVOp2OJuI0KXbxOts3mow0f8cayN8iyWbz/2Pv0uq+XxvbF66nQxavs+nUX4XHhrDq4ila1WxHTMYbbyt/mdCyRAqFCF6+QmZ3Je2vfY2jKUAKKBjAjdAYvNXlJY/viU65b6MaYj4EOQJq19q6cZRWAOUAgcAB42lp7yn0xRa5u89HNhMWFsSl1E53u6MSUdlOoVqaa07FEClyRXKwzE2j7h2VvAsnW2npAcs5zkQJ1KfMSbyW/RVBMEIfPHmZu57nM+995KnPxWdc9QrfWrjTGBP5h8RNASM7jT4EU4A0X5hK5ptUHVxMWF8auX3fxwt0vMOGxCVQoUcHpWCKOyus59JuttakA1tpUY0wVF2YSuapzl88xMHkgU76bQuBNgSQ+l0ibOm2cjiVSKLj9l6LGmEggEqBWrVru3px4sYTdCXSP786hs4eIbhbNyEdGUtq/tNOxRAqN3JxD/yvHjDHVAHLepl1tRWttjLU2yFobVLly5TxuTnzZiYsneH7+87Sb1Y7S/qVZ8/IaJradqDIX+YO8Fnoc8ELO4xeABa6JI/L/WWuZs3UODac05F9b/8WQFkP4vvv3BNcMdjqaSKGUm8sWZ3PlF6CVjDGHgKHAGOArY0wYcBDo7M6Q4nsOnz1Mz8U9idsZR9AtQSwLXUbjmxs7HUukUMvNVS7PXOVduoG0uFy2zSZ2UyyvL32djKwMxj86nuj7oylaRDNwItejnxIpNPac3EPEwghSDqQQEhjC9I7TqVuhrtOxRDyGCl0cl5mdycRvJzJk+RD8/fyJ6RBDeNNwje2L3CAVujhqy7EthMWFseHIBkLrhzK13VSql63udCwRj6RCF0dczrzMyJUjGbNmDOUDyjPnqTl0bthZR+Ui+aBClwK39pe1hMeFs+PEDp5v/DzvP/Y+FUtWdDqWiMdToUuBOZ9+nkHJg/jg3x9Qs1xNErom0LbuH+/7JiJ5pUKXApG0N4nIhZEcPHOQqHujGNVqFGWKl3E6lohXUaGLW5387ST9kvoxc/NM6lesz6qXVtG8VnOnY4l4JRW6uIW1lq93fM2ri1/lxMUTDHpoEINbDCagaIDT0US8lgpdXC71XCpRi6OY/9N8mlZrypLnltCkahOnY4l4PRW6uIy1lk82f0LfxL5czrrMu63fpW9wX43tixQQ/aSJS+w7tY/IhZEk70+mxa0tiO0YS72K9ZyOJeJTVOiSL1nZWUxeP5nBywfjZ/yY1n4aEX+LoIjJ652ZRSSvVOiSZ1vTthIeF876w+tpX6890zpMo0bZGk7HEvFZKnS5YelZ6YxeNZp3Vr1DuYByzHpyFl3u6qKxfRGHqdDlhqw/tJ6wuDC2Hd/Gs42eZeJjE6lcSi8tKFIYqNAlVy6kX2DI8iFM/HYi1ctWJ/6ZeNrf3t7pWCLyOyp0ua7kfclELIxg/+n99AjqwZjWYyhbvKzTsUTkD1ToclWnL52mf1J/Znw/g3oV6rHixRW0uLWF07FE5CpU6PKX5u+YT9TiKNIupPFm8zd5++G3KVGshNOxROQaVOjyX46eP0qvhF7M3T6XJlWbEP9sPE2rNXU6lojkggpdgCtj+5/98Bl9EvtwMeMiox4ZRf8H+lPMr5jT0UQkl1TowoHTB+ge352kvUk0r9mc2NBY7qh0h9OxROQGqdB9WFZ2FlO/m8rA5IEYY5jSbgqvBL2isX0RD6VC91Hbj28nPC6cdYfW8Xjdx5nWYRq1ytVyOpaI5IMK3cekZ6Uzds1YRqwcQRn/Mnze6XO6NuqqsX0RL6BC9yEbjmwgLC6MLce20OWuLkxqO4kqpao4HUtEXESF7gMuZlxk6PKhTPh2AlVLV2VBlwWE1g91OpaIuJgK3culHEghPC6cvaf20v1v3Xm39buUCyjndCwRcQMVupc6c+kMA5YOIGZTDHXK12H5C8sJCQxxOpaIuJEK3QvF7Yyjx6IeHD1/lNcfeJ1hIcMoWayk07FExM1U6F4k7UIavRN6M2fbHBrf3JgFXRYQdEuQ07FEpICo0L2AtZYvf/yS6CXRnE8/z8iWIxnQfIDG9kV8jArdwx08c5BX4l8hYU8CwTWCmRE6gwaVGzgdS0QcoEL3UNk2m2kbpvHGsjew1jK57WR63tsTvyJ+TkcTEYeo0D3QzhM7CV8YzuqDq2lTpw0fdfiIwJsCnY4lIg5ToXuQjKwMxq8dz/AVwylZrCQzn5hJt7u7aWxfRAAVusfYlLqJsLgwNh/dTOeGnZn8+GSqlq7qdCwRKUTyVejGmAPAOSALyLTW6ho5F/st4zf+seIfjFs7jsqlKjPv6Xl0atDJ6VgiUgi54gi9pbX2hAs+j/zOunXr+OSbT1jiv4RfLv5C2D1hjHt0HOVLlHc6mogUUjrlUggtW7WMtuPbktU0C3PcMLn1ZHp16OV0LBEp5PL70jQWSDLGbDTGRLoikK9btGsRT33zFFlNsmAtmI8M538873QsEfEA+T1Cb26tPWKMqQIsNcb8ZK1d+fsVcoo+EqBWLb0iztUcv3Cc1xJfY9aPs6hdqjaXPrxE5s+Z+Pv7ExIS4nQ8EfEA+Sp0a+2RnLdpxpj5wH3Ayj+sEwPEAAQFBdn8bM8bWWv519Z/0XtJb85cOsOwh4cx8KGBbGy+kZSUFEJCQggODnY6poh4gDwXujGmFFDEWnsu53Eb4B8uS+YDDp09RI9FPYjfFU+z6s2YETqDO6vcCUBwcLCKXERuSH6O0G8G5ucMtRQFZllrl7gklZfLttnEbIxhwNIBZNks3n/sfXrd10tj+yKSL3kudGvtPuBuF2bxCbt+3UXEwghW/ryS1re1JqZDDLXL13Y6loh4AV22WEAyszOZsG4CQ1OGElA0gI9DP+bFJi9qbF9EXEaFXgA2H91MWFwYm1I38WSDJ/nn4/+kWplqTscSES+jQnejS5mXGLFiBO+ueZdKJSsxt/Nc/t7w707HEhEvpUJ3k9UHVxMeF87OX3fyUpOXGN9mPBVKVHA6loh4MRW6i527fI6ByQOZ8t0UAm8KJPG5RNrUaeN0LBHxASp0F0rYnUD3+O4cOnuI6GbRjHxkJKX9SzsdS0R8hArdBU5cPEGfxD58seULGlZuyJqX1xBcU0NBIlKwVOj5YK3lq21f0SuhF6cuneLtFm/z1kNvUbxocaejiYgPUqHn0eGzh+m5uCdxO+O495Z7WRa6jMY3N3Y6loj4MBX6Dcq22cRuiuX1pa+TkZXBe23eI7pZtMb2RcRxKvQbsOfkHiIWRpByIIWWgS2Z3nE6dSrUcTqWiAigQr+udevWkbw8mWO1jxG7LxZ/P3+md5xO2D1hGtsXkUJFhX4N69ato2WXllxuexky4KHKDzH7udlUL1vd6WgiIn+S35eg81qXMy8zaNkgLr9wGcpCkblFaHu6rcpcRAotHaH/hbW/rCU8Lpwd2Tvw2+6HTbQUzypOy5YtnY4mInJVKvTfOZ9+nkHJg/jg3x9Qs1xNEromUO54OVL+ppeCE5HCT4WeI2lvEpELIzl45iBR90YxqtUoyhQvA3VRkYuIR/D5Qj/520n6Jvbl0x8+5Y5Kd7DqpVU0r9Xc6VgiIjfMZwvdWsvXO74manEUJ387yaCHBjG4xWACigY4HU1EJE98stBTz6UStTiK+T/Np2m1piQ+l0iTqk2cjiUiki8+VejWWj7Z/Al9E/tyOesyY1uPpU9wH4oW8am/BhHxUj7TZPtO7SNyYSTJ+5N5+NaHmd5xOvUq1nM6loiIy3h9oWdlZzF5/WQGLx+Mn/FjWvtpRPwtgiJGM1Ui4l28utC3pm0lLC6Mfx/+Nx1u78CH7T+kRtkaTscSEXELryz09Kx0Rq0axahVoygXUI7Zf5/N/975v7qZloh4Na8r9PWH1hMWF8a249vo2qgrE9tOpFLJSk7HEhFxO68p9AvpFxiyfAgTv51I9bLVWfTsItrVa+d0LBGRAuMVhZ68L5mIhRHsP72fnkE9Gd16NGWLl3U6lohIgfLoQj/12yn6J/Xn480fc3vF21nx4gpa3NrC6VgiIo7w2EKfv2M+PRf35PiF4wx8cCBvP/y2xvZFxKd5XKEfPX+UXgm9mLt9Lk2qNmHxs4u5p9o9TscSEXGcxxS6tZbPfviMPol9uJhxkdGtRtMvuB/F/Io5HU1EpFDwiEKf98083lz9Jrvtbh6s9SCxHWOpX6m+07FERAqVQj//vm7dOp6e+jS7L+2mWGIxxtQfozIXEfkLhb7QU1JSsEkWpkL2v7NZuWKl05FERAqlQl/oISEhFM8sjt95P/z9/QkJCXE6kohIoVToz6EHBweTnJxMSopeqFlE5FoKfaHDlVJXkYuIXFuhP+UiIiK5k69CN8a0NcbsNMbsMca86apQIiJy4/Jc6MYYP2AK8DjQEHjGGNPQVcFEROTG5OcI/T5gj7V2n7U2HfgX8IRrYomIyI3KT6FXB3753fNDOcv+izEm0hizwRiz4fjx4/nYnIiIXEt+Cv2vXs/N/mmBtTHW2iBrbVDlypXzsTkREbmW/Fy2eAio+bvnNYAj1/qAjRs3njDG/JzH7VUCTuTxYz2V9tk3aJ99Q372+dbcrGSs/dNBda4YY4oCu4BWwGHgO+BZa+22PH3C629vg7U2yB2fu7DSPvsG7bNvKIh9zvMRurU20xjzKpAI+AEfu6vMRUTk+vI1KWqtXQwsdlEWERHJB0+aFI1xOoADtM++QfvsG9y+z3k+hy4iIoWLJx2hi4jINXhEofvaPWOMMTWNMcuNMTuMMduMMdFOZyoIxhg/Y8z3xph4p7MUBGPMTcaYucaYn3K+1l5/S1FjTJ+c7+mtxpjZxpgApzO5mjHmY2NMmjFm6++WVTDGLDXG7M55W94d2y70he6j94zJBPpZaxsA9wNRPrDPANHADqdDFKBJwBJr7R3A3Xj5vhtjqgO9gSBr7V1cuTqui7Op3GIm0PYPy94Ekq219YDknOcuV+gLHR+8Z4y1NtVauynn8Tmu/KD/6bYK3sQYUwNoD8Q6naUgGGPKAi2AGQDW2nRr7WlnUxWIokCJnDmWklxnGNETWWtXAif/sPgJ4NOcx58C/+OObXtCoefqnjHeyhgTCNwDrHc2idtNBAYA2U4HKSC3AceBT3JOM8UaY0o5HcqdrLWHgfHAQSAVOGOtTXI2VYG52VqbClcO2IAq7tiIJxR6ru4Z442MMaWBr4HXrLVnnc7jLsaYDkCatXaj01kKUFGgKfChtfYe4AJu+m94YZFz3vgJoDZwC1DKGPOcs6m8iycU+g3fM8YbGGOKcaXMv7TWznM6j5s1B0KNMQe4ckrtEWPMF85GcrtDwCFr7f/9z2suVwrem7UG9ltrj1trM4B5wAMOZyoox4wx1QBy3qa5YyOeUOjfAfWMMbWNMf5c+SVKnMOZ3MoYY7hybnWHtXaC03nczVo70Fpbw1obyJWv7zfWWq8+crPWHgV+McbUz1nUCtjuYKSCcBC43xhTMud7vBVe/ovg34kDXsh5/AKwwB0bKfQvEu2j94xpDjwP/GiM2Zyz7K2cWy2I9+gFfJlzoLIPeMnhPG5lrV1vjJkLbOLKlVzf44UTo8aY2UAIUMkYcwgYCowBvjLGhHHlH7bObtm2JkVFRLyDJ5xyERGRXFChi4h4CRW6iIiXUKGLiHgJFbqIiJdQoYuIeAkVuoiIl1Chi4h4if8HE/7vYk0vqpgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(X, Y, 'k.')\n",
    "\n",
    "x = np.linspace(0, 10, 10)\n",
    "y = a_new*x+b_new\n",
    "\n",
    "plt.plot(x, y, 'g-')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多元线性回归的梯度下降\n",
    "\n",
    "上面是对一元线性回归的梯度下降法的介绍，回到线性回归的一般形式：\n",
    "\n",
    "$$\n",
    "\\hat{y} = \\theta_0 + \\theta_1x_1 + \\theta_2x_2 + ... + \\theta_nx_n = \\theta^Tx\n",
    "$$\n",
    "\n",
    "其对应的损失函数为：\n",
    "\n",
    "$$\n",
    "loss = J(\\theta_0, \\theta_1, \\dots, \\theta_n) = J(\\theta) = \\frac{1}{2m} \\sum_{i=1}^{m}(y^{(i)} - \\theta^Tx^{(i)})^2\n",
    "$$\n",
    "\n",
    "对 $J(\\theta)$ 进行求导，得到参数 $\\theta$ 的更新公式：\n",
    "\n",
    "$$\n",
    "\\left\\{\n",
    "\\begin{align}\n",
    "\\theta_0 &:= \\theta_0 - \\eta \\frac{1}{m} \\sum_{i=1}^{m}(y^{(i)} - \\theta^Tx^{(i)})x_0^{(i)} \\\\\n",
    "\\theta_1 &:= \\theta_1 - \\eta \\frac{1}{m} \\sum_{i=1}^{m}(y^{(i)} - \\theta^Tx^{(i)})x_1^{(i)} \\\\\n",
    "\\theta_2 &:= \\theta_2 - \\eta \\frac{1}{m} \\sum_{i=1}^{m}(y^{(i)} - \\theta^Tx^{(i)})x_2^{(i)} \\\\\n",
    "\\vdots \\\\\n",
    "\\theta_n &:= \\theta_n - \\eta \\frac{1}{m} \\sum_{i=1}^{m}(y^{(i)} - \\theta^Tx^{(i)})x_n^{(i)} \\\\\n",
    "\\end{align}\n",
    "\\right.\n",
    "$$\n",
    "\n",
    "其中，$m$ 表示样本个数，$n$ 表示特征维度，$x^{(i)}$ 表示第 i 个样本，$x^{(i)}_n$ 表示第 i 个样本的第 n 个特征。\n",
    "为了方便表达和计算，损失函数也可以写成下面的矩阵形式：\n",
    "\n",
    "$$\n",
    "loss = J(\\theta) = \\frac{1}{2m} (\\bf{y}-\\rm{X}\\theta)^T (\\bf{y}-\\rm{X}\\theta)\n",
    "$$\n",
    "\n",
    "根据前面的矩阵求导，得到梯度下降的更新公式为：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\theta &:= \\theta - \\eta \\frac{\\partial}{\\partial\\theta}J(\\theta) \\\\\n",
    "       &= \\theta - \\eta \\frac{1}{m} X^T(\\rm{X}\\theta-\\bf{y})\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "TODO：使用矩阵的迹（Trace）求矩阵导数。\n",
    "\n",
    "我们把上面的代码改成矩阵计算的形式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theta: 0.9333333336913473 2.099999999741958\n",
      "loss: [[0.01]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHyBJREFUeJzt3Xd0VHXC//H3l0CIoUtAEKSKFFkUzCJNCAaRxQIWRF2sj3JclU0BQkIXkCaEWJdlQUARAZUVFzUJSQg19CIgikgTNghIDyUk+T5/kJ8/5LFAMjM3ufN5nbOHJMxyP3OAt8PNzFxjrUVERIq/Ek4PEBERz1DQRURcQkEXEXEJBV1ExCUUdBERl1DQRURcQkEXEXEJBV1ExCUUdBERlyjpy4OFhITYOnXq+PKQIiLF3vr1649Ya6v80e18GvQ6deqwbt06Xx5SRKTYM8bsvZLb6ZSLiIhLKOgiIi6hoIuIuISCLiLiEgq6iIhLKOgiIi6hoIuIuISCLiLiRbuO7SIyMZKcvByvH0tBFxHxglPnTxGXEkfjtxszdcNUNh/c7PVj+vSVoiIibpdn85i5aSYD0wZy8PRBnrzlScaEj+H6ctd7/dgKuoiIh6zYt4KIxAjWZ66nVc1WLHh0AS1rtPTZ8RV0EZFC2ndiHwNSBjBn6xxqlKvBrAdm8fifHscY49MdCrqISAGduXCG8SvGM37FeCyWoe2HEtM2hjKBZRzZo6CLiFwlay1zts4hJiWG/Sf30/PmnozrNI7aFWs7uktBFxG5CmsPrCUyKZKVP6ykRfUWzH5wNnfUvsPpWYCCLiJyRTJPZRKXGsfMzTOpWqYq0+6fxlO3PEVAiQCnp/1MQRcR+R3ncs4xKWMSo5ePJjs3m5g2MQxqP4jypcs7Pe3/UNBFRH6FtZb52+fTf1F/dh/fTfdG3Zlw1wTqX1vf6Wm/SUEXEbnM5oObiUyKJH1POk2rNiXliRTC64U7PesPKegiIvkOZx1myOIh/GvDv6gUVIl3ur7D87c9T8kSxSOVxWOliIgXZedm89aatxixZARZF7Lo07IPwzoMo9I1lZyedlUUdBHxW9ZavvjuC6KTo9nx0w663NiF+M7xNK7S2OlpBaKgi4hf2n54O1FJUSR9n0TDyg35/PHP6dqgq9OzCkVBFxG/cvTsUV5Jf4W3175N2cCyxHeO56WWLxEYEOj0tEJT0EXEL+Tk5TBl/RSGLB7C8XPH6d2iNyM6jqBKmSpOT/MYBV1EXC9lVwpRSVFsPbSVjnU6ktAlgWbXNXN6lscp6CLiWjuP7qRfcj8WfLuAepXqMf+R+XRv1N3nb2vrKwq6iLjOyfMnGbV0FAmrEihdsjRjwscQ2SqSoJJBTk/zKgVdRFwjNy+XGZtmMDBtIIeyDvH0rU8z+s7RVC9X3elpPqGgi4grLNu7jIjECDYe3EibG9rw+eOfE3p9qNOzfEpBF5Fibe/xvcSkxDBv2zxqlq/Jhw99SM+be7r2PPnv+cOgG2PeBe4FDllrm+Z/7VpgLlAH2AM8Yq095r2ZIiK/lJWdxdjlY5mQMQGDYXiH4fRv25/gUsFOT3NMiSu4zQygy2VfiwVSrbUNgNT8z0VEvC7P5jHrq1nc9NZNjFo2igcaPcC3L3/LsLBhfh1zuIJH6NbapcaYOpd9uRsQlv/xTCAdGODBXSIi/8fq/auJSIxg9YHVhF4fykc9PqLNDW2cnlVkFPQc+nXW2kwAa22mMaaqBzeJiPzCgZMHiEuN4/2v3qda2WpM7zadJ295khLmSk4y+A+vf1PUGNMb6A1Qq1Ytbx9ORFzk7IWzxGfEM3r5aHLycohrF0dcuzjKlS7n9LQiqaBB/9EYUz3/0Xl14NBv3dBaOwWYAhAaGmoLeDwR8SPWWj7++mP6L+rP3hN7ebDxg7x212vUq1TP6WlFWkGD/hnwFDA2/8cFHlskIn5tY+ZGIpMiWbp3Kc2ua0ZatzQ61u3o9Kxi4UqetvghF78BGmKM2Q8M42LI5xlj/gfYB/Tw5kgRcb8fT//I4LTBTNs4jcrBlZl8z2Sea/EcASUCnJ5WbFzJs1we+42fKvpXTBWRIi87N5s3Vr/BiCUjOJtzlshWkQztMJSKQRWdnlbs6JWiIuIIay3/2fEf+ib3ZefRnXRt0JX4zvE0DGno9LRiS0EXEZ/bemgrUUlRpOxKoVFII77865d0ufHy1y/K1VLQRcRnfjrzE8PShzF53WTKlS7H611e52+hf6NUQCmnp7mCgi4iXnch9wKT101mWPowTpw/wQu3vcArHV8hJDjE6WmuoqCLiFcl7UwiKimK7Ue2E143nIQuCTSt2tTpWa6koIuIV+z4aQd9k/uycMdC6leqz6c9P+X+hvf75dva+oqCLiIedfzccUYuGcmba94kqGQQ4zqNI+L2CEqXLO30NNdT0EXEI3Lzcpm2cRqD0wZz5MwRnrn1GV4Nf5VqZas5Pc1vKOgiUmjpe9KJTIxk84+baVerHYldEmlRvYXTs/yOgi4iBbb72G76L+rPJ9s/oVaFWsx9eC49mvTQeXKHKOgictVOZ59mzLIxTMyYSECJAEaEjaBfm35cU+oap6f5NQVdRK5Yns3j/c3vE5caR+bpTHo168WY8DHULF/T6WmCgi4iVyjjhwwiEiNY+9+1tKzRkvk959OqZiunZ8klFHQR+V37T+5nQMoAZm+ZTfWy1ZnZfSa9mvXS5d+KIAVdRH7VmQtnmLByAmOXjyXP5jHojkHEtoulbGBZp6fJb1DQReQXrLXM3TaXmEUx/HDyBx5u8jDjO42nbqW6Tk+TP6Cgi8jP1v93PRGJEaz4YQW3VruV9x94nw51Ojg9S66Qgi4iHDx9kEGpg5i+aTohwSFMuXcKzzZ/Vpd/K2YUdBE/dj7nPAmrEnh12aucyzlH39Z9Gdx+MBWCKjg9TQpAQRfxQ9ZaFny7gL7Jfdl1bBf33XQfEztPpEHlBk5Pk0JQ0EX8zJYftxCZFEna7jSaVGlCUq8kOtfv7PQs8QAFXcRPHDlzhKGLh/LP9f+kQukKvPmXN3kh9AVKllAG3EK/kyIudyH3Au+sfYfhS4Zz6vwpXgx9keFhw6kcXNnpaeJhCrqIi3353ZdEJ0fzzZFv6Fy/M/Gd47m56s1OzxIvUdBFXOibI98QnRTNlzu/5MZrb+Q/j/2Hexrco7e1dTkFXcRFjp87zivpr/DW2rcILhXMhLsm0Of2PgQGBDo9TXxAQRdxgZy8HKZumMrgtMEcPXuU51o8x6g7R1G1TFWnp4kPKegixVza7jQiEyPZcmgL7Wu3J+HuBJpXb+70LHGAgi5STO06tot+yf349zf/pk7FOnzU4yMeavyQzpP7sUIF3RgTBTwHWGAL8Iy19pwnhonIrzt1/hSjl40mflU8pUqU4tU7XyW6dTRBJYOcniYOK3DQjTE1gL8DTay1Z40x84BHgRke2iYil8izeczcNJOBaQM5ePogT97yJGPCx3B9ueudniZFRGFPuZQErjHGXACCgf8WfpKIXG7FvhVEJEawPnM9rWq2YsGjC2hZo6XTs6SIKXDQrbUHjDETgH3AWSDZWpvssWUiwr4T+xiQMoA5W+dQo1wNZj0wi8f/9LjOk8uvKswpl0pAN6AucBz4yBjTy1o767Lb9QZ6A9SqVasQU0X8R1Z2FuNXjOe1la9hsQxpP4QBbQdQJrCM09OkCCvMKZdOwG5r7WEAY8x8oA3wi6Bba6cAUwBCQ0NtIY4n4nrWWuZsnUNMSgz7T+6n5809GddpHLUr1nZ6mhQDhQn6PqCVMSaYi6dcwoF1Hlkl4ofWHlhLZFIkK39YSYvqLZj94GzuqH2H07OkGCnMOfTVxpiPgQ1ADrCR/EfiInLlMk9lEpcax8zNM6lapirT7p/GU7c8pcu/yVUr1LNcrLXDgGEe2iLiV87lnGNSxiRGLx9Ndm42MW1iGNR+EOVLl3d6mhRTeqWoiI9Za5m/fT79F/Vn9/HddG/UnQl3TaD+tfWdnibFnIIu4kObD24mMimS9D3pNK3alJQnUgivF+70LHEJBV3EBw5nHWZw2mCmbpxKpaBKvNP1HZ6/7Xld/k08Sn+aRLwoOzebt9a8xYglI8i6kEWfln0Y1mEYla6p5PQ0cSEFXcQLrLV88d0XRCdHs+OnHXS5sQvxneNpXKWx09PExRR0EQ/bfng7UUlRJH2fRMPKDfn88c/p2qCr07PEDyjoIh5y9OxRXkl/hbfXvk3ZwLLEd47npZYv6fJv4jMKukgh5eTl8M91/2Ro+lCOnztO7xa9GdFxBFXKVHF6mvgZBV2kEFJ2pRCZGMm2w9voWKcjCV0SaHZdM6dniZ9S0EUKYOfRnfRN7stn335GvUr1mP/IfLo36q63tRVHKegiV+Hk+ZOMWjqKhFUJlC5ZmjHhY4hsFanLv0mRoKCLXIHcvFxmbJrBwLSBHMo6xDO3PsOrd75K9XLVnZ4m8jMFXeQPLNu7jIjECDYe3EibG9rw+eOfE3p9qNOzRP4PBV3kN+w9vpeYlBjmbZtHzfI1+fChD+l5c0+dJ5ciS0EXuUxWdhbjVozjtZWvYTAM7zCc/m37E1wq2OlpIr9LQRfJl2fzmL1lNrEpsRw4dYDHmj7GuE7juKHCDU5PE7kiCroIsObAGiISI1i1fxWh14cyr8c82tzQxulZIldFQRe/duDkAeJS43j/q/epVrYaM7rN4IlbnqCEKeH0NJGrpqCLXzp74SzxGfGMXj6anLwc4trFEdcujnKlyzk9TaTAFHTxK9ZaPtn+Cf2S+7H3xF4ebPwgr931GvUq1XN6mkihKejiNzYd3EREYgRL9y6l2XXNSOuWRse6HZ2eJeIxCrq43qGsQxcv/7ZhKpWDKzP5nsk81+I5AkoEOD1NxKMUdHGt7Nxs3lj9BiOXjuTMhTNEtYpiSIchVAyq6PQ0Ea9Q0MV1rLUs3LGQ6ORodh7dyT0N7mFi54k0DGno9DQRr1LQxVW2HdpGVFIUi3YtolFII77865d0ubGL07NEfEJBF1c4evYowxYP4x/r/kG50uV4vcvr/C30b5QKKOX0NBGfUdClWMvJy2HyuskMXTyUE+dP8MJtL/BKx1cICQ5xepqIzynoUmwlf59MVFIUXx/+mvC64SR0SaBp1aZOzxJxjIIuxc6On3bQN7kvC3cspH6l+nza81Pub3i/3tZW/F6hgm6MqQhMBZoCFnjWWpvhiWEilztx7gQjl47kjdVvEFQyiPGdxvP32/9O6ZKlnZ4mUiQU9hH660CitfZhY0wgoDeMFo/Lzcvl3Y3vMihtEEfOHOHZ5s8y6s5RVCtbzelpIkVKgYNujCkPtAeeBrDWZgPZnpklctGSPUuITIpk08FNtKvVjsQuibSo3sLpWSJFUmHeI7QecBiYbozZaIyZaowpc/mNjDG9jTHrjDHrDh8+XIjDiT/Zc3wPPT7qQdjMMI6ePcrch+ey9OmlirnI7yhM0EsCLYB/WGubA1lA7OU3stZOsdaGWmtDq1SpUojDiT84nX2aQamDaPRWI7747gtGhI3gm5e+4ZGbH9E3PUX+QGHOoe8H9ltrV+d//jG/EnSRK5Fn85j11SxiU2LJPJ1Jr2a9GBM+hprlazo9TaTYKHDQrbUHjTE/GGMaWmu/BcKBrz03TfzFqv2riEiMYM2BNbSs0ZL5PefTqmYrp2eJFDuFfZZLH+CD/Ge47AKeKfwk8Rf7T+4nNiWWD7Z8QPWy1ZnZfSa9mvXS5d9ECqhQQbfWbgJCPbRF/MTZC2eZsHICY1eMJTcvl0F3DCK2XSxlA8s6PU2kWNMrRcVnrLXM2zaPmJQY9p3Yx8NNHmZ8p/HUrVTX6WkirqCgi09syNxARGIEy/ct59Zqt/Je9/foUKeD07NEXEVBF686ePogg1IHMX3TdEKCQ5hy7xSebf6sLv8m4gUKunjF+ZzzvL76dUYtHcW5nHNEt45mSPshVAiq4PQ0EddS0MWjrLV89u1n9E3uy/fHvue+m+5jYueJNKjcwOlpIq6noIvHbD20lcjESFJ3p9KkShOSeiXRuX5np2eJ+A0FXQrtyJkjDFs8jMnrJ1OhdAXe/MubvBD6AiVL6I+XiC/pb5wU2IXcC7yz9h2GLxnOqfOneDH0RYaHDadycGWnp4n4JQVdCiRxZyJRSVF8c+Qb7qp3F5PunsTNVW92epaIX1PQi6iMjAzS09MJCwujdevWTs/52bdHviU6OZovvvuCG6+9kc8e/Yx7b7pX74QoUgQo6EVQRkYG4eHhZGdnExgYSGpqquNRP37uOCOWjODNNW8SXCqYCXdNoM/tfQgMCHR0l4j8fwp6EZSenk52dja5ublkZ2eTnp7uWNBz83KZumEqgxcP5qczP/Fci+cYdecoqpap6sgeEfltCnoRFBYWRmBg4M+P0MPCwhzZsXj3YiKTIvnqx69oX7s9CXcn0Lx6c0e2iMgfU9CLoNatW5OamurYOfRdx3bRf1F/5m+fT+0Ktfmox0c81PghnScXKeIU9CKqdevWPg/5qfOnGL1sNPGr4ilVohSjOo4iunU015S6xqc7RKRgFHQhz+bx3ub3iEuN4+DpgzzR7AnGhI+hRvkaTk8TkaugoPu5FftWEJkUybr/rqNVzVZ82vNTbq95u9OzRKQAFHQ/te/EPgakDGDO1jnUKFeDWQ/M4rE/PabLv4kUYwq6nzlz4QzjV4xn/IrxWCxD2g9hQNsBlAks4/Q0ESkkBd1PWGuZs3UOMSkx7D+5n0dufoTxncZTu2Jtp6eJiIco6H5g3X/XEZEYwcofVtK8WnNmPzibO2rf4fQsEfEwBd3FMk9lMjBtIDM2zaBqmapMvW8qT9/6tC7/JuJSCroLncs5x6SMSYxePprzOeeJaRPDoPaDKF+6vNPTRMSLFHQXsdby72/+Tb/kfuw+vptuDbsxofMEbrz2RqeniYgPKOgusfngZiKTIknfk07Tqk1JeSKF8HrhTs8SER9S0Iu5w1mHGbJ4CP/a8C8qBlXk7a5v0/u23rr8m4gf0t/6Yio7N5u31rzFiCUjOJ19mpf//DLDwoZx7TXXOj1NRByioBcz1lq++O4LopOj2fHTDu6ufzeT7p5E4yqNnZ4mIg4rdNCNMQHAOuCAtfbewk+S37L98HaikqJI+j6JmyrfxMLHFtK1QVe9ra2IAJ55hB4BbAf0nDgvOXb2GMPTh/P22rcpG1iWiZ0n8nLLl3X5NxH5hUIF3RhTE7gHeBWI9sgi+VlOXg5T1k9h6OKhHDt3jOdbPM/IjiOpUqaK09NEpAgq7CP0BCAGKOeBLXKJlF0pRCVFsfXQVjrW6UhClwSaXdfM6VkiUoQV+L1SjTH3Aoestev/4Ha9jTHrjDHrDh8+XNDD+Y2dR3fSfU537nr/LrKys/jkkU9IfTJVMReRP1SYR+htgfuNMV2BIKC8MWaWtbbXpTey1k4BpgCEhobaQhzP1U6eP8mopaNIWJVAYEAgo+8cTVTrKIJKBjk9TUSKiQIH3VobB8QBGGPCgH6Xx1z+WG5eLjM2zWBg2kAOZR3i6VufZvSdo6lerrrT00SkmNHz0B20bO8yIhIj2HhwI21uaMPCxxby5xp/dnqWiBRTHgm6tTYdSPfEr+UP9h7fS0xKDPO2zaNm+ZrMfnA2jzZ9VM8nF5FC0SN0H8rKzmLcinG8tvI1DIZhHYYR0zaG4FLBTk8TERdQ0H0gz+Yxe8tsYlNiOXDqAI81fYxxncZxQ4UbnJ4mIi6ioHvZmgNriEiMYNX+VdxW/TbmPjyXtrXaOj1LRFxIQfeSAycPEJcax/tfvU+1stWY3m06T97yJCVMgZ/6LyLyuxR0Dzt74SzxGfGMXj6anLwcYtvGMvCOgZQrrRfTioh3KegeYq3lk+2f0C+5H3tP7OWBRg8wofME6lWq5/Q0EfETCroHbDq4iYjECJbuXUqz65qR1i2NjnU7Oj1LRPyMgl4Ih7IOMThtMFM3TKVycGUm3zOZ51o8R0CJAKeniYgfUtALIDs3mzdWv8HIpSM5c+EMka0iGdphKBWDKjo9TUT8mIJ+Fay1LNyxkOjkaHYe3UnXBl2J7xxPw5CGTk8TEVHQf09GRgbp6emEhYVRvn55opKiWLRrEY1CGvHlX7+ky41dnJ4oIvIzBf03ZGRkEB4ezvkS5zEZBkKhXOlyJNydwIt/fpFSAaWcnigi8gsK+m9IXZzKuWbnsGEWSsPt5nYW9llISHCI09NERH6Vgv4rkr9P5t3Ad7F/sbALSi8uzaS5kxRzESnSFPRL7PhpB32T+7Jwx0LqV6rP2AZjyS2VS8cXOtK6dWun54mI/C4FHThx7gQjl47kjdVvEFQyiHGdxhFxewSlS5aGbk6vExG5Mn4d9Ny8XKZtnMbgtMEcOXOEZ5s/y6g7R1GtbDWnp4mIXDW/DfqSPUuISIxg84+baVerHYldEmlRvYXTs0RECszvgr7n+B76L+rPx19/TK0KtZj78Fx6NOmhy7+JSLHnN0E/nX2aMcvGMDFjIgElAhgRNoJ+bfpxTalrnJ4mIuIRrg96ns1j1leziE2JJfN0Jn/9018Z22ksNcvXdHqaiIhHuTroGT9kEJkUyZoDa2hZoyXze86nVc1WTs8SEfEKVwZ9/8n9xKbE8sGWD6hetjozu8+kV7NeuvybiLiaq4J+5sIZJqycwLgV48jNy2XQHYOIbRdL2cCyTk8TEfE6VwTdWsu8bfOISYlh34l9PNzkYcZ3Gk/dSnWdniYi4jPFPugbMjcQkRjB8n3LubXarbzX/T061Ong9CwREZ8rtkE/ePogg1IHMX3TdEKCQ5hy7xSebf6sLv8mIn6r2AX9fM55Xl/9OqOWjuJczjmiW0czpP0QKgRVcHqaiIijik3QrbUs+HYB/ZL78f2x77nvpvuY2HkiDSo3cHqaiEiRUOCgG2NuAN4DqgF5wBRr7eueGnapLT9uISopitTdqTSp0oSkXkl0rt/ZG4cSESm2CvMIPQfoa63dYIwpB6w3xiyy1n7toW0/6/NlH7768Sve/MubvBD6AiVLFJt/WIiI+EyBy2itzQQy8z8+ZYzZDtQAPB70afdPo2JQRSoHV/b0Ly0i4hoeeahrjKkDNAdW/8rP9QZ6A9SqVatAv379a+sXfJyIiJ8o9GvhjTFlgU+ASGvtyct/3lo7xVobaq0NrVKlSmEPJyIiv6FQQTfGlOJizD+w1s73zCQRESmIAgfdXLwixDRgu7U23nOTRESkIArzCL0t8ARwpzFmU/7/unpol4iIXKXCPMtlOaDrtomIFBF6g3AREZdQ0EVEXEJBFxFxCQVdRMQlFHQREZdQ0EVEXEJBFxFxCQVdRMQlFHQREZdQ0EVEXEJBFxFxCQVdRMQlFHQREZcoFkHPyMhgzJgxZGRkOD1FRKTI8sg1Rb0pIyOD8PBwsrOzCQwMJDU1ldatWzs9S0SkyCnyj9DT09PJzs4mNzeX7Oxs0tPTnZ4kIlIkFfmgh4WFERgYSEBAAIGBgYSFhTk9SUSkSCryp1xat25Namoq6enphIWF6XSLiMhvKPJBh4tRV8hFRH5fkT/lIiIiV0ZBFxFxCQVdRMQlFHQREZdQ0EVEXEJBFxFxCWOt9d3BjDkM7C3g/z0EOOLBOcWB7rN/0H12v8Le39rW2ip/dCOfBr0wjDHrrLWhTu/wJd1n/6D77H6+ur865SIi4hIKuoiISxSnoE9xeoADdJ/9g+6z+/nk/habc+giIvL7itMjdBER+R3FIujGmC7GmG+NMTuNMbFO7/EmY8wNxpjFxpjtxphtxpgIpzf5ijEmwBiz0Riz0OktvmCMqWiM+dgY803+77fr31LUGBOV/+d6qzHmQ2NMkNObPM0Y864x5pAxZuslX7vWGLPIGPNd/o+VvHHsIh90Y0wA8DbwF6AJ8Jgxpomzq7wqB+hrrW0MtAJecvn9vVQEsN3pET70OpBorW0E3ILL77sxpgbwdyDUWtsUCAAedXaVV8wAulz2tVgg1VrbAEjN/9zjinzQgZbATmvtLmttNjAH6ObwJq+x1mZaazfkf3yKi3/Jazi7yvuMMTWBe4CpTm/xBWNMeaA9MA3AWpttrT3u7CqfKAlcY4wpCQQD/3V4j8dZa5cCRy/7cjdgZv7HM4Hu3jh2cQh6DeCHSz7fjx8EDsAYUwdoDqx2dolPJAAxQJ7TQ3ykHnAYmJ5/mmmqMaaM06O8yVp7AJgA7AMygRPW2mRnV/nMddbaTLj4oA2o6o2DFIegm1/5muufmmOMKQt8AkRaa086vcebjDH3Aoesteud3uJDJYEWwD+stc2BLLz0z/CiIv+8cTegLnA9UMYY08vZVe5SHIK+H7jhks9r4sJ/pl3KGFOKizH/wFo73+k9PtAWuN8Ys4eLp9TuNMbMcnaS1+0H9ltr/9+/vj7mYuDdrBOw21p72Fp7AZgPtHF4k6/8aIypDpD/4yFvHKQ4BH0t0MAYU9cYE8jFb6J85vAmrzHGGC6eV91urY13eo8vWGvjrLU1rbV1uPj7m2atdfUjN2vtQeAHY0zD/C+FA187OMkX9gGtjDHB+X/Ow3H5N4Iv8RnwVP7HTwELvHGQIn+RaGttjjHmZSCJi98Vf9dau83hWd7UFngC2GKM2ZT/tYHW2i8c3CTe0Qf4IP+Byi7gGYf3eJW1drUx5mNgAxefzbURF75i1BjzIRAGhBhj9gPDgLHAPGPM/3DxP2w9vHJsvVJURMQdisMpFxERuQIKuoiISyjoIiIuoaCLiLiEgi4i4hIKuoiISyjoIiIuoaCLiLjE/wK+8atC/OEbDwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 计算损失函数 J 的梯度\n",
    "def gradient_function(theta, X, y):\n",
    "    diff = np.dot(X, theta) - y\n",
    "    return (1./y.size) * np.dot(np.transpose(X), diff)\n",
    "\n",
    "# 梯度下降\n",
    "def gradient_descent(X, y, eta, theta_init, max_iter):\n",
    "    theta = theta_init\n",
    "    i = 0\n",
    "    gradient = gradient_function(theta, X, y)\n",
    "    while not np.all(np.absolute(gradient) <= 1e-10):\n",
    "        theta = theta - eta * gradient\n",
    "#         print('theta:', theta[0,0], theta[1,0])\n",
    "        i += 1\n",
    "        if i > max_iter:\n",
    "            break\n",
    "        gradient = gradient_function(theta, X, y)\n",
    "#         print('gradient:', gradient[0,0], gradient[1,0])\n",
    "    return theta\n",
    "\n",
    "# 损失函数\n",
    "def loss_function(theta, X, y):\n",
    "    diff = np.dot(X, theta) - y\n",
    "    return (1./2*y.size) * np.dot(np.transpose(diff), diff)\n",
    "\n",
    "# 数据集\n",
    "X0 = np.ones((3, 1))\n",
    "X1 = np.array([0,1,2]).reshape(-1, 1)\n",
    "X = np.hstack((X0, X1))\n",
    "y = np.array([0.9, 3.1, 5.1]).reshape(-1, 1)\n",
    "\n",
    "# 学习率\n",
    "eta = 0.01\n",
    "\n",
    "# 最大迭代次数\n",
    "max_iter = 10000\n",
    "\n",
    "# 初始值\n",
    "theta_init = np.array([0, 0]).reshape(-1, 1)\n",
    "\n",
    "# 使用梯度下降求解线性回归\n",
    "theta = gradient_descent(X, y, eta, theta_init, max_iter)\n",
    "print('theta:', theta[0,0], theta[1,0])\n",
    "print('loss:', loss_function(theta, X, y))\n",
    "\n",
    "# 绘制图形\n",
    "plt.plot(X1, y, 'k.')\n",
    "x = np.linspace(0, 10, 10)\n",
    "y = theta[0,0]*x+theta[1,0]\n",
    "plt.plot(x, y, 'g-')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 疑惑一\n",
    "\n",
    "为什么上面两种方法求出来的解不一样？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 疑惑二\n",
    "\n",
    "在前面介绍正规方程法解一元线性回归时，使用了下面的数据集，这个数据集不知道为什么用梯度下降法无法收敛？\n",
    "\n",
    "```\n",
    "X = np.array([39.93, 42.05, 43.18, 44.68, 49.87, 53.57])\n",
    "Y = np.array([199,   290,   298,   310,   399,   420])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theta: 0.7489393245009788 7.078701795630663\n",
      "loss: [[32892.8042478]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFhdJREFUeJzt3X+MXWWdx/H3h+nc/gIp0Aup/bFlta4/QyEjy4SNGalRQGMxkQ2uuxLSpJrgBo1RwH/EraSYrNY1ZslWQaqrQoMSGsKydgcmrskITqUiUAxVkI7t0mH5IWxrp51+94/73HI7c2fumbkzvXOffl7JzT3nuc+5fE9P+czpd869RxGBmZnl65RWF2BmZjPLQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWVuTqsLAFi8eHGsXLmy1WWYmbWVHTt2vBAR5UbzZkXQr1y5koGBgVaXYWbWViT9ocg8t27MzDLnoDczy5yD3swscw56M7PMOejNzDLnoDczy5yD3sxaor+/n40bN9Lf39/qUrI3K66jN7OTS39/P2vWrGF4eJhSqURvby/d3d2tLitbPqM3sxOur6+P4eFhRkZGGB4epq+vr9UlZc1Bb2YnXE9PD6VSiY6ODkqlEj09Pa0uKWtu3ZjZCdfd3U1vby99fX309PS4bTPDHPRm1hLd3d0O+BPErRszs8w56M3MMuegNzPLXOGgl9Qh6VFJ96X1cyU9LOlpSXdJKqXxuWl9d3p95cyUbmZmRUzmjP46YFfN+leBTRGxCngJWJfG1wEvRcSbgU1pnpmZtUihoJe0DPgg8J20LuAS4O40ZQtwRVpem9ZJr69J882sBfxVA1b08spvAF8ATkvrZwEvR8SRtD4ILE3LS4E9ABFxRNIraf4LtW8oaT2wHmDFihVTrd/MJuCvGjAocEYv6UPA/ojYUTtcZ2oUeO31gYjNEdEVEV3lcsN725rZFPirBgyKndFfDHxY0uXAPOANVM7wF0mak87qlwF70/xBYDkwKGkOcDrw4rRXbmYNVb9qoHpG768aODk1PKOPiBsjYllErASuAh6MiI8DDwEfTdOuBu5Ny9vSOun1ByNizBm9mc286lcNbNiwwW2bk1gzX4FwPXCnpK8AjwK3pfHbgO9L2k3lTP6q5ko0s2b4qwZsUkEfEX1AX1r+PXBhnTl/Bq6chtrMzGwa+JOxZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5orcHHyepEck/VrSE5K+nMbvkPSMpJ3psTqNS9I3Je2W9JikC2Z6J8zMbHxF7jB1CLgkIl6T1An8XNJ/pNc+HxF3j5p/GbAqPf4auDU9m5lZCxS5OXhExGtptTM9JrrZ91rge2m7XwCLJC1pvlQzM5uKQj16SR2SdgL7ge0R8XB66ebUntkkaW4aWwrsqdl8MI2ZmVkLFAr6iBiJiNXAMuBCSe8EbgTeCrwbOBO4Pk1XvbcYPSBpvaQBSQNDQ0NTKt7MzBqb1FU3EfEy0AdcGhH7UnvmEPBd4MI0bRBYXrPZMmBvnffaHBFdEdFVLpenVLyZmTVW5KqbsqRFaXk+8D7gqWrfXZKAK4DH0ybbgE+kq28uAl6JiH0zUr2ZmTVU5KqbJcAWSR1UfjBsjYj7JD0oqUylVbMT+FSafz9wObAbOABcM/1lm5lZUQ2DPiIeA86vM37JOPMDuLb50szMbDr4k7FmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JvV6O/vZ+PGjfT397e6FLNpU+T76M1OCv39/axZs4bh4WFKpRK9vb10d3e3uiyzpvmM3izp6+tjeHiYkZERhoeH6evra3VJZtOiyK0E50l6RNKvJT0h6ctp/FxJD0t6WtJdkkppfG5a351eXzmzu2A2PXp6eiiVSnR0dFAqlejp6Wl1SWbTosgZ/SHgkog4D1gNXJruBftVYFNErAJeAtal+euAlyLizcCmNM9s1uvu7qa3t5cNGza4bWNZKXIrwQBeS6ud6RHAJcDfpfEtwE3ArcDatAxwN/AtSUrvYzardXd3O+AtO4V69JI6JO0E9gPbgd8BL0fEkTRlEFialpcCewDS668AZ01n0WYT8ZUzZscrdNVNRIwAqyUtAu4B3lZvWnrWBK8dI2k9sB5gxYoVhYo1a8RXzpiNNamrbiLiZaAPuAhYJKn6g2IZsDctDwLLAdLrpwMv1nmvzRHRFRFd5XJ5atWbjeIrZ8zGKnLVTTmdySNpPvA+YBfwEPDRNO1q4N60vC2tk15/0P15O1F85YzZWEVaN0uALZI6qPxg2BoR90l6ErhT0leAR4Hb0vzbgO9L2k3lTP6qGajbTjL9/f309fXR09MzYSumeuVMkblmJwvNhpPtrq6uGBgYaHUZNku5725Wn6QdEdHVaJ4/GWuznvvuZs1x0Nus5767WXP8pWY267nvbtYcB721BX9i1Wzq3LoxM8ucg97MLHMOejOzzDnozcwy56A3M8ucg97MLHMOejOzzDnozcwy56A3M8ucg97MLHMOejOzzDnozcwy56A3M8tckXvGLpf0kKRdkp6QdF0av0nSHyXtTI/La7a5UdJuSb+V9IGZ3AEzM5tYka8pPgJ8LiJ+Jek0YIek7em1TRHxz7WTJb2dyn1i3wG8EfgvSW+JiJHpLNzMzIppeEYfEfsi4ldp+VVgF7B0gk3WAndGxKGIeAbYDVw4HcWamdnkTapHL2klcD7wcBr6tKTHJN0u6Yw0thTYU7PZIHV+MEhaL2lA0sDQ0NCkCzczs2IKB72kU4EfA5+JiD8BtwJvAlYD+4CvVafW2TzGDERsjoiuiOgql8uTLtzMzIopFPSSOqmE/A8i4icAEfF8RIxExFHg27zenhkEltdsvgzYO30lm5nZZBS56kbAbcCuiPh6zfiSmmkfAR5Py9uAqyTNlXQusAp4ZPpKNjOzyShy1c3FwD8Av5G0M419EfiYpNVU2jLPAp8EiIgnJG0FnqRyxc61vuLGzKx1GgZ9RPyc+n33+yfY5mbg5ibqMjOzaeJPxpqZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZpkrcivB5ZIekrRL0hOSrkvjZ0raLunp9HxGGpekb0raLekxSRfM9E6Ymdn4ipzRHwE+FxFvAy4CrpX0duAGoDciVgG9aR3gMir3iV0FrAdunfaqzcyssIZBHxH7IuJXaflVYBewFFgLbEnTtgBXpOW1wPei4hfAolE3EreM9ff3s3HjRvr7+1tdipklRW4OfoyklcD5wMPAORGxDyo/DCSdnaYtBfbUbDaYxvY1W6zNbv39/axZs4bh4WFKpRK9vb10d3e3uiyzk17hX8ZKOhX4MfCZiPjTRFPrjEWd91svaUDSwNDQUNEybBbr6+tjeHiYkZERhoeH6evra3VJZkbBoJfUSSXkfxARP0nDz1dbMul5fxofBJbXbL4M2Dv6PSNic0R0RURXuVyeav02i/T09FAqlejo6KBUKtHT09PqksyMYlfdCLgN2BURX695aRtwdVq+Gri3ZvwT6eqbi4BXqi0ey1t3dze9vb1s2LDBbRuzWUQRY7oqx0+Q/gb4b+A3wNE0/EUqffqtwArgOeDKiHgx/WD4FnApcAC4JiIGJvpvdHV1xcDAhFPMzGwUSTsioqvRvIa/jI2In1O/7w6wps78AK5tWKGZmZ0Q/mSsmVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWuSK3Erxd0n5Jj9eM3STpj5J2psflNa/dKGm3pN9K+sBMFW5mZsUUOaO/g8ptAUfbFBGr0+N+AElvB64C3pG2+VdJHdNVrJmZTV7DoI+InwEvFny/tcCdEXEoIp4BdgMXNlGfmZk1qZke/aclPZZaO2eksaXAnpo5g2nMzMxaZKpBfyvwJmA1sA/4WhqvdxPxqPcGktZLGpA0MDQ0NMUyzMyskSkFfUQ8HxEjEXEU+Davt2cGgeU1U5cBe8d5j80R0RURXeVyeSplmJlZAVMKeklLalY/AlSvyNkGXCVprqRzgVXAI82VaGZmzZjTaIKkHwE9wGJJg8CXgB5Jq6m0ZZ4FPgkQEU9I2go8CRwBro2IkZkp3czMilBE3Rb6CdXV1RUDAwOtLsPMrK1I2hERXY3m+ZOxZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZaxj0km6XtF/S4zVjZ0raLunp9HxGGpekb0raLekxSRfMZPFmZtZYkTP6O4BLR43dAPRGxCqgN60DXEblhuCrgPXArdNTppmZTVXDoI+InwEvjhpeC2xJy1uAK2rGvxcVvwAWSVoyXcWamdnkTbVHf05E7ANIz2en8aXAnpp5g2lsDEnrJQ1IGhgaGppiGWZm1sh0/zJWdcai3sSI2BwRXRHRVS6Xp7kMMzOrmmrQP19tyaTn/Wl8EFheM28ZsHfq5ZmZWbOmGvTbgKvT8tXAvTXjn0hX31wEvFJt8ZiZWWvMaTRB0o+AHmCxpEHgS8AtwFZJ64DngCvT9PuBy4HdwAHgmhmo2czMJqFh0EfEx8Z5aU2duQFc22xRZmY2ffzJWDOzzDnozcwy56A3M8ucg97MLHMOejOzzDnozcwy56A3M8ucg97MLHMOejOzzDnozcwy56A3M8ucg97MLHMOejOzzDnozcwy56A3M8ucg97MLHMNbzwyEUnPAq8CI8CRiOiSdCZwF7ASeBb424h4qbkyzcxsqpoK+uS9EfFCzfoNQG9E3CLphrR+/TT8d8zM2taRo0c4ePggBw4f4MDhAxw8cpCDhw9SXlhmxekrZvS/PR1BP9paKveYBdgC9OGgN7NZ6Ggc5eDhgxw88noAHzh84FggV8O4ulzv9drgrn29ul6de/jo4bo1XH/x9dzyvltmdD+bDfoAfiopgH+LiM3AORGxDyAi9kk6u9kizezkEREcGjk0bqjWnhWPF8b1grteUB8aOTSlGjtP6WRB5wLmd85n/pz5LCwtrKzPmU95QfnY+ILOBcfGjy13zj9u7C1nvWWa/wTHajboL46IvSnMt0t6quiGktYD6wFWrJjZf7aYWXMiguGR4fqhenjsWW29sdGhXG+b6utBTLrGDnUcF6S1Ybpo3iKWdC45NlYdHx26o8fmd85nYefCMeMdp3TMwJ/yzGkq6CNib3reL+ke4ELgeUlL0tn8EmD/ONtuBjYDdHV1Tf6omlndvu94rYVCLYkJtj8aRyddn9BxZ761gbmwtJDFCxYff2ZcJ1TrBW/tmXR1vPOUTiTNwJ9y+5ty0EtaCJwSEa+m5fcD/wRsA64GbknP905HoWbtYuToyJggLdLnPe5M98j47Yba9xyv79vI3I65x7UbasN2yalLXg/bOfXPcidqSdTOm9sx1+E7CzRzRn8OcE86iHOAH0bEA5J+CWyVtA54Driy+TLNmhMR/PnInwv1eSf8xVqB7afa9y11lOqexS7oXHCs77ugcwEL5iwY2wOuE7LjBfG8OfM4Rf4IzclkykEfEb8Hzqsz/r/AmmaKspNDbd93oqsVmr3aoTpnKhr2fU9b8voZ8ST7vKPfs936vtY+ZuLySmtzR44eaRi0DX+xNk67YnQYN9P3Ha/3W15YPr6f26DdMLp1UdvS6OzonIE/YbMTy0HfJur1fRtd8TDhmfEEc6fa960Gar0z20XzFr0euHPqh+pE7Yba8VJHyX1fs0lw0Deh2vedbNAWueZ39HsOjwxPqcZSR2ncM9p6fd9G1/2OdwWE+75ms1d2QV/t+9a7PrfI1Q6Ner/T2fetF6SL5i3ijae9cWzYjmovjG491F5mVnv5mfu+ZtbWQf/A7gf47H9+dkxYT+XDFuP1fRd0LuDU0qmcvfDs4wN3EpeZjX5P933N7ERq66A/fe7pvOvsdxW+umGiOe77mlmu2jrou5d30728u9VlmJnNav7tmZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljlFtP4ufpKGgD9McfPFwAvTWE4reV9mp1z2JZf9AO9L1V9ERLnRpFkR9M2QNBARXa2uYzp4X2anXPYll/0A78tkuXVjZpY5B72ZWeZyCPrNrS5gGnlfZqdc9iWX/QDvy6S0fY/ezMwmlsMZvZmZTaCtgl7SPEmPSPq1pCckfTmNnyvpYUlPS7pLUqnVtTYywb7cIekZSTvTY3Wray1CUoekRyXdl9bb7phU1dmXdj0mz0r6Tap5II2dKWl7Oi7bJZ3R6jobGWc/bpL0x5pjcnmr6yxC0iJJd0t6StIuSd0n4pi0VdADh4BLIuI8YDVwqaSLgK8CmyJiFfASsK6FNRY13r4AfD4iVqfHztaVOCnXAbtq1tvxmFSN3hdoz2MC8N5Uc/XyvRuA3nRcetN6Oxi9H1D5+1U9Jve3rLLJ+RfggYh4K3Aelb9nM35M2iroo+K1tNqZHgFcAtydxrcAV7SgvEmZYF/ajqRlwAeB76R10YbHBMbuS4bWUjke0EbHJQeS3gC8B7gNICKGI+JlTsAxaaugh2P/rN4J7Ae2A78DXo6II2nKILC0VfVNxuh9iYiH00s3S3pM0iZJc1tYYlHfAL4AHE3rZ9Gmx4Sx+1LVbscEKicOP5W0Q9L6NHZOROwDSM9nt6y64urtB8Cn0zG5vR1aUMBfAkPAd1Nr8DuSFnICjknbBX1EjETEamAZcCHwtnrTTmxVUzN6XyS9E7gReCvwbuBM4PoWltiQpA8B+yNiR+1wnamz/piMsy/QZsekxsURcQFwGXCtpPe0uqApqrcftwJvotL23Ad8rYX1FTUHuAC4NSLOB/6PE9Q6a7ugr0r/5OkDLgIWSare6HwZsLdVdU1Fzb5cGhH7UlvnEPBdKj/MZrOLgQ9Leha4k0rL5hu05zEZsy+S/r0NjwkAEbE3Pe8H7qFS9/OSlgCk5/2tq7CYevsREc+nE6WjwLdpj2MyCAzW/Mv9birBP+PHpK2CXlJZ0qK0PB94H5VfZjwEfDRNuxq4tzUVFjfOvjxVc8BFpVf3eOuqbCwiboyIZRGxErgKeDAiPk4bHpNx9uXv2+2YAEhaKOm06jLwfip1b6NyPKANjst4+1E9JslHaINjEhH/A+yR9FdpaA3wJCfgmMxpPGVWWQJskdRB5YfU1oi4T9KTwJ2SvgI8Svplxyw33r48KKlMpf2xE/hUK4tswvW03zEZzw/a8JicA9xT+dnEHOCHEfGApF8CWyWtA54DrmxhjUWMtx/fT5e5BvAs8MnWlTgp/0jl71MJ+D1wDen//5k8Jv5krJlZ5tqqdWNmZpPnoDczy5yD3swscw56M7PMOejNzDLnoDczy5yD3swscw56M7PM/T9AESuWJzRV7AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 计算损失函数 J 的梯度\n",
    "def gradient_function(theta, X, y):\n",
    "    diff = np.dot(X, theta) - y\n",
    "    return (1./y.size) * np.dot(np.transpose(X), diff)\n",
    "\n",
    "# 梯度下降\n",
    "def gradient_descent(X, y, eta, theta_init, max_iter):\n",
    "    theta = theta_init\n",
    "    i = 0\n",
    "    gradient = gradient_function(theta, X, y)\n",
    "    while not np.all(np.absolute(gradient) <= 1e-10):\n",
    "        theta = theta - eta * gradient\n",
    "        i += 1\n",
    "        if i > max_iter:\n",
    "            break\n",
    "        gradient = gradient_function(theta, X, y)\n",
    "    return theta\n",
    "\n",
    "# 损失函数\n",
    "def loss_function(theta, X, y):\n",
    "    diff = np.dot(X, theta) - y\n",
    "    return (1./2*y.size) * np.dot(np.transpose(diff), diff)\n",
    "\n",
    "# 数据集\n",
    "X0 = np.ones((6, 1))\n",
    "X1 = np.array([39.93, 42.05, 43.18, 44.68, 49.87, 53.57]).reshape(-1, 1)\n",
    "X = np.hstack((X0, X1))\n",
    "y = np.array([199,   290,   298,   310,   399,   420]).reshape(-1, 1)\n",
    "\n",
    "# 学习率\n",
    "eta = 0.00001\n",
    "\n",
    "# 最大迭代次数\n",
    "max_iter = 10000\n",
    "\n",
    "# 初始值\n",
    "theta_init = np.array([1, 1]).reshape(-1, 1)\n",
    "\n",
    "# 使用梯度下降求解线性回归\n",
    "theta = gradient_descent(X, y, eta, theta_init, max_iter)\n",
    "print('theta:', theta[0,0], theta[1,0])\n",
    "print('loss:', loss_function(theta, X, y))\n",
    "\n",
    "# 绘制图形\n",
    "plt.plot(X1, y, 'k.')\n",
    "x = np.linspace(30, 60, 100)\n",
    "y = theta[0,0]*x+theta[1,0]\n",
    "plt.plot(x, y, 'g-')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}