without_noise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PDE 1 - 2D\n",
    "\n",
    "The problem setup is the same as in *2D_with_multiple_optimizers*.\n",
    "\n",
    "We can calculate that without noise ($s=0$), the minimization problem becomes the following:\n",
    "\\begin{equation}\n",
    "min_{\\theta} \\{\\det((y^T \\tilde{K}^{-1} y) \\tilde{K})\\},\n",
    "\\end{equation}\n",
    "where $K = \\sigma \\tilde{K}$. For stability purposes we re-add the noise parameter to $\\tilde{K}$. \n",
    "\n",
    "The gradient is given by\n",
    "\\begin{equation}\n",
    "tr(A^{-1}\\partial_{\\theta_i}A),\n",
    "\\end{equation}\n",
    "where $A = (y^T \\tilde{K}^{-1} y) \\tilde{K}$. \n",
    "\n",
    "This is equal to:\n",
    "\\begin{equation}\n",
    "det((y^T \\tilde{K}^{-1} y) \\tilde{K}) \\left[ tr(\\tilde{K}^{-1} \\partial_{\\theta_i}\\tilde{K}) - 2n \\frac{y^T \\tilde{K}^{-1} (\\partial_{\\theta_i}\\tilde{K}) \\tilde{K}^{-1} y}{y^T \\tilde{K}^{-1} y} \\right].\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1: Simulate data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 293,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "import sympy as sp\n",
    "from scipy.linalg import solve_triangular\n",
    "from scipy.optimize import minimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 294,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Global variables: x, y, n, y_u, y_f, s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Parameters, that can be modified:*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of data samples:\n",
    "n = 4\n",
    "\n",
    "# Noise of our data:\n",
    "s = 1e-7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 296,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simulate_data():\n",
    "    x = np.random.rand(n)\n",
    "    y = np.random.rand(n)\n",
    "    y_u = np.multiply(x,x) + y\n",
    "    y_f = 2*(np.multiply(x,x) + x + y)\n",
    "    return (x,y,y_u,y_f)\n",
    "(x,y,y_u,y_f) = simulate_data()\n",
    "y_con = np.concatenate((y_u, y_f))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: Evaluate kernels\n",
    "\n",
    "$k_{uu}(X_i, X_j; \\theta) = exp(-\\frac{1}{2l_x}(x_i-x_j)^2 - \\frac{1}{2l_y}(y_i-y_j)^2)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 297,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_i, x_j, y_i, y_j, l_x, l_y, phi = sp.symbols('x_i x_j y_i y_j l_x l_y phi')\n",
    "kuu_sym = sp.exp(-1/(2*l_x)*((x_i - x_j)**2) - 1/(2*l_y)*((y_i - y_j)**2))\n",
    "kuu_fn = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y), kuu_sym, \"numpy\")\n",
    "def kuu(x, y, l_x, l_y):\n",
    "    k = np.zeros((x.size, x.size))\n",
    "    for i in range(x.size):\n",
    "        for j in range(x.size):\n",
    "            k[i,j] = kuu_fn(x[i], x[j], y[i], y[j], l_x, l_y)\n",
    "    return k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$k_{ff}(X_i,X_j;\\theta,\\phi) \\\\\n",
    "= \\mathcal{L}_{X_i}^{\\phi} \\mathcal{L}_{X_j}^{\\phi} k_{uu}(X_i, X_j; \\theta) \\\\\n",
    "= \\phi^2k_{uu} + \\phi \\frac{\\partial}{\\partial x_i}k_{uu} + \\phi \\frac{\\partial^2}{\\partial y_i^2}k_{uu} + \\phi \\frac{\\partial}{\\partial x_j}k_{uu} + \\frac{\\partial^2}{\\partial x_i, x_j}k_{uu} + \\frac{\\partial^3}{\\partial y_i^2 \\partial x_j}k_{uu} + \\phi \\frac{\\partial^2}{\\partial y_j^2}k_{uu} + \\frac{\\partial^3}{\\partial x_i \\partial y_j^2}k_{uu} + \\frac{\\partial^4}{\\partial y_i^2 \\partial y_j^2}k_{uu}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 298,
   "metadata": {},
   "outputs": [],
   "source": [
    "kff_sym = phi**2*kuu_sym \\\n",
    "        + phi*sp.diff(kuu_sym, x_i) \\\n",
    "        + phi*sp.diff(kuu_sym, y_i, y_i) \\\n",
    "        + phi*sp.diff(kuu_sym, x_j) \\\n",
    "        + sp.diff(kuu_sym, x_i, x_j) \\\n",
    "        + sp.diff(kuu_sym, y_i, y_i, x_j) \\\n",
    "        + phi*sp.diff(kuu_sym, y_j, y_j) \\\n",
    "        + sp.diff(kuu_sym, x_i, y_j, y_j) \\\n",
    "        + sp.diff(kuu_sym, y_i, y_i, y_j, y_j)\n",
    "kff_fn = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y, phi), kff_sym, \"numpy\")\n",
    "def kff(x, y, l_x, l_y, phi):\n",
    "    k = np.zeros((x.size, x.size))\n",
    "    for i in range(x.size):\n",
    "        for j in range(x.size):\n",
    "            k[i,j] = kff_fn(x[i], x[j], y[i], y[j], l_x, l_y, phi)\n",
    "    return k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$k_{fu}(X_i,X_j;\\theta,\\phi) \\\\\n",
    "= \\mathcal{L}_{X_i}^{\\phi} k_{uu}(X_i, X_j; \\theta) \\\\\n",
    "= \\phi k_{uu} + \\frac{\\partial}{\\partial x_i}k_{uu} + \\frac{\\partial^2}{\\partial y_i^2}k_{uu}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 299,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfu_sym = phi*kuu_sym \\\n",
    "        + sp.diff(kuu_sym, x_i) \\\n",
    "        + sp.diff(kuu_sym, y_i, y_i)\n",
    "kfu_fn = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y, phi), kfu_sym, \"numpy\")\n",
    "def kfu(x, y, l_x, l_y, phi):\n",
    "    k = np.zeros((x.size, x.size))\n",
    "    for i in range(x.size):\n",
    "        for j in range(x.size):\n",
    "            k[i,j] = kfu_fn(x[i], x[j], y[i], y[j], l_x, l_y, phi)\n",
    "    return k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 300,
   "metadata": {},
   "outputs": [],
   "source": [
    "def kuf(x, y, l_x, l_y, phi):\n",
    "    return kfu(x, y, l_x, l_y, phi).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3: Compute NLML (with Cholesky decomposition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing the covariance matrix K and its inverse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 301,
   "metadata": {},
   "outputs": [],
   "source": [
    "def K(l_x, l_y, phi):\n",
    "    K_mat = np.block([\n",
    "        [kuu(x, y, l_x, l_y) + s*np.eye(n), kuf(x, y, l_x, l_y, phi)],\n",
    "        [kfu(x, y, l_x, l_y, phi), kff(x, y, l_x, l_y, phi) + s*np.eye(n)]\n",
    "    ])\n",
    "    return K_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 302,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nlml(params):\n",
    "    K_inv = np.zeros((2*n, 2*n))\n",
    "    l_x_exp = np.exp(params[0])\n",
    "    l_y_exp = np.exp(params[1]) \n",
    "    # phi = params[2]\n",
    "    \n",
    "    try:\n",
    "        L = np.linalg.cholesky(K(l_x_exp, l_y_exp, params[2]))\n",
    "        L_inv = solve_triangular(L, np.identity(2*n), lower=True) # Slight performance boost \n",
    "                                                                  # over np.linalg.inv\n",
    "        K_inv = (L_inv.T).dot(L_inv)\n",
    "        val = np.linalg.det((y_con @ K_inv @ y_con)*(L @ L.T))\n",
    "        return val\n",
    "    except np.linalg.LinAlgError:\n",
    "        # Inverse of K via SVD\n",
    "        u, s_mat, vt = np.linalg.svd(K(l_x_exp, l_y_exp, params[2]))\n",
    "        K_inv = (vt.T).dot(np.linalg.inv(np.diag(s_mat))).dot(u.T)\n",
    "        val = np.prod((y_con @ K_inv @ y_con)*s_mat)\n",
    "        return val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can alternatively use the block inversion technique (see *2D_with_multiple_optimizers*):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 303,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nlml_block(params):\n",
    "    l_x_exp = np.exp(params[0])\n",
    "    l_y_exp = np.exp(params[1]) \n",
    "    # phi = params[2]\n",
    "    \n",
    "    A = kuu(x, y, l_x_exp, l_y_exp) + s*np.eye(n)\n",
    "    B = kfu(x, y, l_x_exp, l_y_exp, params[2]).T\n",
    "    C = kff(x, y, l_x_exp, l_y_exp, params[2]) + s*np.eye(n)\n",
    "    \n",
    "    # Inversion of A\n",
    "    A_inv = np.zeros((n, n))\n",
    "    \n",
    "    try:\n",
    "        L = np.linalg.cholesky(A)\n",
    "        L_inv = solve_triangular(L, np.identity(n), lower=True) # Slight performance boost \n",
    "                                                                # over np.linalg.inv\n",
    "        A_inv = L_inv.T @ L_inv        \n",
    "        det_A = (np.diag(L)**2).prod()\n",
    "    except np.linalg.LinAlgError:\n",
    "        # Inverse of K via SVD\n",
    "        u, s_mat, vt = np.linalg.svd(A)\n",
    "        A_inv = vt.T @ np.linalg.inv(np.diag(s_mat)) @ u.T\n",
    "        det_A = s_mat.prod()        \n",
    "        \n",
    "    # Inversion of $C-B^T A^{-1} B$\n",
    "    KA_inv = np.zeros((n, n))\n",
    "    KA = C - B.T @ A_inv @ B\n",
    "    \n",
    "    try:\n",
    "        L = np.linalg.cholesky(KA)\n",
    "        L_inv = solve_triangular(L, np.identity(n), lower=True) # Slight performance boost \n",
    "                                                                # over np.linalg.inv\n",
    "        KA_inv = L_inv.T @ L_inv\n",
    "        det_KA = (np.diag(L)**2).prod()\n",
    "    except np.linalg.LinAlgError:\n",
    "        # Inverse of K via SVD\n",
    "        u, s_mat, vt = np.linalg.svd(KA)\n",
    "        KA_inv = vt.T @ np.linalg.inv(np.diag(s_mat)) @ u.T\n",
    "        det_KA = s_mat.prod()\n",
    "        \n",
    "    # Piecing it together\n",
    "    T = A_inv @ B @ KA_inv\n",
    "    factor = y_u @ (A_inv + T @ B.T @ A_inv) @ y_u - 2*y_u @ T @ y_f + y_f @ KA_inv @ y_f\n",
    "    \n",
    "    return np.linalg.det(factor*A)*np.linalg.det(factor*KA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 304,
   "metadata": {},
   "outputs": [],
   "source": [
    "t0 = time.time()\n",
    "m_block = minimize(nlml_block, np.random.rand(3), method=\"Nelder-Mead\", \n",
    "             options={'maxfev':5000, 'fatol':0.001, 'xatol':0.001})\n",
    "t_Nelder_block = time.time() - t0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 305,
   "metadata": {},
   "outputs": [],
   "source": [
    "t0 = time.time()\n",
    "m = minimize(nlml, np.random.rand(3), method=\"Nelder-Mead\",\n",
    "             options={'maxfev':5000, 'fatol':0.001, 'xatol':0.001})\n",
    "t_Nelder = time.time() - t0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Computing the gradient***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 306,
   "metadata": {},
   "outputs": [],
   "source": [
    "kuu_sym_l_x, kuu_sym_l_y, kuu_sym_phi = [sp.diff(kuu_sym, l) for l in [l_x, l_y, phi]]\n",
    "kfu_sym_l_x, kfu_sym_l_y, kfu_sym_phi = [sp.diff(kfu_sym, l) for l in [l_x, l_y, phi]]\n",
    "kff_sym_l_x, kff_sym_l_y, kff_sym_phi = [sp.diff(kff_sym, l) for l in [l_x, l_y, phi]]\n",
    "\n",
    "kuu_l_x = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y), kuu_sym_l_x, \"numpy\")\n",
    "kff_l_x = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y, phi), kff_sym_l_x, \"numpy\")\n",
    "kfu_l_x = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y, phi), kfu_sym_l_x, \"numpy\")\n",
    "\n",
    "kuu_l_y = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y), kuu_sym_l_y, \"numpy\")\n",
    "kff_l_y = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y, phi), kff_sym_l_y, \"numpy\")\n",
    "kfu_l_y = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y, phi), kfu_sym_l_y, \"numpy\")\n",
    "\n",
    "kuu_phi = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y), kuu_sym_phi, \"numpy\")\n",
    "kff_phi = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y, phi), kff_sym_phi, \"numpy\")\n",
    "kfu_phi = sp.lambdify((x_i, x_j, y_i, y_j, l_x, l_y, phi), kfu_sym_phi, \"numpy\")\n",
    "\n",
    "# Derivative of K w.r.t. l_x:\n",
    "def K_l_x(x, y, l_x, l_y, phi):\n",
    "    k = np.zeros([2*n, 2*n])\n",
    "    for i in range(2*n):\n",
    "        for j in range(2*n):\n",
    "            if i<n and j<n:\n",
    "                k[i,j] = kuu_l_x(x[i], x[j], y[i], y[j], l_x, l_y)\n",
    "            if i<n and j>=n:\n",
    "                k[i,j] = kfu_l_x(x[j-n], x[i], y[j-n], y[i], l_x, l_y, phi) # Transpose of kfu\n",
    "            if i>=n and j<n:\n",
    "                k[i,j] = kfu_l_x(x[i-n], x[j], y[i-n], y[j], l_x, l_y, phi)\n",
    "            if i>=n and j>=n:\n",
    "                k[i,j] = kff_l_x(x[i-n], x[j-n], y[i-n], y[j-n], l_x, l_y, phi)\n",
    "    return k\n",
    "\n",
    "# Derivative of K w.r.t. l_y:\n",
    "def K_l_y(x, y, l_x, l_y, phi):\n",
    "    k = np.zeros([2*n, 2*n])\n",
    "    for i in range(2*n):\n",
    "        for j in range(2*n):\n",
    "            if i<n and j<n:\n",
    "                k[i,j] = kuu_l_y(x[i], x[j], y[i], y[j], l_x, l_y)\n",
    "            if i<n and j>=n:\n",
    "                k[i,j] = kfu_l_y(x[j-n], x[i], y[j-n], y[i], l_x, l_y, phi) # Transpose of kfu\n",
    "            if i>=n and j<n:\n",
    "                k[i,j] = kfu_l_y(x[i-n], x[j], y[i-n], y[j], l_x, l_y, phi)\n",
    "            if i>=n and j>=n:\n",
    "                k[i,j] = kff_l_y(x[i-n], x[j-n], y[i-n], y[j-n], l_x, l_y, phi)\n",
    "    return k\n",
    "\n",
    "# Derivative of K w.r.t. phi:\n",
    "def K_phi(x, y, l_x, l_y, phi):\n",
    "    k = np.zeros([2*n, 2*n])\n",
    "    for i in range(2*n):\n",
    "        for j in range(2*n):\n",
    "            if i<n and j<n:\n",
    "                k[i,j] = kuu_phi(x[i], x[j], y[i], y[j], l_x, l_y)\n",
    "            if i<n and j>=n:\n",
    "                k[i,j] = kfu_phi(x[j-n], x[i], y[j-n], y[i], l_x, l_y, phi) # Transpose of kfu\n",
    "            if i>=n and j<n:\n",
    "                k[i,j] = kfu_phi(x[i-n], x[j], y[i-n], y[j], l_x, l_y, phi)\n",
    "            if i>=n and j>=n:\n",
    "                k[i,j] = kff_phi(x[i-n], x[j-n], y[i-n], y[j-n], l_x, l_y, phi)\n",
    "    return k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 307,
   "metadata": {},
   "outputs": [],
   "source": [
    "def K_inv_and_det(l_x, l_y, phi):\n",
    "    \n",
    "    A = kuu(x, y, l_x, l_y) + s*np.eye(n)\n",
    "    B = kfu(x, y, l_x, l_y, phi).T\n",
    "    C = kff(x, y, l_x, l_y, phi) + s*np.eye(n)\n",
    "    \n",
    "    # Inversion of A\n",
    "    A_inv = np.zeros((n, n))\n",
    "    \n",
    "    try:\n",
    "        L = np.linalg.cholesky(A)\n",
    "        L_inv = solve_triangular(L, np.identity(n), lower=True) # Slight performance boost \n",
    "                                                                # over np.linalg.inv\n",
    "        A_inv = L_inv.T @ L_inv\n",
    "    except np.linalg.LinAlgError:\n",
    "        # Inverse of K via SVD\n",
    "        u, s_mat, vt = np.linalg.svd(A)\n",
    "        A_inv = vt.T @ np.linalg.inv(np.diag(s_mat)) @ u.T\n",
    "        \n",
    "    # Inversion of $C-B^T A^{-1} B$\n",
    "    KA_inv = np.zeros((n, n))\n",
    "    KA = C - B.T @ A_inv @ B\n",
    "    \n",
    "    try:\n",
    "        L = np.linalg.cholesky(KA)\n",
    "        L_inv = solve_triangular(L, np.identity(n), lower=True) # Slight performance boost \n",
    "                                                                # over np.linalg.inv\n",
    "        KA_inv = L_inv.T @ L_inv\n",
    "    except np.linalg.LinAlgError:\n",
    "        # Inverse of K via SVD\n",
    "        u, s_mat, vt = np.linalg.svd(KA)\n",
    "        KA_inv = vt.T @ np.linalg.inv(np.diag(s_mat)) @ u.T\n",
    "        \n",
    "    # Piecing it together\n",
    "    T = A_inv @ B @ KA_inv\n",
    "        \n",
    "    K_inv = np.block([\n",
    "        [A_inv + T @ B.T @ A_inv, -T],\n",
    "        [-T.T, KA_inv]\n",
    "    ])\n",
    "    \n",
    "    f = y_con @ K_inv @ y_con\n",
    "    \n",
    "    det_fK = np.linalg.det(f*A)*np.linalg.det(f*KA)\n",
    "    \n",
    "    return K_inv, det_fK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 308,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting the gradient:\n",
    "def grad_L(l_x, l_y, phi):\n",
    "    \n",
    "    K_inv, det_fK = K_inv_and_det(l_x, l_y, phi)\n",
    "    \n",
    "    K_1 = K_l_x(x, y, l_x, l_y, phi)\n",
    "    K_2 = K_l_y(x, y, l_x, l_y, phi)\n",
    "    K_3 = K_phi(x, y, l_x, l_y, phi)\n",
    "    \n",
    "    w = K_inv @ y_con\n",
    "\n",
    "    comp_1 = det_fK*(np.trace(K_inv @ K_1) - (2*n*w.T @ K_1 @ w) / (y_con @ w))\n",
    "    comp_2 = det_fK*(np.trace(K_inv @ K_2) - (2*n*w.T @ K_2 @ w) / (y_con @ w))\n",
    "    comp_3 = det_fK*(np.trace(K_inv @ K_3) - (2*n*w.T @ K_3 @ w) / (y_con @ w))\n",
    "    \n",
    "    return [comp_1, comp_2, comp_3]\n",
    "\n",
    "def grad_nlml(par):\n",
    "    # l_x = par[0]\n",
    "    # l_y = par[1]\n",
    "    # phi = par[2]\n",
    "    M = np.diag([np.exp(par[0]), np.exp(par[1]), 1])\n",
    "    grad = np.array([grad_L(np.exp(par[0]), np.exp(par[1]), par[2])]).dot(M)\n",
    "    \n",
    "    # Scipy.minimize wants an array as the gradient, not a matrix. \n",
    "    # We therefore remove any one-dimensional entries of the grad by squeezing.\n",
    "    return np.squeeze(np.asarray(grad))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running conjugate gradient:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 309,
   "metadata": {},
   "outputs": [],
   "source": [
    "Nfeval = 1\n",
    "def callbackF(Xi):\n",
    "    global Nfeval\n",
    "    print('{0:4d}   {1: 3.6f}   {2: 3.6f}   {3: 3.6f}'.format(Nfeval, Xi[0], Xi[1], Xi[2]))\n",
    "    Nfeval += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 310,
   "metadata": {},
   "outputs": [],
   "source": [
    "import conjugate_gradient._minimize as mi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 311,
   "metadata": {},
   "outputs": [],
   "source": [
    "t1 = time.time()\n",
    "m_CG = mi.minimize(nlml, np.random.rand(3), jac=grad_nlml, method='CG')\n",
    "t_CG = time.time() - t1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 312,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The time it took Nelder-Mead with block-inversion to converge: 0.18849539756774902 seconds\n",
      "The time it took Nelder-Mead to converge:                      0.23704886436462402 seconds\n",
      "The time it took non-linear Conjugate Gradient to converge:    0.5568571090698242 seconds\n"
     ]
    }
   ],
   "source": [
    "print('The time it took Nelder-Mead with block-inversion to converge:', t_Nelder_block, 'seconds')\n",
    "print('The time it took Nelder-Mead to converge:                     ', t_Nelder, 'seconds')\n",
    "print('The time it took non-linear Conjugate Gradient to converge:   ', t_CG, 'seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 313,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The inferred parameter with Nelder-Mead with block-inversion is: 2.0116897972421492\n",
      "The inferred parameter with Nelder-Mead is:                      2.0116696984130775\n",
      "The inferred parameter with non-linear Conjugate Gradient is:    2.0195302066585694\n"
     ]
    }
   ],
   "source": [
    "print('The inferred parameter with Nelder-Mead with block-inversion is:', m_block.x[2])\n",
    "print('The inferred parameter with Nelder-Mead is:                     ', m.x[2])\n",
    "print('The inferred parameter with non-linear Conjugate Gradient is:   ', m_CG.x[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}