notebooks/Using-an-Online-Solver.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using an Online Solver\n",
    "In this section, we will walk through using an Online solver, which takes actions during runtime without any offline pre-processing phase. For more details on POMDPs and offline solvers, please consult Chapter 6 of the DMU textbook [1]. The solver we will use is POMCP or Partially Observable Monte Carlo Planning [2], implemented [here](https://github.com/JuliaPOMDP/BasicPOMCP.jl). We will compare it against a policy that chooses actions at random.\n",
    "\n",
    "[1] Kochenderfer, Mykel J. Decision Making Under Uncertainty: Theory and Application. MIT Press, 2015\n",
    "\n",
    "[2] Silver, David, and Joel Veness. \"Monte-Carlo planning in large POMDPs.\" In Advances in Neural Information Processing Systems, 2010.\n",
    "\n",
    "### POMDP Model\n",
    "For this example we will use the LightDark1D POMDP (defined in [POMDPModels](https://github.com/JuliaPOMDP/POMDPModels.jl)) which is an instance of an explicit POMDP. Please see this [notebook](https://github.com/JuliaPOMDP/POMDPExamples.jl) for how to define a POMDP with the explicit interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using POMDPs\n",
    "using POMDPModels # For the problem\n",
    "using BasicPOMCP # For the solver\n",
    "using POMDPPolicies # For creating a random policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LightDark1D{typeof(POMDPModels.default_sigma)}(0.9, 10.0, -10.0, 1.0, 0.0, POMDPModels.default_sigma)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the POMDP problem with default params\n",
    "pomdp = LightDark1D()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "POMCPSolver\n",
       "  max_depth: Int64 20\n",
       "  c: Float64 10.0\n",
       "  tree_queries: Int64 1000\n",
       "  max_time: Float64 Inf\n",
       "  tree_in_info: Bool false\n",
       "  default_action: ExceptionRethrow ExceptionRethrow()\n",
       "  rng: Random.MersenneTwister\n",
       "  estimate_value: RolloutEstimator\n"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the POMCP solver; use keyword arguments to adjust parameters\n",
    "solver = POMCPSolver(c=10.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the planner\n",
    "\n",
    "A planner is created by calling `solve` on the solver and the pomdp problem.\n",
    "Since POMCP is an online solver, it does not actually do any computation in `solve`; instead the \"planner\" object that is returned by solve has a method of `POMDPs.action` that does planning calculations online during a simulation. \n",
    "Semantically, the planner functions identically to a `POMDPs.Policy` object (and is indeed a subtype of `POMDPs.Policy`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "planner = solve(solver, pomdp);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will compare the planner against a random policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a LightDark1D policy that chooses actions at random\n",
    "rand_policy = RandomPolicy(pomdp);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmarking the planner\n",
    "We will compare the performance of the POMCP planner against that of the Random Policy for the LightDark1D POMDP. Since we only care about the discounted reward, we can use the rollout simulator defined in [POMDPSimulators](https://github.com/JuliaPOMDP/POMDPSimulators.jl). Checkout this [notebook](https://github.com/JuliaPOMDP/POMDPExamples.jl/blob/master/notebooks/Running-Simulations.ipynb) for ways to use the other simulators as well. \n",
    "\n",
    "We will also need a method to update the belief state of the POMDP after taking an action and seeing an observation. More information about that is in [BeliefUpdaters](https://github.com/JuliaPOMDP/BeliefUpdaters.jl). The default updater for a `FunctionPolicy` is the `PreviousObservationUpdater`, which is called when we do not provide an updater argument while calling `simulate`. However, we will use the more sophisticated `ParticleFilters` updater. See [ParticleFilters](https://github.com/JuliaPOMDP/ParticleFilters.jl) for more information. Finally, we can compare the expected discounted rewards and see how the POMCP planner usually does significantly better than random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "using POMDPSimulators \n",
    "using ParticleFilters\n",
    "\n",
    "# Define the specific unweighted particle filter to be used\n",
    "pf = SIRParticleFilter(pomdp, 1000);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create and run the rollout simulator\n",
    "rollout_sim = RolloutSimulator(max_steps=10);\n",
    "r_pomcp = simulate(rollout_sim, pomdp, planner, pf);\n",
    "r_rand = simulate(rollout_sim, pomdp, rand_policy);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r_pomcp = 8.100000000000001\n",
      "r_rand = -9.0\n"
     ]
    }
   ],
   "source": [
    "@show r_pomcp;\n",
    "@show r_rand;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": "86330D22664748BC8AB74365F412488D",
   "lastKernelId": "8301cd31-b23f-4b6d-8bcd-9a536c2be0ea"
  },
  "kernelspec": {
   "display_name": "Julia 1.2.0",
   "language": "julia",
   "name": "julia-1.2"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.2.0"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}