Moonshoot_technologies_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "91bab923ff51b8c4cc2db62df96dceba",
     "grade": false,
     "grade_id": "cell-728287ea719cc025",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "# MoonShot Technologies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "75370e1d01e0e2216d0982ef3c22eb0a",
     "grade": false,
     "grade_id": "cell-e86bb7c59ff0a4a5",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Congratulations! Due to your strong performance in the first three courses, you landed a job as a reinforcement learning engineer at the hottest new non-revenue generating unicorn, MoonShot Technologies (MST). Times are busy at MST, which is preparing for its initial public offering (IPO) at the end of the fiscal year, and your labor is much needed.\n",
    "\n",
    "Like many successful startups, MST is exceedingly concerned with the valuation that it will receive at its IPO (as this valuation determines the price at which its existing venture capitalist shareholders will be able to sell their shares). Accordingly, to whet the appetites of potential investors, MST has set its sights on accomplishing a technological tour de force &mdash; a lunar landing &mdash; before the year is out. But it is not just any mundane lunar landing that MST aspires toward. Rather than the more sensible approach of employing techniques from aerospace engineering to pilot its spacecraft, MST endeavors to wow investors by training an agent to do the job via reinforcement learning.\n",
    "\n",
    "However, it is clearly not practical for a reinforcement learning agent to be trained tabula rasa with real rockets &mdash; even the pockets of venture capitalists have their limits. Instead, MST aims to build a simulator that is realistic enough to train an agent that can be deployed in the real world. This will be a difficult project, and will require building a realistic simulator, choosing the right reinforcement learning algorithm, implementing this algorithm, and optimizing the hyperparameters for this algorithm.\n",
    "\n",
    "Naturally, as the newly hired reinforcement learning engineer, you have been staffed to lead the project. In this notebook, you will take the first steps by building a lunar lander environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c4a738c811626904fa17a90cbfddd80e",
     "grade": false,
     "grade_id": "cell-62c5c402edcd8ae5",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "## Creating an Environment\n",
    "The software engineering team at MST has already set up some infrastructure for your convenience. Specifically, they have provided you with the following functions:\n",
    "\n",
    "* **get_velocity** - returns an array representing the x, y velocity of the lander. Both the x and y velocity are in the range $[0, 60]$.\n",
    "\n",
    "\n",
    "* **get_angle** - returns a scalar representing the angle of the lander. The angle is in the range $[0, 359]$.\n",
    "\n",
    "\n",
    "* **get_position** - returns an array representing the x, y position of the lander. Both the x and y position of the agent are in the range $[0, 100]$.\n",
    "\n",
    "\n",
    "* **get_landing_zone** - returns an array representing the x, y position of the landing zone. Both the x, y coordinates are in the range $[1, 100]$. \n",
    "\n",
    "\n",
    "* **get_fuel** - returns a scalar representing the remaining amount of fuel. Fuel starts at $100$ and is in range $[0, 100]$.\n",
    "\n",
    "Note that these are dummy functions just for this assignment.\n",
    "\n",
    "![Lunar Landar](lunar_landar.png)\n",
    "\n",
    "In this notebook, you will be applying these functions to __structure the reward signal__ based on the following criteria:\n",
    "\n",
    "1. **The lander will crash if** it touches the ground when ``y_velocity < -3`` (the downward velocity is greater than three).\n",
    "\n",
    "\n",
    "2. **The lander will crash if** it touches the ground when ``x_velocity < -10 or 10 < x_velocity`` (horizontal speed is greater than $10$).\n",
    "\n",
    "\n",
    "3. The lander's angle taken values in $[0, 359]$. It is completely vertical at $0$ degrees. **The lander will crash if** it touches the ground when ``5 < angle < 355`` (angle differs from vertical by more than $5$ degrees).\n",
    "\n",
    "\n",
    "4. **The lander will crash if** it has yet to land and ``fuel <= 0`` (it runs out of fuel).\n",
    "\n",
    "\n",
    "5. MST would like to save money on fuel when it is possible **(using less fuel is preferred)**.\n",
    "\n",
    "\n",
    "6. The lander can only land in the landing zone. **The lander will crash if** it touches the ground when ``x_position`` $\\not\\in$ ``landing_zone`` (it lands outside the landing zone).\n",
    "\n",
    "\n",
    "Fill in the methods below to create an environment for the lunar lander."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "193356706350cbad3ee2e968e1f33fce",
     "grade": false,
     "grade_id": "cell-b5475cc072c387ff",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "import environment\n",
    "from utils import get_landing_zone, get_angle, get_velocity, get_position, get_fuel, tests\n",
    "get_landing_zone()\n",
    "# Lunar Lander Environment\n",
    "class LunarLanderEnvironment(environment.BaseEnvironment):\n",
    "    def __init__(self):\n",
    "        self.current_state = None\n",
    "        self.count = 0\n",
    "    \n",
    "    def env_init(self, env_info):\n",
    "        # users set this up\n",
    "        self.state = np.zeros(6) # velocity x, y, angle, distance to ground, landing zone x, y\n",
    "    \n",
    "    def env_start(self):\n",
    "        land_x, land_y = get_landing_zone() # gets the x, y coordinate of the landing zone\n",
    "        # At the start we initialize the agent to the top left hand corner (100, 20) with 0 velocity \n",
    "        # in either any direction. The agent's angle is set to 0 and the landing zone is retrieved and set.\n",
    "        # The lander starts with fuel of 100.\n",
    "        # (vel_x, vel_y, angle, pos_x, pos_y, land_x, land_y, fuel)\n",
    "        self.current_state = (0, 0, 0, 100, 20, land_x, land_y, 100)\n",
    "        return self.current_state\n",
    "    \n",
    "    def env_step(self, action):\n",
    "        \n",
    "        land_x, land_y = get_landing_zone() # gets the x, y coordinate of the landing zone\n",
    "        vel_x, vel_y = get_velocity(action) # gets the x, y velocity of the lander\n",
    "        angle = get_angle(action) # gets the angle the lander is positioned in\n",
    "        pos_x, pos_y = get_position(action) # gets the x, y position of the lander\n",
    "        fuel = get_fuel(action) # get the amount of fuel remaining for the lander\n",
    "        \n",
    "        terminal = False\n",
    "        reward = 0.0\n",
    "        observation = (vel_x, vel_y, angle, pos_x, pos_y, land_x, land_y, fuel)\n",
    "        \n",
    "        # use the above observations to decide what the reward will be, and if the\n",
    "        # agent is in a terminal state.\n",
    "        # Recall - if the agent crashes or lands terminal needs to be set to True\n",
    "        \n",
    "        # your code here\n",
    "        \n",
    "        \n",
    "        self.reward_obs_term = (reward, observation, terminal)\n",
    "        return self.reward_obs_term\n",
    "    \n",
    "    def env_cleanup(self):\n",
    "        return None\n",
    "    \n",
    "    def env_message(self):\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "6f960ff843630e06b23aa258113d6e6b",
     "grade": false,
     "grade_id": "cell-9c57ff0d2b96ac51",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "## Evaluating your reward function\n",
    "\n",
    "Designing the best reward function for an objective is a challenging task - it is not clear what the term “best reward function” even means, let alone how to find it. Consequently, rather than evaluating your reward function by quantitative metrics, we merely ask that you check that its behavior is qualitatively reasonable. For this purpose, we provide a series of test cases below. In each case we show a transition and explain how a reward function that we implemented behaves. As you read, check how your own reward behaves in each scenario and judge for yourself whether it acts appropriately. (For the latter parts of the capstone you will use our implementation of the lunar lander environment, so don’t worry if your reward function isn’t exactly the same as ours. The purpose of this of this notebook is to gain experience implementing environments and reward functions.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8ea30a25d9a5f478e892745463ddacbc",
     "grade": false,
     "grade_id": "cell-2d30dbf6446d5afc",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "### Case 1: Uncertain Future\n",
    "The lander is in the top left corner of the screen moving at a velocity of (12, 15) with 10 units of fuel &mdash; whether this landing will be successful remains to be seen.\n",
    "\n",
    "![Lunar Landar](lunar_landar_1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "09b03ba3c243b29134f5bc04dcdc10f4",
     "grade": true,
     "grade_id": "cell-99abd81376335339",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward: 0.0, Terminal: False\n"
     ]
    }
   ],
   "source": [
    "tests(LunarLanderEnvironment, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ca2d70140549a4b7eb3c52787154953b",
     "grade": false,
     "grade_id": "cell-89911113f764c447",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "In this case we gave the agent no reward, as it neither achieved the objective nor crashed. One alternative is giving the agent a positive reward for moving closer to the goal. Another is to give a negative reward for fuel consumption. What did your reward function do?\n",
    "\n",
    "Also check to make sure that ``Terminal`` is set to ``False``. Your agent has not landed, crashed, or ran out of fuel. The episode is not over."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f6c45e927b165cdb95744caaf3309aed",
     "grade": false,
     "grade_id": "cell-b19c487e9da05800",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "### Case 2: Imminent Crash!\n",
    "\n",
    "The lander is positioned in the target landing zone at a 45 degree angle, but its landing gear can only handle an angular offset of five degrees &mdash; it is about to crash!\n",
    "\n",
    "![Lunar Landar](lunar_landar_2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5f07c252a2f60904323ad2fb02675221",
     "grade": true,
     "grade_id": "cell-9b3900153803f78e",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward: 0.0, Terminal: False\n"
     ]
    }
   ],
   "source": [
    "tests(LunarLanderEnvironment, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "48870c2ad1832288abe07ea16cc88ba2",
     "grade": false,
     "grade_id": "cell-4731b02a8f54214b",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "We gave the agent a reward of -10000 to punish it for crashing. How did your reward function handle the crash?\n",
    "\n",
    "Also check to make sure that ``Terminal`` is set to ``True``. Your agent has crashed and the episode is over."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a06bb6be22b15cb43cad3378583b188c",
     "grade": false,
     "grade_id": "cell-af000f1895c6bd69",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "### Case 3: Nice Landing!\n",
    "The lander is vertically oriented and positioned in the target landing zone with five units of remaining fuel. The landing is being completed successfully!\n",
    "\n",
    "![Lunar Landar](lunar_landar_3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b2013e0485824e18311b02cfda0c857b",
     "grade": true,
     "grade_id": "cell-6a53769313d85b0b",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward: 0.0, Terminal: False\n"
     ]
    }
   ],
   "source": [
    "tests(LunarLanderEnvironment, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7fcd0832a9a99dd4d6ae3cd1f2a99666",
     "grade": false,
     "grade_id": "cell-e23d284c1e7c6865",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "To encourage the agent to conserve as much fuel as possible, we reward successful landings proportionally to the amount of fuel remaining. Here, we gave the agent a reward of five since it landed with five units of fuel remaining. How did you incentivize the agent to be fuel efficient?\n",
    "\n",
    "Also check to make sure that ``Terminal`` is set to ``True``. Your agent has landed and the episode is over."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "144e1370f8a1146808c367d077c8a6db",
     "grade": false,
     "grade_id": "cell-21cc28d788b4455d",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "### Case 4: Dark Times Ahead!\n",
    "The lander is directly above the target landing zone but has no fuel left. The future does not look good for the agent &mdash; without fuel there is no way for it to avoid crashing!\n",
    "\n",
    "![Lunar Landar](lunar_landar_4.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4ca9f4cf5b7d498584d814a0ec14fbbb",
     "grade": true,
     "grade_id": "cell-86ece2998b73491a",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward: 0.0, Terminal: False\n"
     ]
    }
   ],
   "source": [
    "tests(LunarLanderEnvironment, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "14711bd7c29c70cfc7f2779a4a4548ff",
     "grade": false,
     "grade_id": "cell-e1cc048da1ecc920",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "We gave the agent a reward of -10000 to punish it for crashing. Did your reward function treat all crashes equally, as ours did? Or did you penalize some crashes more than others? What reasoning did you use to make this decision?\n",
    "\n",
    "Also check to make sure that ``Terminal`` is set to ``True``. Your agent has crashed and the episode is over."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "abf49fa9dd8069cd57474df9bafe03e8",
     "grade": false,
     "grade_id": "cell-29610cb5c650ab8b",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Case 5: Where's The Landing Zone?!\n",
    "\n",
    "The lander is touching down at a vertical angle with fuel to spare. But it is not in the landing zone and the surface is uneven &mdash; it is going to crash!\n",
    "\n",
    "![Lunar Landar](lunar_landar_5.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4198a382067136602a638dd1690df307",
     "grade": true,
     "grade_id": "cell-b7cee5ebb3ab91bf",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward: 0.0, Terminal: False\n"
     ]
    }
   ],
   "source": [
    "tests(LunarLanderEnvironment, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "63a597e957db543772b510123497c5f8",
     "grade": false,
     "grade_id": "cell-aead9765209963cd",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "We gave the agent a reward of -10000 to punish it for landing in the wrong spot. An alternative is to scale the negative reward by distance from the landing zone. What approach did you take?\n",
    "\n",
    "Also check to make sure that ``Terminal`` is set to ``True``. Your agent has crashed and the episode is over."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "847d71c3274e0abf69bb00349ad82069",
     "grade": false,
     "grade_id": "cell-228ffe4d4fd5d55f",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "## Wrapping Up\n",
    "Excellent! The lunar lander simulator is complete and the project can commence. In the next module, you will build upon your work here by implementing an agent to train in the environment. Don’t dally! The team at MST is eagerly awaiting your solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}