diff --git a/notebooks/unit2/unit2.ipynb b/notebooks/unit2/unit2.ipynb index e9ae624c..47edcea7 100644 --- a/notebooks/unit2/unit2.ipynb +++ b/notebooks/unit2/unit2.ipynb @@ -1,1772 +1,1780 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "view-in-github", - "colab_type": "text" - }, - "source": [ - "\"Open" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "njb_ProuHiOe" - }, - "source": [ - "# Unit 2: Q-Learning with FrozenLake-v1 โ›„ and Taxi-v3 ๐Ÿš•\n", - "\n", - "\"Unit\n", - "\n", - "In this notebook, **you'll code your first Reinforcement Learning agent from scratch** to play FrozenLake โ„๏ธ using Q-Learning, share it with the community, and experiment with different configurations.\n", - "\n", - "โฌ‡๏ธ Here is an example of what **you will achieve in just a couple of minutes.** โฌ‡๏ธ\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "vRU_vXBrl1Jx" - }, - "source": [ - "\"Environments\"/" - ] - }, - { - "cell_type": "markdown", - "source": [ - "###๐ŸŽฎ Environments:\n", - "\n", - "- [FrozenLake-v1](https://gymnasium.farama.org/environments/toy_text/frozen_lake/)\n", - "- [Taxi-v3](https://gymnasium.farama.org/environments/toy_text/taxi/)\n", - "\n", - "###๐Ÿ“š RL-Library:\n", - "\n", - "- Python and NumPy\n", - "- [Gymnasium](https://gymnasium.farama.org/)\n", - "\n", - "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)." - ], - "metadata": { - "id": "DPTBOv9HYLZ2" - } - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4i6tjI2tHQ8j" - }, - "source": [ - "## Objectives of this notebook ๐Ÿ†\n", - "\n", - "At the end of the notebook, you will:\n", - "\n", - "- Be able to use **Gymnasium**, the environment library.\n", - "- Be able to code a Q-Learning agent from scratch.\n", - "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score ๐Ÿ”ฅ.\n", - "\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "source": [ - "## This notebook is from the Deep Reinforcement Learning Course\n", - "\n", - "\"Deep" - ], - "metadata": { - "id": "viNzVbVaYvY3" - } - }, - { - "cell_type": "markdown", - "metadata": { - "id": "6p5HnEefISCB" - }, - "source": [ - "In this free course, you will:\n", - "\n", - "- ๐Ÿ“– Study Deep Reinforcement Learning in **theory and practice**.\n", - "- ๐Ÿง‘โ€๐Ÿ’ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n", - "- ๐Ÿค– Train **agents in unique environments**\n", - "\n", - "And more check ๐Ÿ“š the syllabus ๐Ÿ‘‰ https://simoninithomas.github.io/deep-rl-course\n", - "\n", - "Donโ€™t forget to **sign up to the course** (we are collecting your email to be able toย **send you the links when each Unit is published and give you information about the challenges and updates).**\n", - "\n", - "\n", - "The best way to keep in touch is to join our discord server to exchange with the community and with us ๐Ÿ‘‰๐Ÿป https://discord.gg/ydHrjt3WP5" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Y-mo_6rXIjRi" - }, - "source": [ - "## Prerequisites ๐Ÿ—๏ธ\n", - "\n", - "Before diving into the notebook, you need to:\n", - "\n", - "๐Ÿ”ฒ ๐Ÿ“š **Study [Q-Learning by reading Unit 2](https://huggingface.co/deep-rl-course/unit2/introduction)** ๐Ÿค— " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "f2ONOODsyrMU" - }, - "source": [ - "## A small recap of Q-Learning" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "V68VveLacfxJ" - }, - "source": [ - "*Q-Learning* **is the RL algorithm that**:\n", - "\n", - "- Trains *Q-Function*, an **action-value function** that encoded, in internal memory, by a *Q-table* **that contains all the state-action pair values.**\n", - "\n", - "- Given a state and action, our Q-Function **will search the Q-table for the corresponding value.**\n", - " \n", - "\"Q\n", - "\n", - "- When the training is done,**we have an optimal Q-Function, so an optimal Q-Table.**\n", - " \n", - "- And if we **have an optimal Q-function**, we\n", - "have an optimal policy, since we **know for, each state, the best action to take.**\n", - "\n", - "\"Link\n", - "\n", - "\n", - "But, in the beginning,ย our **Q-Table is useless since it gives arbitrary value for each state-action pairย (most of the time we initialize the Q-Table to 0 values)**. But, as weโ€™llย explore the environment and update our Q-Table it will give us better and better approximations\n", - "\n", - "\"q-learning.jpeg\"\n", - "\n", - "This is the Q-Learning pseudocode:\n", - "\n", - "\"Q-Learning\"\n" - ] - }, - { - "cell_type": "markdown", - "source": [ - "# Let's code our first Reinforcement Learning algorithm ๐Ÿš€" - ], - "metadata": { - "id": "HEtx8Y8MqKfH" - } - }, - { - "cell_type": "markdown", - "source": [ - "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push your trained Taxi model to the Hub and **get a result of >= 4.5**.\n", - "\n", - "To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward**\n", - "\n", - "For more information about the certification process, check this section ๐Ÿ‘‰ https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" - ], - "metadata": { - "id": "Kdxb1IhzTn0v" - } - }, - { - "cell_type": "markdown", - "source": [ - "## Install dependencies and create a virtual display ๐Ÿ”ฝ\n", - "\n", - "In the notebook, we'll need to generate a replay video. To do so, with Colab, **we need to have a virtual screen to render the environment** (and thus record the frames).\n", - "\n", - "Hence the following cell will install the libraries and create and run a virtual screen ๐Ÿ–ฅ\n", - "\n", - "Weโ€™ll install multiple ones:\n", - "\n", - "- `gymnasium`: Contains the FrozenLake-v1 โ›„ and Taxi-v3 ๐Ÿš• environments.\n", - "- `pygame`: Used for the FrozenLake-v1 and Taxi-v3 UI.\n", - "- `numpy`: Used for handling our Q-table.\n", - "\n", - "The Hugging Face Hub ๐Ÿค— works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.\n", - "\n", - "You can see here all the Deep RL models available (if they use Q Learning) here ๐Ÿ‘‰ https://huggingface.co/models?other=q-learning" - ], - "metadata": { - "id": "4gpxC1_kqUYe" - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "9XaULfDZDvrC" - }, - "outputs": [], - "source": [ - "!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit2/requirements-unit2.txt" - ] - }, - { - "cell_type": "code", - "source": [ - "!sudo apt-get update\n", - "!sudo apt-get install -y python3-opengl\n", - "!apt install ffmpeg xvfb\n", - "!pip3 install pyvirtualdisplay" - ], - "metadata": { - "id": "n71uTX7qqzz2" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "To make sure the new installed libraries are used, **sometimes it's required to restart the notebook runtime**. The next cell will force the **runtime to crash, so you'll need to connect again and run the code starting from here**. Thanks to this trick, **we will be able to run our virtual screen.**" - ], - "metadata": { - "id": "K6XC13pTfFiD" - } - }, - { - "cell_type": "code", - "source": [ - "import os\n", - "os.kill(os.getpid(), 9)" - ], - "metadata": { - "id": "3kuZbWAkfHdg" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "source": [ - "# Virtual display\n", - "from pyvirtualdisplay import Display\n", - "\n", - "virtual_display = Display(visible=0, size=(1400, 900))\n", - "virtual_display.start()" - ], - "metadata": { - "id": "DaY1N4dBrabi" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "W-7f-Swax_9x" - }, - "source": [ - "## Import the packages ๐Ÿ“ฆ\n", - "\n", - "In addition to the installed libraries, we also use:\n", - "\n", - "- `random`: To generate random numbers (that will be useful for epsilon-greedy policy).\n", - "- `imageio`: To generate a replay video." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "VcNvOAQlysBJ" - }, - "outputs": [], - "source": [ - "import numpy as np\n", - "import gymnasium as gym\n", - "import random\n", - "import imageio\n", - "import os\n", - "import tqdm\n", - "\n", - "import pickle5 as pickle\n", - "from tqdm.notebook import tqdm" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "xp4-bXKIy1mQ" - }, - "source": [ - "We're now ready to code our Q-Learning algorithm ๐Ÿ”ฅ" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "xya49aNJWVvv" - }, - "source": [ - "# Part 1: Frozen Lake โ›„ (non slippery version)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "NAvihuHdy9tw" - }, - "source": [ - "## Create and understand [FrozenLake environment โ›„]((https://gymnasium.farama.org/environments/toy_text/frozen_lake/)\n", - "---\n", - "\n", - "๐Ÿ’ก A good habit when you start to use an environment is to check its documentation\n", - "\n", - "๐Ÿ‘‰ https://gymnasium.farama.org/environments/toy_text/frozen_lake/\n", - "\n", - "---\n", - "\n", - "We're going to train our Q-Learning agent **to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H)**.\n", - "\n", - "We can have two sizes of environment:\n", - "\n", - "- `map_name=\"4x4\"`: a 4x4 grid version\n", - "- `map_name=\"8x8\"`: a 8x8 grid version\n", - "\n", - "\n", - "The environment has two modes:\n", - "\n", - "- `is_slippery=False`: The agent always moves **in the intended direction** due to the non-slippery nature of the frozen lake (deterministic).\n", - "- `is_slippery=True`: The agent **may not always move in the intended direction** due to the slippery nature of the frozen lake (stochastic)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "UaW_LHfS0PY2" - }, - "source": [ - "For now let's keep it simple with the 4x4 map and non-slippery.\n", - "We add a parameter called `render_mode` that specifies how the environment should be visualised. In our case because we **want to record a video of the environment at the end, we need to set render_mode to rgb_array**.\n", - "\n", - "As [explained in the documentation](https://gymnasium.farama.org/api/env/#gymnasium.Env.render) โ€œrgb_arrayโ€: Return a single frame representing the current state of the environment. A frame is a np.ndarray with shape (x, y, 3) representing RGB values for an x-by-y pixel image." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "IzJnb8O3y8up" - }, - "outputs": [], - "source": [ - "# Create the FrozenLake-v1 environment using 4x4 map and non-slippery version and render_mode=\"rgb_array\"\n", - "env = gym.make() # TODO use the correct parameters" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Ji_UrI5l2zzn" - }, - "source": [ - "### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "jNxUbPMP0akP" - }, - "outputs": [], - "source": [ - "env = gym.make(\"FrozenLake-v1\", map_name=\"4x4\", is_slippery=False, render_mode=\"rgb_array\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "KASNViqL4tZn" - }, - "source": [ - "You can create your own custom grid like this:\n", - "\n", - "```python\n", - "desc=[\"SFFF\", \"FHFH\", \"FFFH\", \"HFFG\"]\n", - "gym.make('FrozenLake-v1', desc=desc, is_slippery=True)\n", - "```\n", - "\n", - "but we'll use the default environment for now." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "SXbTfdeJ1Xi9" - }, - "source": [ - "### Let's see what the Environment looks like:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ZNPG0g_UGCfh" - }, - "outputs": [], - "source": [ - "# We create our environment with gym.make(\"\")- `is_slippery=False`: The agent always moves in the intended direction due to the non-slippery nature of the frozen lake (deterministic).\n", - "print(\"_____OBSERVATION SPACE_____ \\n\")\n", - "print(\"Observation Space\", env.observation_space)\n", - "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2MXc15qFE0M9" - }, - "source": [ - "We see with `Observation Space Shape Discrete(16)` that the observation is an integer representing the **agentโ€™s current position as current_row * ncols + current_col (where both the row and col start at 0)**.\n", - "\n", - "For example, the goal position in the 4x4 map can be calculated as follows: 3 * 4 + 3 = 15. The number of possible observations is dependent on the size of the map. **For example, the 4x4 map has 16 possible observations.**\n", - "\n", - "\n", - "For instance, this is what state = 0 looks like:\n", - "\n", - "\"FrozenLake\"" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "We5WqOBGLoSm" - }, - "outputs": [], - "source": [ - "print(\"\\n _____ACTION SPACE_____ \\n\")\n", - "print(\"Action Space Shape\", env.action_space.n)\n", - "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "MyxXwkI2Magx" - }, - "source": [ - "The action space (the set of possible actions the agent can take) is discrete with 4 actions available ๐ŸŽฎ:\n", - "- 0: GO LEFT\n", - "- 1: GO DOWN\n", - "- 2: GO RIGHT\n", - "- 3: GO UP\n", - "\n", - "Reward function ๐Ÿ’ฐ:\n", - "- Reach goal: +1\n", - "- Reach hole: 0\n", - "- Reach frozen: 0" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1pFhWblk3Awr" - }, - "source": [ - "## Create and Initialize the Q-table ๐Ÿ—„๏ธ\n", - "\n", - "(๐Ÿ‘€ Step 1 of the pseudocode)\n", - "\n", - "\"Q-Learning\"\n", - "\n", - "\n", - "It's time to initialize our Q-table! To know how many rows (states) and columns (actions) to use, we need to know the action and observation space. We already know their values from before, but we'll want to obtain them programmatically so that our algorithm generalizes for different environments. Gym provides us a way to do that: `env.action_space.n` and `env.observation_space.n`\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "y3ZCdluj3k0l" - }, - "outputs": [], - "source": [ - "state_space =\n", - "print(\"There are \", state_space, \" possible states\")\n", - "\n", - "action_space =\n", - "print(\"There are \", action_space, \" possible actions\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "rCddoOXM3UQH" - }, - "outputs": [], - "source": [ - "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros. np.zeros needs a tuple (a,b)\n", - "def initialize_q_table(state_space, action_space):\n", - " Qtable =\n", - " return Qtable" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "9YfvrqRt3jdR" - }, - "outputs": [], - "source": [ - "Qtable_frozenlake = initialize_q_table(state_space, action_space)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "67OdoKL63eDD" - }, - "source": [ - "### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "HuTKv3th3ohG" - }, - "outputs": [], - "source": [ - "state_space = env.observation_space.n\n", - "print(\"There are \", state_space, \" possible states\")\n", - "\n", - "action_space = env.action_space.n\n", - "print(\"There are \", action_space, \" possible actions\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "lnrb_nX33fJo" - }, - "outputs": [], - "source": [ - "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros\n", - "def initialize_q_table(state_space, action_space):\n", - " Qtable = np.zeros((state_space, action_space))\n", - " return Qtable" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Y0WlgkVO3Jf9" - }, - "outputs": [], - "source": [ - "Qtable_frozenlake = initialize_q_table(state_space, action_space)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Atll4Z774gri" - }, - "source": [ - "## Define the greedy policy ๐Ÿค–\n", - "\n", - "Remember we have two policies since Q-Learning is an **off-policy** algorithm. This means we're using a **different policy for acting and updating the value function**.\n", - "\n", - "- Epsilon-greedy policy (acting policy)\n", - "- Greedy-policy (updating policy)\n", - "\n", - "The greedy policy will also be the final policy we'll have when the Q-learning agent completes training. The greedy policy is used to select an action using the Q-table.\n", - "\n", - "\"Q-Learning\"\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "E3SCLmLX5bWG" - }, - "outputs": [], - "source": [ - "def greedy_policy(Qtable, state):\n", - " # Exploitation: take the action with the highest state, action value\n", - " action =\n", - "\n", - " return action" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "B2_-8b8z5k54" - }, - "source": [ - "#### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "se2OzWGW5kYJ" - }, - "outputs": [], - "source": [ - "def greedy_policy(Qtable, state):\n", - " # Exploitation: take the action with the highest state, action value\n", - " action = np.argmax(Qtable[state][:])\n", - "\n", - " return action" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "flILKhBU3yZ7" - }, - "source": [ - "##Define the epsilon-greedy policy ๐Ÿค–\n", - "\n", - "Epsilon-greedy is the training policy that handles the exploration/exploitation trade-off.\n", - "\n", - "The idea with epsilon-greedy:\n", - "\n", - "- With *probability 1โ€Š-โ€Šษ›* : **we do exploitation** (i.e. our agent selects the action with the highest state-action pair value).\n", - "\n", - "- With *probability ษ›*: we do **exploration** (trying a random action).\n", - "\n", - "As the training continues, we progressively **reduce the epsilon value since we will need less and less exploration and more exploitation.**\n", - "\n", - "\"Q-Learning\"\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "6Bj7x3in3_Pq" - }, - "outputs": [], - "source": [ - "def epsilon_greedy_policy(Qtable, state, epsilon):\n", - " # Randomly generate a number between 0 and 1\n", - " random_num =\n", - " # if random_num > greater than epsilon --> exploitation\n", - " if random_num > epsilon:\n", - " # Take the action with the highest value given a state\n", - " # np.argmax can be useful here\n", - " action =\n", - " # else --> exploration\n", - " else:\n", - " action = # Take a random action\n", - "\n", - " return action" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8R5ej1fS4P2V" - }, - "source": [ - "#### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "cYxHuckr4LiG" - }, - "outputs": [], - "source": [ - "def epsilon_greedy_policy(Qtable, state, epsilon):\n", - " # Randomly generate a number between 0 and 1\n", - " random_num = random.uniform(0,1)\n", - " # if random_num > greater than epsilon --> exploitation\n", - " if random_num > epsilon:\n", - " # Take the action with the highest value given a state\n", - " # np.argmax can be useful here\n", - " action = greedy_policy(Qtable, state)\n", - " # else --> exploration\n", - " else:\n", - " action = env.action_space.sample()\n", - "\n", - " return action" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hW80DealcRtu" - }, - "source": [ - "## Define the hyperparameters โš™๏ธ\n", - "\n", - "The exploration related hyperparamters are some of the most important ones.\n", - "\n", - "- We need to make sure that our agent **explores enough of the state space** to learn a good value approximation. To do that, we need to have progressive decay of the epsilon.\n", - "- If you decrease epsilon too fast (too high decay_rate), **you take the risk that your agent will be stuck**, since your agent didn't explore enough of the state space and hence can't solve the problem." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Y1tWn0tycWZ1" - }, - "outputs": [], - "source": [ - "# Training parameters\n", - "n_training_episodes = 10000 # Total training episodes\n", - "learning_rate = 0.7 # Learning rate\n", - "\n", - "# Evaluation parameters\n", - "n_eval_episodes = 100 # Total number of test episodes\n", - "\n", - "# Environment parameters\n", - "env_id = \"FrozenLake-v1\" # Name of the environment\n", - "max_steps = 99 # Max steps per episode\n", - "gamma = 0.95 # Discounting rate\n", - "eval_seed = [] # The evaluation seed of the environment\n", - "\n", - "# Exploration parameters\n", - "max_epsilon = 1.0 # Exploration probability at start\n", - "min_epsilon = 0.05 # Minimum exploration probability\n", - "decay_rate = 0.0005 # Exponential decay rate for exploration prob" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "cDb7Tdx8atfL" - }, - "source": [ - "## Create the training loop method\n", - "\n", - "\"Q-Learning\"\n", - "\n", - "The training loop goes like this:\n", - "\n", - "```\n", - "For episode in the total of training episodes:\n", - "\n", - "Reduce epsilon (since we need less and less exploration)\n", - "Reset the environment\n", - "\n", - " For step in max timesteps: \n", - " Choose the action At using epsilon greedy policy\n", - " Take the action (a) and observe the outcome state(s') and reward (r)\n", - " Update the Q-value Q(s,a) using Bellman equation Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n", - " If done, finish the episode\n", - " Our next state is the new state\n", - "```" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "paOynXy3aoJW" - }, - "outputs": [], - "source": [ - "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n", - " for episode in tqdm(range(n_training_episodes)):\n", - " # Reduce epsilon (because we need less and less exploration)\n", - " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n", - " # Reset the environment\n", - " state, info = env.reset()\n", - " step = 0\n", - " terminated = False\n", - " truncated = False\n", - "\n", - " # repeat\n", - " for step in range(max_steps):\n", - " # Choose the action At using epsilon greedy policy\n", - " action =\n", - "\n", - " # Take action At and observe Rt+1 and St+1\n", - " # Take the action (a) and observe the outcome state(s') and reward (r)\n", - " new_state, reward, terminated, truncated, info =\n", - "\n", - " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n", - " Qtable[state][action] =\n", - "\n", - " # If terminated or truncated finish the episode\n", - " if terminated or truncated:\n", - " break\n", - "\n", - " # Our next state is the new state\n", - " state = new_state\n", - " return Qtable" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Pnpk2ePoem3r" - }, - "source": [ - "#### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "IyZaYbUAeolw" - }, - "outputs": [], - "source": [ - "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n", - " for episode in tqdm(range(n_training_episodes)):\n", - " # Reduce epsilon (because we need less and less exploration)\n", - " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n", - " # Reset the environment\n", - " state, info = env.reset()\n", - " step = 0\n", - " terminated = False\n", - " truncated = False\n", - "\n", - " # repeat\n", - " for step in range(max_steps):\n", - " # Choose the action At using epsilon greedy policy\n", - " action = epsilon_greedy_policy(Qtable, state, epsilon)\n", - "\n", - " # Take action At and observe Rt+1 and St+1\n", - " # Take the action (a) and observe the outcome state(s') and reward (r)\n", - " new_state, reward, terminated, truncated, info = env.step(action)\n", - "\n", - " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n", - " Qtable[state][action] = Qtable[state][action] + learning_rate * (reward + gamma * np.max(Qtable[new_state]) - Qtable[state][action])\n", - "\n", - " # If terminated or truncated finish the episode\n", - " if terminated or truncated:\n", - " break\n", - "\n", - " # Our next state is the new state\n", - " state = new_state\n", - " return Qtable" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "WLwKQ4tUdhGI" - }, - "source": [ - "## Train the Q-Learning agent ๐Ÿƒ" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "DPBxfjJdTCOH" - }, - "outputs": [], - "source": [ - "Qtable_frozenlake = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_frozenlake)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "yVeEhUCrc30L" - }, - "source": [ - "## Let's see what our Q-Learning table looks like now ๐Ÿ‘€" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "nmfchsTITw4q" - }, - "outputs": [], - "source": [ - "Qtable_frozenlake" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "pUrWkxsHccXD" - }, - "source": [ - "## The evaluation method ๐Ÿ“\n", - "\n", - "- We defined the evaluation method that we're going to use to test our Q-Learning agent." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "jNl0_JO2cbkm" - }, - "outputs": [], - "source": [ - "def evaluate_agent(env, max_steps, n_eval_episodes, Q, seed):\n", - " \"\"\"\n", - " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n", - " :param env: The evaluation environment\n", - " :param max_steps: Maximum number of steps per episode\n", - " :param n_eval_episodes: Number of episode to evaluate the agent\n", - " :param Q: The Q-table\n", - " :param seed: The evaluation seed array (for taxi-v3)\n", - " \"\"\"\n", - " episode_rewards = []\n", - " for episode in tqdm(range(n_eval_episodes)):\n", - " if seed:\n", - " state, info = env.reset(seed=seed[episode])\n", - " else:\n", - " state, info = env.reset()\n", - " step = 0\n", - " truncated = False\n", - " terminated = False\n", - " total_rewards_ep = 0\n", - "\n", - " for step in range(max_steps):\n", - " # Take the action (index) that have the maximum expected future reward given that state\n", - " action = greedy_policy(Q, state)\n", - " new_state, reward, terminated, truncated, info = env.step(action)\n", - " total_rewards_ep += reward\n", - "\n", - " if terminated or truncated:\n", - " break\n", - " state = new_state\n", - " episode_rewards.append(total_rewards_ep)\n", - " mean_reward = np.mean(episode_rewards)\n", - " std_reward = np.std(episode_rewards)\n", - "\n", - " return mean_reward, std_reward" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0jJqjaoAnxUo" - }, - "source": [ - "## Evaluate our Q-Learning agent ๐Ÿ“ˆ\n", - "\n", - "- Usually, you should have a mean reward of 1.0\n", - "- The **environment is relatively easy** since the state space is really small (16). What you can try to do is [to replace it with the slippery version](https://gymnasium.farama.org/environments/toy_text/frozen_lake/), which introduces stochasticity, making the environment more complex." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "fAgB7s0HEFMm" - }, - "outputs": [], - "source": [ - "# Evaluate our Agent\n", - "mean_reward, std_reward = evaluate_agent(env, max_steps, n_eval_episodes, Qtable_frozenlake, eval_seed)\n", - "print(f\"Mean_reward={mean_reward:.2f} +/- {std_reward:.2f}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "yxaP3bPdg1DV" - }, - "source": [ - "## Publish our trained model to the Hub ๐Ÿ”ฅ\n", - "\n", - "Now that we saw good results after the training, **we can publish our trained model to the Hub ๐Ÿค— with one line of code**.\n", - "\n", - "Here's an example of a Model Card:\n", - "\n", - "\"Model\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "kv0k1JQjpMq3" - }, - "source": [ - "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "QZ5LrR-joIHD" - }, - "source": [ - "#### Do not modify this code" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Jex3i9lZ8ksX" - }, - "outputs": [], - "source": [ - "from huggingface_hub import HfApi, snapshot_download\n", - "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n", - "\n", - "from pathlib import Path\n", - "import datetime\n", - "import json" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Qo57HBn3W74O" - }, - "outputs": [], - "source": [ - "def record_video(env, Qtable, out_directory, fps=1):\n", - " \"\"\"\n", - " Generate a replay video of the agent\n", - " :param env\n", - " :param Qtable: Qtable of our agent\n", - " :param out_directory\n", - " :param fps: how many frame per seconds (with taxi-v3 and frozenlake-v1 we use 1)\n", - " \"\"\"\n", - " images = []\n", - " terminated = False\n", - " truncated = False\n", - " state, info = env.reset(seed=random.randint(0,500))\n", - " img = env.render()\n", - " images.append(img)\n", - " while not terminated or truncated:\n", - " # Take the action (index) that have the maximum expected future reward given that state\n", - " action = np.argmax(Qtable[state][:])\n", - " state, reward, terminated, truncated, info = env.step(action) # We directly put next_state = state for recording logic\n", - " img = env.render()\n", - " images.append(img)\n", - " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)" - ] - }, - { - "cell_type": "code", - "source": [ - "def push_to_hub(\n", - " repo_id, model, env, video_fps=1, local_repo_path=\"hub\"\n", - "):\n", - " \"\"\"\n", - " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n", - " This method does the complete pipeline:\n", - " - It evaluates the model\n", - " - It generates the model card\n", - " - It generates a replay video of the agent\n", - " - It pushes everything to the Hub\n", - "\n", - " :param repo_id: repo_id: id of the model repository from the Hugging Face Hub\n", - " :param env\n", - " :param video_fps: how many frame per seconds to record our video replay\n", - " (with taxi-v3 and frozenlake-v1 we use 1)\n", - " :param local_repo_path: where the local repository is\n", - " \"\"\"\n", - " _, repo_name = repo_id.split(\"/\")\n", - "\n", - " eval_env = env\n", - " api = HfApi()\n", - "\n", - " # Step 1: Create the repo\n", - " repo_url = api.create_repo(\n", - " repo_id=repo_id,\n", - " exist_ok=True,\n", - " )\n", - "\n", - " # Step 2: Download files\n", - " repo_local_path = Path(snapshot_download(repo_id=repo_id))\n", - "\n", - " # Step 3: Save the model\n", - " if env.spec.kwargs.get(\"map_name\"):\n", - " model[\"map_name\"] = env.spec.kwargs.get(\"map_name\")\n", - " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n", - " model[\"slippery\"] = False\n", - "\n", - " # Pickle the model\n", - " with open((repo_local_path) / \"q-learning.pkl\", \"wb\") as f:\n", - " pickle.dump(model, f)\n", - "\n", - " # Step 4: Evaluate the model and build JSON with evaluation metrics\n", - " mean_reward, std_reward = evaluate_agent(\n", - " eval_env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"]\n", - " )\n", - "\n", - " evaluate_data = {\n", - " \"env_id\": model[\"env_id\"],\n", - " \"mean_reward\": mean_reward,\n", - " \"n_eval_episodes\": model[\"n_eval_episodes\"],\n", - " \"eval_datetime\": datetime.datetime.now().isoformat()\n", - " }\n", - "\n", - " # Write a JSON file called \"results.json\" that will contain the\n", - " # evaluation results\n", - " with open(repo_local_path / \"results.json\", \"w\") as outfile:\n", - " json.dump(evaluate_data, outfile)\n", - "\n", - " # Step 5: Create the model card\n", - " env_name = model[\"env_id\"]\n", - " if env.spec.kwargs.get(\"map_name\"):\n", - " env_name += \"-\" + env.spec.kwargs.get(\"map_name\")\n", - "\n", - " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n", - " env_name += \"-\" + \"no_slippery\"\n", - "\n", - " metadata = {}\n", - " metadata[\"tags\"] = [env_name, \"q-learning\", \"reinforcement-learning\", \"custom-implementation\"]\n", - "\n", - " # Add metrics\n", - " eval = metadata_eval_result(\n", - " model_pretty_name=repo_name,\n", - " task_pretty_name=\"reinforcement-learning\",\n", - " task_id=\"reinforcement-learning\",\n", - " metrics_pretty_name=\"mean_reward\",\n", - " metrics_id=\"mean_reward\",\n", - " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n", - " dataset_pretty_name=env_name,\n", - " dataset_id=env_name,\n", - " )\n", - "\n", - " # Merges both dictionaries\n", - " metadata = {**metadata, **eval}\n", - "\n", - " model_card = f\"\"\"\n", - " # **Q-Learning** Agent playing1 **{env_id}**\n", - " This is a trained model of a **Q-Learning** agent playing **{env_id}** .\n", - "\n", - " ## Usage\n", - "\n", - " ```python\n", - "\n", - " model = load_from_hub(repo_id=\"{repo_id}\", filename=\"q-learning.pkl\")\n", - "\n", - " # Don't forget to check if you need to add additional attributes (is_slippery=False etc)\n", - " env = gym.make(model[\"env_id\"])\n", - " ```\n", - " \"\"\"\n", - "\n", - " evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])\n", - "\n", - " readme_path = repo_local_path / \"README.md\"\n", - " readme = \"\"\n", - " print(readme_path.exists())\n", - " if readme_path.exists():\n", - " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n", - " readme = f.read()\n", - " else:\n", - " readme = model_card\n", - "\n", - " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n", - " f.write(readme)\n", - "\n", - " # Save our metrics to Readme metadata\n", - " metadata_save(readme_path, metadata)\n", - "\n", - " # Step 6: Record a video\n", - " video_path = repo_local_path / \"replay.mp4\"\n", - " record_video(env, model[\"qtable\"], video_path, video_fps)\n", - "\n", - " # Step 7. Push everything to the Hub\n", - " api.upload_folder(\n", - " repo_id=repo_id,\n", - " folder_path=repo_local_path,\n", - " path_in_repo=\".\",\n", - " )\n", - "\n", - " print(\"Your model is pushed to the Hub. You can view your model here: \", repo_url)" - ], - "metadata": { - "id": "U4mdUTKkGnUd" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "81J6cet_ogSS" - }, - "source": [ - "### .\n", - "\n", - "By using `push_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the Hub**.\n", - "\n", - "This way:\n", - "- You can **showcase our work** ๐Ÿ”ฅ\n", - "- You can **visualize your agent playing** ๐Ÿ‘€\n", - "- You can **share an agent with the community that others can use** ๐Ÿ’พ\n", - "- You can **access a leaderboard ๐Ÿ† to see how well your agent is performing compared to your classmates** ๐Ÿ‘‰ https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "cWnFC0iZooTw" - }, - "source": [ - "To be able to share your model with the community there are three more steps to follow:\n", - "\n", - "1๏ธโƒฃ (If it's not already done) create an account to HF โžก https://huggingface.co/join\n", - "\n", - "2๏ธโƒฃ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", - "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", - "\n", - "\n", - "\"Create\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "QB5nIcxR8paT" - }, - "outputs": [], - "source": [ - "from huggingface_hub import notebook_login\n", - "notebook_login()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "GyWc1x3-o3xG" - }, - "source": [ - "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login` (or `login`)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Gc5AfUeFo3xH" - }, - "source": [ - "3๏ธโƒฃ We're now ready to push our trained agent to the ๐Ÿค— Hub ๐Ÿ”ฅ using `push_to_hub()` function\n", - "\n", - "- Let's create **the model dictionary that contains the hyperparameters and the Q_table**." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "FiMqxqVHg0I4" - }, - "outputs": [], - "source": [ - "model = {\n", - " \"env_id\": env_id,\n", - " \"max_steps\": max_steps,\n", - " \"n_training_episodes\": n_training_episodes,\n", - " \"n_eval_episodes\": n_eval_episodes,\n", - " \"eval_seed\": eval_seed,\n", - "\n", - " \"learning_rate\": learning_rate,\n", - " \"gamma\": gamma,\n", - "\n", - " \"max_epsilon\": max_epsilon,\n", - " \"min_epsilon\": min_epsilon,\n", - " \"decay_rate\": decay_rate,\n", - "\n", - " \"qtable\": Qtable_frozenlake\n", - "}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "9kld-AEso3xH" - }, - "source": [ - "Let's fill the `push_to_hub` function:\n", - "\n", - "- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `\n", - "(repo_id = {username}/{repo_name})`\n", - "๐Ÿ’ก A good `repo_id` is `{username}/q-{env_id}`\n", - "- `model`: our model dictionary containing the hyperparameters and the Qtable.\n", - "- `env`: the environment.\n", - "- `commit_message`: message of the commit" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "5sBo2umnXpPd" - }, - "outputs": [], - "source": [ - "model" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "RpOTtSt83kPZ" - }, - "outputs": [], - "source": [ - "username = \"\" # FILL THIS\n", - "repo_name = \"q-FrozenLake-v1-4x4-noSlippery\"\n", - "push_to_hub(\n", - " repo_id=f\"{username}/{repo_name}\",\n", - " model=model,\n", - " env=env)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "E2875IGsprzq" - }, - "source": [ - "Congrats ๐Ÿฅณ you've just implemented from scratch, trained, and uploaded your first Reinforcement Learning agent.\n", - "FrozenLake-v1 no_slippery is very simple environment, let's try a harder one ๐Ÿ”ฅ." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "18lN8Bz7yvLt" - }, - "source": [ - "# Part 2: Taxi-v3 ๐Ÿš–\n", - "\n", - "## Create and understand [Taxi-v3 ๐Ÿš•](https://gymnasium.farama.org/environments/toy_text/taxi/)\n", - "---\n", - "\n", - "๐Ÿ’ก A good habit when you start to use an environment is to check its documentation\n", - "\n", - "๐Ÿ‘‰ https://gymnasium.farama.org/environments/toy_text/taxi/\n", - "\n", - "---\n", - "\n", - "In `Taxi-v3` ๐Ÿš•, there are four designated locations in the grid world indicated by R(ed), G(reen), Y(ellow), and B(lue).\n", - "\n", - "When the episode starts, **the taxi starts off at a random square** and the passenger is at a random location. The taxi drives to the passengerโ€™s location, **picks up the passenger**, drives to the passengerโ€™s destination (another one of the four specified locations), and then **drops off the passenger**. Once the passenger is dropped off, the episode ends.\n", - "\n", - "\n", - "\"Taxi\"\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "gL0wpeO8gpej" - }, - "outputs": [], - "source": [ - "env = gym.make(\"Taxi-v3\", render_mode=\"rgb_array\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "gBOaXgtsrmtT" - }, - "source": [ - "There are **500 discrete states since there are 25 taxi positions, 5 possible locations of the passenger** (including the case when the passenger is in the taxi), and **4 destination locations.**\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "_TPNaGSZrgqA" - }, - "outputs": [], - "source": [ - "state_space = env.observation_space.n\n", - "print(\"There are \", state_space, \" possible states\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "CdeeZuokrhit" - }, - "outputs": [], - "source": [ - "action_space = env.action_space.n\n", - "print(\"There are \", action_space, \" possible actions\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "R1r50Advrh5Q" - }, - "source": [ - "The action space (the set of possible actions the agent can take) is discrete with **6 actions available ๐ŸŽฎ**:\n", - "\n", - "- 0: move south\n", - "- 1: move north\n", - "- 2: move east\n", - "- 3: move west\n", - "- 4: pickup passenger\n", - "- 5: drop off passenger\n", - "\n", - "Reward function ๐Ÿ’ฐ:\n", - "\n", - "- -1 per step unless other reward is triggered.\n", - "- +20 delivering passenger.\n", - "- -10 executing โ€œpickupโ€ and โ€œdrop-offโ€ actions illegally." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "US3yDXnEtY9I" - }, - "outputs": [], - "source": [ - "# Create our Q table with state_size rows and action_size columns (500x6)\n", - "Qtable_taxi = initialize_q_table(state_space, action_space)\n", - "print(Qtable_taxi)\n", - "print(\"Q-table shape: \", Qtable_taxi .shape)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "gUMKPH0_LJyH" - }, - "source": [ - "## Define the hyperparameters โš™๏ธ\n", - "\n", - "โš  DO NOT MODIFY EVAL_SEED: the eval_seed array **allows us to evaluate your agent with the same taxi starting positions for every classmate**" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "AB6n__hhg7YS" - }, - "outputs": [], - "source": [ - "# Training parameters\n", - "n_training_episodes = 25000 # Total training episodes\n", - "learning_rate = 0.7 # Learning rate\n", - "\n", - "# Evaluation parameters\n", - "n_eval_episodes = 100 # Total number of test episodes\n", - "\n", - "# DO NOT MODIFY EVAL_SEED\n", - "eval_seed = [16,54,165,177,191,191,120,80,149,178,48,38,6,125,174,73,50,172,100,148,146,6,25,40,68,148,49,167,9,97,164,176,61,7,54,55,\n", - " 161,131,184,51,170,12,120,113,95,126,51,98,36,135,54,82,45,95,89,59,95,124,9,113,58,85,51,134,121,169,105,21,30,11,50,65,12,43,82,145,152,97,106,55,31,85,38,\n", - " 112,102,168,123,97,21,83,158,26,80,63,5,81,32,11,28,148] # Evaluation seed, this ensures that all classmates agents are trained on the same taxi starting position\n", - " # Each seed has a specific starting state\n", - "\n", - "# Environment parameters\n", - "env_id = \"Taxi-v3\" # Name of the environment\n", - "max_steps = 99 # Max steps per episode\n", - "gamma = 0.95 # Discounting rate\n", - "\n", - "# Exploration parameters\n", - "max_epsilon = 1.0 # Exploration probability at start\n", - "min_epsilon = 0.05 # Minimum exploration probability\n", - "decay_rate = 0.005 # Exponential decay rate for exploration prob\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1TMORo1VLTsX" - }, - "source": [ - "## Train our Q-Learning agent ๐Ÿƒ" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "WwP3Y2z2eS-K" - }, - "outputs": [], - "source": [ - "Qtable_taxi = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_taxi)\n", - "Qtable_taxi" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wPdu0SueLVl2" - }, - "source": [ - "## Create a model dictionary ๐Ÿ’พ and publish our trained model to the Hub ๐Ÿ”ฅ\n", - "\n", - "- We create a model dictionary that will contain all the training hyperparameters for reproducibility and the Q-Table.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "0a1FpE_3hNYr" - }, - "outputs": [], - "source": [ - "model = {\n", - " \"env_id\": env_id,\n", - " \"max_steps\": max_steps,\n", - " \"n_training_episodes\": n_training_episodes,\n", - " \"n_eval_episodes\": n_eval_episodes,\n", - " \"eval_seed\": eval_seed,\n", - "\n", - " \"learning_rate\": learning_rate,\n", - " \"gamma\": gamma,\n", - "\n", - " \"max_epsilon\": max_epsilon,\n", - " \"min_epsilon\": min_epsilon,\n", - " \"decay_rate\": decay_rate,\n", - "\n", - " \"qtable\": Qtable_taxi\n", - "}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "dhQtiQozhOn1" - }, - "outputs": [], - "source": [ - "username = \"\" # FILL THIS\n", - "repo_name = \"\" # FILL THIS\n", - "push_to_hub(\n", - " repo_id=f\"{username}/{repo_name}\",\n", - " model=model,\n", - " env=env)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ZgSdjgbIpRti" - }, - "source": [ - "Now that it's on the Hub, you can compare the results of your Taxi-v3 with your classmates using the leaderboard ๐Ÿ† ๐Ÿ‘‰ https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n", - "\n", - "\n", - "\"Taxi" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bzgIO70c0bu2" - }, - "source": [ - "# Part 3: Load from Hub ๐Ÿ”ฝ\n", - "\n", - "What's amazing with Hugging Face Hub ๐Ÿค— is that you can easily load powerful models from the community.\n", - "\n", - "Loading a saved model from the Hub is really easy:\n", - "\n", - "1. You go https://huggingface.co/models?other=q-learning to see the list of all the q-learning saved models.\n", - "2. You select one and copy its repo_id\n", - "\n", - "\"Copy" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "gTth6thRoC6X" - }, - "source": [ - "3. Then we just need to use `load_from_hub` with:\n", - "- The repo_id\n", - "- The filename: the saved model inside the repo." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "EtrfoTaBoNrd" - }, - "source": [ - "#### Do not modify this code" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Eo8qEzNtCaVI" - }, - "outputs": [], - "source": [ - "from urllib.error import HTTPError\n", - "\n", - "from huggingface_hub import hf_hub_download\n", - "\n", - "\n", - "def load_from_hub(repo_id: str, filename: str) -> str:\n", - " \"\"\"\n", - " Download a model from Hugging Face Hub.\n", - " :param repo_id: id of the model repository from the Hugging Face Hub\n", - " :param filename: name of the model zip file from the repository\n", - " \"\"\"\n", - " # Get the model from the Hub, download and cache the model on your local disk\n", - " pickle_model = hf_hub_download(\n", - " repo_id=repo_id,\n", - " filename=filename\n", - " )\n", - "\n", - " with open(pickle_model, 'rb') as f:\n", - " downloaded_model_file = pickle.load(f)\n", - "\n", - " return downloaded_model_file" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "b_sM2gNioPZH" - }, - "source": [ - "### ." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "JUm9lz2gCQcU" - }, - "outputs": [], - "source": [ - "model = load_from_hub(repo_id=\"ThomasSimonini/q-Taxi-v3\", filename=\"q-learning.pkl\") # Try to use another model\n", - "\n", - "print(model)\n", - "env = gym.make(model[\"env_id\"])\n", - "\n", - "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "O7pL8rg1MulN" - }, - "outputs": [], - "source": [ - "model = load_from_hub(repo_id=\"ThomasSimonini/q-FrozenLake-v1-no-slippery\", filename=\"q-learning.pkl\") # Try to use another model\n", - "\n", - "env = gym.make(model[\"env_id\"], is_slippery=False)\n", - "\n", - "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "BQAwLnYFPk-s" - }, - "source": [ - "## Some additional challenges ๐Ÿ†\n", - "\n", - "The best way to learn **is to try things on your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results!\n", - "\n", - "In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n", - "\n", - "Here are some ideas to climb up the leaderboard:\n", - "\n", - "* Train more steps\n", - "* Try different hyperparameters by looking at what your classmates have done.\n", - "* **Push your new trained model** on the Hub ๐Ÿ”ฅ\n", - "\n", - "Are walking on ice and driving taxis too boring to you? Try to **change the environment**, why not use FrozenLake-v1 slippery version? Check how they work [using the gymnasium documentation](https://gymnasium.farama.org/) and have fun ๐ŸŽ‰." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "p-fW-EU5WejJ" - }, - "source": [ - "_____________________________________________________________________\n", - "Congrats ๐Ÿฅณ, you've just implemented, trained, and uploaded your first Reinforcement Learning agent.\n", - "\n", - "Understanding Q-Learning is an **important step to understanding value-based methods.**\n", - "\n", - "In the next Unit with Deep Q-Learning, we'll see that while creating and updating a Q-table was a good strategy โ€” **however, it is not scalable.**\n", - "\n", - "For instance, imagine you create an agent that learns to play Doom.\n", - "\n", - "\"Doom\"/\n", - "\n", - "Doom is a large environment with a huge state space (millions of different states). Creating and updating a Q-table for that environment would not be efficient.\n", - "\n", - "That's why we'll study Deep Q-Learning in the next unit, an algorithm **where we use a neural network that approximates, given a state, the different Q-values for each action.**\n", - "\n", - "\"Environments\"/\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "BjLhT70TEZIn" - }, - "source": [ - "See you in Unit 3! ๐Ÿ”ฅ\n", - "\n", - "## Keep learning, stay awesome ๐Ÿค—" - ] - } - ], - "metadata": { - "colab": { - "private_outputs": true, - "provenance": [], - "collapsed_sections": [ - "67OdoKL63eDD", - "B2_-8b8z5k54", - "8R5ej1fS4P2V", - "Pnpk2ePoem3r" - ], - "include_colab_link": true - }, - "gpuClass": "standard", - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - }, - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "view-in-github" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "njb_ProuHiOe" + }, + "source": [ + "# Unit 2: Q-Learning with FrozenLake-v1 โ›„ and Taxi-v3 ๐Ÿš•\n", + "\n", + "\"Unit\n", + "\n", + "In this notebook, **you'll code your first Reinforcement Learning agent from scratch** to play FrozenLake โ„๏ธ using Q-Learning, share it with the community, and experiment with different configurations.\n", + "\n", + "โฌ‡๏ธ Here is an example of what **you will achieve in just a couple of minutes.** โฌ‡๏ธ\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vRU_vXBrl1Jx" + }, + "source": [ + "\"Environments\"/" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DPTBOv9HYLZ2" + }, + "source": [ + "###๐ŸŽฎ Environments:\n", + "\n", + "- [FrozenLake-v1](https://gymnasium.farama.org/environments/toy_text/frozen_lake/)\n", + "- [Taxi-v3](https://gymnasium.farama.org/environments/toy_text/taxi/)\n", + "\n", + "###๐Ÿ“š RL-Library:\n", + "\n", + "- Python and NumPy\n", + "- [Gymnasium](https://gymnasium.farama.org/)\n", + "\n", + "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4i6tjI2tHQ8j" + }, + "source": [ + "## Objectives of this notebook ๐Ÿ†\n", + "\n", + "At the end of the notebook, you will:\n", + "\n", + "- Be able to use **Gymnasium**, the environment library.\n", + "- Be able to code a Q-Learning agent from scratch.\n", + "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score ๐Ÿ”ฅ.\n", + "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "viNzVbVaYvY3" + }, + "source": [ + "## This notebook is from the Deep Reinforcement Learning Course\n", + "\n", + "\"Deep" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6p5HnEefISCB" + }, + "source": [ + "In this free course, you will:\n", + "\n", + "- ๐Ÿ“– Study Deep Reinforcement Learning in **theory and practice**.\n", + "- ๐Ÿง‘โ€๐Ÿ’ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n", + "- ๐Ÿค– Train **agents in unique environments**\n", + "\n", + "And more check ๐Ÿ“š the syllabus ๐Ÿ‘‰ https://simoninithomas.github.io/deep-rl-course\n", + "\n", + "Donโ€™t forget to **sign up to the course** (we are collecting your email to be able toย **send you the links when each Unit is published and give you information about the challenges and updates).**\n", + "\n", + "\n", + "The best way to keep in touch is to join our discord server to exchange with the community and with us ๐Ÿ‘‰๐Ÿป https://discord.gg/ydHrjt3WP5" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Y-mo_6rXIjRi" + }, + "source": [ + "## Prerequisites ๐Ÿ—๏ธ\n", + "\n", + "Before diving into the notebook, you need to:\n", + "\n", + "๐Ÿ”ฒ ๐Ÿ“š **Study [Q-Learning by reading Unit 2](https://huggingface.co/deep-rl-course/unit2/introduction)** ๐Ÿค— " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f2ONOODsyrMU" + }, + "source": [ + "## A small recap of Q-Learning" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "V68VveLacfxJ" + }, + "source": [ + "*Q-Learning* **is the RL algorithm that**:\n", + "\n", + "- Trains *Q-Function*, an **action-value function** that encoded, in internal memory, by a *Q-table* **that contains all the state-action pair values.**\n", + "\n", + "- Given a state and action, our Q-Function **will search the Q-table for the corresponding value.**\n", + " \n", + "\"Q\n", + "\n", + "- When the training is done,**we have an optimal Q-Function, so an optimal Q-Table.**\n", + " \n", + "- And if we **have an optimal Q-function**, we\n", + "have an optimal policy, since we **know for, each state, the best action to take.**\n", + "\n", + "\"Link\n", + "\n", + "\n", + "But, in the beginning,ย our **Q-Table is useless since it gives arbitrary value for each state-action pairย (most of the time we initialize the Q-Table to 0 values)**. But, as weโ€™llย explore the environment and update our Q-Table it will give us better and better approximations\n", + "\n", + "\"q-learning.jpeg\"\n", + "\n", + "This is the Q-Learning pseudocode:\n", + "\n", + "\"Q-Learning\"\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HEtx8Y8MqKfH" + }, + "source": [ + "# Let's code our first Reinforcement Learning algorithm ๐Ÿš€" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Kdxb1IhzTn0v" + }, + "source": [ + "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push your trained Taxi model to the Hub and **get a result of >= 4.5**.\n", + "\n", + "To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward**\n", + "\n", + "For more information about the certification process, check this section ๐Ÿ‘‰ https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4gpxC1_kqUYe" + }, + "source": [ + "## Install dependencies and create a virtual display ๐Ÿ”ฝ\n", + "\n", + "In the notebook, we'll need to generate a replay video. To do so, with Colab, **we need to have a virtual screen to render the environment** (and thus record the frames).\n", + "\n", + "Hence the following cell will install the libraries and create and run a virtual screen ๐Ÿ–ฅ\n", + "\n", + "Weโ€™ll install multiple ones:\n", + "\n", + "- `gymnasium`: Contains the FrozenLake-v1 โ›„ and Taxi-v3 ๐Ÿš• environments.\n", + "- `pygame`: Used for the FrozenLake-v1 and Taxi-v3 UI.\n", + "- `numpy`: Used for handling our Q-table.\n", + "\n", + "The Hugging Face Hub ๐Ÿค— works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.\n", + "\n", + "You can see here all the Deep RL models available (if they use Q Learning) here ๐Ÿ‘‰ https://huggingface.co/models?other=q-learning" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9XaULfDZDvrC" + }, + "outputs": [], + "source": [ + "!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit2/requirements-unit2.txt" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "n71uTX7qqzz2" + }, + "outputs": [], + "source": [ + "!sudo apt-get update\n", + "!sudo apt-get install -y python3-opengl\n", + "!apt install ffmpeg xvfb\n", + "!pip3 install pyvirtualdisplay" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "K6XC13pTfFiD" + }, + "source": [ + "To make sure the new installed libraries are used, **sometimes it's required to restart the notebook runtime**. The next cell will force the **runtime to crash, so you'll need to connect again and run the code starting from here**. Thanks to this trick, **we will be able to run our virtual screen.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3kuZbWAkfHdg" + }, + "outputs": [], + "source": [ + "import os\n", + "os.kill(os.getpid(), 9)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DaY1N4dBrabi" + }, + "outputs": [], + "source": [ + "# Virtual display\n", + "from pyvirtualdisplay import Display\n", + "\n", + "virtual_display = Display(visible=0, size=(1400, 900))\n", + "virtual_display.start()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "W-7f-Swax_9x" + }, + "source": [ + "## Import the packages ๐Ÿ“ฆ\n", + "\n", + "In addition to the installed libraries, we also use:\n", + "\n", + "- `random`: To generate random numbers (that will be useful for epsilon-greedy policy).\n", + "- `imageio`: To generate a replay video." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "VcNvOAQlysBJ" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import gymnasium as gym\n", + "import random\n", + "import imageio\n", + "import os\n", + "import tqdm\n", + "\n", + "import pickle5 as pickle\n", + "from tqdm.notebook import tqdm" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xp4-bXKIy1mQ" + }, + "source": [ + "We're now ready to code our Q-Learning algorithm ๐Ÿ”ฅ" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xya49aNJWVvv" + }, + "source": [ + "# Part 1: Frozen Lake โ›„ (non slippery version)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NAvihuHdy9tw" + }, + "source": [ + "## Create and understand [FrozenLake environment โ›„]((https://gymnasium.farama.org/environments/toy_text/frozen_lake/)\n", + "---\n", + "\n", + "๐Ÿ’ก A good habit when you start to use an environment is to check its documentation\n", + "\n", + "๐Ÿ‘‰ https://gymnasium.farama.org/environments/toy_text/frozen_lake/\n", + "\n", + "---\n", + "\n", + "We're going to train our Q-Learning agent **to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H)**.\n", + "\n", + "We can have two sizes of environment:\n", + "\n", + "- `map_name=\"4x4\"`: a 4x4 grid version\n", + "- `map_name=\"8x8\"`: a 8x8 grid version\n", + "\n", + "\n", + "The environment has two modes:\n", + "\n", + "- `is_slippery=False`: The agent always moves **in the intended direction** due to the non-slippery nature of the frozen lake (deterministic).\n", + "- `is_slippery=True`: The agent **may not always move in the intended direction** due to the slippery nature of the frozen lake (stochastic)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UaW_LHfS0PY2" + }, + "source": [ + "For now let's keep it simple with the 4x4 map and non-slippery.\n", + "We add a parameter called `render_mode` that specifies how the environment should be visualised. In our case because we **want to record a video of the environment at the end, we need to set render_mode to rgb_array**.\n", + "\n", + "As [explained in the documentation](https://gymnasium.farama.org/api/env/#gymnasium.Env.render) โ€œrgb_arrayโ€: Return a single frame representing the current state of the environment. A frame is a np.ndarray with shape (x, y, 3) representing RGB values for an x-by-y pixel image." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "IzJnb8O3y8up" + }, + "outputs": [], + "source": [ + "# Create the FrozenLake-v1 environment using 4x4 map and non-slippery version and render_mode=\"rgb_array\"\n", + "env = gym.make() # TODO use the correct parameters" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ji_UrI5l2zzn" + }, + "source": [ + "### Solution" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "jNxUbPMP0akP" + }, + "outputs": [], + "source": [ + "env = gym.make(\"FrozenLake-v1\", map_name=\"4x4\", is_slippery=False, render_mode=\"rgb_array\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KASNViqL4tZn" + }, + "source": [ + "You can create your own custom grid like this:\n", + "\n", + "```python\n", + "desc=[\"SFFF\", \"FHFH\", \"FFFH\", \"HFFG\"]\n", + "gym.make('FrozenLake-v1', desc=desc, is_slippery=True)\n", + "```\n", + "\n", + "but we'll use the default environment for now." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SXbTfdeJ1Xi9" + }, + "source": [ + "### Let's see what the Environment looks like:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ZNPG0g_UGCfh" + }, + "outputs": [], + "source": [ + "# We create our environment with gym.make(\"\")- `is_slippery=False`: The agent always moves in the intended direction due to the non-slippery nature of the frozen lake (deterministic).\n", + "print(\"_____OBSERVATION SPACE_____ \\n\")\n", + "print(\"Observation Space\", env.observation_space)\n", + "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2MXc15qFE0M9" + }, + "source": [ + "We see with `Observation Space Shape Discrete(16)` that the observation is an integer representing the **agentโ€™s current position as current_row * ncols + current_col (where both the row and col start at 0)**.\n", + "\n", + "For example, the goal position in the 4x4 map can be calculated as follows: 3 * 4 + 3 = 15. The number of possible observations is dependent on the size of the map. **For example, the 4x4 map has 16 possible observations.**\n", + "\n", + "\n", + "For instance, this is what state = 0 looks like:\n", + "\n", + "\"FrozenLake\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "We5WqOBGLoSm" + }, + "outputs": [], + "source": [ + "print(\"\\n _____ACTION SPACE_____ \\n\")\n", + "print(\"Action Space Shape\", env.action_space.n)\n", + "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MyxXwkI2Magx" + }, + "source": [ + "The action space (the set of possible actions the agent can take) is discrete with 4 actions available ๐ŸŽฎ:\n", + "- 0: GO LEFT\n", + "- 1: GO DOWN\n", + "- 2: GO RIGHT\n", + "- 3: GO UP\n", + "\n", + "Reward function ๐Ÿ’ฐ:\n", + "- Reach goal: +1\n", + "- Reach hole: 0\n", + "- Reach frozen: 0" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1pFhWblk3Awr" + }, + "source": [ + "## Create and Initialize the Q-table ๐Ÿ—„๏ธ\n", + "\n", + "(๐Ÿ‘€ Step 1 of the pseudocode)\n", + "\n", + "\"Q-Learning\"\n", + "\n", + "\n", + "It's time to initialize our Q-table! To know how many rows (states) and columns (actions) to use, we need to know the action and observation space. We already know their values from before, but we'll want to obtain them programmatically so that our algorithm generalizes for different environments. Gym provides us a way to do that: `env.action_space.n` and `env.observation_space.n`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "y3ZCdluj3k0l" + }, + "outputs": [], + "source": [ + "state_space =\n", + "print(\"There are \", state_space, \" possible states\")\n", + "\n", + "action_space =\n", + "print(\"There are \", action_space, \" possible actions\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "rCddoOXM3UQH" + }, + "outputs": [], + "source": [ + "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros. np.zeros needs a tuple (a,b)\n", + "def initialize_q_table(state_space, action_space):\n", + " Qtable =\n", + " return Qtable" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9YfvrqRt3jdR" + }, + "outputs": [], + "source": [ + "Qtable_frozenlake = initialize_q_table(state_space, action_space)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "67OdoKL63eDD" + }, + "source": [ + "### Solution" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HuTKv3th3ohG" + }, + "outputs": [], + "source": [ + "state_space = env.observation_space.n\n", + "print(\"There are \", state_space, \" possible states\")\n", + "\n", + "action_space = env.action_space.n\n", + "print(\"There are \", action_space, \" possible actions\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lnrb_nX33fJo" + }, + "outputs": [], + "source": [ + "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros\n", + "def initialize_q_table(state_space, action_space):\n", + " Qtable = np.zeros((state_space, action_space))\n", + " return Qtable" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Y0WlgkVO3Jf9" + }, + "outputs": [], + "source": [ + "Qtable_frozenlake = initialize_q_table(state_space, action_space)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Atll4Z774gri" + }, + "source": [ + "## Define the greedy policy ๐Ÿค–\n", + "\n", + "Remember we have two policies since Q-Learning is an **off-policy** algorithm. This means we're using a **different policy for acting and updating the value function**.\n", + "\n", + "- Epsilon-greedy policy (acting policy)\n", + "- Greedy-policy (updating policy)\n", + "\n", + "The greedy policy will also be the final policy we'll have when the Q-learning agent completes training. The greedy policy is used to select an action using the Q-table.\n", + "\n", + "\"Q-Learning\"\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "E3SCLmLX5bWG" + }, + "outputs": [], + "source": [ + "def greedy_policy(Qtable, state):\n", + " # Exploitation: take the action with the highest state, action value\n", + " action =\n", + "\n", + " return action" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "B2_-8b8z5k54" + }, + "source": [ + "#### Solution" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "se2OzWGW5kYJ" + }, + "outputs": [], + "source": [ + "def greedy_policy(Qtable, state):\n", + " # Exploitation: take the action with the highest state, action value\n", + " action = np.argmax(Qtable[state][:])\n", + "\n", + " return action" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "flILKhBU3yZ7" + }, + "source": [ + "##Define the epsilon-greedy policy ๐Ÿค–\n", + "\n", + "Epsilon-greedy is the training policy that handles the exploration/exploitation trade-off.\n", + "\n", + "The idea with epsilon-greedy:\n", + "\n", + "- With *probability 1โ€Š-โ€Šษ›* : **we do exploitation** (i.e. our agent selects the action with the highest state-action pair value).\n", + "\n", + "- With *probability ษ›*: we do **exploration** (trying a random action).\n", + "\n", + "As the training continues, we progressively **reduce the epsilon value since we will need less and less exploration and more exploitation.**\n", + "\n", + "\"Q-Learning\"\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6Bj7x3in3_Pq" + }, + "outputs": [], + "source": [ + "def epsilon_greedy_policy(Qtable, state, epsilon):\n", + " # Randomly generate a number between 0 and 1\n", + " random_num =\n", + " # if random_num > greater than epsilon --> exploitation\n", + " if random_num > epsilon:\n", + " # Take the action with the highest value given a state\n", + " # np.argmax can be useful here\n", + " action =\n", + " # else --> exploration\n", + " else:\n", + " action = # Take a random action\n", + "\n", + " return action" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8R5ej1fS4P2V" + }, + "source": [ + "#### Solution" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cYxHuckr4LiG" + }, + "outputs": [], + "source": [ + "def epsilon_greedy_policy(Qtable, state, epsilon):\n", + " # Randomly generate a number between 0 and 1\n", + " random_num = random.uniform(0,1)\n", + " # if random_num > greater than epsilon --> exploitation\n", + " if random_num > epsilon:\n", + " # Take the action with the highest value given a state\n", + " # np.argmax can be useful here\n", + " action = greedy_policy(Qtable, state)\n", + " # else --> exploration\n", + " else:\n", + " action = env.action_space.sample()\n", + "\n", + " return action" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hW80DealcRtu" + }, + "source": [ + "## Define the hyperparameters โš™๏ธ\n", + "\n", + "The exploration related hyperparamters are some of the most important ones.\n", + "\n", + "- We need to make sure that our agent **explores enough of the state space** to learn a good value approximation. To do that, we need to have progressive decay of the epsilon.\n", + "- If you decrease epsilon too fast (too high decay_rate), **you take the risk that your agent will be stuck**, since your agent didn't explore enough of the state space and hence can't solve the problem." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Y1tWn0tycWZ1" + }, + "outputs": [], + "source": [ + "# Training parameters\n", + "n_training_episodes = 10000 # Total training episodes\n", + "learning_rate = 0.7 # Learning rate\n", + "\n", + "# Evaluation parameters\n", + "n_eval_episodes = 100 # Total number of test episodes\n", + "\n", + "# Environment parameters\n", + "env_id = \"FrozenLake-v1\" # Name of the environment\n", + "max_steps = 99 # Max steps per episode\n", + "gamma = 0.95 # Discounting rate\n", + "eval_seed = [] # The evaluation seed of the environment\n", + "\n", + "# Exploration parameters\n", + "max_epsilon = 1.0 # Exploration probability at start\n", + "min_epsilon = 0.05 # Minimum exploration probability\n", + "decay_rate = 0.0005 # Exponential decay rate for exploration prob" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cDb7Tdx8atfL" + }, + "source": [ + "## Create the training loop method\n", + "\n", + "\"Q-Learning\"\n", + "\n", + "The training loop goes like this:\n", + "\n", + "```\n", + "For episode in the total of training episodes:\n", + "\n", + "Reduce epsilon (since we need less and less exploration)\n", + "Reset the environment\n", + "\n", + " For step in max timesteps: \n", + " Choose the action At using epsilon greedy policy\n", + " Take the action (a) and observe the outcome state(s') and reward (r)\n", + " Update the Q-value Q(s,a) using Bellman equation Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n", + " If done, finish the episode\n", + " Our next state is the new state\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "paOynXy3aoJW" + }, + "outputs": [], + "source": [ + "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n", + " for episode in tqdm(range(n_training_episodes)):\n", + " # Reduce epsilon (because we need less and less exploration)\n", + " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n", + " # Reset the environment\n", + " state, info = env.reset()\n", + " step = 0\n", + " terminated = False\n", + " truncated = False\n", + "\n", + " # repeat\n", + " for step in range(max_steps):\n", + " # Choose the action At using epsilon greedy policy\n", + " action =\n", + "\n", + " # Take action At and observe Rt+1 and St+1\n", + " # Take the action (a) and observe the outcome state(s') and reward (r)\n", + " new_state, reward, terminated, truncated, info =\n", + "\n", + " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n", + " Qtable[state][action] =\n", + "\n", + " # If terminated or truncated finish the episode\n", + " if terminated or truncated:\n", + " break\n", + "\n", + " # Our next state is the new state\n", + " state = new_state\n", + " return Qtable" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Pnpk2ePoem3r" + }, + "source": [ + "#### Solution" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "IyZaYbUAeolw" + }, + "outputs": [], + "source": [ + "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n", + " for episode in tqdm(range(n_training_episodes)):\n", + " # Reduce epsilon (because we need less and less exploration)\n", + " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) # steps-independent\n", + "# epsilon = max_epsilon * ((min_epsilon/max_epsilon)**(1/(n_training_episodes-1))) ** episode # steps-dependent\n", + " # Reset the environment\n", + " state, _ = env.reset()\n", + "\n", + " # repeat\n", + " for _ in range(max_steps):\n", + " # Choose the action At using epsilon greedy policy\n", + " action = epsilon_greedy_policy(Qtable, state, epsilon)\n", + "\n", + " # Take action At and observe Rt+1 and St+1\n", + " # Take the action (a) and observe the outcome state(s') and reward (r)\n", + " new_state, reward, terminated, truncated, _ = env.step(action)\n", + "\n", + " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n", + " Qtable[state][action] = Qtable[state][action] + learning_rate * (reward + gamma * np.max(Qtable[new_state]) - Qtable[state][action])\n", + "\n", + " # If terminated or truncated finish the episode\n", + " if terminated or truncated:\n", + " break\n", + "\n", + " # Our next state is the new state\n", + " state = new_state\n", + " return Qtable" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WLwKQ4tUdhGI" + }, + "source": [ + "## Train the Q-Learning agent ๐Ÿƒ" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DPBxfjJdTCOH" + }, + "outputs": [], + "source": [ + "Qtable_frozenlake = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_frozenlake)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yVeEhUCrc30L" + }, + "source": [ + "## Let's see what our Q-Learning table looks like now ๐Ÿ‘€" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "nmfchsTITw4q" + }, + "outputs": [], + "source": [ + "Qtable_frozenlake" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pUrWkxsHccXD" + }, + "source": [ + "## The evaluation method ๐Ÿ“\n", + "\n", + "- We defined the evaluation method that we're going to use to test our Q-Learning agent." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "jNl0_JO2cbkm" + }, + "outputs": [], + "source": [ + "def evaluate_agent(env, max_steps, n_eval_episodes, Q, seed):\n", + " \"\"\"\n", + " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n", + " :param env: The evaluation environment\n", + " :param max_steps: Maximum number of steps per episode\n", + " :param n_eval_episodes: Number of episode to evaluate the agent\n", + " :param Q: The Q-table\n", + " :param seed: The evaluation seed array (for taxi-v3)\n", + " \"\"\"\n", + " episode_rewards = []\n", + " for episode in tqdm(range(n_eval_episodes)):\n", + " if seed:\n", + " state, info = env.reset(seed=seed[episode])\n", + " else:\n", + " state, info = env.reset()\n", + " step = 0\n", + " truncated = False\n", + " terminated = False\n", + " total_rewards_ep = 0\n", + "\n", + " for step in range(max_steps):\n", + " # Take the action (index) that have the maximum expected future reward given that state\n", + " action = greedy_policy(Q, state)\n", + " new_state, reward, terminated, truncated, info = env.step(action)\n", + " total_rewards_ep += reward\n", + "\n", + " if terminated or truncated:\n", + " break\n", + " state = new_state\n", + " episode_rewards.append(total_rewards_ep)\n", + " mean_reward = np.mean(episode_rewards)\n", + " std_reward = np.std(episode_rewards)\n", + "\n", + " return mean_reward, std_reward" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0jJqjaoAnxUo" + }, + "source": [ + "## Evaluate our Q-Learning agent ๐Ÿ“ˆ\n", + "\n", + "- Usually, you should have a mean reward of 1.0\n", + "- The **environment is relatively easy** since the state space is really small (16). What you can try to do is [to replace it with the slippery version](https://gymnasium.farama.org/environments/toy_text/frozen_lake/), which introduces stochasticity, making the environment more complex." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "fAgB7s0HEFMm" + }, + "outputs": [], + "source": [ + "# Evaluate our Agent\n", + "mean_reward, std_reward = evaluate_agent(env, max_steps, n_eval_episodes, Qtable_frozenlake, eval_seed)\n", + "print(f\"Mean_reward={mean_reward:.2f} +/- {std_reward:.2f}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yxaP3bPdg1DV" + }, + "source": [ + "## Publish our trained model to the Hub ๐Ÿ”ฅ\n", + "\n", + "Now that we saw good results after the training, **we can publish our trained model to the Hub ๐Ÿค— with one line of code**.\n", + "\n", + "Here's an example of a Model Card:\n", + "\n", + "\"Model\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kv0k1JQjpMq3" + }, + "source": [ + "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QZ5LrR-joIHD" + }, + "source": [ + "#### Do not modify this code" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Jex3i9lZ8ksX" + }, + "outputs": [], + "source": [ + "from huggingface_hub import HfApi, snapshot_download\n", + "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n", + "\n", + "from pathlib import Path\n", + "import datetime\n", + "import json" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Qo57HBn3W74O" + }, + "outputs": [], + "source": [ + "def record_video(env, Qtable, out_directory, fps=1):\n", + " \"\"\"\n", + " Generate a replay video of the agent\n", + " :param env\n", + " :param Qtable: Qtable of our agent\n", + " :param out_directory\n", + " :param fps: how many frame per seconds (with taxi-v3 and frozenlake-v1 we use 1)\n", + " \"\"\"\n", + " images = []\n", + " terminated = False\n", + " truncated = False\n", + " state, info = env.reset(seed=random.randint(0,500))\n", + " img = env.render()\n", + " images.append(img)\n", + " while not terminated or truncated:\n", + " # Take the action (index) that have the maximum expected future reward given that state\n", + " action = np.argmax(Qtable[state][:])\n", + " state, reward, terminated, truncated, info = env.step(action) # We directly put next_state = state for recording logic\n", + " img = env.render()\n", + " images.append(img)\n", + " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "U4mdUTKkGnUd" + }, + "outputs": [], + "source": [ + "def push_to_hub(\n", + " repo_id, model, env, video_fps=1, local_repo_path=\"hub\"\n", + "):\n", + " \"\"\"\n", + " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n", + " This method does the complete pipeline:\n", + " - It evaluates the model\n", + " - It generates the model card\n", + " - It generates a replay video of the agent\n", + " - It pushes everything to the Hub\n", + "\n", + " :param repo_id: repo_id: id of the model repository from the Hugging Face Hub\n", + " :param env\n", + " :param video_fps: how many frame per seconds to record our video replay\n", + " (with taxi-v3 and frozenlake-v1 we use 1)\n", + " :param local_repo_path: where the local repository is\n", + " \"\"\"\n", + " _, repo_name = repo_id.split(\"/\")\n", + "\n", + " eval_env = env\n", + " api = HfApi()\n", + "\n", + " # Step 1: Create the repo\n", + " repo_url = api.create_repo(\n", + " repo_id=repo_id,\n", + " exist_ok=True,\n", + " )\n", + "\n", + " # Step 2: Download files\n", + " repo_local_path = Path(snapshot_download(repo_id=repo_id))\n", + "\n", + " # Step 3: Save the model\n", + " if env.spec.kwargs.get(\"map_name\"):\n", + " model[\"map_name\"] = env.spec.kwargs.get(\"map_name\")\n", + " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n", + " model[\"slippery\"] = False\n", + "\n", + " # Pickle the model\n", + " with open((repo_local_path) / \"q-learning.pkl\", \"wb\") as f:\n", + " pickle.dump(model, f)\n", + "\n", + " # Step 4: Evaluate the model and build JSON with evaluation metrics\n", + " mean_reward, std_reward = evaluate_agent(\n", + " eval_env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"]\n", + " )\n", + "\n", + " evaluate_data = {\n", + " \"env_id\": model[\"env_id\"],\n", + " \"mean_reward\": mean_reward,\n", + " \"n_eval_episodes\": model[\"n_eval_episodes\"],\n", + " \"eval_datetime\": datetime.datetime.now().isoformat()\n", + " }\n", + "\n", + " # Write a JSON file called \"results.json\" that will contain the\n", + " # evaluation results\n", + " with open(repo_local_path / \"results.json\", \"w\") as outfile:\n", + " json.dump(evaluate_data, outfile)\n", + "\n", + " # Step 5: Create the model card\n", + " env_name = model[\"env_id\"]\n", + " if env.spec.kwargs.get(\"map_name\"):\n", + " env_name += \"-\" + env.spec.kwargs.get(\"map_name\")\n", + "\n", + " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n", + " env_name += \"-\" + \"no_slippery\"\n", + "\n", + " metadata = {}\n", + " metadata[\"tags\"] = [env_name, \"q-learning\", \"reinforcement-learning\", \"custom-implementation\"]\n", + "\n", + " # Add metrics\n", + " eval = metadata_eval_result(\n", + " model_pretty_name=repo_name,\n", + " task_pretty_name=\"reinforcement-learning\",\n", + " task_id=\"reinforcement-learning\",\n", + " metrics_pretty_name=\"mean_reward\",\n", + " metrics_id=\"mean_reward\",\n", + " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n", + " dataset_pretty_name=env_name,\n", + " dataset_id=env_name,\n", + " )\n", + "\n", + " # Merges both dictionaries\n", + " metadata = {**metadata, **eval}\n", + "\n", + " model_card = f\"\"\"\n", + " # **Q-Learning** Agent playing1 **{env_id}**\n", + " This is a trained model of a **Q-Learning** agent playing **{env_id}** .\n", + "\n", + " ## Usage\n", + "\n", + " ```python\n", + "\n", + " model = load_from_hub(repo_id=\"{repo_id}\", filename=\"q-learning.pkl\")\n", + "\n", + " # Don't forget to check if you need to add additional attributes (is_slippery=False etc)\n", + " env = gym.make(model[\"env_id\"])\n", + " ```\n", + " \"\"\"\n", + "\n", + " evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])\n", + "\n", + " readme_path = repo_local_path / \"README.md\"\n", + " readme = \"\"\n", + " print(readme_path.exists())\n", + " if readme_path.exists():\n", + " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n", + " readme = f.read()\n", + " else:\n", + " readme = model_card\n", + "\n", + " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n", + " f.write(readme)\n", + "\n", + " # Save our metrics to Readme metadata\n", + " metadata_save(readme_path, metadata)\n", + "\n", + " # Step 6: Record a video\n", + " video_path = repo_local_path / \"replay.mp4\"\n", + " record_video(env, model[\"qtable\"], video_path, video_fps)\n", + "\n", + " # Step 7. Push everything to the Hub\n", + " api.upload_folder(\n", + " repo_id=repo_id,\n", + " folder_path=repo_local_path,\n", + " path_in_repo=\".\",\n", + " )\n", + "\n", + " print(\"Your model is pushed to the Hub. You can view your model here: \", repo_url)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "81J6cet_ogSS" + }, + "source": [ + "### .\n", + "\n", + "By using `push_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the Hub**.\n", + "\n", + "This way:\n", + "- You can **showcase our work** ๐Ÿ”ฅ\n", + "- You can **visualize your agent playing** ๐Ÿ‘€\n", + "- You can **share an agent with the community that others can use** ๐Ÿ’พ\n", + "- You can **access a leaderboard ๐Ÿ† to see how well your agent is performing compared to your classmates** ๐Ÿ‘‰ https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cWnFC0iZooTw" + }, + "source": [ + "To be able to share your model with the community there are three more steps to follow:\n", + "\n", + "1๏ธโƒฃ (If it's not already done) create an account to HF โžก https://huggingface.co/join\n", + "\n", + "2๏ธโƒฃ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", + "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", + "\n", + "\n", + "\"Create\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "QB5nIcxR8paT" + }, + "outputs": [], + "source": [ + "from huggingface_hub import notebook_login\n", + "notebook_login()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GyWc1x3-o3xG" + }, + "source": [ + "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login` (or `login`)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Gc5AfUeFo3xH" + }, + "source": [ + "3๏ธโƒฃ We're now ready to push our trained agent to the ๐Ÿค— Hub ๐Ÿ”ฅ using `push_to_hub()` function\n", + "\n", + "- Let's create **the model dictionary that contains the hyperparameters and the Q_table**." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "FiMqxqVHg0I4" + }, + "outputs": [], + "source": [ + "model = {\n", + " \"env_id\": env_id,\n", + " \"max_steps\": max_steps,\n", + " \"n_training_episodes\": n_training_episodes,\n", + " \"n_eval_episodes\": n_eval_episodes,\n", + " \"eval_seed\": eval_seed,\n", + "\n", + " \"learning_rate\": learning_rate,\n", + " \"gamma\": gamma,\n", + "\n", + " \"max_epsilon\": max_epsilon,\n", + " \"min_epsilon\": min_epsilon,\n", + " \"decay_rate\": decay_rate,\n", + "\n", + " \"qtable\": Qtable_frozenlake\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9kld-AEso3xH" + }, + "source": [ + "Let's fill the `push_to_hub` function:\n", + "\n", + "- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `\n", + "(repo_id = {username}/{repo_name})`\n", + "๐Ÿ’ก A good `repo_id` is `{username}/q-{env_id}`\n", + "- `model`: our model dictionary containing the hyperparameters and the Qtable.\n", + "- `env`: the environment.\n", + "- `commit_message`: message of the commit" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5sBo2umnXpPd" + }, + "outputs": [], + "source": [ + "model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "RpOTtSt83kPZ" + }, + "outputs": [], + "source": [ + "username = \"\" # FILL THIS\n", + "repo_name = \"q-FrozenLake-v1-4x4-noSlippery\"\n", + "push_to_hub(\n", + " repo_id=f\"{username}/{repo_name}\",\n", + " model=model,\n", + " env=env)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "E2875IGsprzq" + }, + "source": [ + "Congrats ๐Ÿฅณ you've just implemented from scratch, trained, and uploaded your first Reinforcement Learning agent.\n", + "FrozenLake-v1 no_slippery is very simple environment, let's try a harder one ๐Ÿ”ฅ." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "18lN8Bz7yvLt" + }, + "source": [ + "# Part 2: Taxi-v3 ๐Ÿš–\n", + "\n", + "## Create and understand [Taxi-v3 ๐Ÿš•](https://gymnasium.farama.org/environments/toy_text/taxi/)\n", + "---\n", + "\n", + "๐Ÿ’ก A good habit when you start to use an environment is to check its documentation\n", + "\n", + "๐Ÿ‘‰ https://gymnasium.farama.org/environments/toy_text/taxi/\n", + "\n", + "---\n", + "\n", + "In `Taxi-v3` ๐Ÿš•, there are four designated locations in the grid world indicated by R(ed), G(reen), Y(ellow), and B(lue).\n", + "\n", + "When the episode starts, **the taxi starts off at a random square** and the passenger is at a random location. The taxi drives to the passengerโ€™s location, **picks up the passenger**, drives to the passengerโ€™s destination (another one of the four specified locations), and then **drops off the passenger**. Once the passenger is dropped off, the episode ends.\n", + "\n", + "\n", + "\"Taxi\"\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "gL0wpeO8gpej" + }, + "outputs": [], + "source": [ + "env = gym.make(\"Taxi-v3\", render_mode=\"rgb_array\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gBOaXgtsrmtT" + }, + "source": [ + "There are **500 discrete states since there are 25 taxi positions, 5 possible locations of the passenger** (including the case when the passenger is in the taxi), and **4 destination locations.**\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_TPNaGSZrgqA" + }, + "outputs": [], + "source": [ + "state_space = env.observation_space.n\n", + "print(\"There are \", state_space, \" possible states\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "CdeeZuokrhit" + }, + "outputs": [], + "source": [ + "action_space = env.action_space.n\n", + "print(\"There are \", action_space, \" possible actions\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "R1r50Advrh5Q" + }, + "source": [ + "The action space (the set of possible actions the agent can take) is discrete with **6 actions available ๐ŸŽฎ**:\n", + "\n", + "- 0: move south\n", + "- 1: move north\n", + "- 2: move east\n", + "- 3: move west\n", + "- 4: pickup passenger\n", + "- 5: drop off passenger\n", + "\n", + "Reward function ๐Ÿ’ฐ:\n", + "\n", + "- -1 per step unless other reward is triggered.\n", + "- +20 delivering passenger.\n", + "- -10 executing โ€œpickupโ€ and โ€œdrop-offโ€ actions illegally." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "US3yDXnEtY9I" + }, + "outputs": [], + "source": [ + "# Create our Q table with state_size rows and action_size columns (500x6)\n", + "Qtable_taxi = initialize_q_table(state_space, action_space)\n", + "print(Qtable_taxi)\n", + "print(\"Q-table shape: \", Qtable_taxi .shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gUMKPH0_LJyH" + }, + "source": [ + "## Define the hyperparameters โš™๏ธ\n", + "\n", + "โš  DO NOT MODIFY EVAL_SEED: the eval_seed array **allows us to evaluate your agent with the same taxi starting positions for every classmate**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "AB6n__hhg7YS" + }, + "outputs": [], + "source": [ + "# Training parameters\n", + "n_training_episodes = 25000 # Total training episodes\n", + "learning_rate = 0.7 # Learning rate\n", + "\n", + "# Evaluation parameters\n", + "n_eval_episodes = 100 # Total number of test episodes\n", + "\n", + "# DO NOT MODIFY EVAL_SEED\n", + "eval_seed = [16,54,165,177,191,191,120,80,149,178,48,38,6,125,174,73,50,172,100,148,146,6,25,40,68,148,49,167,9,97,164,176,61,7,54,55,\n", + " 161,131,184,51,170,12,120,113,95,126,51,98,36,135,54,82,45,95,89,59,95,124,9,113,58,85,51,134,121,169,105,21,30,11,50,65,12,43,82,145,152,97,106,55,31,85,38,\n", + " 112,102,168,123,97,21,83,158,26,80,63,5,81,32,11,28,148] # Evaluation seed, this ensures that all classmates agents are trained on the same taxi starting position\n", + " # Each seed has a specific starting state\n", + "\n", + "# Environment parameters\n", + "env_id = \"Taxi-v3\" # Name of the environment\n", + "max_steps = 99 # Max steps per episode\n", + "gamma = 0.95 # Discounting rate\n", + "\n", + "# Exploration parameters\n", + "max_epsilon = 1.0 # Exploration probability at start\n", + "min_epsilon = 0.05 # Minimum exploration probability\n", + "decay_rate = 0.005 # Exponential decay rate for exploration prob\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1TMORo1VLTsX" + }, + "source": [ + "## Train our Q-Learning agent ๐Ÿƒ" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "WwP3Y2z2eS-K" + }, + "outputs": [], + "source": [ + "Qtable_taxi = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_taxi)\n", + "Qtable_taxi" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wPdu0SueLVl2" + }, + "source": [ + "## Create a model dictionary ๐Ÿ’พ and publish our trained model to the Hub ๐Ÿ”ฅ\n", + "\n", + "- We create a model dictionary that will contain all the training hyperparameters for reproducibility and the Q-Table.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0a1FpE_3hNYr" + }, + "outputs": [], + "source": [ + "model = {\n", + " \"env_id\": env_id,\n", + " \"max_steps\": max_steps,\n", + " \"n_training_episodes\": n_training_episodes,\n", + " \"n_eval_episodes\": n_eval_episodes,\n", + " \"eval_seed\": eval_seed,\n", + "\n", + " \"learning_rate\": learning_rate,\n", + " \"gamma\": gamma,\n", + "\n", + " \"max_epsilon\": max_epsilon,\n", + " \"min_epsilon\": min_epsilon,\n", + " \"decay_rate\": decay_rate,\n", + "\n", + " \"qtable\": Qtable_taxi\n", + "}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "dhQtiQozhOn1" + }, + "outputs": [], + "source": [ + "username = \"\" # FILL THIS\n", + "repo_name = \"\" # FILL THIS\n", + "push_to_hub(\n", + " repo_id=f\"{username}/{repo_name}\",\n", + " model=model,\n", + " env=env)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZgSdjgbIpRti" + }, + "source": [ + "Now that it's on the Hub, you can compare the results of your Taxi-v3 with your classmates using the leaderboard ๐Ÿ† ๐Ÿ‘‰ https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n", + "\n", + "\n", + "\"Taxi" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bzgIO70c0bu2" + }, + "source": [ + "# Part 3: Load from Hub ๐Ÿ”ฝ\n", + "\n", + "What's amazing with Hugging Face Hub ๐Ÿค— is that you can easily load powerful models from the community.\n", + "\n", + "Loading a saved model from the Hub is really easy:\n", + "\n", + "1. You go https://huggingface.co/models?other=q-learning to see the list of all the q-learning saved models.\n", + "2. You select one and copy its repo_id\n", + "\n", + "\"Copy" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gTth6thRoC6X" + }, + "source": [ + "3. Then we just need to use `load_from_hub` with:\n", + "- The repo_id\n", + "- The filename: the saved model inside the repo." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EtrfoTaBoNrd" + }, + "source": [ + "#### Do not modify this code" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Eo8qEzNtCaVI" + }, + "outputs": [], + "source": [ + "from urllib.error import HTTPError\n", + "\n", + "from huggingface_hub import hf_hub_download\n", + "\n", + "\n", + "def load_from_hub(repo_id: str, filename: str) -> str:\n", + " \"\"\"\n", + " Download a model from Hugging Face Hub.\n", + " :param repo_id: id of the model repository from the Hugging Face Hub\n", + " :param filename: name of the model zip file from the repository\n", + " \"\"\"\n", + " # Get the model from the Hub, download and cache the model on your local disk\n", + " pickle_model = hf_hub_download(\n", + " repo_id=repo_id,\n", + " filename=filename\n", + " )\n", + "\n", + " with open(pickle_model, 'rb') as f:\n", + " downloaded_model_file = pickle.load(f)\n", + "\n", + " return downloaded_model_file" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b_sM2gNioPZH" + }, + "source": [ + "### ." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "JUm9lz2gCQcU" + }, + "outputs": [], + "source": [ + "model = load_from_hub(repo_id=\"ThomasSimonini/q-Taxi-v3\", filename=\"q-learning.pkl\") # Try to use another model\n", + "\n", + "print(model)\n", + "env = gym.make(model[\"env_id\"])\n", + "\n", + "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "O7pL8rg1MulN" + }, + "outputs": [], + "source": [ + "model = load_from_hub(repo_id=\"ThomasSimonini/q-FrozenLake-v1-no-slippery\", filename=\"q-learning.pkl\") # Try to use another model\n", + "\n", + "env = gym.make(model[\"env_id\"], is_slippery=False)\n", + "\n", + "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BQAwLnYFPk-s" + }, + "source": [ + "## Some additional challenges ๐Ÿ†\n", + "\n", + "The best way to learn **is to try things on your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results!\n", + "\n", + "In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n", + "\n", + "Here are some ideas to climb up the leaderboard:\n", + "\n", + "* Train more steps\n", + "* Try different hyperparameters by looking at what your classmates have done.\n", + "* **Push your new trained model** on the Hub ๐Ÿ”ฅ\n", + "\n", + "Are walking on ice and driving taxis too boring to you? Try to **change the environment**, why not use FrozenLake-v1 slippery version? Check how they work [using the gymnasium documentation](https://gymnasium.farama.org/) and have fun ๐ŸŽ‰." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "p-fW-EU5WejJ" + }, + "source": [ + "_____________________________________________________________________\n", + "Congrats ๐Ÿฅณ, you've just implemented, trained, and uploaded your first Reinforcement Learning agent.\n", + "\n", + "Understanding Q-Learning is an **important step to understanding value-based methods.**\n", + "\n", + "In the next Unit with Deep Q-Learning, we'll see that while creating and updating a Q-table was a good strategy โ€” **however, it is not scalable.**\n", + "\n", + "For instance, imagine you create an agent that learns to play Doom.\n", + "\n", + "\"Doom\"/\n", + "\n", + "Doom is a large environment with a huge state space (millions of different states). Creating and updating a Q-table for that environment would not be efficient.\n", + "\n", + "That's why we'll study Deep Q-Learning in the next unit, an algorithm **where we use a neural network that approximates, given a state, the different Q-values for each action.**\n", + "\n", + "\"Environments\"/\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BjLhT70TEZIn" + }, + "source": [ + "See you in Unit 3! ๐Ÿ”ฅ\n", + "\n", + "## Keep learning, stay awesome ๐Ÿค—" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [ + "67OdoKL63eDD", + "B2_-8b8z5k54", + "8R5ej1fS4P2V", + "Pnpk2ePoem3r" + ], + "include_colab_link": true, + "private_outputs": true, + "provenance": [] + }, + "gpuClass": "standard", + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +}