MDP_with_PI_VI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reinforcement Learning Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is the area of Machine learning that deal with how an agent behave in an enviorment. This notebook covers two fundamental algorithms to solve MDPs namely **Value Iteration** and **Policy Iteration**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Markov Decision Process - MDP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In MDP, there is an agent. The agent choose an action $a_{t}$ at time $t$ and as a consequance the enviorment changes.\n",
    "Here The evniorment is world around the agent. After the action the enviorment state changes to $s_{t+1}$.\n",
    "A reward might be emitted assciated with what just happened and then this process repeats. ![](nb_images/mdp.png)\n",
    "\n",
    "So, there is a feedback cycle in that the next action you take, the next decision you make is in a situation that's the consiquence of what you did before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import gym\n",
    "import gym.spaces as spaces\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "← ↓ → ↑\n"
     ]
    }
   ],
   "source": [
    "# action mapping for display the final result\n",
    "action_mapping = {\n",
    "    3: '\\u2191', # UP\n",
    "    2: '\\u2192', # RIGHT\n",
    "    1: '\\u2193', # DOWN\n",
    "    0: '\\u2190' # LEFT\n",
    "}\n",
    "print(' '.join([action_mapping[i] for i in range(4)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Setup GYM Env for playing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we define a faction that will take a GYM enviorment and plays number of games according to given policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_episodes(enviorment, n_episodes, policy, random = False):\n",
    "    \"\"\"\n",
    "    This fucntion plays the given number of episodes given by following a policy or sample randomly from action_space.\n",
    "    \n",
    "    Parameters:\n",
    "        enviorment: openAI GYM object\n",
    "        n_episodes: number of episodes to run\n",
    "        policy: Policy to follow while playing an episode\n",
    "        random: Flag for taking random actions. if True no policy would be followed and action will be taken randomly\n",
    "        \n",
    "    Return:\n",
    "        wins: Total number of wins playing n_episodes\n",
    "        total_reward: Total reward of n_episodes\n",
    "        avg_reward: Average reward of n_episodes\n",
    "    \n",
    "    \"\"\"\n",
    "    # intialize wins and total reward\n",
    "    wins = 0\n",
    "    total_reward = 0\n",
    "    \n",
    "    # loop over number of episodes to play\n",
    "    for episode in range(n_episodes):\n",
    "        \n",
    "        # flag to check if the game is finished\n",
    "        terminated = False\n",
    "        \n",
    "        # reset the enviorment every time when playing a new episode\n",
    "        state = enviorment.reset()\n",
    "        \n",
    "        while not terminated:\n",
    "            \n",
    "            # check if the random flag is not true then follow the given policy other wise take random action\n",
    "            if random:\n",
    "                action = enviorment.action_space.sample()\n",
    "            else:\n",
    "                action = policy[state]\n",
    "\n",
    "            # take the next step\n",
    "            next_state, reward,  terminated, info = enviorment.step(action)\n",
    "            \n",
    "            enviorment.render()\n",
    "            \n",
    "            # accumalate total reward\n",
    "            total_reward += reward\n",
    "            \n",
    "            # change the state\n",
    "            state = next_state\n",
    "            \n",
    "            # if game is over with positive reward then add 1.0 in wins\n",
    "            if terminated and reward == 1.0:\n",
    "                wins += 1\n",
    "                \n",
    "    # calculate average reward\n",
    "    average_reward = total_reward / n_episodes\n",
    "    \n",
    "    return wins, total_reward, average_reward\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Solve for Value Iteration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](nb_images/value_iter.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def one_step_lookahead(env, state, V , discount_factor = 0.99):\n",
    "    \"\"\"\n",
    "    Helper function to  calculate state-value function\n",
    "    \n",
    "    Arguments:\n",
    "        env: openAI GYM Enviorment object\n",
    "        state: state to consider\n",
    "        V: Estimated Value for each state. Vector of length nS\n",
    "        discount_factor: MDP discount factor\n",
    "        \n",
    "    Return:\n",
    "        action_values: Expected value of each action in a state. Vector of length nA\n",
    "    \"\"\"\n",
    "    \n",
    "    # initialize vector of action values\n",
    "    action_values = np.zeros(env.nA)\n",
    "    \n",
    "    # loop over the actions we can take in an enviorment \n",
    "    for action in range(env.nA):\n",
    "        # loop over the P_sa distribution.\n",
    "        for probablity, next_state, reward, info in env.P[state][action]:\n",
    "             #if we are in state s and take action a. then sum over all the possible states we can land into.\n",
    "            action_values[action] += probablity * (reward + (discount_factor * V[next_state]))\n",
    "            \n",
    "    return action_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_policy(env, policy, V, discount_factor):\n",
    "    \n",
    "    \"\"\"\n",
    "    Helper function to update a given policy based on given value function.\n",
    "    \n",
    "    Arguments:\n",
    "        env: openAI GYM Enviorment object.\n",
    "        policy: policy to update.\n",
    "        V: Estimated Value for each state. Vector of length nS.\n",
    "        discount_factor: MDP discount factor.\n",
    "    Return:\n",
    "        policy: Updated policy based on the given state-Value function 'V'.\n",
    "    \"\"\"\n",
    "    \n",
    "    for state in range(env.nS):\n",
    "        # for a given state compute state-action value.\n",
    "        action_values = one_step_lookahead(env, state, V, discount_factor)\n",
    "        \n",
    "        # choose the action which maximizez the state-action value.\n",
    "        policy[state] =  np.argmax(action_values)\n",
    "        \n",
    "    return policy\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def value_iteration(env, discount_factor = 0.999, max_iteration = 1000):\n",
    "    \"\"\"\n",
    "    Algorithm to solve MPD.\n",
    "    \n",
    "    Arguments:\n",
    "        env: openAI GYM Enviorment object.\n",
    "        discount_factor: MDP discount factor.\n",
    "        max_iteration: Maximum No.  of iterations to run.\n",
    "        \n",
    "    Return:\n",
    "        V: Optimal state-Value function. Vector of lenth nS.\n",
    "        optimal_policy: Optimal policy. Vector of length nS.\n",
    "    \n",
    "    \"\"\"\n",
    "    # intialize value fucntion\n",
    "    V = np.zeros(env.nS)\n",
    "    \n",
    "    # iterate over max_iterations\n",
    "    for i in range(max_iteration):\n",
    "        \n",
    "        #  keep track of change with previous value function\n",
    "        prev_v = np.copy(V) \n",
    "    \n",
    "        # loop over all states\n",
    "        for state in range(env.nS):\n",
    "            \n",
    "            # Asynchronously update the state-action value\n",
    "            #action_values = one_step_lookahead(env, state, V, discount_factor)\n",
    "            \n",
    "            # Synchronously update the state-action value\n",
    "            action_values = one_step_lookahead(env, state, prev_v, discount_factor)\n",
    "            \n",
    "            # select best action to perform based on highest state-action value\n",
    "            best_action_value = np.max(action_values)\n",
    "            \n",
    "            # update the current state-value fucntion\n",
    "            V[state] =  best_action_value\n",
    "            \n",
    "        # if policy not changed over 10 iterations it converged.\n",
    "        if i % 10 == 0:\n",
    "            # if values of 'V' not changing after one iteration\n",
    "            if (np.all(np.isclose(V, prev_v))):\n",
    "                print('Value converged at iteration %d' %(i+1))\n",
    "                break\n",
    "\n",
    "    # intialize optimal policy\n",
    "    optimal_policy = np.zeros(env.nS, dtype = 'int8')\n",
    "    \n",
    "    # update the optimal polciy according to optimal value function 'V'\n",
    "    optimal_policy = update_policy(env, optimal_policy, V, discount_factor)\n",
    "    \n",
    "    return V, optimal_policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the Algorithim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Value converged at iteration 341\n",
      "Time to converge:  67.9 ms\n",
      "Optimal Value function: \n",
      "[[0.78538826 0.77836049 0.77368481 0.7713498 ]\n",
      " [0.78775777 0.         0.50562724 0.        ]\n",
      " [0.79250312 0.79963699 0.74472318 0.        ]\n",
      " [0.         0.86409247 0.93114742 0.        ]]\n",
      "Final Policy: \n",
      "[0 3 3 3 0 0 0 0 3 1 0 0 0 2 1 0]\n",
      "← ↑ ↑ ↑ ← ← ← ← ↑ ↓ ← ← ← → ↓ ←\n"
     ]
    }
   ],
   "source": [
    "enviorment = gym.make('FrozenLake-v0')\n",
    "tic = time.time()\n",
    "opt_V, opt_Policy = value_iteration(enviorment.env, max_iteration = 1000)\n",
    "toc = time.time()\n",
    "elapsed_time = (toc - tic) * 1000\n",
    "print (f\"Time to converge: {elapsed_time: 0.3} ms\")\n",
    "print('Optimal Value function: ')\n",
    "print(opt_V.reshape((4, 4)))\n",
    "print('Final Policy: ')\n",
    "print(opt_Policy)\n",
    "print(' '.join([action_mapping[int(action)] for action in opt_Policy]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FH\u001b[41mF\u001b[0mH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FH\u001b[41mF\u001b[0mH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFF\u001b[41mF\u001b[0m\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "n_episode = 10\n",
    "wins, total_reward, avg_reward = play_episodes(enviorment, n_episode, opt_Policy, random = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total wins with value iteration: 8\n",
      "Average rewards with value iteration: 0.8\n"
     ]
    }
   ],
   "source": [
    "print(f'Total wins with value iteration: {wins}')\n",
    "print(f\"Average rewards with value iteration: {avg_reward}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Solve for Policy Iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](nb_images/policy_iter.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def policy_eval(env, policy, V, discount_factor):\n",
    "    \"\"\"\n",
    "    Helper function to evaluate a policy.\n",
    "    \n",
    "    Arguments:\n",
    "        env: openAI GYM Enviorment object.\n",
    "        policy: policy to evaluate.\n",
    "        V: Estimated Value for each state. Vector of length nS.\n",
    "        discount_factor: MDP discount factor.\n",
    "    Return:\n",
    "        policy_value: Estimated value of each state following a given policy and state-value 'V'. \n",
    "        \n",
    "    \"\"\"\n",
    "    policy_value = np.zeros(env.nS)\n",
    "    for state, action in enumerate(policy):\n",
    "        for probablity, next_state, reward, info in env.P[state][action]:\n",
    "            policy_value[state] += probablity * (reward + (discount_factor * V[next_state]))\n",
    "            \n",
    "    return policy_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def policy_iteration(env, discount_factor = 0.999, max_iteration = 1000):\n",
    "    \"\"\"\n",
    "    Algorithm to solve MPD.\n",
    "    \n",
    "    Arguments:\n",
    "        env: openAI GYM Enviorment object.\n",
    "        discount_factor: MDP discount factor.\n",
    "        max_iteration: Maximum No.  of iterations to run.\n",
    "        \n",
    "    Return:\n",
    "        V: Optimal state-Value function. Vector of lenth nS.\n",
    "        new_policy: Optimal policy. Vector of length nS.\n",
    "    \n",
    "    \"\"\"\n",
    "    # intialize the state-Value function\n",
    "    V = np.zeros(env.nS)\n",
    "    \n",
    "    # intialize a random policy\n",
    "    policy = np.random.randint(0, 4, env.nS)\n",
    "    policy_prev = np.copy(policy)\n",
    "    \n",
    "    for i in range(max_iteration):\n",
    "        \n",
    "        # evaluate given policy\n",
    "        V = policy_eval(env, policy, V, discount_factor)\n",
    "        \n",
    "        # improve policy\n",
    "        policy = update_policy(env, policy, V, discount_factor)\n",
    "        \n",
    "        # if policy not changed over 10 iterations it converged.\n",
    "        if i % 10 == 0:\n",
    "            if (np.all(np.equal(policy, policy_prev))):\n",
    "                print('policy converged at iteration %d' %(i+1))\n",
    "                break\n",
    "            policy_prev = np.copy(policy)\n",
    "            \n",
    "\n",
    "            \n",
    "    return V, policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Policy Iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "policy converged at iteration 31\n",
      "Time to converge:  7.97 ms\n",
      "Optimal Value function: \n",
      "[[0.35417188 0.29789248 0.27323702 0.24828299]\n",
      " [0.39178738 0.         0.27180699 0.        ]\n",
      " [0.46403122 0.56508154 0.55525204 0.        ]\n",
      " [0.         0.69646823 0.84379216 0.        ]]\n",
      "Final Policy: \n",
      "[0 3 0 3 0 0 0 0 3 1 0 0 0 2 1 0]\n",
      "← ↑ ← ↑ ← ← ← ← ↑ ↓ ← ← ← → ↓ ←\n"
     ]
    }
   ],
   "source": [
    "enviorment2 = gym.make('FrozenLake-v0')\n",
    "tic = time.time()\n",
    "opt_V2, opt_policy2 = policy_iteration(enviorment2.env, discount_factor = 0.999, max_iteration = 10000)\n",
    "toc = time.time()\n",
    "elapsed_time = (toc - tic) * 1000\n",
    "print (f\"Time to converge: {elapsed_time: 0.3} ms\")\n",
    "print('Optimal Value function: ')\n",
    "print(opt_V2.reshape((4, 4)))\n",
    "print('Final Policy: ')\n",
    "print(opt_policy2)\n",
    "print(' '.join([action_mapping[(action)] for action in opt_policy2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FH\u001b[41mF\u001b[0mH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FH\u001b[41mF\u001b[0mH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FH\u001b[41mF\u001b[0mH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FH\u001b[41mF\u001b[0mH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FH\u001b[41mF\u001b[0mH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FF\u001b[41mF\u001b[0mH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "n_episode = 10\n",
    "wins, total_reward, avg_reward = play_episodes(enviorment2, n_episode, opt_policy2, random = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total wins with Policy iteration: 8\n",
      "Average rewards with Policy iteration: 0.8\n"
     ]
    }
   ],
   "source": [
    "print(f'Total wins with Policy iteration: {wins}')\n",
    "print(f\"Average rewards with Policy iteration: {avg_reward}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remarks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Policy Iteration converge faster but takes more computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}