RL_Acrobot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reinforcement Learning for Acrobot game"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\python36\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import gym\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining hyperparameters:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define all the hyperparameters for the model, this will allow as to easily iterate and find the values that give best results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "num_episodes = 250      # Number of episodes for training\n",
    "learning_rate = 0.001\n",
    "gamma = 0.99           # Discount factor for reward\n",
    "num_Hidden = 100       # number of nodes in the hidden layer\n",
    "\n",
    "env = gym.make('Acrobot-v1')    # Choose a game and create an environment\n",
    "env = env.unwrapped              # The wrapper limits the number of steps in an episode, let's get rid of it\n",
    "# env._max_episode_steps = 10000\n",
    "obs_dim = env.reset().shape      # obervation dimension\n",
    "num_actions = env.action_space.n # number of actions (this works only for descrete action space, which is the case here)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discount reward function**: We we'll train our agent based on the reward gained for his actions. For each action we'll define the episode reward as the total reward gained in all the next steps of the current episode. Since a reward gained further away in the future has less correlation to the present action, we will give it less weight by discounting future rewards.\n",
    "\n",
    "The formula for the discounted rewards is given by:\n",
    "\n",
    "$$ R_t = \\sum_k \\gamma^k r_{t+k} $$\n",
    "\n",
    "Where $r_t$ is the reward gained in the step $t$ and $\\gamma \\in [0,1]$ is a hyperparameter called the discount factor.\n",
    "Here we define a function that takes a vector of rewards in consequent steps and returns the discounted reward vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def discount_rewards(r):\n",
    "    '''Takes a 1D rewards (one episode) and discounts it and also standardize\n",
    "    the rewards to be unit normal (helps control the gradient estimator variance)'''\n",
    "    \n",
    "    # Discounting\n",
    "    dis_r = np.zeros_like(r)\n",
    "    running_sum = 0\n",
    "    for t in reversed(range(len(r))):\n",
    "        running_sum = gamma*running_sum + r[t]\n",
    "        dis_r[t] = running_sum\n",
    "    \n",
    "    # Normailzing\n",
    "    dis_r = (dis_r - np.mean(dis_r))/np.std(dis_r)\n",
    "        \n",
    "    return dis_r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep order in our model, we use name scopes which basically group the layers of our model in a simple to follow way. Eventually, when using [TensorBoard](https://www.tensorflow.org/guide/summaries_and_tensorboard) for visualizing the model, the graph is more readable and makes it easy to understand the model and find errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''Create placeholders for inputs'''\n",
    "# A place holder for input observations\n",
    "input_ = tf.placeholder(tf.float32, shape = (None, obs_dim[0]), name = \"input\")\n",
    "# A place holder for actions in a full episode\n",
    "actions = tf.placeholder(tf.float32, shape = (None, num_actions), name = \"actions\")\n",
    "# A place holder for discounted rewards in a full episode\n",
    "dis_rewards = tf.placeholder(tf.float32, shape = (None, ), name = \"dis_rewards\")\n",
    "\n",
    "'''Fully connected layers'''\n",
    "with tf.name_scope(\"FC1\"):\n",
    "    fc1 = tf.layers.dense(inputs = input_, units = num_Hidden ,activation = tf.nn.relu, name = \"fc1\" )\n",
    "\n",
    "with tf.name_scope(\"FC2\"):\n",
    "    fc2 = tf.layers.dense(inputs = fc1, units = num_actions ,activation = None, name = \"fc2\" )\n",
    "    \n",
    "with tf.name_scope(\"Action_PDF\"):\n",
    "    # Operate with softmax on fc2 outputs to get an action probability distribution\n",
    "    action_prob_dist = tf.nn.softmax(logits = fc2, name = \"softamx\")\n",
    "\n",
    "with tf.name_scope(\"Loss\"):\n",
    "    '''Define loss'''\n",
    "    # Fist define reular softmax cross entropy loss\n",
    "    CE_loss = tf.nn.softmax_cross_entropy_with_logits_v2(labels = actions, logits = fc2, name = \"CE_loss\")\n",
    "    # Modulate the loss based on our discounted reward - this is where reinforcment learning happens,\n",
    "    # we favor actions that produced high reward\n",
    "    loss = tf.reduce_mean(CE_loss * dis_rewards)\n",
    "\n",
    "with tf.name_scope(\"Training\"):\n",
    "    '''Define optimizer'''\n",
    "    training_opt = tf.train.AdamOptimizer(learning_rate).minimize(loss)\n",
    "\n",
    "'''Define saver for saving and restoring model'''\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a writer for saving summaries to TensorBoard:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# writer = tf.summary.FileWriter(\"./tensorboard/1/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example of a graph visualization:\n",
    "\n",
    "<img src=\"./img/model_graph.png\" width=\"500\">\n",
    "\n",
    "Each block in the graph is expandable and let you see the content inside, for example see an [image](./img/model_graph_loss.png) with expanded loss block."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a model ready, we can start training it.\n",
    "\n",
    "Our goal is to achieve a model that uses the current observation to create the best probability distribution function (PDF) for the next action to be taken. This is **not a deterministic model** - the agent has a fine probability to take any action after receiving an observation, with larger probability to take favorable actions.\n",
    "\n",
    "To achieve that, in each step, we use our model to generate a PDF of actions, draw an action out of it and take the next step.\n",
    "When we reach the end of the episode, we compute the episode loss by feeding the observation, action and discounted rewards vectors (which we kept track of). Our optimizer minimizes the loss which makes favorable (unfavorable) actions more probable (less probable) due to the multiplicative factor of the discounted reward. By repeating these steps to each episode, our agent gradually improves its PDF and becomes a better player!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------\n",
      "Episode 0\n",
      "Episode ended after 42243 steps\n",
      "Accumulated reward in this episode -42242.0\n",
      "Mean reward so far -42242.00\n",
      "Maximal reward so far -42242.0\n",
      "-------------------------------------------------\n",
      "Episode 1\n",
      "Episode ended after 4189 steps\n",
      "Accumulated reward in this episode -4188.0\n",
      "Mean reward so far -23215.00\n",
      "Maximal reward so far -4188.0\n",
      "-------------------------------------------------\n",
      "Episode 2\n",
      "Episode ended after 1785 steps\n",
      "Accumulated reward in this episode -1784.0\n",
      "Mean reward so far -16071.33\n",
      "Maximal reward so far -1784.0\n",
      "-------------------------------------------------\n",
      "Episode 3\n",
      "Episode ended after 11085 steps\n",
      "Accumulated reward in this episode -11084.0\n",
      "Mean reward so far -14824.50\n",
      "Maximal reward so far -1784.0\n",
      "-------------------------------------------------\n",
      "Episode 4\n",
      "Episode ended after 4041 steps\n",
      "Accumulated reward in this episode -4040.0\n",
      "Mean reward so far -12667.60\n",
      "Maximal reward so far -1784.0\n",
      "-------------------------------------------------\n",
      "Episode 5\n",
      "Episode ended after 14926 steps\n",
      "Accumulated reward in this episode -14925.0\n",
      "Mean reward so far -13043.83\n",
      "Maximal reward so far -1784.0\n",
      "-------------------------------------------------\n",
      "Episode 6\n",
      "Episode ended after 1633 steps\n",
      "Accumulated reward in this episode -1632.0\n",
      "Mean reward so far -11413.57\n",
      "Maximal reward so far -1632.0\n",
      "-------------------------------------------------\n",
      "Episode 7\n",
      "Episode ended after 5520 steps\n",
      "Accumulated reward in this episode -5519.0\n",
      "Mean reward so far -10676.75\n",
      "Maximal reward so far -1632.0\n",
      "-------------------------------------------------\n",
      "Episode 8\n",
      "Episode ended after 2057 steps\n",
      "Accumulated reward in this episode -2056.0\n",
      "Mean reward so far -9718.89\n",
      "Maximal reward so far -1632.0\n",
      "-------------------------------------------------\n",
      "Episode 9\n",
      "Episode ended after 3214 steps\n",
      "Accumulated reward in this episode -3213.0\n",
      "Mean reward so far -9068.30\n",
      "Maximal reward so far -1632.0\n",
      "-------------------------------------------------\n",
      "Episode 10\n",
      "Episode ended after 842 steps\n",
      "Accumulated reward in this episode -841.0\n",
      "Mean reward so far -8320.36\n",
      "Maximal reward so far -841.0\n",
      "-------------------------------------------------\n",
      "Episode 11\n",
      "Episode ended after 3475 steps\n",
      "Accumulated reward in this episode -3474.0\n",
      "Mean reward so far -7916.50\n",
      "Maximal reward so far -841.0\n",
      "-------------------------------------------------\n",
      "Episode 12\n",
      "Episode ended after 933 steps\n",
      "Accumulated reward in this episode -932.0\n",
      "Mean reward so far -7379.23\n",
      "Maximal reward so far -841.0\n",
      "-------------------------------------------------\n",
      "Episode 13\n",
      "Episode ended after 406 steps\n",
      "Accumulated reward in this episode -405.0\n",
      "Mean reward so far -6881.07\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 14\n",
      "Episode ended after 1887 steps\n",
      "Accumulated reward in this episode -1886.0\n",
      "Mean reward so far -6548.07\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 15\n",
      "Episode ended after 513 steps\n",
      "Accumulated reward in this episode -512.0\n",
      "Mean reward so far -6170.81\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 16\n",
      "Episode ended after 521 steps\n",
      "Accumulated reward in this episode -520.0\n",
      "Mean reward so far -5838.41\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 17\n",
      "Episode ended after 1886 steps\n",
      "Accumulated reward in this episode -1885.0\n",
      "Mean reward so far -5618.78\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 18\n",
      "Episode ended after 1437 steps\n",
      "Accumulated reward in this episode -1436.0\n",
      "Mean reward so far -5398.63\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 19\n",
      "Episode ended after 760 steps\n",
      "Accumulated reward in this episode -759.0\n",
      "Mean reward so far -5166.65\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 20\n",
      "Episode ended after 2303 steps\n",
      "Accumulated reward in this episode -2302.0\n",
      "Mean reward so far -5030.24\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 21\n",
      "Episode ended after 790 steps\n",
      "Accumulated reward in this episode -789.0\n",
      "Mean reward so far -4837.45\n",
      "Maximal reward so far -405.0\n",
      "-------------------------------------------------\n",
      "Episode 22\n",
      "Episode ended after 335 steps\n",
      "Accumulated reward in this episode -334.0\n",
      "Mean reward so far -4641.65\n",
      "Maximal reward so far -334.0\n",
      "-------------------------------------------------\n",
      "Episode 23\n",
      "Episode ended after 1241 steps\n",
      "Accumulated reward in this episode -1240.0\n",
      "Mean reward so far -4499.92\n",
      "Maximal reward so far -334.0\n",
      "-------------------------------------------------\n",
      "Episode 24\n",
      "Episode ended after 504 steps\n",
      "Accumulated reward in this episode -503.0\n",
      "Mean reward so far -4340.04\n",
      "Maximal reward so far -334.0\n",
      "-------------------------------------------------\n",
      "Episode 25\n",
      "Episode ended after 659 steps\n",
      "Accumulated reward in this episode -658.0\n",
      "Mean reward so far -4198.42\n",
      "Maximal reward so far -334.0\n",
      "-------------------------------------------------\n",
      "Episode 26\n",
      "Episode ended after 494 steps\n",
      "Accumulated reward in this episode -493.0\n",
      "Mean reward so far -4061.19\n",
      "Maximal reward so far -334.0\n",
      "-------------------------------------------------\n",
      "Episode 27\n",
      "Episode ended after 470 steps\n",
      "Accumulated reward in this episode -469.0\n",
      "Mean reward so far -3932.89\n",
      "Maximal reward so far -334.0\n",
      "-------------------------------------------------\n",
      "Episode 28\n",
      "Episode ended after 906 steps\n",
      "Accumulated reward in this episode -905.0\n",
      "Mean reward so far -3828.48\n",
      "Maximal reward so far -334.0\n",
      "-------------------------------------------------\n",
      "Episode 29\n",
      "Episode ended after 1034 steps\n",
      "Accumulated reward in this episode -1033.0\n",
      "Mean reward so far -3735.30\n",
      "Maximal reward so far -334.0\n",
      "-------------------------------------------------\n",
      "Episode 30\n",
      "Episode ended after 247 steps\n",
      "Accumulated reward in this episode -246.0\n",
      "Mean reward so far -3622.74\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 31\n",
      "Episode ended after 514 steps\n",
      "Accumulated reward in this episode -513.0\n",
      "Mean reward so far -3525.56\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 32\n",
      "Episode ended after 414 steps\n",
      "Accumulated reward in this episode -413.0\n",
      "Mean reward so far -3431.24\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 33\n",
      "Episode ended after 361 steps\n",
      "Accumulated reward in this episode -360.0\n",
      "Mean reward so far -3340.91\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 34\n",
      "Episode ended after 276 steps\n",
      "Accumulated reward in this episode -275.0\n",
      "Mean reward so far -3253.31\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 35\n",
      "Episode ended after 338 steps\n",
      "Accumulated reward in this episode -337.0\n",
      "Mean reward so far -3172.31\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 36\n",
      "Episode ended after 272 steps\n",
      "Accumulated reward in this episode -271.0\n",
      "Mean reward so far -3093.89\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 37\n",
      "Episode ended after 341 steps\n",
      "Accumulated reward in this episode -340.0\n",
      "Mean reward so far -3021.42\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 38\n",
      "Episode ended after 423 steps\n",
      "Accumulated reward in this episode -422.0\n",
      "Mean reward so far -2954.77\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 39\n",
      "Episode ended after 302 steps\n",
      "Accumulated reward in this episode -301.0\n",
      "Mean reward so far -2888.43\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 40\n",
      "Episode ended after 322 steps\n",
      "Accumulated reward in this episode -321.0\n",
      "Mean reward so far -2825.80\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 41\n",
      "Episode ended after 344 steps\n",
      "Accumulated reward in this episode -343.0\n",
      "Mean reward so far -2766.69\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 42\n",
      "Episode ended after 302 steps\n",
      "Accumulated reward in this episode -301.0\n",
      "Mean reward so far -2709.35\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 43\n",
      "Episode ended after 281 steps\n",
      "Accumulated reward in this episode -280.0\n",
      "Mean reward so far -2654.14\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 44\n",
      "Episode ended after 306 steps\n",
      "Accumulated reward in this episode -305.0\n",
      "Mean reward so far -2601.93\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 45\n",
      "Episode ended after 284 steps\n",
      "Accumulated reward in this episode -283.0\n",
      "Mean reward so far -2551.52\n",
      "Maximal reward so far -246.0\n",
      "-------------------------------------------------\n",
      "Episode 46\n",
      "Episode ended after 211 steps\n",
      "Accumulated reward in this episode -210.0\n",
      "Mean reward so far -2501.70\n",
      "Maximal reward so far -210.0\n",
      "-------------------------------------------------\n",
      "Episode 47\n",
      "Episode ended after 248 steps\n",
      "Accumulated reward in this episode -247.0\n",
      "Mean reward so far -2454.73\n",
      "Maximal reward so far -210.0\n",
      "-------------------------------------------------\n",
      "Episode 48\n",
      "Episode ended after 240 steps\n",
      "Accumulated reward in this episode -239.0\n",
      "Mean reward so far -2409.51\n",
      "Maximal reward so far -210.0\n",
      "-------------------------------------------------\n",
      "Episode 49\n",
      "Episode ended after 203 steps\n",
      "Accumulated reward in this episode -202.0\n",
      "Mean reward so far -2365.36\n",
      "Maximal reward so far -202.0\n",
      "-------------------------------------------------\n",
      "Episode 50\n",
      "Episode ended after 221 steps\n",
      "Accumulated reward in this episode -220.0\n",
      "Mean reward so far -2323.29\n",
      "Maximal reward so far -202.0\n",
      "-------------------------------------------------\n",
      "Episode 51\n",
      "Episode ended after 383 steps\n",
      "Accumulated reward in this episode -382.0\n",
      "Mean reward so far -2285.96\n",
      "Maximal reward so far -202.0\n",
      "-------------------------------------------------\n",
      "Episode 52\n",
      "Episode ended after 248 steps\n",
      "Accumulated reward in this episode -247.0\n",
      "Mean reward so far -2247.49\n",
      "Maximal reward so far -202.0\n",
      "-------------------------------------------------\n",
      "Episode 53\n",
      "Episode ended after 184 steps\n",
      "Accumulated reward in this episode -183.0\n",
      "Mean reward so far -2209.26\n",
      "Maximal reward so far -183.0\n",
      "-------------------------------------------------\n",
      "Episode 54\n",
      "Episode ended after 179 steps\n",
      "Accumulated reward in this episode -178.0\n",
      "Mean reward so far -2172.33\n",
      "Maximal reward so far -178.0\n",
      "-------------------------------------------------\n",
      "Episode 55\n",
      "Episode ended after 191 steps\n",
      "Accumulated reward in this episode -190.0\n",
      "Mean reward so far -2136.93\n",
      "Maximal reward so far -178.0\n",
      "-------------------------------------------------\n",
      "Episode 56\n",
      "Episode ended after 176 steps\n",
      "Accumulated reward in this episode -175.0\n",
      "Mean reward so far -2102.51\n",
      "Maximal reward so far -175.0\n",
      "-------------------------------------------------\n",
      "Episode 57\n",
      "Episode ended after 201 steps\n",
      "Accumulated reward in this episode -200.0\n",
      "Mean reward so far -2069.71\n",
      "Maximal reward so far -175.0\n",
      "-------------------------------------------------\n",
      "Episode 58\n",
      "Episode ended after 234 steps\n",
      "Accumulated reward in this episode -233.0\n",
      "Mean reward so far -2038.58\n",
      "Maximal reward so far -175.0\n",
      "-------------------------------------------------\n",
      "Episode 59\n",
      "Episode ended after 221 steps\n",
      "Accumulated reward in this episode -220.0\n",
      "Mean reward so far -2008.27\n",
      "Maximal reward so far -175.0\n",
      "-------------------------------------------------\n",
      "Episode 60\n",
      "Episode ended after 353 steps\n",
      "Accumulated reward in this episode -352.0\n",
      "Mean reward so far -1981.11\n",
      "Maximal reward so far -175.0\n",
      "-------------------------------------------------\n",
      "Episode 61\n",
      "Episode ended after 347 steps\n",
      "Accumulated reward in this episode -346.0\n",
      "Mean reward so far -1954.74\n",
      "Maximal reward so far -175.0\n",
      "-------------------------------------------------\n",
      "Episode 62\n",
      "Episode ended after 233 steps\n",
      "Accumulated reward in this episode -232.0\n",
      "Mean reward so far -1927.40\n",
      "Maximal reward so far -175.0\n",
      "-------------------------------------------------\n",
      "Episode 63\n",
      "Episode ended after 170 steps\n",
      "Accumulated reward in this episode -169.0\n",
      "Mean reward so far -1899.92\n",
      "Maximal reward so far -169.0\n",
      "-------------------------------------------------\n",
      "Episode 64\n",
      "Episode ended after 157 steps\n",
      "Accumulated reward in this episode -156.0\n",
      "Mean reward so far -1873.09\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 65\n",
      "Episode ended after 244 steps\n",
      "Accumulated reward in this episode -243.0\n",
      "Mean reward so far -1848.39\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 66\n",
      "Episode ended after 202 steps\n",
      "Accumulated reward in this episode -201.0\n",
      "Mean reward so far -1823.81\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 67\n",
      "Episode ended after 233 steps\n",
      "Accumulated reward in this episode -232.0\n",
      "Mean reward so far -1800.40\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 68\n",
      "Episode ended after 294 steps\n",
      "Accumulated reward in this episode -293.0\n",
      "Mean reward so far -1778.55\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 69\n",
      "Episode ended after 258 steps\n",
      "Accumulated reward in this episode -257.0\n",
      "Mean reward so far -1756.81\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 70\n",
      "Episode ended after 224 steps\n",
      "Accumulated reward in this episode -223.0\n",
      "Mean reward so far -1735.21\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 71\n",
      "Episode ended after 231 steps\n",
      "Accumulated reward in this episode -230.0\n",
      "Mean reward so far -1714.31\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 72\n",
      "Episode ended after 195 steps\n",
      "Accumulated reward in this episode -194.0\n",
      "Mean reward so far -1693.48\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 73\n",
      "Episode ended after 299 steps\n",
      "Accumulated reward in this episode -298.0\n",
      "Mean reward so far -1674.62\n",
      "Maximal reward so far -156.0\n",
      "-------------------------------------------------\n",
      "Episode 74\n",
      "Episode ended after 138 steps\n",
      "Accumulated reward in this episode -137.0\n",
      "Mean reward so far -1654.12\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 75\n",
      "Episode ended after 210 steps\n",
      "Accumulated reward in this episode -209.0\n",
      "Mean reward so far -1635.11\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 76\n",
      "Episode ended after 190 steps\n",
      "Accumulated reward in this episode -189.0\n",
      "Mean reward so far -1616.32\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 77\n",
      "Episode ended after 195 steps\n",
      "Accumulated reward in this episode -194.0\n",
      "Mean reward so far -1598.09\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 78\n",
      "Episode ended after 189 steps\n",
      "Accumulated reward in this episode -188.0\n",
      "Mean reward so far -1580.24\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 79\n",
      "Episode ended after 211 steps\n",
      "Accumulated reward in this episode -210.0\n",
      "Mean reward so far -1563.11\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 80\n",
      "Episode ended after 233 steps\n",
      "Accumulated reward in this episode -232.0\n",
      "Mean reward so far -1546.68\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 81\n",
      "Episode ended after 256 steps\n",
      "Accumulated reward in this episode -255.0\n",
      "Mean reward so far -1530.93\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 82\n",
      "Episode ended after 164 steps\n",
      "Accumulated reward in this episode -163.0\n",
      "Mean reward so far -1514.45\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 83\n",
      "Episode ended after 215 steps\n",
      "Accumulated reward in this episode -214.0\n",
      "Mean reward so far -1498.96\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 84\n",
      "Episode ended after 222 steps\n",
      "Accumulated reward in this episode -221.0\n",
      "Mean reward so far -1483.93\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 85\n",
      "Episode ended after 410 steps\n",
      "Accumulated reward in this episode -409.0\n",
      "Mean reward so far -1471.43\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 86\n",
      "Episode ended after 161 steps\n",
      "Accumulated reward in this episode -160.0\n",
      "Mean reward so far -1456.36\n",
      "Maximal reward so far -137.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------\n",
      "Episode 87\n",
      "Episode ended after 147 steps\n",
      "Accumulated reward in this episode -146.0\n",
      "Mean reward so far -1441.47\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 88\n",
      "Episode ended after 158 steps\n",
      "Accumulated reward in this episode -157.0\n",
      "Mean reward so far -1427.03\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 89\n",
      "Episode ended after 156 steps\n",
      "Accumulated reward in this episode -155.0\n",
      "Mean reward so far -1412.90\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 90\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -1399.04\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 91\n",
      "Episode ended after 199 steps\n",
      "Accumulated reward in this episode -198.0\n",
      "Mean reward so far -1385.99\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 92\n",
      "Episode ended after 156 steps\n",
      "Accumulated reward in this episode -155.0\n",
      "Mean reward so far -1372.75\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 93\n",
      "Episode ended after 193 steps\n",
      "Accumulated reward in this episode -192.0\n",
      "Mean reward so far -1360.19\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 94\n",
      "Episode ended after 162 steps\n",
      "Accumulated reward in this episode -161.0\n",
      "Mean reward so far -1347.57\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 95\n",
      "Episode ended after 201 steps\n",
      "Accumulated reward in this episode -200.0\n",
      "Mean reward so far -1335.61\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 96\n",
      "Episode ended after 294 steps\n",
      "Accumulated reward in this episode -293.0\n",
      "Mean reward so far -1324.87\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 97\n",
      "Episode ended after 196 steps\n",
      "Accumulated reward in this episode -195.0\n",
      "Mean reward so far -1313.34\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 98\n",
      "Episode ended after 217 steps\n",
      "Accumulated reward in this episode -216.0\n",
      "Mean reward so far -1302.25\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 99\n",
      "Episode ended after 177 steps\n",
      "Accumulated reward in this episode -176.0\n",
      "Mean reward so far -1290.99\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 100\n",
      "Episode ended after 341 steps\n",
      "Accumulated reward in this episode -340.0\n",
      "Mean reward so far -1281.57\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 101\n",
      "Episode ended after 165 steps\n",
      "Accumulated reward in this episode -164.0\n",
      "Mean reward so far -1270.62\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 102\n",
      "Episode ended after 227 steps\n",
      "Accumulated reward in this episode -226.0\n",
      "Mean reward so far -1260.48\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 103\n",
      "Episode ended after 246 steps\n",
      "Accumulated reward in this episode -245.0\n",
      "Mean reward so far -1250.71\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 104\n",
      "Episode ended after 250 steps\n",
      "Accumulated reward in this episode -249.0\n",
      "Mean reward so far -1241.17\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 105\n",
      "Episode ended after 201 steps\n",
      "Accumulated reward in this episode -200.0\n",
      "Mean reward so far -1231.35\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 106\n",
      "Episode ended after 224 steps\n",
      "Accumulated reward in this episode -223.0\n",
      "Mean reward so far -1221.93\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 107\n",
      "Episode ended after 219 steps\n",
      "Accumulated reward in this episode -218.0\n",
      "Mean reward so far -1212.63\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 108\n",
      "Episode ended after 169 steps\n",
      "Accumulated reward in this episode -168.0\n",
      "Mean reward so far -1203.05\n",
      "Maximal reward so far -137.0\n",
      "-------------------------------------------------\n",
      "Episode 109\n",
      "Episode ended after 136 steps\n",
      "Accumulated reward in this episode -135.0\n",
      "Mean reward so far -1193.34\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 110\n",
      "Episode ended after 216 steps\n",
      "Accumulated reward in this episode -215.0\n",
      "Mean reward so far -1184.52\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 111\n",
      "Episode ended after 186 steps\n",
      "Accumulated reward in this episode -185.0\n",
      "Mean reward so far -1175.60\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 112\n",
      "Episode ended after 168 steps\n",
      "Accumulated reward in this episode -167.0\n",
      "Mean reward so far -1166.67\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 113\n",
      "Episode ended after 176 steps\n",
      "Accumulated reward in this episode -175.0\n",
      "Mean reward so far -1157.97\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 114\n",
      "Episode ended after 198 steps\n",
      "Accumulated reward in this episode -197.0\n",
      "Mean reward so far -1149.62\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 115\n",
      "Episode ended after 155 steps\n",
      "Accumulated reward in this episode -154.0\n",
      "Mean reward so far -1141.03\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 116\n",
      "Episode ended after 201 steps\n",
      "Accumulated reward in this episode -200.0\n",
      "Mean reward so far -1132.99\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 117\n",
      "Episode ended after 252 steps\n",
      "Accumulated reward in this episode -251.0\n",
      "Mean reward so far -1125.52\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 118\n",
      "Episode ended after 200 steps\n",
      "Accumulated reward in this episode -199.0\n",
      "Mean reward so far -1117.73\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 119\n",
      "Episode ended after 147 steps\n",
      "Accumulated reward in this episode -146.0\n",
      "Mean reward so far -1109.63\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 120\n",
      "Episode ended after 154 steps\n",
      "Accumulated reward in this episode -153.0\n",
      "Mean reward so far -1101.73\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 121\n",
      "Episode ended after 166 steps\n",
      "Accumulated reward in this episode -165.0\n",
      "Mean reward so far -1094.05\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 122\n",
      "Episode ended after 136 steps\n",
      "Accumulated reward in this episode -135.0\n",
      "Mean reward so far -1086.25\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 123\n",
      "Episode ended after 313 steps\n",
      "Accumulated reward in this episode -312.0\n",
      "Mean reward so far -1080.01\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 124\n",
      "Episode ended after 155 steps\n",
      "Accumulated reward in this episode -154.0\n",
      "Mean reward so far -1072.60\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 125\n",
      "Episode ended after 138 steps\n",
      "Accumulated reward in this episode -137.0\n",
      "Mean reward so far -1065.17\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 126\n",
      "Episode ended after 205 steps\n",
      "Accumulated reward in this episode -204.0\n",
      "Mean reward so far -1058.39\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 127\n",
      "Episode ended after 154 steps\n",
      "Accumulated reward in this episode -153.0\n",
      "Mean reward so far -1051.32\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 128\n",
      "Episode ended after 219 steps\n",
      "Accumulated reward in this episode -218.0\n",
      "Mean reward so far -1044.86\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 129\n",
      "Episode ended after 155 steps\n",
      "Accumulated reward in this episode -154.0\n",
      "Mean reward so far -1038.01\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 130\n",
      "Episode ended after 185 steps\n",
      "Accumulated reward in this episode -184.0\n",
      "Mean reward so far -1031.49\n",
      "Maximal reward so far -135.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------\n",
      "Episode 131\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -1024.83\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 132\n",
      "Episode ended after 189 steps\n",
      "Accumulated reward in this episode -188.0\n",
      "Mean reward so far -1018.53\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 133\n",
      "Episode ended after 138 steps\n",
      "Accumulated reward in this episode -137.0\n",
      "Mean reward so far -1011.96\n",
      "Maximal reward so far -135.0\n",
      "-------------------------------------------------\n",
      "Episode 134\n",
      "Episode ended after 135 steps\n",
      "Accumulated reward in this episode -134.0\n",
      "Mean reward so far -1005.45\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 135\n",
      "Episode ended after 174 steps\n",
      "Accumulated reward in this episode -173.0\n",
      "Mean reward so far -999.33\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 136\n",
      "Episode ended after 163 steps\n",
      "Accumulated reward in this episode -162.0\n",
      "Mean reward so far -993.22\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 137\n",
      "Episode ended after 292 steps\n",
      "Accumulated reward in this episode -291.0\n",
      "Mean reward so far -988.13\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 138\n",
      "Episode ended after 180 steps\n",
      "Accumulated reward in this episode -179.0\n",
      "Mean reward so far -982.31\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 139\n",
      "Episode ended after 149 steps\n",
      "Accumulated reward in this episode -148.0\n",
      "Mean reward so far -976.35\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 140\n",
      "Episode ended after 144 steps\n",
      "Accumulated reward in this episode -143.0\n",
      "Mean reward so far -970.44\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 141\n",
      "Episode ended after 226 steps\n",
      "Accumulated reward in this episode -225.0\n",
      "Mean reward so far -965.19\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 142\n",
      "Episode ended after 193 steps\n",
      "Accumulated reward in this episode -192.0\n",
      "Mean reward so far -959.78\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 143\n",
      "Episode ended after 147 steps\n",
      "Accumulated reward in this episode -146.0\n",
      "Mean reward so far -954.13\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 144\n",
      "Episode ended after 266 steps\n",
      "Accumulated reward in this episode -265.0\n",
      "Mean reward so far -949.38\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 145\n",
      "Episode ended after 155 steps\n",
      "Accumulated reward in this episode -154.0\n",
      "Mean reward so far -943.93\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 146\n",
      "Episode ended after 163 steps\n",
      "Accumulated reward in this episode -162.0\n",
      "Mean reward so far -938.61\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 147\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -933.30\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 148\n",
      "Episode ended after 186 steps\n",
      "Accumulated reward in this episode -185.0\n",
      "Mean reward so far -928.28\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 149\n",
      "Episode ended after 147 steps\n",
      "Accumulated reward in this episode -146.0\n",
      "Mean reward so far -923.06\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 150\n",
      "Episode ended after 182 steps\n",
      "Accumulated reward in this episode -181.0\n",
      "Mean reward so far -918.15\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 151\n",
      "Episode ended after 144 steps\n",
      "Accumulated reward in this episode -143.0\n",
      "Mean reward so far -913.05\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 152\n",
      "Episode ended after 161 steps\n",
      "Accumulated reward in this episode -160.0\n",
      "Mean reward so far -908.12\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 153\n",
      "Episode ended after 167 steps\n",
      "Accumulated reward in this episode -166.0\n",
      "Mean reward so far -903.31\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 154\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -898.46\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 155\n",
      "Episode ended after 142 steps\n",
      "Accumulated reward in this episode -141.0\n",
      "Mean reward so far -893.60\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 156\n",
      "Episode ended after 173 steps\n",
      "Accumulated reward in this episode -172.0\n",
      "Mean reward so far -889.01\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 157\n",
      "Episode ended after 172 steps\n",
      "Accumulated reward in this episode -171.0\n",
      "Mean reward so far -884.46\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 158\n",
      "Episode ended after 138 steps\n",
      "Accumulated reward in this episode -137.0\n",
      "Mean reward so far -879.76\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 159\n",
      "Episode ended after 175 steps\n",
      "Accumulated reward in this episode -174.0\n",
      "Mean reward so far -875.35\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 160\n",
      "Episode ended after 160 steps\n",
      "Accumulated reward in this episode -159.0\n",
      "Mean reward so far -870.90\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 161\n",
      "Episode ended after 149 steps\n",
      "Accumulated reward in this episode -148.0\n",
      "Mean reward so far -866.44\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 162\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -862.06\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 163\n",
      "Episode ended after 148 steps\n",
      "Accumulated reward in this episode -147.0\n",
      "Mean reward so far -857.70\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 164\n",
      "Episode ended after 147 steps\n",
      "Accumulated reward in this episode -146.0\n",
      "Mean reward so far -853.38\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 165\n",
      "Episode ended after 280 steps\n",
      "Accumulated reward in this episode -279.0\n",
      "Mean reward so far -849.92\n",
      "Maximal reward so far -134.0\n",
      "-------------------------------------------------\n",
      "Episode 166\n",
      "Episode ended after 122 steps\n",
      "Accumulated reward in this episode -121.0\n",
      "Mean reward so far -845.56\n",
      "Maximal reward so far -121.0\n",
      "-------------------------------------------------\n",
      "Episode 167\n",
      "Episode ended after 175 steps\n",
      "Accumulated reward in this episode -174.0\n",
      "Mean reward so far -841.56\n",
      "Maximal reward so far -121.0\n",
      "-------------------------------------------------\n",
      "Episode 168\n",
      "Episode ended after 181 steps\n",
      "Accumulated reward in this episode -180.0\n",
      "Mean reward so far -837.64\n",
      "Maximal reward so far -121.0\n",
      "-------------------------------------------------\n",
      "Episode 169\n",
      "Episode ended after 156 steps\n",
      "Accumulated reward in this episode -155.0\n",
      "Mean reward so far -833.63\n",
      "Maximal reward so far -121.0\n",
      "-------------------------------------------------\n",
      "Episode 170\n",
      "Episode ended after 176 steps\n",
      "Accumulated reward in this episode -175.0\n",
      "Mean reward so far -829.78\n",
      "Maximal reward so far -121.0\n",
      "-------------------------------------------------\n",
      "Episode 171\n",
      "Episode ended after 158 steps\n",
      "Accumulated reward in this episode -157.0\n",
      "Mean reward so far -825.87\n",
      "Maximal reward so far -121.0\n",
      "-------------------------------------------------\n",
      "Episode 172\n",
      "Episode ended after 120 steps\n",
      "Accumulated reward in this episode -119.0\n",
      "Mean reward so far -821.78\n",
      "Maximal reward so far -119.0\n",
      "-------------------------------------------------\n",
      "Episode 173\n",
      "Episode ended after 111 steps\n",
      "Accumulated reward in this episode -110.0\n",
      "Mean reward so far -817.69\n",
      "Maximal reward so far -110.0\n",
      "-------------------------------------------------\n",
      "Episode 174\n",
      "Episode ended after 185 steps\n",
      "Accumulated reward in this episode -184.0\n",
      "Mean reward so far -814.07\n",
      "Maximal reward so far -110.0\n",
      "-------------------------------------------------\n",
      "Episode 175\n",
      "Episode ended after 147 steps\n",
      "Accumulated reward in this episode -146.0\n",
      "Mean reward so far -810.27\n",
      "Maximal reward so far -110.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------\n",
      "Episode 176\n",
      "Episode ended after 115 steps\n",
      "Accumulated reward in this episode -114.0\n",
      "Mean reward so far -806.34\n",
      "Maximal reward so far -110.0\n",
      "-------------------------------------------------\n",
      "Episode 177\n",
      "Episode ended after 147 steps\n",
      "Accumulated reward in this episode -146.0\n",
      "Mean reward so far -802.63\n",
      "Maximal reward so far -110.0\n",
      "-------------------------------------------------\n",
      "Episode 178\n",
      "Episode ended after 150 steps\n",
      "Accumulated reward in this episode -149.0\n",
      "Mean reward so far -798.98\n",
      "Maximal reward so far -110.0\n",
      "-------------------------------------------------\n",
      "Episode 179\n",
      "Episode ended after 114 steps\n",
      "Accumulated reward in this episode -113.0\n",
      "Mean reward so far -795.17\n",
      "Maximal reward so far -110.0\n",
      "-------------------------------------------------\n",
      "Episode 180\n",
      "Episode ended after 96 steps\n",
      "Accumulated reward in this episode -95.0\n",
      "Mean reward so far -791.30\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 181\n",
      "Episode ended after 219 steps\n",
      "Accumulated reward in this episode -218.0\n",
      "Mean reward so far -788.15\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 182\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -784.67\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 183\n",
      "Episode ended after 171 steps\n",
      "Accumulated reward in this episode -170.0\n",
      "Mean reward so far -781.33\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 184\n",
      "Episode ended after 160 steps\n",
      "Accumulated reward in this episode -159.0\n",
      "Mean reward so far -777.97\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 185\n",
      "Episode ended after 185 steps\n",
      "Accumulated reward in this episode -184.0\n",
      "Mean reward so far -774.77\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 186\n",
      "Episode ended after 163 steps\n",
      "Accumulated reward in this episode -162.0\n",
      "Mean reward so far -771.50\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 187\n",
      "Episode ended after 210 steps\n",
      "Accumulated reward in this episode -209.0\n",
      "Mean reward so far -768.51\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 188\n",
      "Episode ended after 217 steps\n",
      "Accumulated reward in this episode -216.0\n",
      "Mean reward so far -765.58\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 189\n",
      "Episode ended after 134 steps\n",
      "Accumulated reward in this episode -133.0\n",
      "Mean reward so far -762.25\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 190\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -759.06\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 191\n",
      "Episode ended after 254 steps\n",
      "Accumulated reward in this episode -253.0\n",
      "Mean reward so far -756.42\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 192\n",
      "Episode ended after 150 steps\n",
      "Accumulated reward in this episode -149.0\n",
      "Mean reward so far -753.27\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 193\n",
      "Episode ended after 185 steps\n",
      "Accumulated reward in this episode -184.0\n",
      "Mean reward so far -750.34\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 194\n",
      "Episode ended after 137 steps\n",
      "Accumulated reward in this episode -136.0\n",
      "Mean reward so far -747.19\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 195\n",
      "Episode ended after 166 steps\n",
      "Accumulated reward in this episode -165.0\n",
      "Mean reward so far -744.22\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 196\n",
      "Episode ended after 151 steps\n",
      "Accumulated reward in this episode -150.0\n",
      "Mean reward so far -741.20\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 197\n",
      "Episode ended after 182 steps\n",
      "Accumulated reward in this episode -181.0\n",
      "Mean reward so far -738.37\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 198\n",
      "Episode ended after 156 steps\n",
      "Accumulated reward in this episode -155.0\n",
      "Mean reward so far -735.44\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 199\n",
      "Episode ended after 157 steps\n",
      "Accumulated reward in this episode -156.0\n",
      "Mean reward so far -732.54\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 200\n",
      "Episode ended after 184 steps\n",
      "Accumulated reward in this episode -183.0\n",
      "Mean reward so far -729.81\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 201\n",
      "Episode ended after 170 steps\n",
      "Accumulated reward in this episode -169.0\n",
      "Mean reward so far -727.03\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 202\n",
      "Episode ended after 208 steps\n",
      "Accumulated reward in this episode -207.0\n",
      "Mean reward so far -724.47\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 203\n",
      "Episode ended after 149 steps\n",
      "Accumulated reward in this episode -148.0\n",
      "Mean reward so far -721.65\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 204\n",
      "Episode ended after 113 steps\n",
      "Accumulated reward in this episode -112.0\n",
      "Mean reward so far -718.67\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 205\n",
      "Episode ended after 159 steps\n",
      "Accumulated reward in this episode -158.0\n",
      "Mean reward so far -715.95\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 206\n",
      "Episode ended after 136 steps\n",
      "Accumulated reward in this episode -135.0\n",
      "Mean reward so far -713.14\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 207\n",
      "Episode ended after 149 steps\n",
      "Accumulated reward in this episode -148.0\n",
      "Mean reward so far -710.43\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 208\n",
      "Episode ended after 165 steps\n",
      "Accumulated reward in this episode -164.0\n",
      "Mean reward so far -707.81\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 209\n",
      "Episode ended after 174 steps\n",
      "Accumulated reward in this episode -173.0\n",
      "Mean reward so far -705.27\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 210\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -702.64\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 211\n",
      "Episode ended after 96 steps\n",
      "Accumulated reward in this episode -95.0\n",
      "Mean reward so far -699.78\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 212\n",
      "Episode ended after 135 steps\n",
      "Accumulated reward in this episode -134.0\n",
      "Mean reward so far -697.12\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 213\n",
      "Episode ended after 97 steps\n",
      "Accumulated reward in this episode -96.0\n",
      "Mean reward so far -694.31\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 214\n",
      "Episode ended after 159 steps\n",
      "Accumulated reward in this episode -158.0\n",
      "Mean reward so far -691.82\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 215\n",
      "Episode ended after 259 steps\n",
      "Accumulated reward in this episode -258.0\n",
      "Mean reward so far -689.81\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 216\n",
      "Episode ended after 112 steps\n",
      "Accumulated reward in this episode -111.0\n",
      "Mean reward so far -687.14\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 217\n",
      "Episode ended after 140 steps\n",
      "Accumulated reward in this episode -139.0\n",
      "Mean reward so far -684.63\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 218\n",
      "Episode ended after 153 steps\n",
      "Accumulated reward in this episode -152.0\n",
      "Mean reward so far -682.20\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 219\n",
      "Episode ended after 105 steps\n",
      "Accumulated reward in this episode -104.0\n",
      "Mean reward so far -679.57\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 220\n",
      "Episode ended after 133 steps\n",
      "Accumulated reward in this episode -132.0\n",
      "Mean reward so far -677.09\n",
      "Maximal reward so far -95.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------\n",
      "Episode 221\n",
      "Episode ended after 152 steps\n",
      "Accumulated reward in this episode -151.0\n",
      "Mean reward so far -674.72\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 222\n",
      "Episode ended after 139 steps\n",
      "Accumulated reward in this episode -138.0\n",
      "Mean reward so far -672.31\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 223\n",
      "Episode ended after 185 steps\n",
      "Accumulated reward in this episode -184.0\n",
      "Mean reward so far -670.13\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 224\n",
      "Episode ended after 235 steps\n",
      "Accumulated reward in this episode -234.0\n",
      "Mean reward so far -668.20\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 225\n",
      "Episode ended after 120 steps\n",
      "Accumulated reward in this episode -119.0\n",
      "Mean reward so far -665.77\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 226\n",
      "Episode ended after 131 steps\n",
      "Accumulated reward in this episode -130.0\n",
      "Mean reward so far -663.41\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 227\n",
      "Episode ended after 136 steps\n",
      "Accumulated reward in this episode -135.0\n",
      "Mean reward so far -661.09\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 228\n",
      "Episode ended after 171 steps\n",
      "Accumulated reward in this episode -170.0\n",
      "Mean reward so far -658.94\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 229\n",
      "Episode ended after 149 steps\n",
      "Accumulated reward in this episode -148.0\n",
      "Mean reward so far -656.72\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 230\n",
      "Episode ended after 116 steps\n",
      "Accumulated reward in this episode -115.0\n",
      "Mean reward so far -654.38\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 231\n",
      "Episode ended after 145 steps\n",
      "Accumulated reward in this episode -144.0\n",
      "Mean reward so far -652.18\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 232\n",
      "Episode ended after 148 steps\n",
      "Accumulated reward in this episode -147.0\n",
      "Mean reward so far -650.01\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 233\n",
      "Episode ended after 137 steps\n",
      "Accumulated reward in this episode -136.0\n",
      "Mean reward so far -647.81\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 234\n",
      "Episode ended after 122 steps\n",
      "Accumulated reward in this episode -121.0\n",
      "Mean reward so far -645.57\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 235\n",
      "Episode ended after 157 steps\n",
      "Accumulated reward in this episode -156.0\n",
      "Mean reward so far -643.50\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 236\n",
      "Episode ended after 120 steps\n",
      "Accumulated reward in this episode -119.0\n",
      "Mean reward so far -641.28\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 237\n",
      "Episode ended after 123 steps\n",
      "Accumulated reward in this episode -122.0\n",
      "Mean reward so far -639.10\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 238\n",
      "Episode ended after 155 steps\n",
      "Accumulated reward in this episode -154.0\n",
      "Mean reward so far -637.07\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 239\n",
      "Episode ended after 112 steps\n",
      "Accumulated reward in this episode -111.0\n",
      "Mean reward so far -634.88\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 240\n",
      "Episode ended after 129 steps\n",
      "Accumulated reward in this episode -128.0\n",
      "Mean reward so far -632.78\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 241\n",
      "Episode ended after 131 steps\n",
      "Accumulated reward in this episode -130.0\n",
      "Mean reward so far -630.70\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 242\n",
      "Episode ended after 102 steps\n",
      "Accumulated reward in this episode -101.0\n",
      "Mean reward so far -628.52\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 243\n",
      "Episode ended after 134 steps\n",
      "Accumulated reward in this episode -133.0\n",
      "Mean reward so far -626.49\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 244\n",
      "Episode ended after 129 steps\n",
      "Accumulated reward in this episode -128.0\n",
      "Mean reward so far -624.45\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 245\n",
      "Episode ended after 118 steps\n",
      "Accumulated reward in this episode -117.0\n",
      "Mean reward so far -622.39\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 246\n",
      "Episode ended after 134 steps\n",
      "Accumulated reward in this episode -133.0\n",
      "Mean reward so far -620.41\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 247\n",
      "Episode ended after 140 steps\n",
      "Accumulated reward in this episode -139.0\n",
      "Mean reward so far -618.47\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 248\n",
      "Episode ended after 127 steps\n",
      "Accumulated reward in this episode -126.0\n",
      "Mean reward so far -616.49\n",
      "Maximal reward so far -95.0\n",
      "-------------------------------------------------\n",
      "Episode 249\n",
      "Episode ended after 130 steps\n",
      "Accumulated reward in this episode -129.0\n",
      "Mean reward so far -614.54\n",
      "Maximal reward so far -95.0\n"
     ]
    }
   ],
   "source": [
    "ep_action, ep_obs, ep_reward = [], [], []  # Allocate space for episode actions, observations and rewards\n",
    "tot_ep_reward = [] # Total episode reward\n",
    "mean_reward = []\n",
    "\n",
    "''' Run TF session '''\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    ''' Run episodes '''\n",
    "    for ep in range(num_episodes): \n",
    "        obs = env.reset()  # Reset and save first observation\n",
    "        ep_obs.append(obs) # append observation\n",
    "\n",
    "        ''' Run steps '''\n",
    "        while True:\n",
    "            # Propagate forward to compute action probability distribution\n",
    "            apd = np.squeeze(sess.run(action_prob_dist, feed_dict = {input_ : obs.reshape((1,obs_dim[0]))}))\n",
    "            action = np.random.choice(np.arange(num_actions), p = apd)   # Sample an action based on the pdf\n",
    "            obs, reward, done, info = env.step(action)  # Take action and save observation, reward and done boolean\n",
    "            \n",
    "            # Convert action to one hot\n",
    "            action_oh = np.zeros((1,num_actions))\n",
    "            action_oh[0,action] = 1\n",
    "            \n",
    "            ep_action.append(action_oh)  # append action\n",
    "            ep_obs.append(obs)           # append observation\n",
    "            ep_reward.append(reward)     # append reward\n",
    "\n",
    "            if done: \n",
    "                # Stack vertically episode parameters to one np.array\n",
    "                ep_action = np.vstack(ep_action)\n",
    "                ep_obs = np.vstack(ep_obs)\n",
    "                ep_reward = np.hstack(ep_reward)\n",
    "\n",
    "                # Discount rewards\n",
    "                dis_rewards_arr = discount_rewards(ep_reward)\n",
    "                # Compute loss and optimize\n",
    "                sess.run([loss, training_opt],\n",
    "                         feed_dict = {input_ : ep_obs[:-1], actions : ep_action, dis_rewards : dis_rewards_arr})\n",
    "                \n",
    "                tot_ep_reward.append(np.sum(ep_reward))  # Compute total reward for episode\n",
    "                mean_reward.append(np.mean(tot_ep_reward))\n",
    "                \n",
    "                 # print info\n",
    "                print(\"-------------------------------------------------\")\n",
    "                print(\"Episode {}\".format(ep))\n",
    "                print(\"Episode ended after {} steps\".format(ep_action.shape[0]))\n",
    "                print(\"Accumulated reward in this episode {}\".format(tot_ep_reward[ep]))\n",
    "                print(\"Mean reward so far {:0.2f}\".format(np.mean(tot_ep_reward)))\n",
    "                print(\"Maximal reward so far {}\".format(np.max(tot_ep_reward)))\n",
    "                \n",
    "                ep_action, ep_obs, ep_reward = [], [], []  # Clear episode values for next episode\n",
    "                      \n",
    "                break\n",
    "                \n",
    "    saver.save(sess, \"models/Acrobot/model.ckpt\") # save model for later\n",
    "#     writer.add_graph(sess.graph) # Save graph for displaying with TensorBoard\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assess model performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to get a sense of how good our model is, and to see if our agent improves in the training process, we can plot the mean rewards gained in one episode over the episode number. This value would grow as our agent becomes more skilled and gains more rewards in each episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhEAAAEcCAYAAAB56OZRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcHFW5//HPM0t2QhKykJCEAAnIonAhQqKIgywGXEDFa1AEFUW9cHFXEBdE8brr5YIoCgYUBUS98FMEwjICF8K+y5IQspFIErKQmUlm6X5+f5zTMzVNz0zPTE/3dM/3/Xr1q6tOnao6dbq66+lTp6rM3RERERHprapSF0BERETKk4IIERER6RMFESIiItInCiJERESkTxREiIiISJ8oiBAREZE+URBRgczsI2bm8bV3jul1ielHl6KMxZaok1lFWt+suL6PFGN95cbMzjczXV/eB8Wuu8TvRV2x1llMZrbIzNaUuhzlSkFEZdsGfDhH+qlx2lDyN2A+sK7UBRHpp18T9mWRklMQUdn+DJxiZpZJMLORwPuAP5WsVCXg7hvcfYm7N5e6LIViZsNLXYaumFm1mdWUuhzFUszPwt3XuPuSYq1P+m8wf1f7S0FEZfstsDtweCLtPUA1XQQRZvZWM7vdzLaZWaOZ3WJmB2TlOdbMbjKzdWbWZGZPmdkXzKw6K98KM/udmS00s2fi8h4ys8PJg5kdaGY3mtlmM9tuZv9nZm/JyrPIzNaY2ZvM7EEz2xHX+59Z+V5zOsPMPmhmj5pZg5ltNbMnzeyTWfOdYmaPx+VuNLPfmtnUrDyjzOznZvZKXNaNwPS+1m8X82W2c76Z3Wtm24EfJKZ/Iqucl5vZhMT0v5rZbYlxM7MNZtZsZqMS6Veb2QOJ8YVmdkfM2xDr67Qc5XMzu9DMzjGzF4EW4PVx2r+Z2d2xbC+Z2dcBy7GMz8T9ZHv8zB8ys/fkUTfdfkZxX304x3xTzazNzD6bSNsj1kGmbh7LLoPF0wlmdkD8/BqA63ooYz7fq3ozu8fMTojfqWYze9bM/j3X+ntTd/Hz/pyZPWdmLRa+uxeb2dis5Uwys9+b2atmtsXMrgLGdbFN7zWzJRZ+A7aY2R/NbGZ39ZC1nUeb2SPW8RtyYla+RWa2oov56xPjmdMtJ5rZL81sU6yDn1oIZt8Y19doZk+b2du7KFe3vyExz4DsH2XN3fWqsBfwEcCB2UA9cFli2s2E4KIu5jk6Me0dQBtwA3BCfN0LbAZmJPJ9CvgCcBxwJPAlwumR72WVYwWwEngQOAl4J/AosAUY18M2HAw0AvfEeY8HbgSagUMS+RYBrwKrgbOABTHNgY/kqJNZcfxwIA38DDgaOBY4G/hKYp4z4jzXxPV/HFgPPA+MSeT7LeGgeV5czg+BVTnKkFf9dlEfi2IdrwT+M35+h8Vp3wNagR/H9X8UeAm4H6iOeT4PNAHD4/iBcft3AMcm1rMW+H5i/KvAf8TlHg1cENf1qazyeVzn3YSWrgXAFGBi3L5ngA8AJwL/Fz8vT8z/oVg33yDsU8cD5wCn91AvPX5GwMKYZ7+seb8Q1zkljs+I8z4FnAK8Hbgi1tO7E/OdH5f3QqyftwF13ZQx3+9VPfCv+Bl/NM7317j+I7PX35u6A74by3xx3K7PAQ3x86pK5Lub8H06K7H9q+O8dYl8n4ppV8T1fSB+xi8CO/XwmdUTTis+Het5AbA4bsPsrH1+RRfz1yfG62JZVgA/AY4Bvh3T/ieW62Nxe+4m/K5M7MNvyIDsH+X+KnkB9BqAD7VzEPGx+GM1Apgav6jHkDuIWAbcnrWsscBG4GddrMuAGsIBdHPWD9KKmDY+kTY3rveDPWzD7fHLPyyRVh3T/jeRlvmyL8yafzHhx9iy6mRWHP8isKmb9VcDLwN3ZqUfHpdzdhzfB0gB52TluzTHj1Cv6zfHdp6QlT4rrv8bWelvjvlPjOP/FsffGsc/CzwR6+m/YtrrYp4FXZShKn7WvwIez5rmhABkZFb6hYQAa2YibXTcZk+kXQw80sv9PN/PaCSwNbOdiXyPATclxi8HNgC75NiXHkuMnx+X/5k8y5nX5044ODowL2sbnwXuzl5/vnUHTCAEi4uy0k+J63t3HD+G3N+lv5MIIoAxsT6vyLEvtgCf7aE+6gmB6JxE2uS4H381a59f0cX89Ynxuli+7PI8EtMPT6S9IaadluO71dNvyIDsH+X+0umMyvdHYDjwLsI/ln8RDtCdmNkcYC/gajOrybwI/17vA45I5J0amw1XEn40WoHvEJo9J2ct+j5335wYfzK+d9nsaaHfxltj2dOJshhwW7IsUYrXnp65Jq5jty5W8yAw3sLplneaWXaT7T5xW65OJrr7PYQflrfGpMMIB9fs5sprsrYp7/rtRhvhn2nSMXH92cu9n/DvKrPcx4FNhH9FxPc74iuZ1kr4t9ZebjP7g5m9FKe1Ev7t75OjfDe7+/astPnAEndflUlw90bg/2XlexA4yMz+JzZzj6JneX1GsUx/Aj5kFvoHmdnrCa0xVyVmXQDcBGzNqstbgAOzm/6Bv/RUwD587qs90d/B3VOE78GhZtbV73VPdTeP8Bvwu6z0awj7VGZfnk/X36Wk+YQgKHub1hACnnz25aXuvjQz4u7rCf/yezwd0o2/Z40/CzTG/SGZBqFVISmf35CC7x+VQEFEhXP3bcD/Eq7SOBW42t3TObJmDv6X03GwyLzeCewCEH/Iboxp3yEceN5I+McJocUjaVNWeZq7yJc0gfAP7Os5ynIW4eCf3Hc3u3tr1jJeju85gwh3/wfwfsKPyV+ADWZ2m5m9IVEGyH01x78S0zPn3l/OypM9nlf99mB9PKjkWu6yHMsdm1lu/Mz/ARxpoe/KEcCd8XVI/AE8EngwHuQxszGEf1kHEprH30L4rK8gHJSy5aqrqby2LsiRdhXwaUJQdguwycz+bN1fkpvvZ5RZ/gzCv1YI34dthFMMGZMJ35HsevxhnJ79GeVzpU9vP/eu6moYMKmLdfRUdznryd3bgFfovC93913K3qbbcmzT63NsUy6bcqQ10/3vQk82Z423EE6dtnP3ljiYvZ58fkMGYv8oe0Om9/QQdxXhEscq4OQu8rwS388l/Dhky3z59iKckviwu7f/szGzdxWmqED44qeBS+j8T7FdViA03sxqs34EpsT3l7paibtfD1wfD5Z1wPeBm81sOh0/crvmmHVX4KE4nPmhmAIsz7H+jHzrtzueIy2z3GN57Y9ocjqEgOFHhOb+nQhBxTbCOeK3Eurgl4n88wkdc9+S/DdnXV91kat863htXZCd5qEd+JfAL81sfNyeHwPXEg6OueT7GUHY1lWEq5X+QfgeXJ/VcvIKoRXm+12sb23WeK7tzdbbz72rumohNKW/Rh51l6ynpzPzxc9xl0QZ19H9dyl7mz6SXF5CoS4f30EInrIly1wo+fyGDMT+UfYURAwNiwnN7VvcPdeXHuA5Qh+G/d39e90sK9NU2v5lM7NawqmSgnD3RjO7m/AP+JEuWk6Sqgmd+ZLNrgsJB40ug4jE+hqAv5rZnsB/E36kniP8E1lI+BcJhB7chAPrj2PS/YSA598JHRyT60/Kt357a3Fc/0x3X9xD3jsJP8pfJ9TrFoBY158hdIK8I5E/12c9ntAxMF/3AV8ysxnuvjouYzTh9FpO8fTXtWZ2GPDJrvKR/2eEu7uZXQ2cSWh5ms5rA9SbCYHT0zlOy/RVbz/3GWY2L3NKI7YavR94II/vQVd1t4TwL38hnU9lfoBwDPhHHL+Prr9LSfcSAoXZ7n5lHtvUVyuBKWY20d03ApjZXoTTWPcWeF35/IYMxP5R9hREDAGxCbyrFohMHjezM4EbzGwYIejYSIjG3wSscvefEDo2rgQuNLMU4QDzuQEo9ueBu4BbzOxywr+kiYSrNqrd/ZxE3m3AD8xsIrCUsK1HEzo15vw3YGYXxG27k/APYjrh6ozH3H1DzPMNwr+73xHOJ+9GOG2zFPgNgLs/Z2a/By6Ip1geJPRTOD65vl7Ub6+4+wtm9n3gYjPbh3BA2EFouj8G+LW73xnzPmVm64Gj6GiChY4WimbCgSTjXkK/ikvM7JuEDpFfi+XeOc8i/pRwdcetZnZ+XMeXgE4/wmZ2GeFzvI9wbnxvwimHW7vZ9lQ+n1HCVYQWgV8QeuL/I2v6N4AHgLvM7GLCwX88cACwp7t/LM9tTpaxt5/7y4Qg4JuElodPE+ri012to6e6c/dNZvYT4FwzaySc19+XcDryHkIrJe6+2MzuIdRn5rv0gbj9yW161cy+RNgvJhH6Imwl1P1bCZ0ef9/busrhj4SrLK6O5Z9I+Pw2FmDZ2fL5DSn4/lERSt2zU6/Cv0hcndFNnjqyrs6I6fMJnfc2Ew5GKwjR+fxEnoMIPz5NhM5UFxA627Vf/RDzrQB+l2PdDpyfx3bsG9e9nnDwWUPoj3F8Is+imP4mwgF8ByHIObuLOpkVx99BOH+8Li57NeHf7LSs+U4hdEpsJjRn/haYmpVnFOFqjE2Ey+ZupOPqiI/0tn67qItFwJpupn+Y8I+zMZbhGUKv/elZ+a4l6woMOq7cqM+x3LcRLsvdTrhk7Wyyrg5IfKbf6aJsBxOagXcQ/tV9HfhWchnAaYRe95nP+kVCADI2j/2kx88okffBWNbvdjF9OuGOkC8RTiGsI7T2nJLIc35cRk0vvpP5fK/qCd+rdxMuI2wmtGR8IGtZneo/n7ojdEr+XFxeZrsuya5fQr+LPxAOqlsIgdcJZF3iGfMeTwhAX437xzJCf5n9eqiLeuCeHOkreO0VJCfGutgeP+Nj6frqjOzfskXk+M5k76vk+RsykPtHOb8yl66IlCUzW0T48ch5cyeRcmHhBko17p7XzdhEBgNdnSEiIiJ9oiBCRERE+kSnM0RERKRP1BIhIiIifaJLPHswceJEnzVrVsGW19jYyOjRowu2vKFK9dh/qsPCUD32n+qwMApVjw8//PBGd+/qDqmdKIjowaxZs3jooYd6zpin+vp66urqCra8oUr12H+qw8JQPfaf6rAwClWP8blIedHpDBEREekTBREiIiLSJ0MuiDCzBWb2nJktM7Nzep5DREREchlSQUR8mM0lwHHAfsDJZrZfaUslIiJSnoZUEAEcCixz9+Uenit/Db17IqGIiIhEQ+pmU2Z2EuHBQx+P4x8GDnP3s7LynQGcATBlypRDrrnmmtcsq68aGhoYM2ZMwZY3VKke+091WBiqx/5THRZGoerxyCOPfNjd5+aTd6hd4mk50l4TRbn7ZcBlAHPnzvVCXnqkS5kKQ/XYf6rDwlA99p/qsDBKUY9DLYhYA8xIjE8H1paoLCIi0o3wuGlIuZOOw2l3Umkn7WF62iGV9vbhzPRMXof2eTN5HCedDunE6cnleWa+dEf+9uUlluseltOxjsz0rDxZ6WSVLR0ydsyXeHc6p3eUsWOdk3YazofnzyrJZzTUgogHgTlmtgfhefALgQ+WtkgiUi48HqDa0vE95WxrcdZv20E6HQ926Y48mQNaKms4lfaYN8yTSqdJpemUL+1h+e3LjO9tnZbXcdBsS3fOl12G5ME3HcczB6e0dxxUO6V5xza/drzzQbzTQT4x3HnZudff2tqG3X5z+zKSQYP07IDdxiqIKAZ3bzOzs4BbgGrgCnd/usTFEqlY7uFA1ppK09rmNKdStKac1rY0rak0Lak0LW3pkBbHW9vieyqkt6XCQbYtHrTDQTRMS6Wd1nSaVExvS6dDWqrjINqWSnd6T7Uvp2OZ7ctJTOu8nHT7unO64/biVmwXqgyqq4wqM6qrEi8zquJ7dZVhiXxmhPyJ4aoq4rhR1Z4Wh6uqGF6TGE/mifNaXF4m3cyoruoYrspa/9o1a9h995kd67fk+jPrtc7ryl5/Ynqn+eI6Mus1OvIk12eJ+YyOdDLLgPZ1Qudlt6+DsP2Z/NapPCGduJxc60iu26pI5MlsA53zxGWX0pAKIgDc/SbgplKXQ2SgtKXSNLdlXimaWzuGd7R2pD3yrzY2PbImTGtNtc+zo304RWtb4uCeOOB3HOQTaW0daZnxllR6QLc1c5Csje811VXUVBk1VUZ1tVFbVdUpvbrKqK0O76Nra8K0KqOmqirmN6qr4jKq47TEvO3Lj9Oqq6p48YVl7LPP3vEATfcH8OqOA3lHPqiuqor5aJ8nM39V1nBNVVxe+7Khpqqq/WBVjurrN1BXt2+piyF9MOSCCJFScg8H4O0tKZriKwy30dQahre3pOJwW2J6zNva9pr5trek2BEDgR1t4R903h57/DVJNVXG8JoqhsVXbXUcrg7DtdVGbXUVY4bXdKTVhPThNZk8cb5qa19GJl92WmYZw2o6lt0xPR7gEwf/2uqqTgfnUqtvW0ndvN1LXQyRklAQIZKnVNp5dXsr23a0sa05vDfE4fDe1pG2o5WGOL5tR1scDmmtqd6d6B1eU8WoYdWMGlbDyGHVjBpWzcjaanYZM4wZw0YysraGkcOqGFFTzfDaKobXVDO8porhNVWMqM1Oy4xX8cSjj3D4/MNCnpoqhteGA3pN9VC7fYyI9JWCCBly3J3GlhRbmlrY0tTK5qYWNje1sjW+b47pW+J45v3VHa09dvSqqTJ2GlHDTiNqGTO8hp1G1DBt3IhO46OH1zCyNgYDMTjoGA4BQiZ9ZG011QP0b3vTsipmTdTjl0Wk7xRESMVIp51XGlt4+dUdrN+2g5dfbeblV3fEV3NMb2ZrU2u35+pHD6tm3KhhjB9dy/hRw5g+fiTjRw1j/Khadh41jLExSAjBQk0MDsL48Jqqsj0vLSLSWwoipCyk0876bc2s2tTEqk1N3PdCC3dsfapTgLBhW3PO3vMTxwxj8k4jmDJ2OAdM25nxo4cxblQt40fVMm7UMMaNrG1PGzdyGMNq1JwvIpIPBREyaLSl0qza1MQLGxpZtamJ1TFgWPlKI6s3b6elrXPrwc5r1jJl7HCmjB3B7MkT24czAcOUsSOYOGa4ggIRkQGiIEKKbntLihc2NPDChgaWre94rXilsVOnwzHDa5g5YRRzJu/EUftOYeaEUcycMIoZE0ax9PEHOPaoI0u4FSIioiBCBtTW7a08vXYrT7/0Kk+t3cpTL21l+cbG9g6KVQYzJ4xi9uQxvG3fycyeNIbZk8cwa5fRjBtV22X/gpXV6ncgIlJqCiKkYNydZesbWPLiJh54cROPr97Cqk1N7dOn7TyC/abtzDvfMI05UzqChRG11SUstYiI9JWCCOmzdNp59l/buP/FV7h/+SYeWLGJTY0tAOw6dgQH7z6OhYfO4IBpO7P/tLHsMmZ4iUssIiKFpCBCeqWhuY17lm7gjmfXc+dzG9iwrRmA3caNpG6fSczbYxcO23MCMyeM0qWOIiIVTkGEdMvdWb6xkTufXc+dz63ngRc30ZpydhpRwxF7T6Ju70nM32sXpo8fVeqiiohIkSmIkJzWbd3OHx5YzY2PvcSKV0K/hjmTx/CxN+/Bka+bzCG7j6dWt0cWERnSFERIu3TaufeFV/jtkhXc9sx60u4cPnsiHzt8D47cZzIzJqi1QUREOiiIELY2tfLHh1dz9f2reHFjIxNGD+MTb9mTDx02U4GDiIh0SUHEEPb8y9v41V3LufHxtTS3pTlk9/GcfdRsjjtgqi67FBGRHimIGII2NjTzk8XPc80DqxheU817D57OKfNmsv+0nUtdNBERKSMKIoaQDduaueaBVfzyruXsaE1x6vxZfOaoOYwfPazURRMRkTKkIGIIaGhu44c3P8vV96+iLe0cve9kzj1+X/aaNKbURRMRkTKmIKLC/eP5DXz1z0+ydut2PnjoTD765j2YPVnBg4iI9J+CiAq1pamFb//1Gf70yBr2mjSa6z81n0N2n1DqYomISAVREFGB7l/+Cmf94VE2NbZw5pF78Z9vm6OrLUREpOAURFSYu57fwBm/fYhp40bym4+8kQN20xUXIiIyMBREVJAly1/h41c9xJ4TR/O7jx/GRD01U0REBpCCiArxz7Wv8okrH2LmhFH8/hPzmKDLNkVEZIDpCUoV4Om1Wznl8vsZM6KGqz52qAIIEREpCgURZW7py9s4+bIljKip4g+fmMe0cSNLXSQRERkidDqjjLWl0nzhj49TU13FtZ+cr4dliYhIUSmIKGOX3b2cJ9Zs5ZIPHqwAQkREik6nM8rU0pe38bPFSzn+9bvyjjdMLXVxRERkCFIQUYbaUmm+eP0TjB5ezQUnHFDq4oiIyBCl0xll6Of1L/D46i1cdPK/6V4QIiJSMmqJKDM3P7WOnyx+nhMOmsa7dBpDRERKSEFEGVmxsZHPXvsYB88cx/ff9wbMrNRFEhGRIUxBRBn55V3LSTtcesoheqCWiIiUnIKIMrFhWzN/emQN7zt4OlPGjih1cURERBRElIsr711BayrNJ96yR6mLIiIiAiiIKAtbmlq48r4VHLvfFPacNKbUxREREQEURJSFX/xjOQ3NbXz26L1LXRQREZF2gyKIMLP3m9nTZpY2s7lZ0841s2Vm9pyZvT2RviCmLTOzcxLpe5jZ/Wa21MyuNbNhMX14HF8Wp88q1vb1x/pXd7Do3hd594HT2Hfq2FIXR0REpN2gCCKAp4D3AnclE81sP2AhsD+wAPi5mVWbWTVwCXAcsB9wcswL8H3gp+4+B9gMnB7TTwc2u/ts4Kcx36D3m3tX0JpyPn+MWiFERGRwGRRBhLs/4+7P5Zh0AnCNuze7+4vAMuDQ+Frm7svdvQW4BjjBwo0T3gZcH+e/Ejgxsawr4/D1wFE2yG+0kE47Nzz6Em+ZM5Hddxld6uKIiIh0Mthve70bsCQxviamAazOSj8M2AXY4u5tOfLvlpnH3dvMbGvMvzF7pWZ2BnAGwJQpU6ivry/EtgDQ0NCQ9/Ke3ZRi7dYdvGt3L2gZKkFv6lFyUx0Whuqx/1SHhVGKeixaEGFmtwG75ph0nrvf0NVsOdKc3C0o3k3+7pb12kT3y4DLAObOnet1dXVdFK/36uvryXd5N//pCUYNW8tnTqpj1LDBHu8VV2/qUXJTHRaG6rH/VIeFUYp6LNqRyd2P7sNsa4AZifHpwNo4nCt9IzDOzGpia0Qyf2ZZa8ysBtgZ2NSHMhXFjtYUf3tyHQv231UBhIiIDEqDok9EN24EFsYrK/YA5gAPAA8Cc+KVGMMInS9vdHcH7gROivOfBtyQWNZpcfgk4I6Yf1C6/uE1bNvRxvvnzug5s4iISAkMiiDCzN5jZmuA+cDfzOwWAHd/GrgO+CdwM3Cmu6diK8NZwC3AM8B1MS/AV4DPm9kyQp+Hy2P65cAuMf3zQPtloYNNKu386u7lHDh9Z+btOaHUxREREclpULSTu/tfgL90Me1C4MIc6TcBN+VIX064eiM7fQfw/n4Xtgj+/tQ6Vr7SxLmnHKwndYqIyKA1KFoipLM/PLCKWbuM4pj9cvVDFRERGRwURAwyqbTz+OqtHLH3JKqr1AohIiKDl4KIQWb5hgYamts4cPq4UhdFRESkWwoiBpnHVm8B4MAZCiJERGRw67JjpZnNzHch7r6qMMWRx9dsYafhNew5Ube5FhGRwa27qzNW0MUdHXOo7n9RhrYXNzYydecRPL56K6+fvjNV6g8hIiKDXHdBxBsTw3sDPwB+AdwX0+YDnyTcl0H6oaUtzfH/fTf7TxvLM+te5RNH7FnqIomIiPSoyyDC3R/ODJvZT4DPufv1iSx3mNlzwGeAPwxcESvfylca2d6a4qGVmwHUqVJERMpCvh0rDwWeyJH+BHBI4YozNC1b3wDABw+bycQxw5k7a3yJSyQiItKzfIOIFcB/5Ej/D2BlwUozRGWCiK+9Y18ePO8oJo4ZXuISiYiI9Czf215/DviLmS0AlsS0w4BZwHsHoFxDyrINDew2bqSe1ikiImUlr5YId7+Z8ATNPwNjCY/R/jOwt7v/feCKNzS8sKGBPSfpkk4RESkvPf71NbNawgOwLnH3rw58kYaWdNp5YX0jCw/V0zpFRKS89NgS4e6thL4PunHBAFi7dTvbW1PMnjym1EURERHplXw7Vt4CvG0gCzJUZTpVzp6kIEJERMpLvj35bge+a2ZvAB4GGpMT3f3PhS7YUJEJIvZSS4SIiJSZfIOIi+P72TmmObrtdZ8tfbmBcaNq2WX0sFIXRUREpFfyCiLcXU/7HADuzr3LNzJ39wmYqcuJiIiUFwUHJbTylSZWb9rOEXtPLHVRREREei3vuxuZ2QRgATAT6NT27u4XFLhcQ8JdSzcAcMScSSUuiYiISO/lFUSY2Tzgb0AzMAl4CZgax1cACiL64K7nNzJjwkh232VUqYsiIiLSa/mezvghcDWwG7CDcLnnTOAh4PsDU7TK1ppKc98LGzliziT1hxARkbKUbxDxBuBid3cgBQx395eBrwDnD1DZKtpTL22lsSXF4bPVH0JERMpTvkFES2L4ZWD3ONwATCtoiYaITY2hSqeOG1nikoiIiPRNvh0rHwHeCDwP1APfMbMpwCnAEwNTtMrW2JICYMxw3WJDRETKU74tEecBa+Pw14ANwP8A44EzBqBcFa+puQ1Aj/8WEZGyle/Nph5KDG8AjhuwEg0RDTGIGK0gQkREylReLRFmdrKZ7TrQhRlKmuLpjFE6nSEiImUq37/BPwCmmdkyQp+IeqDe3dcNULkqXmNLG8Nqqqit1k1DRUSkPOV1BHP3GcDrgB8BowlBxRoze87MfjGA5atYTc0pRg9TK4SIiJSvvP8Gu/tSd/8VcBrw78BVwJ7AJwaobBWtsbmN0cPVH0JERMpXvre9fiNwZHy9GdgI3EUIIO4csNJVsMaWNnWqFBGRspbvUex+wmWdPwY+6e6rBq5IQ0NTS0qdKkVEpKzlezrjv4ClhAdt3WRm/2Nm7zOzXQauaJWtoVktESIiUt7y7Vh5nrsfTri51GeBrfF9rZk9PoDlq1hNzSlGqyVCRETKWG+vLxwL7EJ4HPgUoBbQE6T6QH0iRESk3OV7s6mfm9k/Cbe+/hmwM/ATYD93320Ay1ex1CdCRETKXb5/hScAFxFuMPXsAJZnyGjQJZ5iHzPAAAAWH0lEQVQiIlLm8u0TsdDdfzFQAYSZ/dDMnjWzJ8zsL2Y2LjHtXDNbFm9s9fZE+oKYtszMzkmk72Fm95vZUjO71syGxfThcXxZnD5rILYlH62pNC1taZ3OEBGRspZ3nwgzO87M/mpmz5jZjJj2cTM7qgDlWAwc4O5vIDxu/Ny4/P2AhcD+wALg52ZWbWbVwCWEB4HtB5wc8wJ8H/ipu88BNgOnx/TTgc3uPhv4acxXEu3PzdAdK0VEpIzl2yfiQ8B1hMs8ZxE6VAJUA1/ubyHc/VZ3b4ujS4DpcfgE4Bp3b3b3F4FlwKHxtczdl7t7C3ANcIKZGfA24Po4/5XAiYllXRmHrweOivmLrjHzBE+dzhARkTKW71Hsy8An3P0aM/t4In0J4d4RhfQx4No4vFtcR8aamAawOiv9MMKVI1sSAUky/26Zedy9zcy2xvwbswtgZmcAZwBMmTKF+vr6/m1RQkNDA/X33AfAyheep75pecGWPZQ0NDQU9HMZilSHhaF67D/VYWGUoh7zDSLmAPflSG8gXPbZIzO7Dcj1OPHz3P2GmOc8oA24OjNbjvxO7hYU7yZ/d8t6baL7ZcBlAHPnzvW6urpc2fqkvr6eGXsdBPf8H2886PXU7TulYMseSurr6ynk5zIUqQ4LQ/XYf6rDwihFPeYbRKwF9gZWZqUfAbyQzwLc/ejuppvZacA7gaPcPXNwXwPMSGSbHstCF+kbgXFmVhNbI5L5M8taY2Y1hMtUN+VT9kJrbAkNJaPUsVJERMpYvh0rLwMuMrM3x/EZ8aD/A+DS/hbCzBYAXwHe7e5NiUk3AgvjlRV7EFpEHgAeBObEKzGGETpf3hiDjzuBk+L8pwE3JJZ1Whw+CbgjEawUVWNz6Fg5Rn0iRESkjOV1FHP3H5jZzoSrKEYQDtTNwI/c/ZIClONiYDiwOPZ1XOLun3L3p83sOuCfhNMcZ7p7CsDMzgJuIXTuvMLdn47L+gpwjZl9B3gUuDymXw781syWEVogFhag3H3SlGmJ0M2mRESkjOX7KPBRwDeACwmXVFYB/3T3hkIUIl522dW0C+N6s9NvAm7Kkb6ccPVGdvoO4P39K2lhZFoidJ8IEREpZz0exeI9GbYCB7r7P4GHBrxUFa7jEk+1RIiISPnqsU9EPH2wEhg28MUZGtSxUkREKkG+HSu/DXzPzPTEzgJoakkxoraK6qqS3OtKRESkIPL9K/xFYA/gJTNbAzQmJ8bbVUueGpv1GHARESl/+R7Jru85i+SrUU/wFBGRCpDvJZ7fGuiCDCWNLSk9fEtERMpe3k/xlMJpalFLhIiIlD8FESXQ0JxSECEiImVPQUQJNDW3MVqnM0REpMwpiCiBppYUIxVEiIhImVMQUQKtqTTDqlX1IiJS3vI+MW9mhwFHAZPJCj7c/ewCl6uitaWdmmrdaEpERMpbvg/g+iLhsd/LgLVA8hHaJXmcdjlrS6WpqVJLhIiIlLd8WyI+A5zt7hcPZGGGira0U6NbXouISJnL9+/wWHI8dlv6pi3l1KhPhIiIlLl8j2R/ABYMZEGGkrZ0mlr1iRARkTKX7+mM1cC3zOzNwBNAa3Kiu/+k0AWrVGl30o6e4CkiImUv3yDi40AD8Kb4SnJAQUSeUrEbaq1OZ4iISJnL9wFcewx0QYaKdDq8qyVCRETKnf4OF1lbbInQ1RkiIlLuenOzqb2Bk4CZwLDkNHf/WIHLVbHSOp0hIiIVIt+bTb0D+BPwKHAI8CCwFzAcuHvASleBUjGK0OkMEREpd/n+Hb4A+Ja7zweagQ8Ds4DbgPoBKVmF6uhYqSBCRETKW75BxD7AtXG4FRjl7jsIwcVnB6JglSoTRFTrttciIlLm8j2SbQNGxOF1wOw4XAOML3ShKlkqXp2hlggRESl3+XasvB84HPgn8Dfgx2Z2IPAe4L4BKltFSrVfnaGWCBERKW/5BhGfB8bE4fOBnYD3Ac/HaZKnlKtjpYiIVIZ8bza1PDHcBHx6wEpU4dSxUkREKkXebepmNsLMTjKzr5jZuJi2l5lNGLjiVZ7MHSv1FE8RESl3+d4nYjbhcs4xwDjgj8AWQovEOMKzNSQPKd2xUkREKkS+f4d/BtwKTAG2J9JvBI4sdKEqmYIIERGpFPl2rHwTMM/dU2adDn6rgGkFL1UFy9yxskZ9IkREpMz15sR8bY60mcDWApVlSNAlniIiUinyPZLdSudLOd3MxgLfItw3QvLUHkSoJUJERMpcb+4TcaeZPUe4c+W1hLtWvgz8+wCVrSKpJUJERCpFvveJWGtmBwEnAwcTWjAuA6529+3dziydpNov8VRLhIiIlLd8WyKIwcIV8SV9lI53rKxVS4SIiJS5vIMIM9uVcJXGZLL6Urj7zwtcrorVlnmKp1oiRESkzOV7s6lTgF8DBmwGPDHZAQUReWp/iqfuEyEiImUu3zb1C4EfAKPdfVd3n5p49fs+EWb2bTN7wsweM7NbzWxaTDczu8jMlsXpByfmOc3MlsbXaYn0Q8zsyTjPRRZvbGFmE8xsccy/2MxK8gjzdKYlQkGEiIiUuXyDiLHAIndvG6By/NDd3+DuBwF/Bb4R048D5sTXGcClEAIC4JvAYcChwDcTQcGlMW9mvgUx/RzgdnefA9wex4uu4xJP9YkQEZHylu+R7GrgHQNVCHd/NTE6mo7TJScAV3mwBBhnZlOBtwOL3X2Tu28GFgML4rSx7n6fuztwFXBiYllXxuErE+lFlXkUuJ7iKSIi5a4394n4XzM7CngSaE1OdPcL+lsQM7sQOJVwB8zM8zh2A1Ynsq2Jad2lr8mRDjDF3dfF8q4zs8ndlOUMQmsGU6ZMob6+vm8blcP2HS2A8X/33K1+Ef3Q0NBQ0M9lKFIdFobqsf9Uh4VRinrMN4j4JOG0wEbCTaayO1b2GESY2W3ArjkmnefuN7j7ecB5ZnYucBbhdEWuo6z3Ib1X3P0ywn0wmDt3rtfV1fV2EV3632W3Aq0cVVdHlYKIPquvr6eQn8tQpDosDNVj/6kOC6MU9ZhvEPF14Avu/tO+rsjdj84z6+8Jt9L+JqElYUZi2nRgbUyvy0qvj+nTc+QHeNnMpsZWiKnA+l5uQkGkHKoMBRAiIlL28u0TUU147PeAMLM5idF3A8/G4RuBU+NVGvOArfGUxC3AsWY2PnaoPBa4JU7bZmbz4lUZpwI3JJaVuYrjtER6UaXSuuW1iIhUhnxbIn4DfIg8Tlv00ffMbB8gDawEPhXTbwKOB5YBTcBHAdx9k5l9G3gw5rvA3TfF4U8Di4CRwN/jC+B7wHVmdjrhEebvH6Bt6VbKXbe8FhGRipBvEDEK+LiZvR14gtd2rDy7P4Vw9/d1ke7AmV1My3kLbnd/CDggR/orwFH9KWchpF33iBARkcqQbxCxL/BoHH5d1rRed1wcylJpqNU9IkREpALk+xTPI3vOJflIOdSoJUJERCqA/hIXmYIIERGpFAoiiiyVdt3yWkREKoKOZkWWcnR1hoiIVAQFEUWm0xkiIlIpFEQUmW42JSIilUJHsyJL63SGiIhUCAURRZZy1+kMERGpCAoiiix0rFS1i4hI+dPRrMhCnwi1RIiISPlTEFFkaokQEZFKoaNZkekSTxERqRQKIooslVbHShERqQwKIoos5XqKp4iIVAYdzYos7VCtlggREakACiKKrC2tm02JiEhlUBBRZGmHWt32WkREKoCOZkWWcqdaLREiIlIBFEQUWcqhVn0iRESkAiiIKLJUGqp1OkNERCqAjmZFFi7xVEuEiIiUPwURRaZHgYuISKVQEFFE7k7KdTpDREQqg45mRdSWdkAdK0VEpDIoiCiiVAwi9BRPERGpBDqaFVFrKg3oKZ4iIlIZFEQUUVsq0xKhIEJERMqfgogiyvSJUEuEiIhUAgURRdSWjqcz1CdCREQqgI5mRdR+OkMtESIiUgEURBRR++kM9YkQEZEKoCCiiNrar85QtYuISPnT0ayI2m82pZYIERGpAAoiiijTJ0K3vRYRkUqgo1kRtbZfnaGWCBERKX8KIooopftEiIhIBVEQUUSt6lgpIiIVZFAdzczsi2bmZjYxjpuZXWRmy8zsCTM7OJH3NDNbGl+nJdIPMbMn4zwXmZnF9AlmtjjmX2xm44u9fSl1rBQRkQoyaIIIM5sBHAOsSiQfB8yJrzOAS2PeCcA3gcOAQ4FvJoKCS2PezHwLYvo5wO3uPge4PY4XVUfHSgURIiJS/gZNEAH8FPgy4Im0E4CrPFgCjDOzqcDbgcXuvsndNwOLgQVx2lh3v8/dHbgKODGxrCvj8JWJ9KLJnM6o1W2vRUSkAtSUugAAZvZu4CV3fzyefcjYDVidGF8T07pLX5MjHWCKu68DcPd1Zja5m/KcQWjNYMqUKdTX1/dhq17r8X+1AfDoIw+zcakCif5oaGgo2OcyVKkOC0P12H+qw8IoRT0WLYgws9uAXXNMOg/4KnBsrtlypHkf0nvF3S8DLgOYO3eu19XV9XYROb36+Fp47FHmH/ZGZk/eqSDLHKrq6+sp1OcyVKkOC0P12H+qw8IoRT0WLYhw96NzpZvZ64E9gEwrxHTgETM7lNCSMCORfTqwNqbXZaXXx/TpOfIDvGxmU2MrxFRgfT83qdd022sREakkJT+aufuT7j7Z3We5+yxCIHCwu/8LuBE4NV6lMQ/YGk9J3AIca2bjY4fKY4Fb4rRtZjYvXpVxKnBDXNWNQOYqjtMS6UWTue21OlaKiEglGBR9IrpxE3A8sAxoAj4K4O6bzOzbwIMx3wXuvikOfxpYBIwE/h5fAN8DrjOz0wlXgLy/GBuQlLk6Qx0rRUSkEgy6ICK2RmSGHTizi3xXAFfkSH8IOCBH+ivAUQUraB+kdNtrERGpIPpLXEStKd32WkREKoeCiCJqa2+JULWLiEj509GsiNr0AC4REakgCiKKqE2nM0REpIIoiCgiXeIpIiKVREFEEc2ZPIZDd60m69beIiIiZWnQXeJZyd514DR22vx8qYshIiJSEGqJEBERkT5RECEiIiJ9oiBCRERE+kRBhIiIiPSJgggRERHpEwURIiIi0icKIkRERKRPFESIiIhIn5i7l7oMg5qZbQBWFnCRE4GNBVzeUKV67D/VYWGoHvtPdVgYharH3d19Uj4ZFUQUmZk95O5zS12Ocqd67D/VYWGoHvtPdVgYpahHnc4QERGRPlEQISIiIn2iIKL4Lit1ASqE6rH/VIeFoXrsP9VhYRS9HtUnQkRERPpELREiIiLSJwoiREREpE8URBSRmS0ws+fMbJmZnVPq8pQLM1thZk+a2WNm9lBMm2Bmi81saXwfX+pyDjZmdoWZrTezpxJpOevNgovivvmEmR1cupIPHl3U4flm9lLcHx8zs+MT086Ndficmb29NKUefMxshpndaWbPmNnTZvaZmK79MU/d1GFJ90cFEUViZtXAJcBxwH7AyWa2X2lLVVaOdPeDEtdAnwPc7u5zgNvjuHS2CFiQldZVvR0HzImvM4BLi1TGwW4Rr61DgJ/G/fEgd78JIH6fFwL7x3l+Hr/3Am3AF9x9X2AecGasL+2P+euqDqGE+6OCiOI5FFjm7svdvQW4BjihxGUqZycAV8bhK4ETS1iWQcnd7wI2ZSV3VW8nAFd5sAQYZ2ZTi1PSwauLOuzKCcA17t7s7i8Cywjf+yHP3de5+yNxeBvwDLAb2h/z1k0ddqUo+6OCiOLZDVidGF9D9zuAdHDgVjN72MzOiGlT3H0dhC8XMLlkpSsvXdWb9s/eOSs2s1+ROJWmOsyDmc0C/g24H+2PfZJVh1DC/VFBRPFYjjRdX5ufN7v7wYQmzjPN7IhSF6gCaf/M36XAXsBBwDrgxzFdddgDMxsD/An4rLu/2l3WHGmqS3LWYUn3RwURxbMGmJEYnw6sLVFZyoq7r43v64G/EJrkXs40b8b39aUrYVnpqt60f+bJ3V9295S7p4Ff0dFErDrshpnVEg5+V7v7n2Oy9sdeyFWHpd4fFUQUz4PAHDPbw8yGETq83FjiMg16ZjbazHbKDAPHAk8R6u60mO004IbSlLDsdFVvNwKnxl7x84CtmWZm6Szr3Px7CPsjhDpcaGbDzWwPQqfAB4pdvsHIzAy4HHjG3X+SmKT9MU9d1WGp98eaQi9QcnP3NjM7C7gFqAaucPenS1yscjAF+Ev4/lAD/N7dbzazB4HrzOx0YBXw/hKWcVAysz8AdcBEM1sDfBP4Hrnr7SbgeELnqybgo0Uv8CDURR3WmdlBhKbhFcAnAdz9aTO7DvgnoSf9me6eKkW5B6E3Ax8GnjSzx2LaV9H+2Btd1eHJpdwfddtrERER6ROdzhAREZE+URAhIiIifaIgQkRERPpEQYSIiIj0iYIIERER6RMFESJSVGY2y8zczOb2nLvP6zjJzHTpmcgA030iRKTYVgNTgY2lLoiI9I+CCBEpqnjDm3+Vuhwi0n86nSEivRJvRfxlM3vBzLab2ZNmdkqcljlV8UEzu8fMdpjZs2Z2bGL+TqczzKzWzC4ys7Vm1mxmq83se4n8483sSjPbHNd3m5ntn1WmU81spZk1mdlfCXc6zS73u+KTYHeY2YtmdmG8Bb2I9JGCCBHpre8ApwNnAvsB/wX80szekcjzA+AiwpMFFwM3mFlXjyE+m3DP/4WE+/t/AHguMX0RcBhwAuHhQk3AzWY2EsDMDot5Lovr+3/ABckVmNnbgauBi4H9gY8BJwHf7eW2i0iCbnstInmLD0HbCBzr7ncn0n8G7A38B/Ai8DV3vzBOqwKeBa5z96+Z2ayY543u/pCZXUQ4sB/tWT9IZjYHeB54q7vfFdN2Jjxn4Qvu/msz+z0wyd2PScz3a+B0d7c4fhew2N2/nchzIvA7YKfs9YpIftQnQkR6Yz9gBKElIHngrSU8/CfjvsyAu6fN7P44by6LCK0Vz5vZrYSHL/09Ptp4XyCdtbytZvZkYnn7Elofku4jtJZkHAIcamZfSaRVASOBXYEh/YRIkb5SECEivZE5BfouQmtAUitgvV2guz8SWycWAG8DrgQeN7NjelheJojJZ51VwLeAP+aYtiHvwopIJwoiRKQ3/gk0A7u7+x3ZE2MwADAPuCOmGaEvw/VdLdTdtxEO8H80s0XAEmB2XF8VMB/InM4YC7we+E2iTPOyFpk9/gjwOndf1vMmiki+FESISN7cfZuZ/Qj4UQwO7gLGEA7aaeDWmPXTZvY88CShn8TuwKW5lmlmnyecTniM0JrxQeBVYI27N5nZDYSOm2cAW4AL4/Tfx0VcBNxrZucSApU6QkfNpAuAv5rZSuA6oA04ADjU3b/c9xoRGdp0dYaI9NbXgfOBLwJPE/ozvI/QWTLjHODzwOOE0xTvcfc1XSxvG/Al4AFCi8FBwHHu3hSnfzROuzG+jwIWuPt2AHdfQuj/8GngCeC9sXzt3P0W4B3AkXEZD8QyZp+SEZFe0NUZIlIw2VdelLY0IjLQ1BIhIiIifaIgQkRERPpEpzNERESkT9QSISIiIn2iIEJERET6REGEiIiI9ImCCBEREekTBREiIiLSJ/8fYqep183mElgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,4))\n",
    "plt.plot(mean_reward)\n",
    "plt.xlabel(\"episode\", fontsize=14)\n",
    "plt.ylabel(\"mean reward\", fontsize = 14)\n",
    "plt.title(\"Mean episode rewards over episode number\", fontsize = 16)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Watch the agent play an episode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the fun part!\n",
    "Now we get to see how good our agent really is by watching it play an episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "INFO:tensorflow:Restoring parameters from models/Acrobot/model.ckpt\n",
      "Game ended after 133 steps\n",
      "Total reward earned in this episode: -131.00\n"
     ]
    }
   ],
   "source": [
    "ep_reward = []\n",
    "env = gym.make('Acrobot-v1')\n",
    "env._max_episode_steps = 1000\n",
    "# env = gym.wrappers.Monitor(env, \"recording/LunarLander\")\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"models/Acrobot/model.ckpt\") # load model\n",
    "    obs = env.reset() # Reset env and save observation\n",
    "    t = 0\n",
    "    while True:\n",
    "        env.render() # Render game\n",
    "        # Use our model to create a probability distribution of actions based on observation\n",
    "        apd = np.squeeze(sess.run(action_prob_dist, feed_dict={input_ : obs.reshape((1,obs_dim[0]))}))\n",
    "        # Choose an action out of the PDF and take action\n",
    "        action = np.random.choice(np.arange(num_actions), p = apd)\n",
    "        obs, reward, done, info = env.step(action)\n",
    "        ep_reward.append(reward)\n",
    "        t = t+1\n",
    "        if done:\n",
    "            print(\"Game ended after {} steps\".format(t+1))\n",
    "            print(\"Total reward earned in this episode: {:0.2f}\".format(sum(ep_reward)))\n",
    "            break\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the trained model performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "INFO:tensorflow:Restoring parameters from models/Acrobot/model.ckpt\n",
      "-------------------------------------------\n",
      "Game ended after 160 steps\n",
      "Total reward earned in this episode: -158.00\n",
      "-------------------------------------------\n",
      "Game ended after 181 steps\n",
      "Total reward earned in this episode: -179.00\n",
      "-------------------------------------------\n",
      "Game ended after 120 steps\n",
      "Total reward earned in this episode: -118.00\n",
      "-------------------------------------------\n",
      "Game ended after 121 steps\n",
      "Total reward earned in this episode: -119.00\n",
      "-------------------------------------------\n",
      "Game ended after 151 steps\n",
      "Total reward earned in this episode: -149.00\n",
      "-------------------------------------------\n",
      "Game ended after 137 steps\n",
      "Total reward earned in this episode: -135.00\n",
      "-------------------------------------------\n",
      "Game ended after 165 steps\n",
      "Total reward earned in this episode: -163.00\n",
      "-------------------------------------------\n",
      "Game ended after 110 steps\n",
      "Total reward earned in this episode: -108.00\n",
      "-------------------------------------------\n",
      "Game ended after 147 steps\n",
      "Total reward earned in this episode: -145.00\n",
      "-------------------------------------------\n",
      "Game ended after 119 steps\n",
      "Total reward earned in this episode: -117.00\n",
      "===================================================\n",
      "Average reward in batch is -139.10\n"
     ]
    }
   ],
   "source": [
    "num_ep = 10\n",
    "ep_reward = []\n",
    "batch_reward = []\n",
    "env = gym.make('Acrobot-v1')\n",
    "env._max_episode_steps = 1000\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"models/Acrobot/model.ckpt\") # load model\n",
    "    for ep in range(num_ep):\n",
    "        obs = env.reset() # Reset env and save observation\n",
    "        t = 0\n",
    "        while True:\n",
    "            # Use our model to create a probability distribution of actions based on observation\n",
    "            apd = np.squeeze(sess.run(action_prob_dist, feed_dict={input_ : obs.reshape((1,obs_dim[0]))}))\n",
    "            # Choose an action out of the PDF and take action\n",
    "            action = np.random.choice(np.arange(num_actions), p = apd)\n",
    "            obs, reward, done, info = env.step(action)\n",
    "            ep_reward.append(reward)\n",
    "            t = t+1\n",
    "            if done:\n",
    "                print(\"-------------------------------------------\")\n",
    "                print(\"Game ended after {} steps\".format(t+1))\n",
    "                print(\"Total reward earned in this episode: {:0.2f}\".format(sum(ep_reward)))\n",
    "                batch_reward.append(sum(ep_reward))\n",
    "                ep_reward = []\n",
    "                break\n",
    "print(\"===================================================\")\n",
    "print(\"Average reward in batch is {:0.2f}\".format(np.mean(batch_reward)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}