FrozenLake_v0_Q_Learning_with_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FrozenLake-V0\n",
    "#### $Q$-Learning with neural networks approach to solve FrozenLake-v0 enivronment from OpenAI-Gym  \n",
    "\n",
    "***FrozenLake-v0 environment***\n",
    "\n",
    "The agent controls the movement of a character in a grid world. Some tiles of the grid are walkable, and others lead to the agent falling into the water. Additionally, the movement direction of the agent is uncertain and only partially depends on the chosen direction. The agent is rewarded for finding a walkable path to a goal tile.\n",
    "\n",
    "Winter is here. You and your friends were tossing around a frisbee at the park when you made a wild throw that left the frisbee out in the middle of the lake. The water is mostly frozen, but there are a few holes where the ice has melted. If you step into one of those holes, you'll fall into the freezing water. At this time, there's an international frisbee shortage, so it's absolutely imperative that you navigate across the lake and retrieve the disc. However, the ice is slippery, so you won't always move in the direction you intend.\n",
    "\n",
    "The surface is described using a grid like the following:\n",
    "```\n",
    "SFFF       (S: starting point, safe)\n",
    "FHFH       (F: frozen surface, safe)\n",
    "FFFH       (H: hole, fall to your doom)\n",
    "HFFG       (G: goal, where the frisbee is located)\n",
    "\n",
    "```\n",
    "The episode ends when you reach the goal or fall in a hole. You receive a reward of 1 if you reach the goal, and zero otherwise. \n",
    "\n",
    "\n",
    "http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a.pdf\n",
    "\n",
    "***$Q$-Learning :***\n",
    "\n",
    "```\n",
    "                                  - - - < -  Action  - - < - - -\n",
    "                                 |                              | \n",
    "                                 |                              | \n",
    "                            _____________                    _______\n",
    "                           | Environment |                  | Agent |\n",
    "                           |_____________|                  |_______|\n",
    "                                 |                              |\n",
    "                                 |                              |\n",
    "                                  - - - > -  State  - - > - - -  \n",
    "\n",
    "```\n",
    "Consider an environment which has a vehicle that wants to reach a known location. An agent helps the vehicle in reaching its desitnation by providing instructions. After following the instruction the vehicle sends back its current state in the environment to the agent, so that the agent can make next decision.\n",
    "Lets call an instruction as an action and vehicles current state in the environment as a state. An action could be as simple as move up, down, right or left. A state can be combination of gps coordinates, sensory information from the vehicle like pixel data from camera, lidar information etc.\n",
    "\n",
    "In order to provide actions to the vehicle the agent should either be intelligent or should fake it until it becomes intelligent, that is, may be to start with it can provide random actions, learn based on the feedback from the environment and finally refine its learning.\n",
    "There are many ways for an agent to learn, Reinforcement Learning is one of them. The figure below depicts what happens in Reinforcement learning. Everything that has been explained above stays the same about environment and agent, but there is an additional input that an agent receives from the environment, that is, Reward.  \n",
    "\n",
    "```\n",
    "                                  - - - < -  Action  - - < - - -\n",
    "                                 |                              | \n",
    "                                 |                              | \n",
    "                            _____________                    _______                \n",
    "                           | Environment |                  | Agent |\n",
    "                           |_____________|                  |_______|               \n",
    "                                 |  |                          |  |\n",
    "                                 |  |                          |  |\n",
    "                                 |   - - > - - Reward - - > - -   |\n",
    "                                 |                                |                                 \n",
    "                                  - - > - - - State  - - > - - - - \n",
    "\n",
    "```\n",
    "A reward can be considered as a feedback, but more of a way to motivate the agent to learn to do the right thing. For example agent would receive a positive reward if the action instructed by the agent has moved the vehicle to next state safely without causing an accident and a negative reward if it has caused an accident.\n",
    "\n",
    "```\n",
    "                          - - - < -  Action  - - < - - -\n",
    "                         |                              | \n",
    "                         |                              | \n",
    "                    _____________                    _______                  _______ \n",
    "                   | Environment |                  | Agent |- < - - - - > - |Q-Table|\n",
    "                   |_____________|                  |_______|                |_______|\n",
    "                         |  |                          |  |\n",
    "                         |  |                          |  |\n",
    "                         |   - - > - - Reward - - > - -   |\n",
    "                         |                                |                                 \n",
    "                          - - > - - - State  - - > - - - - \n",
    "\n",
    "                                                Q-Learning \n",
    "```\n",
    "\n",
    "$Q$-Learning is a basic Reinforcement learning techinique. It is simple and easy to implement, especially if our states are discrete and the state space is small. To start with the agent provides random actions as input to the environment and bulids a table which eventually helps it in making the right decisions. The table is called as a Q table. The table would look similar to the one below.\n",
    "\n",
    "|States|Action,Q-value pairs\n",
    "|------|--------|---------\n",
    "|State-1| {'Action-1': -20, 'Action-2': 10}\n",
    "|State-2| {'Action-1': 20, 'Action-2': 0}\n",
    "\n",
    "The table has two columns, namely, states and action,Q-value pairs. Each row in the table has information about a unique state and its corresponding action Q-value pairs, for all possible actions when the vehicle is in that state. The Q values are values that are learned by the agent from the feedback provided by the environment. \n",
    "\n",
    "Assume that the agent has a Q Table as above and it wants to send an input to vehicle which is at 'state-1'. The agent first looks at all the action Q value pairs for that state, then chooses the action with highest Q value and sends it to the vehicle. The vehicle then sends back a reward and next state so that the agent can update the Q value for the action. The equation for updating the Q value is given by\n",
    "\n",
    "$$ Q(s,a) = R(s,a) + \\gamma(\\max_{a'}Q(s',a')) $$\n",
    "\n",
    "We want to learn the Q value over time, the Q value learning rule or the $Q$-Learning rule is give by :\n",
    "\n",
    "$$ Q(s,a) := Q(s,a) + \\alpha(R(s,a) + \\gamma(\\max_{a'}Q(s',a')) - Q(s,a)) $$\n",
    "\n",
    "Where   \n",
    "  $Q$ is $Q$ function  \n",
    "  $R$ is reward function   \n",
    "  $\\alpha$ is Learning rate      \n",
    "  $\\gamma$ is Discount factor  \n",
    "  $s$ is current state  \n",
    "  $s'$ is future state  \n",
    "  $a$ is current action  \n",
    "  $a'$ is future action  \n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.\n",
      "  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "from gym import wrappers\n",
    "import tensorflow as tf\n",
    "from matplotlib import pyplot as plt\n",
    "import json\n",
    "from os import listdir\n",
    "import numpy as np\n",
    "import random\n",
    "import math\n",
    "\n",
    "'''\n",
    "if there is import error with tf \n",
    "\n",
    "export LC_ALL=en_US.UTF-8\n",
    "export LANG=en_US.UTF-8\n",
    "\n",
    "'''\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**$Q$-Learning with Neural Networks :**\n",
    "\n",
    "In $Q$-Learning the intelligence aquired by the agent is stored in the form of a table. This approach is not scalable  and is good for learning environment which has smaller size state space. To make it scalable a neural network can be used to replace the table. \n",
    "\n",
    "```\n",
    "                      - - - < -  Action  - - < - - -\n",
    "                     |                              | \n",
    "                     |                              | \n",
    "                _____________                    _______                  ______________\n",
    "               | Environment |                  | Agent |- < - - - - > - |Neural Network|\n",
    "               |_____________|                  |_______|                |______________|\n",
    "                     |  |                          |  |\n",
    "                     |  |                          |  |\n",
    "                     |   - - > - - Reward - - > - -   |\n",
    "                     |                                |                                 \n",
    "                      - - > - - - State  - - > - - - - \n",
    "\n",
    "                                     Q-Learning with Neural Networks\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "RESULTS_PATH = 'tmp/results'\n",
    "\n",
    "class Agent(object):\n",
    "    \n",
    "    def __init__(self,environment,alpha=0.1,gamma=0.5,epsilon=0.1):\n",
    "        \n",
    "        \n",
    "        self.env = environment \n",
    "        self.alpha = alpha\n",
    "        self.gamma = gamma\n",
    "        self.epsilon = epsilon\n",
    "        self.graph = tf.Graph()\n",
    "        \n",
    "        self.epsilon_0 = epsilon # initial value of epsilon\n",
    "        self.alpha_0 = alpha     # initial value of alpha\n",
    "        self.trail_count = 0\n",
    "        self.build_tf_graph()\n",
    "        \n",
    "    def set_session(self,session):\n",
    "        self.session = session\n",
    "    \n",
    "    def get_graph(self):\n",
    "        return self.graph\n",
    "    \n",
    "    def build_tf_graph(self):\n",
    "        with self.graph.as_default():\n",
    "            # first layer \n",
    "            self._input = tf.placeholder(shape=[1,16],dtype=tf.float32)\n",
    "            self.W1 = tf.Variable(tf.random_uniform([16,4],0,0.01))\n",
    "            self.Qout = tf.matmul(self._input,self.W1)\n",
    "\n",
    "            # hidden layer  TODO        \n",
    "\n",
    "            # final layer TODO\n",
    "\n",
    "            self.predict = tf.argmax(self.Qout,1)\n",
    "            #Below we obtain the loss by taking the sum of squares difference\n",
    "            #between the target and prediction Q values.\n",
    "            self.nextQ = tf.placeholder(shape=[1,4],dtype=tf.float32)\n",
    "            self.loss = tf.reduce_sum(tf.square(self.nextQ - self.Qout))\n",
    "            self.trainer = tf.train.GradientDescentOptimizer(learning_rate=self.alpha)\n",
    "            self.updateModel = self.trainer.minimize(self.loss)\n",
    "        \n",
    "    def reset(self):\n",
    "        self.current_state = self.env.reset()\n",
    "        \n",
    "        # Decay function for epsilon\n",
    "        constant = 0.0003\n",
    "\n",
    "        self.epsilon =  self.epsilon_0 * math.exp(-constant*self.trail_count)\n",
    "        #self.epsilon -= 0.0001\n",
    "        #self.epsilon = 1./((self.trail_count/50) + 10)\n",
    "        \n",
    "\n",
    "        # Decay function for alpha (not usual to implement this)\n",
    "        #self.alpha = self.alpha_0 * math.exp(-constant*self.trail_count)\n",
    "        #self.alpha = max(self.alpha-0.001, 0.05 )\n",
    "\n",
    "        self.trail_count += 1\n",
    "\n",
    "\n",
    "        \n",
    "    def choose_action(self,state):\n",
    "        action, allActionsQvalues = self.session.run(\n",
    "                                                    [self.predict,self.Qout],\n",
    "                                                    feed_dict={\n",
    "                                                            self._input:np.identity(16)[state:state+1]\n",
    "                                                        }\n",
    "                                                    )\n",
    "        if random.random() < self.epsilon:\n",
    "            action[0] = self.env.action_space.sample()\n",
    "        return action, allActionsQvalues\n",
    "    \n",
    "    \n",
    "    def learn(self, current_state, action, allActionsQvalues, reward, next_state):\n",
    "        \n",
    "        Q1 = self.session.run(\n",
    "                            self.Qout,\n",
    "                            feed_dict={\n",
    "                                self._input:np.identity(16)[next_state:next_state+1]\n",
    "                                }\n",
    "                            )\n",
    "        maxQ1 = np.max(Q1)\n",
    "        targetQ = allActionsQvalues\n",
    "        targetQ[0,action[0]] = reward + self.gamma*maxQ1\n",
    "        \n",
    "        _,W1 = self.session.run(\n",
    "                            [self.updateModel,self.W1],\n",
    "                            feed_dict={\n",
    "                                self._input:np.identity(16)[current_state:current_state+1],\n",
    "                                self.nextQ:targetQ\n",
    "                            }\n",
    "                        )\n",
    "    \n",
    "    def update(self):\n",
    "        \n",
    "        action , allActionsQvalues = self.choose_action(self.current_state)\n",
    "        \n",
    "        next_state, reward, self.done, info = self.env.step(action[0])\n",
    "        \n",
    "        self.learn(\n",
    "            current_state = self.current_state,\n",
    "            action=action,\n",
    "            allActionsQvalues=allActionsQvalues,\n",
    "            reward=reward,\n",
    "            next_state=next_state\n",
    "        )\n",
    "        \n",
    "        self.current_state = next_state\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-07 22:56:34,549] Making new env: FrozenLake-v0\n",
      "[2017-03-07 22:56:34,568] Clearing 26 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-3-9beb4d35baa1>:5: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-07 22:56:34,645] From <ipython-input-3-9beb4d35baa1>:5: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n",
      "[2017-03-07 22:56:34,665] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000000.json\n",
      "[2017-03-07 22:56:34,707] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000001.json\n",
      "[2017-03-07 22:56:34,771] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000008.json\n",
      "[2017-03-07 22:56:34,904] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000027.json\n",
      "[2017-03-07 22:56:35,409] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000064.json\n",
      "[2017-03-07 22:56:36,113] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000125.json\n",
      "[2017-03-07 22:56:37,209] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000216.json\n",
      "[2017-03-07 22:56:38,426] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000343.json\n",
      "[2017-03-07 22:56:40,303] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000512.json\n",
      "[2017-03-07 22:56:42,253] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video000729.json\n",
      "[2017-03-07 22:56:46,474] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video001000.json\n",
      "[2017-03-07 22:57:06,675] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video002000.json\n",
      "[2017-03-07 22:57:33,309] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video003000.json\n",
      "[2017-03-07 22:58:02,271] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.11585.video004000.json\n",
      "[2017-03-07 22:58:02,623] Finished writing results. You can upload them to the scoreboard via gym.upload('/Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results')\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('FrozenLake-v0')\n",
    "env = wrappers.Monitor(env, RESULTS_PATH,force=True)\n",
    "agent = Agent(environment=env,alpha=0.1,gamma=0.99,epsilon=0.1)\n",
    "with tf.Session(graph=agent.graph) as session:\n",
    "    tf.initialize_all_variables().run()\n",
    "    agent.set_session(session=session)\n",
    "    agent.reset()\n",
    "    count = 0\n",
    "    while agent.epsilon > 0.03:\n",
    "    #while count < 2000:\n",
    "        agent.update()\n",
    "        \n",
    "        #print 'step'\n",
    "        if agent.done :\n",
    "            agent.reset()\n",
    "            count += 1\n",
    "            #print count \n",
    "            #count = 0\n",
    "\n",
    "    env.close()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_episode_batch_file_name(results_path):\n",
    "    for file_name in listdir(results_path):\n",
    "        if 'openaigym.episode_batch' in file_name:\n",
    "            return results_path+ '/'+ file_name\n",
    "        \n",
    "def element_wise_average(rewards):\n",
    "    reward_avg = [sum(rewards[:i])/(i+1) for i in range(len(rewards))]\n",
    "    return reward_avg\n",
    "\n",
    "def get_consecutive_100_episode_average_list(rewards):\n",
    "    consecutive_average = [sum(rewards[:100])/100.0]\n",
    "    for i in range(100,len(rewards)):\n",
    "        consecutive_average.append(sum(rewards[i-100 : i])/100.0)\n",
    "    return consecutive_average\n",
    "\n",
    "# Loading the results\n",
    "result = json.load(open(get_episode_batch_file_name(RESULTS_PATH)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEACAYAAACznAEdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYlHX9//HnGwQ8oKKRJ06ieADNTAX5atakpqgo/tAS\nO4hddoUpZloKosmSWlDf1NLUDkrpV8GwlIOWmDqWhrAIiHHaRQtYEBA5qQgsy/v3x+deGIdZdhZm\n5p6deT2ua66573vu+5439yzzns/xNndHRESkRdwBiIhIcVBCEBERQAlBREQiSggiIgIoIYiISEQJ\nQUREgCwTgpn1MbP5ZlZlZkN2st+lZrbVzE5K2XaLmVWb2TwzOycXQYuISO7t0dgOZtYCuB84C1gG\nVJrZeHefn7ZfW+A64PWUbd2BrwLdgY7A383sKNfgBxGRopNNCaEXUO3ui9y9FhgL9Muw3x3AKGBT\nyrZ+wFh33+Lu/wWqo/OJiEiRySYhdACWpKzXRNu2MbMTgY7u/lwjxy5NP1ZERIpDo1VGgGXYtq3K\nx8wMuAcY2NRjRUSkeGSTEGqAzinrHQltCfX2BY4DklFyOASYYGYXZXEsAGamJCEisgvcPdMP710+\n2U4fQEtgIdAFaA3MArrvZP+Xgc9Fyz2AmdFxXaPzWIZjvBgNHz487hB2oJiyo5iyV4xxKabsRN+d\njX6PZ/totITg7nVmNhiYTGhzeNjd55nZCKDS3SelH0JUVeTuc83sT8BcoBa4JvpHiIhIkcmmygh3\n/xtwTNq24Q3se2ba+k+Bn+5qgCIiUhgaqbwTiUQi7hB2oJiyo5iyV4xxKaZ4WDHU4JiZapJERJrI\nzHLaqKwSgoiIAEoIIiISUUIQEWlEdTVceincf3/ckeSXEoKIyE6MHw9HHw2bN8PFF8cdTX5l1e1U\nRKRcuMPKlTBtGowdC088AZMmwQUXxB1Z/ikhiEhZW74cXnwxVAd16QJTpsDixdCuHfTtG5Y7dYo7\nysJQt1MRKTtLl8I778CPfgSvvBK2XXUV7LUXHHQQfP/7sO++8caYjVx3O1UJQUTKwpYtoT1g/Hh4\n7LGw7ZvfhHnz4LDDYL/94o2vGCghiEhJWrwY3ngDXngBksnwxQ/wrW+F9oGePWMNryipykhEmpWt\nW2H06FCls2oV/PWv8MEHsHEjnH8+DBoE3/52aAg+9FDo1g0uuQROPx1OOglalFDfSlUZiUjZ2rAB\nBgyAiRO3V/H07QunnQZ77gkTJsDw4fDlL4exA926xRtvc6MSgog0CzNnhl/4XbrAq69Cx4477uMe\nGouPOAIsd7eNKVqay0hEys5rr0EiAXfdBf/5T+ZkACEJHHlkeSSDfFBCEJGitWEDXHklfP7zoYvo\nsGH6ss8ntSGISFF69VU44ww4/vjQU+jss+OOqPSphCAiReP992H16lASOOMMGDw4tB0oGRSGSggi\nEruPPoLbboN77w3rLVrAs8+GbqRSOFmVEMysj5nNN7MqMxuS4fVBZjbbzGaa2T/M7Nhoexcz22Bm\nM6LHA7n+B4hI87Z6NZxwQkgGv/89rFsHtbVKBnFotNupmbUAqoCzgGVAJTDA3een7NPW3T+Mli8E\nrnH388ysCzDR3U9o5D3U7VSkDD39dBhE9tnPwvPPQ6tWcUfUvMTR7bQXUO3ui9y9FhgL9EvdoT4Z\nRNoCW1PW1SdARHZw//3Qvz985jPw3HNKBsUgmzaEDsCSlPUaQpL4BDO7BrgRaAWcmfLS4Wb2BrAe\n+JG7v7rr4YpIc7dwYWgwXr48jC847bS4I5J62ZQQMv3C36F+x90fcPduwBDgR9Hmd4HO7n4y8APg\nCTNru6vBikjzs3UrLFoU5h269VY45hjo0QOWLVMyKDbZlBBqgM4p6x0JbQkNeRJ4CMDdNwObo+UZ\nZvY2cDQwI/2gioqKbcuJRIJEIpFFaCJSzD74INyLePLk7dtuuw3uuCO+mJqzZDJJMpnM2/mzaVRu\nCSwgNCq/C0wDLnf3eSn7dHP3hdHyhYSqoV5m1h5Y7e5bzewI4BXgM+6+Nu091KgsUmJuuQVGjoS2\nbeHtt2HFijDITCONc6fgs526e52ZDQYmE6qYHnb3eWY2Aqh090nAYDM7m1AaWAMMjA7/AvBjM6sF\n6oBB6clARErL5s1w4YWhVDBuXJh62izciUyKm2Y7FZGc+c1v4OqrwwRzU6bApz8dd0SlTbOdikhR\ncYcFC+AHPwjJ4He/g9mzlQyaI01dISJN9t57YUDZ2rVQVRW6kAKMGRNuYCPNkxKCiGRlw4YwA+nU\nqXD77WFbx47QqVOYjfT44+ONT3afEoKINOp734P77tu+fscdofuolBYlBBHJyD20CUyaFAaRPfgg\nXHEFtGkDLVvGHZ3kgxKCiOxg5Uo49lhYsyZ0Gx05UjesLwfqZSQi27jD9dfDwQeHKanfew+eekrJ\noFyohCAifPhhmHH0ssvC+lNPwcUXq2qo3KiEIFKmRo8OA8jMYN99QzIYOHB7NZGSQfnRSGWRMrNl\nS+g++qUvhVlHTzgBrr02LB94YNzRSVMUfC4jESkNc+fCd74T7kEA8KtfwXXXxRuTFBeVEETKQE0N\ndO0aSge33hoajjW1RPOnuYxEpEl+//swmvjSS0MvojvvVDKQzFRlJFKC3ENJYMIEmDMHnn0Wzj8/\n7qik2CkhiJSYjz4Kt6lcujSsV1QoGUh2lBBESkhdHRx3XLh/8dy50L173BFJc6KEIFIiNm2Cww+H\nffYJYwn22ivuiKS5UUIQKQG1tXDIIeH+BMuWKRnIrlEvI5FmqqYGbrwREglo3TokgzVr4NBD445M\nmqusEoKZ9TGz+WZWZWZDMrw+yMxmm9lMM/uHmR2b8totZlZtZvPM7JxcBi9SjtavD6OKO3WCe+6B\nt94KXUo//hjatYs7OmnOGh2YZmYtgCrgLGAZUAkMcPf5Kfu0dfcPo+ULgWvc/Twz6wE8DvQEOgJ/\nB45KH4WmgWki2amthZ494c034Z//hNNPD3MRSXmKY+qKXkC1uy+KAhgL9AO2JYT6ZBBpC2yNli8C\nxrr7FuC/ZlYdnW9qDmIXKTvf/jYsWRJKCfvuG3c0UmqySQgdgCUp6zWEL/VPMLNrgBuBVsCZKcdO\nSdltabRNRJpg1arQVjBnTkgISgaSD9kkhEzFkR3qd9z9AeABMxsA/Ai4MttjASoqKrYtJxIJEolE\nFqGJlL4nn4QBA8LylCnhxvZSnpLJJMlkMm/nz6YNoTdQ4e59ovWhgLv7qAb2N2CNu7dL39fM/gYM\nd/epaceoDUEkzYoVcPPN8Oij0LkzLFgAe+4Zd1RSTOKY3K4S6GZmXcysNTAAmJAWVOoN9voSGqGJ\n9htgZq3NrCvQDZi2+2GLlLZNm6BDh5AMkklYtEjJQPKv0Sojd68zs8HAZEICedjd55nZCKDS3ScB\ng83sbGAzsAYYGB0718z+BMwFagm9j1QUENmJVatCl9ITToA33lAvIikc3Q9BpIhMmwanngrt28PC\nhbD//nFHJMVMd0wTKUEffri951DXrlBdrXsaS+Fp6gqRInDhheH58cfhnXeUDCQeKiGIxOjZZ6Fv\n37C8ejUccEC88Uh5UwlBJCZVVSEZ9O8fJqVTMpC4KSGIFFBdHXzta6Hn0DHHwLBh8Oc/a1I6KQ6q\nMhIpkGeegcsvh40b4bzz4JvfhEsuiTsqke2UEERy6OOP4V//CrOQjhsXJqPbvPmT+yxYAEcfHU98\nIjujhCCSI2vXhi/699775PZLLw23tbzmGui1w7SQIsVDCUFkN732Gpx1VphuAsKAsrfeggsugFat\n4o1NpCk0UllkNyxZEiaeM4Pnn4ezz9ZUE1I4uR6prIQgsos2bgy9g/r1C1NUixSaEoJIzNzh7bfh\nqKNgjz3go4/CTe5FCi2O6a9Fyt7mzaHL6PTp0KJFSAYdOoQBZUoGUipUQhDJwte+BmPGhOWDD4a/\n/Q1OPDHemEQ026lIAa1YEdoHxoyBN9+Egw6Ctm3DQ6TUKCGINODll+HMM8Py978fblgjUsqUEERS\n1NTAI4/AL34B69eHEsGSJRpPIOVBbQgikU2bPnnf4hdf3F5CEClGakMQyYOamnAf4099KtzTWKQc\nZdXt1Mz6mNl8M6sysyEZXr/BzOaY2Swze8HMOqW8VmdmM8xsppk9k8vgRXKh/qb2n/40zJsXdzQi\n8Wm0ysjMWgBVwFnAMqASGODu81P2+SIw1d03mtnVQMLdB0SvrXf3/Rp5D1UZSSzcw7gCgJUrQ1IQ\naS7iGJjWC6h290XuXguMBfql7uDur7j7xmj1daBDysua2UWK1rBh4XnDBiUDkWwSQgdgScp6DZ/8\nwk93FfDXlPU2ZjbNzP5lZv0aOkik0B59FEaODIPM9tor7mhE4pdNo3KmX/gZ63fM7BvAycAXUzZ3\ndvflZtYVeMnMZrv7f9KPraio2LacSCRIJBJZhCbSdHPmwHXXhXEGv/41nHtu3BGJZCeZTJJMJvN2\n/mzaEHoDFe7eJ1ofCri7j0rb72zgl8AX3P39Bs41Gpjo7n9J2642BCmId9+Fww4Ly//4B5xxRrzx\niOyOONoQKoFuZtbFzFoDA4AJaUF9DngIuCg1GZhZu+gYzKw9cBowN1fBizTFQw+FZNCyJWzZomQg\nkq7RhODudcBgYDIwBxjr7vPMbISZ9Y12+xmwDzAurXtpd2C6mc0EXgR+mto7SaRQVq+G734Xzj8/\nNCC3bBl3RCLFRyOVpSz07QvvvANzVT6VEqKRyiJNsHYtfOlLMGsWLF4cdzQixU0JQUrW1q1wwAFh\nediwMBpZRBqmhCAl65ZbwvOWLWozEMmG2hCkJK1aFUYea8ZSKWW6p7JII+bNg86d4atfVTIQaQol\nBCkZq1fDzTdDjx7w8ccwYkTcEYk0L2pDkJIwZw4cf3xYvu8+GDw43nhEmiOVEKTZmz07JIP+/WH5\nciUDkV2lRmVp1p54Ar7+9bC8dOn2eYpEyoEalUUivXuHZDBqVOhaqmQgsnuUEKTZ+fnPwQymTg0D\nzm6+WeMMRHJBVUbSrNTWQuvWcNFF8LOfwTHHxB2RSHw0l5GUteuuC89PP739XsgikhsqIUizUVsL\ne+8NySScfnrc0YjET43KUpZmzAhVRS1bKhmI5IsSghS92bPh5JPhyCNhvm6vJJI3qjKSonfUUbDv\nvvCvf8Gee8YdjUjxUKOylJWnn4aFC2HlSiUDkXxTlZEUrS1bwnQUN90UprIWkfzKKiGYWR8zm29m\nVWY2JMPrN5jZHDObZWYvmFmnlNcGRsctMLMrchm8lK4NG6BVK9h//zDeQETyr9GEYGYtgPuBc4Hj\ngMvN7Ni03WYAJ7v7icCfgZ9Hxx4A3A70BE4FhpvZ/rkLX0rRfffBPvuE5RdeiDcWkXKSTQmhF1Dt\n7ovcvRYYC/RL3cHdX3H3jdHq60CHaPlcYLK7r3P3tcBkoE9uQpdS9Npr8L3vwRFHQE0N9OwZd0Qi\n5SObRuUOwJKU9RpCkmjIVcBfGzh2KduThcgnfPABfP7zYbm6WiORRQotm4SQqUtTxj6iZvYN4GTg\ni009tqKiYttyIpEgkUhkEZqUkjFj4LzzYNIkJQORTJLJJMlkMm/nb3Qcgpn1BircvU+0PhRwdx+V\ntt/ZwC+BL7j7+9G2AUDC3a+O1h8CXnb3J9OO1TiEMrdqVehJNHo0XHll3NGINA+5HoeQTUJoCSwA\nzgLeBaYBl7v7vJR9PgeMA85197dTth8ATAdOIrRXTCc0Pq9New8lhDK2efP2bqVr14aprUWkcQUf\nmObudWY2mNAg3AJ42N3nmdkIoNLdJwE/A/YBxpmZAYvc/WJ3X2NmdxASgQMj0pOBlLfKSugVtUit\nWqVkIBInTV0hsamvJgJ47z1o3z7eeESaG01dISXjy18Oz5s2hZlMRSRe6sshsRg0CGbNCg8lA5Hi\noIQgBXffffDb38K118JnPxt3NCJST20IUnCHHQbXXw9DdpgVS0SaouDdTgtBCaF8LF4cbnazfHm4\n+5mI7DrdQlOatbvvhjPPVDIQKUYqIUjBbNoUbnIzYQJceGHc0Yg0fyohSLP129/CoYcqGYgUK41D\nkIJ45ZUwrfXdd8cdiYg0RFVGkndjx8Lll8ORR8LcuRp3IJIr6mUkzcry5aGaqFOnkAzato07IpHS\noTYEaTYefzwkA4C331YyECl2SgiSF5s3wze+ERqQ16+HVq3ijkhEGqNGZcmLu+4Kz+PHa0prkeZC\nbQiSc+6wxx7wyCMwcGDc0YiULjUqS9Hr2ROmT4etW1U6EMknNSpLUXv00ZAMJk5UMhBpblRCkJyp\nqoJjjoHzzoPnnos7GpHSpyojKVqXXQYLF4YSgkoHIvkXS5WRmfUxs/lmVmVmO8xib2ZnmNkbZlZr\nZv3TXqszsxlmNtPMnslV4FJctmyBP/0JHnpIyUCkuWq026mZtQDuB84ClgGVZjbe3een7LYIGAj8\nMMMpPnL3k3IRrBSvUaPCWINTTok7EhHZVdmMQ+gFVLv7IgAzGwv0A7YlBHdfHL2Wqd5HvxdLnDvc\ndlu4NaZKByLNVzZVRh2AJSnrNdG2bLUxs2lm9i8z69ek6KRZ+MMfwvO118YahojspmxKCJl+8zWl\nBbizuy83s67AS2Y2293/k75TRUXFtuVEIkEikWjCW0hc+vULN7y5+GKVDkTyLZlMkkwm83b+RnsZ\nmVlvoMLd+0TrQwF391EZ9h0NTHT3vzRwroyvq5dR8/T++9C+PdxzD3z/+3FHI1J+4uhlVAl0M7Mu\nZtYaGABM2Mn+24Izs3bRMZhZe+A0YO5uxCtFoj4ZtGmjZCBSKhpNCO5eBwwGJgNzgLHuPs/MRphZ\nXwAzO8XMlgCXAg+Z2VvR4d2B6WY2E3gR+Gla7yRppr7znfA8V+ldpGRoYJo02bp10K4dTJsW5i0S\nkXhoLiOJ3V13wVFHKRmIlBolBGmyiRPhppvijkJEck1VRtIkK1fCwQfDxx/DnnvGHY1IeVOVkcTq\nnnuga1clA5FSpBKCZM0d9t8fnnwyTHEtIvFSCUFiM2wYfPABnH123JGISD4oIUhW/v53GDkyTG/d\nqlXc0YhIPqjKSLJiBp/7HMyYEXckIlJPVUZScP/8Z3jO45xaIlIEVEKQnXKHFi0gkYCXX447GhFJ\npRKCFNT//V94Hj8+3jhEJP+UEKRBa9bAFVfA7bfDfvvFHY2I5JuqjCSjTZu2Dz7bulU3vxEpRqoy\nkoL43/8Nz//9r5KBSLlQQpCMHn88zGrapUvckYhIoajKSHawfn2YomLt2vAsIsVJVUaSV2++uT0J\nKBmIlBclBNnm7rvhxBPD8qJF8cYiIoWnKiMB4PnnoU8f6NwZqqqgTZu4IxKRxsRSZWRmfcxsvplV\nmdmQDK+fYWZvmFmtmfVPe21gdNwCM7siV4FL7qxbF5JB27ahV5GSgUh52qOxHcysBXA/cBawDKg0\ns/HuPj9lt0XAQOCHacceANwOnAQY8EZ07LocxS858Mgj4XnJEnUxFSln2ZQQegHV7r7I3WuBsUC/\n1B3cfbG7/xtIr/c5F5js7uvcfS0wGeiTg7glR+rq4MYb4bbboF27uKMRkThlkxA6AEtS1muibdlI\nP3ZpE46VAhg+PDz/+MfxxiEi8Wu0yohQ1ZMu2xbgrI+tqKjYtpxIJEgkElm+heyq6dPD4LMRI1RV\nJNIcJJNJknmch77RXkZm1huocPc+0fpQwN19VIZ9RwMT3f0v0foAIOHuV0frDwEvu/uTacepl1EM\nfvITmDkTxo2LOxIR2RVx9DKqBLqZWRczaw0MACbsZP/U4J4Hvmxm+0cNzF+OtkkRuPVWuOyyuKMQ\nkWLRaEJw9zpgMKFBeA4w1t3nmdkIM+sLYGanmNkS4FLgITN7Kzp2DXAHMB2YCoyIGpclZlOnhudL\nLok3DhEpHhqYVobq74LWuzdMmRJ3NCKyqzSXkey2WbPC87PPxhuHiBQXlRDKUP/+8OGHMHly3JGI\nyO7IdQkhm26nUkJWrICnnw5zF4mIpFIJocyccw689FK4RWbLlnFHIyK7QyUE2WWvvQYvvBDueaBk\nICLpVEIoEx9/DHvvDWefHZKCiDR/6mUku2T8eDjgACUDEWmYEkKZ+PWv4etfjzsKESlmakMoA889\nB6++Cg8/HHckIlLM1IZQ4pYtgw4d4DOfgdmz445GRHJJbQjSJLfdBp06hZ5FIiI7oyqjEvXRR3Df\nfTB6NDz4oO53ICKNU5VRCXrsMRg6NFQXHXdcqCpqobKgSMlRlZE0aPNmqKqCK64I65Mmwb//rWQg\nItlRCaFEvPkmnHgi7LcftG0LNTWqJhIpdSohSEY//zkcfDD85jehVKBkICJNpRJCCXj3XTjsMKis\nhFNOiTsaESkUlRBkBxMnQufOSgYisnuUEJq5tWth0KDwEBHZHVklBDPrY2bzzazKzIZkeL21mY01\ns2ozm2JmnaPtXcxsg5nNiB4P5PofUO4efDA8X3VVvHGISPPX6MA0M2sB3A+cBSwDKs1svLvPT9nt\nKmC1ux9lZpcBPwMGRK8tdPeTchy3AO+/D8OGwYgRoUFZRGR3ZFNC6AVUu/sid68FxgL90vbpB/wx\nWn6KkDzqqb9Lntx5Z3gePDjeOESkNGSTEDoAS1LWa6JtGfdx9zpgrZkdGL12uJm9YWYvm9nndzdg\nCe69NzwefxwOPLDx/UVEGpPNXEaZfuGn9xFN38eifd4FOrv7GjM7CXjGzHq4+4fpJ6yoqNi2nEgk\nSCQSWYRWnv79b7jhBjj5ZLjoorijEZFCSSaTJJPJvJ2/0XEIZtYbqHD3PtH6UMDdfVTKPn+N9plq\nZi2Bd939oAznehn4gbvPSNuucQhZev99aN8+JIPp0+OORkTiFMc4hEqgW9RjqDWhsXhC2j4TgYHR\n8leAl6Jg20eN0pjZEUA34J1cBF6O/vKXkAzM4Omn445GREpNo1VG7l5nZoOByYQE8rC7zzOzEUCl\nu08CHgYeM7Nq4H229zD6AvBjM6sF6oBB7r42H/+QUrdyJVxyCXTtCq+9BoceGndEIlJqNHVFMzFo\nEPz2t7B1q+YpEpFAU1eUmQ0b4PrrQzIYN07JQETyR3dMK3JXXAF//jPcfDP07x93NCJSypQQitit\nt4Zk8PLLoF64IpJvakMoUh99FG50M2pUKB2IiKTLdRuCEkKRqqyEU0+FjRuhdeu4oxGRYqSEUAa2\nbIF27eC442Dq1LijEZFipV5GZeDqq0OV0WOPxR2JiJQTlRCKjDu0aAGPPALf+lbc0YhIMVMJocT9\n4hfh+corYw1DRMqQEkKR2LgRhg6Fm26CW27RADQRKTyNQygCNTUwZkzoYjpsGNx1V9wRiUg5UkKI\n0dtvw9e/vr0n0ciRMGSHO1aLiBSGEkKMfvhDWLQolA769g0D0URE4qJeRjHZsCEkgNmz4fjj445G\nRJoj9TJq5rZuhRdegH32CV1Mjz467ohERIKiTgg1NWHUbilwhyVLQnvBOedAjx5hXdNSiEixKOqE\n0KlT+OJcvz7uSHbfmDHQuTN88Yvwla/AnDnQsWPcUYmIbFfUCaF7dzj44FDH/uSTUFcXd0S7Zty4\n0IA8eHAYbzBmTNwRiYjsKKuEYGZ9zGy+mVWZ2Q4dI82stZmNNbNqM5tiZp1TXrsl2j7PzM5pSnBm\n8OCD8Ic/wD33wLHHwp13wrx5oQqmOXjnHfjqV+E73wnjDFq0gJYt445KRGRHjSYEM2sB3A+cCxwH\nXG5mx6btdhWw2t2PAu4FfhYd2wP4KtAdOA94wCz7Mbh1deEL9MwzYcoUePRReO+9UAd/2GHhpvMj\nR8L48bBgAdTWZnvm7CSTySYfs24dLF8OH3wAN9wA/frBWWdBRQXsvXc8MeWbYspOMcYExRmXYopH\nNiWEXkC1uy9y91pgLNAvbZ9+wB+j5aeAM6Pli4Cx7r7F3f8LVEfny0pd3fZf02bwP/8Dv/wlLF4c\nEsQll8DKleF+wxdcELpxdugAPXuGL+Lvfhduvx3uvhtGj4annw53H5s1K/xyX748tE80lEia8gew\nejU8/jj06hWquA49NExQd8cdIZHlSjH+USqm7BRjTFCccSmmeGQzMK0DsCRlvYYdv9S37ePudWa2\nzswOjLZPSdlvabQtK6kJIZUZHH54eHzta9u319bCihWwdCksWxYeq1aF3jxvvQVr18KaNeF53Tr4\n+OMwHmDDhnDOvfcOj732Cs+rV4cuoq1bQ6tWmZ/rl6uqQqLq3TskqDZtsv1XiogUh2wSQqYqnvQa\n/Ib2yeZYAC68cMdt774bvmyz1apV6LmzK713amu3J4cNG0Ky+OUvYeDA8NrmzdufMy137hxKKT16\nNP29RUSKgrvv9AH0Bv6Wsj4UGJK2z1+BU6PllsDKTPsCf6vfL+1410MPPfTQo+mPxr7Dm/LIpoRQ\nCXQzsy7Au8AA4PK0fSYCA4GpwFeAl6LtE4DHzeweQlVRN2Ba+hvkcui1iIjsmkYTQtQmMBiYTGiE\nftjd55nZCKDS3ScBDwOPmVk18D4haeDuc83sT8BcoBa4puwmLRIRaSaKYnI7ERGJX+wjlRsb9Jbn\n9/6vmb1pZjPNbFq07QAzm2xmC8zseTPbP2X/X0WD7GaZ2Yk5iuFhM1thZrNTtjU5BjMbGF3DBWZ2\nRZ7iGm5mNWY2I3r0SXkt4wDEXH2+ZtbRzF4ys7lm9paZfS/aHuu1yhDXddH2OK9VGzObGv1dv2Vm\nw6Pth5vZ69G/e4yZ7RFtz8vA0ixjGm1m70TbZ5jZCSnHFOpvvUX03hOi9diuU1pMM1Ni+kNBrlMu\nGySa+iAkpIVAF6AVMAs4toDv/w5wQNq2UcDN0fIQYGS0fB7wbLR8KvB6jmL4PHAiMHtXYwAOAN4G\n9gfa1S/nIa7hwI0Z9u0OzCRUQR4efaaWy88XOAQ4MVpuCywAjo37Wu0krtiuVfQ+e0fPLYHXo2vw\nJPCVaPuDwKBo+bvAA9HyZYSxQwA9MsWa45hGA/0z7FvIv/UbgP8DJkTrsV6nBmIaDfy/fF+nuEsI\n2Qx6y6ds7AVxAAADw0lEQVT6/4ipUgfZ/TElnn7AowDuPhXY38wO3t0A3P1VYM1uxnAuMNnd17n7\nWkJ7Tx92QwNxQeauxP3IPAAxZ5+vuy9391nR8ofAPKAjMV+rBuKqH2sTy7WKYtkQLbYhfFE58CXg\nz9H2PwIXp8SU84GlWcS0NVpv6Drl/fMzs47A+cDvUzafSYzXqYGYIHONTk6vU9wJIdOgt6wHruWA\nA8+bWaWZfTvadrC7r4Dwnx04KNqeHmuTBtk10UFZxlB/vQoZ27VR0fT3KdUzDb1/Xj5fMzucUHp5\nnew/r7xfq5S4opuixnet6qscgOXAC4RfiGvdvf5LOPX8nxhYCqQOLM3ZtUqPyd0ro5fujK7TL8ys\nfuRRoT6/e4CbCN8FmNmngDVxXqf0mFLk/TrFnRCyHriWJ6e5+ymEbHytmZ2xk/ePO9ZMMVgUQ6Fi\newA40t1PJPyn/kUDcdW/f87jMrO2hF9n10e/yLP9vPJ6rTLEFeu1cvet7v45QimqF6GqqqHzxxKT\nhbnOhrp7d6An8ClCtV+mmHL++ZnZBcCKqIRXf17L8B4Fu04NxAQFuk5xJ4QaoHPKekdgWaHePPpF\nibu/BzxD+I+zor4qyMwOAVamxNqpQLE2NYaCXEd3f8+jCkrgd2wvFhckrqhx7yngMXcfH22O/Vpl\niivua1XP3dcDrxAGmLazMFll+vm3xWRmLQl1zWt2EmuuYuqTUrqrJdSTF/I6nQ5cZGbvAGMIVUD3\nEqpd4rpOO8RkZo8W7DrtTsPH7j4IjUv1DWmtCQ1p3Qv03nsDbaPlfYDXgHMIjZRDou1D2d5IeT7b\nG296k6NG5eh8hwNvpaw3KQY+2YBUv9wuD3EdkrJ8A/BEtFzfqNYa6Mr2htKcfr6EutK707bFfq0a\niCu2awW0J2pABPYC/hFdjyeBy6LtDwJXR8vXsL2xdAA7NpZ+ItYcx3RItM0IVSU/ielv/Yt8slE5\nluu0k5gKcp1y8oW2m//oPoSeGdWEYlGh3rdr9J9uJvBW/XsDBwJ/j2J6IfUiEqYBXwi8CZyUozie\nIGTuTcBi4FvRB9ikGIAro2tYBVyRp7geBWZH1+0ZQv19/f63RHHNA87J9edL+OVUl/KZzYjO3eTP\nK5fXaidxxXmtPhPFMSuK4daUv/mp0b/7SaBVtL0N8KfofV8HDm8s1hzG9GL0+cyOrtnehf5bj86Z\n+uUb23XaSUwFuU4amCYiIkD8bQgiIlIklBBERARQQhARkYgSgoiIAEoIIiISUUIQERFACUFERCJK\nCCIiAsD/B4tgy+Pb3lwVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1194cc790>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Average of rewards till that episode vs episodes\n",
    "plt.plot(element_wise_average(result['episode_rewards']))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXm4FNW19t/FfAaRScQI4kAYBBUHcNajEkSvSq4xBuIc\nk+A1qAnRoGLkcP2i0cQhxinXKVETcZ5iVHA4cYSg4ICCgCiCosjMGYHD/v5Yva3q6uru6nOqq6tP\nv7/n6aemXVWrq7tWrVp77bXEGANCCCGlQbtCC0AIISQ6qPQJIaSEoNInhJASgkqfEEJKCCp9Qggp\nIaj0CSGkhAik9EVkjIgsFJFFIjLZZ3s/EXlZROaKyLsiclz4ohJCCGktki1OX0TaAVgE4BgAXwKY\nA2CcMWahq81fAMw1xvxFRIYA+JcxZrf8iU0IIaQlBLH0RwJYbIxZZozZAmA6gLGeNtsAdE3MdwPw\nRXgiEkIICYsOAdrsDGC5a3kF9EHgZhqAGSJyIYByAKPCEY8QQkiYBLH0xWed1yc0HsC9xph+AP4L\nwAOtFYwQQkj4BLH0VwDYxbXcF+rbd3MugGMBwBgzS0S6iEgvY8xqdyMRYaIfQghpAcYYPwM8Z4JY\n+nMADBCR/iLSCcA4AE972ixDwqWT6Mjt7FX4FmNM7D9Tp04tuAyUkzJSTsppP2GSVekbY5oBTAQw\nA8CHAKYbYxaIyDQROSHR7GIAPxORdwH8HcBZoUpJCCEkFIK4d2CMeR7AIM+6qa75BQAOC1c0Qggh\nYcMRuT5UVVUVWoRAUM7wKAYZAcoZNsUiZ5hkHZwV6slETJTnI4SQtoCIwETYkUsIIaSNQKVPCCEl\nBJU+IYSUEFT6hBBSQlDpE0JICUGlTwghJQSVPiGElBBU+oQQUkJQ6RNCSAlBpU8IISUElT4hhJQQ\nVPqEEFJCUOkTQkqW1auBmppCSxEtVPqEkJLl1luBo44qtBTRQqVPCClZmpoKLUH0UOkTQkqWdgkN\nuHx5YeWIEip9QkjJ0tys02uuKawcURJI6YvIGBFZKCKLRGSyz/YbRGSeiMwVkY9FZG34ohJCSLjU\n1+t0w4bCyhElWQuji0g7ALcAOAbAlwDmiMhTxpiFto0xZpKr/UQAw/MgKyGEhMb8+cDNN+v8tm2F\nlSVKglj6IwEsNsYsM8ZsATAdwNgM7ccDeDAM4QghJF/MnOnMH3BA4eSImiBKf2cA7m6OFYl1KYjI\nLgB2BfByqyUjhJA80iHh59hzT8fNUwoEUfp+FdhNmrbjADxqjEm3nRBC8sqWLYAI8OGHmdtdeKFO\nd9oJeO+9/MsVF7L69KGW/S6u5b5Q374f4wCcn+lg1dXV385XVVWhqqoqgAiEEBKMujqdLl4MDB2a\nvf055wD/+ld+ZcqVmpoa1ORpqLBkM8pFpD2Aj6EduSsB/AfAeGPMAk+7QQCeM8bsnuFYfAkghOSV\nDz8Ehg0D/vEPYPz49O0k4cN49FHgppuA116LRr6WICIwxvh5XXImq3vHGNMMYCKAGQA+BDDdGLNA\nRKaJyAmupuOgnbyEEFIwhg3T6XbbZW970EHA++8Dr7+eX5niRBD3DowxzwMY5Fk31bM8LUS5CCGk\nVbTLYtIedRQwZQowY0Y08sQFjsglhLQZ3G7whob07YwB5s4FKiqALl3yLlasoNInhLQZ3BkzMyn9\nNWt0FO6QIcBPf5p/ueJEIPcOIYQUG5mUfn090K8fsP32QGWlduoa43TutmVo6RNC2iTZlH5Zmc63\nb68K/9VXo5Gr0FDpE0LaHMcdl13pl5cnr7v88vzKFBeo9AkhbY4RIzIr/YaGVKXf2JhfmeIClT4h\npE3gHvdZVhbcvWMJEtffFqDSJ4S0CRobgU6dgG++ya70vZb+DTcAu6fNJdC2oNInhLQJrPXeq1d2\npb9xY7Kl/53vlE4hFSp9QkjsEQHeeCNzm2XLnGRr8+YBd93l384Y4IwzgIcfdtZ17w48/XQ4ssYd\nKn1CSFHw8ceZt9fWAgcfrPNfpssDDOfB4GbECI3XLwWo9AkhSRgDvP12csdoHFibpfJ2ba2mVQAc\n182yZTpdtAh46SUti7h6deq+XbsC69cDW7eGJ29codInhCSxZIlavpms5UKwZk3m7XV1jrV+zjk6\n3XVXnQ4aBIwapWmXly7VdSef7Ozbvr12Ar/zTqgixxIqfUJIErW1Oo1bCcFsxcvdlv6BB/q32bRJ\nO3iPOw547LHkbYcc4u/6aWtQ6RNCkrBRL01NhZXD0tys0z/+MfODaP58R+l37uysHz3ama+tBR5/\nHFi1KnX/8vL4PejyAZU+ISQJq/TjMkLVyrNtW+YInpUrgR131Hl3uuSZM535ujrgnnv83TgVFVT6\nhJASJG5KP6gi3rgR2HdfnRcBfvzj1DbWdeVHeTndO4SQIueNN4A//Sm3fazSP/xwYL/9gPvvD1+u\noHzyCXDBBc5yJpfTRx8lp1LYZZfUNmeemX7/lSuBn/ykMOmVf/lLJ9Io31DpE9KGmTpVFUouuEey\nzpuXWVHmm7//XQdRDRyokTiZLPG1a5MV/eWX6/7XXAPccgtw2mnONj/3zsKFoYmdM889l3kEcZgE\nKqIiImMA3AR9SNxtjLnWp82pAKYC2AbgPWPM6WEKSgjJnQ4tKJMUlfIJgrW6KyqAQw/N7J5p317j\n7S3bbZfs4nHH+fvl2dmypXWytoampujKNma19EWkHYBbABwLYCiA8SIy2NNmAIDJAA42xuwFIEfb\nghCSD154Qae5ROJkUvpLlwJ9+rROpqDU1gJXXqnzjY36AMhU6KShITVzppu+fZ15G+Xj5qCDWiZn\nGDQ2xkjpAxgJYLExZpkxZguA6QDGetr8DMCtxpiNAGCM8RnzRggpFNkGNrlpaAB+9Sv/bV98AXz9\ntRNGmU/cMldXq6W/ebN/W2NUcWZS+mefDey5p8537Ji6/ZFHgM8+Sw73jIrGxujOG0Tp7wxguWt5\nRWKdm4EABonI6yLypogcG5aAhJCW4c4amYuibmjQ2rHptgHRRLm4XTk77wz06OEfydPUpJE7HTsC\n7TJoNJHM1nT79pptc+vW6FNQROneCeLx8+vL9l6SDgAGADgCwC4AXhORodbyd1NdXf3tfFVVFaqq\nqoLKSggJyHPPAccf7yzvt58uP/ts9n0bGoAddgBOPVU7Udu3d7ZZZV9bm+w/zwfuB0u/fqrw/R42\nY8boW0EmK99yxhnA3nun324fHFu2aFqGKDBGlb7b0q+pqUFNTU1ezhdE6a+AKnJLXwDerBwrALxl\njNkG4DMR+RjAdwGk9JG7lT4hJD+sWOHM33EHcN55wCuvBNvX+sYfegi48ELgkkucbdbSjsrSr6py\n5P7iC/+OXKsbg/Q1BIlksiNzo1L6mzenvqV4DeJp06aFdr4g7p05AAaISH8R6QRgHABv5uknARwN\nACLSC6rwl4YmJSHEl82bVektXgzcd58qRiDZVWCTkGWLylm4UEe9ujtEy8qSXSp2/tNPg8v3zDOZ\no27SsXJl8veoqAAWLEh2vbi/UxBLPwibN0ebbM5r5eebrErfGNMMYCKAGQA+BDDdGLNARKaJyAmJ\nNi8AWCMiHwJ4CcDFxph1eZSbEALgrbeAo47SOPazznIiVGx0ymOPJUeqZPJVDxmihUTcSr+8PFmx\nWqU/a1Yw+Z5+GjjpJGDixGDt3axalazId91Vfffutxh3IZSgD6Js7L575iihsIkycgcIGKdvjHke\nwCDPuqme5V8D+HV4ohFCsrFpk//6rVuBH/5Q0we/+KKzvrY2cwHwDRtSLX230rdunaAhoLa9Xw77\nbDQ0aEpkS9euuux2LeUjKdyoUdGOVYha6XNELiEA3n1Xc60XG35uk6Ym4Ec/cgYbuS19t5Xsx9at\nGrboVvrLlwMzZujyBx9oZE9QpWgHVz37rHOMoLhTJVtsfpzFi9U/n48om8rKlrmjWsrHH0eb0ZRK\nnxBooq4DDii0FLnjp5w++0ynTz6p0733Bq6/XsMev/468/G2btWRqz166LJV/scmgrC/+goYMCC4\n0ncr5WNzDOR2F0Wx2E7WgQM1p1B9PdCtW27HzUZlZbSJ1z78EBg2LLrzUekTkqCQw/Bbip9y8q6r\nqAAmTVLln02Zbd2qnbk77aTL3s7RzZv1ARk082VrMnXW1flb+u5z19UBv/hFy8/hR0VFtEq/ri59\n0Zd8QKVPSIIoRpmGjV8I4pFH6rR79+T1zc3ACSdkPt7EieoCssrWm7unthbo1Su4pd+a/PSzZqX6\numtqNC7fcu+94fvDu3QJL/na1q3AEUdkbnPFFRqpFBVU+oQg1Y1QDNg3E2uNH364Tq3Lx6u4li9H\nWrwFwdNdj7o6HbgVVOm7LWbrMgrKqlVOfnyL921s6VKncEpYfOc74RVTqasDXnst+1vk4sXhnC8I\nVPqEINroibCwinfkSJ2Wlydv7907edlu96s16+1ITDcwqbY2N6XvVp7eN48g7LBD9jY77RRuDvze\nvcPrWLXH2ZiSm0CxfR7p0l7kAyp9UtJUVACvv96ykMJCYxXqccfp9P33M7cfMUKn7rQKFq9l76dE\nL7pIO4L79Amu9OfPdyzxXMIgZ84EvvnGPxuml4oKzZU/blzw42c7Xlg+/TPO0Oknn/hvt6NwDzss\nnPMFgUqflDT19TrAqRiprwd22w2YPFmXV65MLhTi5dZbgx131KjkZWul3nyzWqQ9e+bm/rj9ds2N\nk4vSt4O/vG8vNuXCE08Av/+9zldWAn/7mxZMCYMwQzZtmGq2h8hll4VzviBQ6ZOipq6u9Z1gX32V\num7jRue4zc3auRl15sVs+OWPz2QZu3O7WB/zpk2p/mava8d9jsrK1AFbftTX6zVsbFSZvOkc0inV\n5mZ9QFgXlDdrpv1+3bs78xUV2i5Ths1cqKjQ/oSgrF3r7w5y95PEqeA6lT4paiorteOtNdxwQ+q6\n7bfX427bBvzud5rl8e23W3eesKmvT7aEJ050HlTZEostXaqJzLp2Be65x1n/gx9oOgc37giesrLU\n1Ax+nHKKupOsjF26aLinVebbbZec+tly++0aHZSuc9Z+3169NDd+9+5OeGlYdO2qstrxDtno2dO/\nT2jSJGc+TtXIWlBMjZC2iV/Sq82bnZwu62KWTaqhwVGC9i3EFj+58cbM+9bVAZ9/rvPffOOsf/TR\nzPuVlwez9N9+2/HJl5drH0HnzsmFTvysYxthVFbm+MO95wc0D8/QocklEMOiY0cd07B+feuOM3++\nM5/O0t9tt+Q0GVFAS5+ExkcfAcuWBWu7cKHzij93LjBnjs6/807L3CizZ/tHpeRCU5MOiXezcaOj\nFINafvmkvh647Tb9rtddl5p7J+hgqFde0ZTLAHD33cHPX1amn2++Sf2d3n7bKThur9mCBY6ibmwE\nXnoJePllXfYLY7S/4Rtv+FvPtuKV19cfNq3x68+dCzz4oH4Hi5/SX7xYDYrIK3UZYyL76OlIW6W8\n3JiqqmBtAWN+8Qtn3v41AGPefz/4Oe2+gDH//Gdu8m7dmry/Vw7AmF//WuUEjJk8Obfj54N771VZ\nnnwyWV7LggXG3HRT+v3//GdjundP/c4TJhjz1FPp97PtDj3UmOZmnV+92r+Nex4w5vPPU9cBxixZ\nknqeSZOc7eedl7p982ZjLrggvZxhMXq0Mc8/H6wtYEzXrsnL3s8NN6Tu99OfGrPHHsY0NQU5B4wJ\nSQ/T0iehUV+fWx5yr5Vqrbxchu7b0adA7pZZpvP066fTVavUuuzXLx5+WStzurjvwYM1tDIdEyc6\ndWLd/O//agrkdFir3hjtMO3XL/j1TmeV+11/99ua3/aOHTWKKN9UVOT2f9prr/TbzjjD39KvrdXr\nHlWxFguVPgkFm8LAr+B0Oryx4L9OJOYeOVI79ILgVsS5RvEsWpR+m1U4998P3HSTujT+/W9Vmpn2\nyzf2+rYmp40fQQcH2bEAy5erX/1Pf8q+T7riJn4+/f/8x5kvZLRUeXn62Ho39n+/fLn+nx96KLVN\n166O0n/8cec7+uUWigIqfRIK1of73e8G38cbYnfTTc78+ecHO4Zb6eeaO2eqqyLECy8kb/Mq1UWL\ngPfe01j3Cy7I7TxhYhWh7cB8992WH8Oyzz7B/cpe69cdJeRW7l26OA8Sa+lffXXyvn4PLnd66ylT\ngsmUD7p1C2bp2/+9ffPyDhC7807trLX/0x/8QEtQAnr8QqT/oNInoWAHn+SieMMYOu9W+rlmyXTv\nO3p08javFeouPBK2lZ0L9jqvWgUMH64KO1e8Sv83vwm2X7YatFaWbdv0+p1ySvL2730vednvOlrZ\nBg7MzYAImwED/ENKvdTX63/D676xD7rTTkvNDGofELT0SdEh4mQ8tBEgzzyTmrzLix0Mdc89uflN\n163Tc4oAjzyi6/xK+QUl3Q339tsaqulm+HBn/tVXs3/HfGEt6xtuCB4p5cUqVmuZe3P0pGPgwMy5\ncOw1qa1VS98bP++t2HXllcnLK1c6CrGQLjRAFbU75DId9fV6Tbz/F1uboUsXHefwl784Ro51G33w\nQf6jkPyg0ietwrpFrrnGWZfNQnIrq2yVnNy4s0Ra10xDg3PuXPOlHH+8phq2fQH/+Y9mgrSx6uvX\nO2GNNtWBpZDWvqW14wZWrtTPMccEa//cc8CSJTr/6aeaEthd/MM+gNesUWV25ZXJOY0GDtSpdYG4\nQxoBHTA2eHDu3yMf9O8frOO+vj41e+iJJ2r6haVLVdHbusUW+zbc0KCun6gJpPRFZIyILBSRRSIy\n2Wf7WSKySkTmJj4/CV9UUixkU/p+NVf98Lpr3BEr1mpqaNARkUDuln5Tk97c1m0xcKBaq+7Mhwcf\nnH7fQpNLp7kb9/fr0ye4m628XDslAe3EPeWU5H3t77p6tbbt2NH5bQCnbbrRtuvX63HjQLduwd7m\n6utT3xgHD9Y+EqvQvZ3kNpBTJKaWvoi0A3ALgGMBDAUwXkT8nsfTjTH7JT73+GwnbYT165Nv9rlz\nk7fbV/PDDnPcMbZ0H5CsnGfPTt73Rz9y5r2WlruTr107PW5DgxMuZ61yALj0UuDMMzN/j+nTk5cr\nKvTBct11zjqroHbZJbltr16Zj20RUQvZsnhx6/sybHnATAXOM3Hiian+9ZbgTsdw661q2QIafZVu\nNGunTsllKX/3O2d+/vz45Dfy+uHT4U2FAaR2rnsHmW3bpi4hY/wznuabIJb+SACLjTHLjDFbAEwH\nMNanXYgZrUmc8dZZXbBAp8OGqUKxVrD79d2O1ARUUfz3f2uJuBUr1M2ybZt+HnzQuSm8Sr+5WdsC\nyTfLQQcBYz3/yPvv108mOnVSOSwdOgA//rHOWyt6xx315txrL+D55zMfLx3uB1u2GrXZGDTIyQra\n0uIhU6akRiu1BHcStVdfTd7mjsRy09QEnH46cNdduvzss842Y7Qz2FrChSRIqgnAsfS3bXMyg3r7\nI2zuovvuc9atWROOnC0hiNLfGYC75s6KxDovJ4vIuyLysIj09dlO2gCff57qi7Vul44dHQvJ+2rs\nTm/gzg45e7ZaQvaNwH78brr6ekfRecM9rbVlHy6ZMi7aMDt37hqLDaFzuyUsLe28tX7txkYN+2zp\nsRoa9DqGEeYXRuSUtfQ3bkytypXtLcT6tT/6yLkWfllDC0VQS3/5cuf/m67EpO3kta6xQhMk4Zrf\n38P7HH4awD+MMVtEZAKAvwHw7R6qrq7+dr6qqgpVVVWBBCXxoH//1HU2pv7884E339SbxWtJ2mgb\nwLm5va4dN35Kv67OiQjxVmEaP17fEg44QDuKM7029+6tr+B+r+avv65Tt9vBss8+Wr5v3jxHniAh\nd9YtdfHFTk77urrcqyXZwWs9ewKXXOI/sjZKrKXv9z2yXRdbN3bDBuCppzR+3UbCxIEgmUQBzSNk\nXX822+tvf5vcZrfdgCFDcvPf19TUoMYWDwiZIEp/BQC3R7MvgKTB9sYYdxzBnQCuTXcwt9InbYtT\nTlFl2tCgPt0RI5xEaoDTeeVnYXvx5l8HnEiJZ54Brr8+eZu7+HRjY/bc6hs2+FuWthP6Jz6hCH37\nav/Fiy+qT3zDhmBK31rV7tTMtbW5K317LcvKkvsdCoVNl2y58EInRUK2wV6DB2vU1WWXOb9znCx9\nv/9fOmzYco8e/m6pnj31jcbb95UJr0E8bdq04DtnIYh7Zw6AASLSX0Q6ARgHtey/RUTcwzbGAvgo\nNAlJ0dC5s94sU6aoUvP6nG0q3yA3d3m5ZmS0xk5Dgyq6Dh10X2+op1vJNDamt/TffFOnZ5+t8ngf\nPt5Xcz+sO+LahGkzY4bjz/XDxmW7yxnmUqTDErd8/iLJSi7IYCY/XntNp3FS+p06qQsqmxuuvj64\nzK3NAhsWWZW+MaYZwEQAMwB8CI3SWSAi00TkhESzC0VkvojMS7Q9O18Ck3jQp0/q4JsuXbSDauNG\nfe3dfXeNoLFl7ezIT3uj2ME5fvlKyso0Nv6oo3TZuoseeUS3LV2qVrafZVlbm95/euihOv30U7VS\nvUr/4Yezu00OP1yn9tzHHps+tBNIfthZWlvtKy643WwbN2pE1M9/7tTjzYR9YNx5p07jpPTT9St5\n8XMRpmOvvfReeOEFfWv8619bLWaLCFRExRjzPIBBnnVTXfOXA7g8XNFI3HDHzd96K3DyyY7r4sEH\ndd5ayuvWady7Tch16aXOSFwbW3/++Zpl0L4eu/He/O6Ebl26qNU0YoSTB8danaNGqb88qPLwths5\nMjk01I+KCr2BP/ggcztrJXpdHaee2nKr+OijW7ZfvhgwwHE7tW+vIbfusNtMeF0huSjQKLCduZk6\npXN5UHXu7Azy83Z8RwlH5JLAuK0emw7WWtQ2xNH60mfPTr2BRfQm+sMfVCFaZejnF3f75N3VkXr0\ncG4yvyiWzp01tbCNMHI/qPysa79CHUHwjsL0w/ZdbN6cHLfetau6rnLB7u/twC40Cxc687lGp7h/\n9ylTtK8mLpY+oFFeDz6YuU3cHlRBoNIngamr0xv7//7PiZe3oZg2N771W27cqO4dy8knq2Vu/drv\nvKMPjlmz/P3vbivwo4+ckbsPPJBZ6e+5pzNuAEh+UN17rzNvR0u2NHTRRiNl8tNahVBWpm8FPXtq\nqod99829CpeN/b4nZsMe7e/+wQfZSzR6+clPnPrEV1+t1zJOSh9wyk+mI0hQQtyg0ieBqa9XxfWz\nnzmWuE1hYBW3Wwm6lf7RR+v+Vsnam/vAA/3PtcceOh0xQl0htbXAeefpq7bd1+8NwZ0LBkhW+u7O\n09b6U3fYQd8SbA4evweQVQjl5XruPffU7zN8eO55gmprgaqq+MR6exk2LHfZtttOB2q5iZvSz0Yu\nHblxgYXRSWAmTHCKhHuxnbpuRe9Wyh07am1Xa2EPGZL5XDat7qefalI0wEnVa28yv0gbb+Uud9id\n7VAFwnGTVFQ4x/ez+G+80TnnXXc516NXL33D8Ua/ZOKYY1o+AjfOeP3lxWQ1P/usuh6LSWaAlj7J\ngXR+6PXrtQMVSM5G6Vb6NsPivHmq8C+7LPO5JkzQjI5upWhjwq3S90vO5R4C36NHsqU/dKjmN1+7\nVjti0+WHCUp5uWOx+yl9d1m/5593rkdL88S3NoVDPqitdUY4t4QuXfQNzhJ5kfAM/OEP/uM1LA8/\nrNOW9gsVCip90mrcg4zc/nm30rdpDbZu1ZGt2RJNtW+vLh53h649hu009hvc5F7Xs2eypV9Xp/50\na+XnOjjKS+fOqX0ZmYrI2OsRRgqEuFBRETz5XDrcob/ZBtVFSe/embOp2t88TjIHocjEJcWE289t\nXRObNuX2OuwuP+cN7fQLe3RXwOrRI1np33hjywZFpaO52cnwaRVAhw46iGrChNT2Nu9Orti0D97q\nXm0Fd4RVkKioqCgry1w3oaVFbAoNlT4JzNFHAzNnZm9nsyS6X3t799YMlrbARlBuvtlxBZ16avb2\nxx/vFJ7u2jV1cE1L4+P9cJcZbG5Orl9rXWEPPOAkgXMP+lq1KriFbPsp7CCmtoZ71GucQlKzDc7a\nY4/4RVMFgUq/hFm1KrcKUK0NTysrUys41xzi6YqFpDuO9Qt37pzqb7bjC8LA7R4yxhlP0NTk+N8r\nKpx27hqz5eWafTNIR66tkNWW3EJuClV6MhuZlP7bb6sBU4gat62FSr+E2XHH7HHIblobnlZWpm6Q\nbHnug3DqqZq734/dd9cCLl26AF98oeusYjnrrNaf2+K2Stu103w+gPYd2Nq/lZVO9kW7HXCu4xNP\nZD+PtfRb6zuPK+PGaXTSSScVWpJk0in95mYNvX3mmXDSXEcNQzZLnFyGg7c2N4p9S/AWkc6GX+Fu\nv3w9lspKTeJVXZ2cwbGyEth//9zOnQk7lqCpSVM6W9+7ux+hokKvmdeit51/6UJg3TQ2arreYosH\nD8r++2vm0riRTum719HSJ0WD7VTMJfNfa5W+3TdXv22miJhMbL+9jvRcv16t77Djqe336dRJHyg2\nt9BFFzltslmCQTLm/v3v8cnQWEp4lf6cORoM4F5XjJY+lX6JMny4TnOJMW6tT98q7yCdwW7OPjuz\nZZ+OXXdVt84DD/gXsG4t3/mO05FXUeEkQ7OundGjNSFZJmzIZyYWLQJeeaXlcpKW4VX6I0cCkyYl\nv8kNGpS6X9yh0i9xcvETt9bSt0o/1xula9dgkTtebLRMU1N+EmO1awecc47OV1amugKeeCLz9bIV\nx4JQbKM+2wLp3DtupU9Ln8SexsbkKJA779TlbOmEZ8/OLWWxH1G7KKwbaeVKtbrzmcO+S5fUOPxs\n18qOKLaFzjPRr1+LxCKtIJ3SP+yw6GUJEyr9EsNbccryxz9m3s9Wucw13NJNS33zLWWHHbRoSrt2\nqvDdKZrDpmdPddVceKFm87znnuwhlr/8pU5tMRk/bAewrUtAosMqffsb7LefTteu1VHltiJascHo\nnRLDPWT8kEOc8oFr1mTeL2hisExEbemLaJI2GyHTt2/+zmUt8f79k0MzM2HHH2Ty62/Zou0yFfIg\n+aFDB72c5tV/AAAWVUlEQVRftmzRznr3vbPDDsnJBYsJWvolhrvak7sT95lnMu8XJLQwGy1NNNYa\nunRxEp+1NtdOJuwgt5a4vzINGCvG1L1tia1bnfrH7hrFcQwxDQqVfonhziyZizLZZx/gz39u3bkn\nTMheczRs3A+2fCr9jRt1mm70cDouuMDJUOpHMRbpaEuccIIzOE5ECwgVO4GUvoiMEZGFIrJIRCZn\naHeKiGwTkf3CE5GEiQ0nBFLDNdO5X7Zt0xh0OxippYhEn4bWfb58Kk/bX5HrwLM+fdI/CBsb9SFN\npV84evbU36epSV2cbWFUdFalLyLtANwC4FgAQwGMF5HBPu0qAVwAYFbYQpLwcFdsOu44TWtrSx+m\nGyg0fryOcPXLXx933KlxWxL2GZRjjtFptuIwXjLldykr07BTuncKhy2Obou9D07RfMVHEEt/JIDF\nxphlxpgtAKYDGOvT7ioA1wLIkIGaFJraWif/zGmn6aur9Xmni0Z44QWd5qrQ4oB9szFGyzzmixNP\n1HMcdVRu+2XL5AjQ0i8k9vd56ildHjLEySJbrARR+jsDcGdoWZFY9y0iMhxAX2PMv0KUjeSBL75w\nRqZa14cdYLJ+vZb3e/ddp/3WreGmI46aTEUw4kBZmWY7tYnhLO4RuK0JkyWtI8hDudgIErLpF238\n7XNORATAjQDc+QvTRihX24BvAFVVVaiqqgogAgmLVauAI48ETj7ZWWcLV9TVacgh4FgydsDRoYdG\nJ2OYjB0bTuRRvigrA558Uj/2mm/a5KR0AOKVY77UKCtzXKJR3gM1NTWoqanJy7GDKP0VAHZxLfcF\n4C4/vR3U11+TeAD0AfCUiJxkjJnrPZhb6ZPo2bIFGDYMuPhiZ13HjkBNjWZy9FJbCxxxBPDvf0cm\nYqjssUfro47yiZ/rxlu7d+TIaGQhqZSXa02GHXcEHnkkuvN6DeJpQTLzBSSIe2cOgAEi0l9EOgEY\nB+Bpu9EYs9EY09sYs7sxZjdoR+6JfgqfFJ50OWgqKpI7eV96Sf/s+chZQxz8Omk//zx5uQOHUBaM\nsjItYvP118WZZ8ePrErfGNMMYCKAGQA+BDDdGLNARKaJyAl+uyCDe4cUlnRKvLIyOZxz1ChgypT8\npCQmDn5Kf/Xq5OV85gwimSkrA+67T+fbyn0QyIYwxjwPYJBn3dQ0bY/2W0/iQVBLH9BOXFr6+cVP\n6Xs7DuNaTrAUcP8+baVDnSNyS4i6OmDBAv+88tttlxpBcu+9Gt6ZLS8PaTl+LgOr9O3D1na0k+hp\niwYPlX4J8dlnOvX7I3frln6/OEe/FDsDBjg1g230TkMD8OMfA8uWaTlLxj4UDmvpDxxYWDnChEq/\nBGmL1kuxIgKcfrq+fdk+lYYGTc/Qq5dmBu3cubAyljJW6duCPG0BKv0S4uuvdZrrsP7Ro8OXhSRT\nV6cVwkS0wDpTL8QDayDFfZBfLlDplxD19arA26X51dPVkL3xxvzJRBR3bvbPPqPSjws239SwYQUV\nI1So9EuIVasypxdOFw+e7iFBwsOdU3/NGir9uGANoa5dCytHmPB2LiHOPTdzJM5vfgNMmpS8rhCF\nT0qRhQud+dWr2e8SF2x9hFzrJMQZjvUrMWyFJz8uv1yn118fjSzEH1r6JJ/Q0i8RbDhgptJ8JB4Y\nQ6VP8geVfolgh/JfcUVh5SD+3HGHTm2xDndKDFJYrroK+P73Cy1FeFDplwi1teqftxWeSLyYMEEt\n/F0S+WyLuUhHW+OKK4BBg7K3Kxao9EuEt94CFi8utBQkKFT6JF9Q6ZcIZ59daAlIEGx47OmnF1YO\n0nah0ickRvTpo1NbypKQsKHSJyRG7LRToSUgbR3G6ZcI554L9OtXaClINk49VctTEpIvaOmXCFu2\nOEXPSXwRobVP8guVfonwzjttayg5IaRlUOmXCJ07t61YY0JIywik9EVkjIgsFJFFIjLZZ/sEEXlf\nROaJyKsiMjh8UUlraGhgEi9CSAClLyLtANwC4FgAQwGM91HqfzfG7G2M2RfAHwAwA3vMYIFzQggQ\nzNIfCWCxMWaZMWYLgOkAxrobGGPcmUIqAWwLT0QSBnV16YukEEJKhyAhmzsDWO5aXgF9ECQhIucD\nmASgI4CjQ5GOhMKbbwKbNtHSJ4QEU/risy4lM4gx5jYAt4nIOAC/BXC238Gqq6u/na+qqkJVVVUA\nEUhrOPRQnTJdLyHFQU1NDWpqavJybDFZMjuJyEEAqo0xYxLLlwIwxphr07QXAOuMMd18tpls5yPh\nI4nHNi89IcWJiMAY42eA50wQn/4cAANEpL+IdAIwDsDTHoEGuBZPALAoDOEIIYSES1b3jjGmWUQm\nApgBfUjcbYxZICLTAMwxxvwTwEQRGQVgM4B1AM7Kp9AkONvYpU4IcZHVvRPqyejeiZy6OqCyUud5\n6QkpTqJ275AY8sADWm0pGxMn5l8WQkjxQEu/SDnsMOCNN7Jb77YTd9Mmx+InhBQXtPQJOnfOrT0V\nPiEEoNIvWl5+Wad//ata89XVjlVvWb06aqkIIXGHSr/Iee89nT71VOq2VauAIUPYgUsIcaDSL0Le\nesuZt8p+wYLUdl99xdQLhJBkqPSLkEMO0elFFwGffqrzTU2p7T79FNhuu+jkIoTEHyr9IsamMfr+\n9/2319UBw4ZFJg4hpAig0i8y3P55a8W7C54bA8ycCTQ3A0uWANtvH618hJB4EyTLJokRmzfr9IEH\ngPbtgeuvB44/XnPl//73QGMjMHo08OKLwIYNLLJNCEmGln6RUV8PdOsGnHaaLk+aBAweDFxzDdCj\nh24H1OJvbAR2261wshJC4geVfpFRV5c+ImftWuCII3RehHVxCSGpUOkXGfX1mcsefvSRTrdu1bYs\nnEIIcUOlX2RkKnDeq1dyOxZDJ4R4odKPIRs3qntGBFi3zllvDLDvvs4oXC82fh8AzjxT3Tu09Akh\nbqj0Y8jKlc78l18687aTNp0if+wxZ762lpY+ISQVKv0YsmqVM795s0bhABqCCQDdu/vv18ETgLtp\nE5U+ISQZKv0Y8vOfO/P77aeW/dq1wKJFqdu9DBzozK9cSfcOISQZKv0YIgJ88EHyuvXr1V0zZgww\ndWr6fT/+GLjySmeZlj4hxE0gpS8iY0RkoYgsEpHJPtt/JSIfisi7IjJTRPr5HYdk5733NGNmt27J\n69etA15/PVgxFOsOAmjpE0KSyar0RaQdgFsAHAtgKIDxIjLY02wugP2NMcMBPAbgD2ELWioMH67T\n3r2B665z1j/xhI66zRSjb7GjdQGgHd/lCCEugqiEkQAWG2OWGWO2AJgOYKy7gTHm38YYa1/OArBz\nuGKWHp06AZdc4izbzt0glv7ee+vUPkAIIcQSROnvDGC5a3kFMiv1cwE81xqhSpHaWuCKK9Jvt+UR\ng1j6lo4dWycTIaTtESTLpl8Fdt8CfCJyOoD9ARyZ7mDVNgk8gKqqKlRVVQUQoe0zfz7wu9+lrp85\nE7jlFqdC1u67Bzve7NnAjjuGJx8hJDpqampQU1OTl2OLyVJAVUQOAlBtjBmTWL4UgDHGXOtpNwrA\nnwAcYYxZk+ZYJtv5SpWXXgJGjXKW3ZfpxReB731P5994I3nkLSGk7SMiMMb4GeA5E8S9MwfAABHp\nLyKdAIwD8LRHoH0B3AHgpHQKn2Smtjb9Nrcfn0VRCCGtIavSN8Y0A5gIYAaADwFMN8YsEJFpInJC\notl1ACoAPCIi80TkybxJ3EapqwP22QdYtsype2tx+/GHDo1WLkJI2yJQ5SxjzPMABnnWTXXNfy9k\nuUqO2lpgxAhgl11StwWJ2CGEkCAwijsG3HYbMGFC+tGzNtfOoYdGJxMhpG2StSM31JOxI9eXMWOA\nF17QXDl9+vi3sRWzJJSuHEJIMRFmRy4Lo8eA9u112qNH+ja5xOcTQkg66N6JAV276rRTp8LKQQhp\n+1Dpx4BOnYB77y20FISQUoBKv8A8/LCOtqX7hhASBezILSAbNjgplJcsAfbYo7DyEELiSdQjckme\nWL/emafCJ4REAZV+AZkypdASEEJKDbp3Cog75p6XhRCSDrp3CCGEtAgq/QKxZUuhJSCElCIckVsg\nbCfupk2Z0yoTQkiYUOkXiLo6zahZWcksmoSQ6KB7pwAsWQLsthuVPSEkeqj0C8CSJTrlKFxCSNRQ\n6eeZ+npg3jydWr76SqdMsEYIiRoq/Txz443AfvslW/V33KFTWvqEkKih0s8zGzakrmtu1ukBB0Qr\nCyGEBFL6IjJGRBaKyCIRmeyz/XAReUdEtojIyeGLWby43Tp/+Qvw5pvAihW63IGxU4SQiMmqdkSk\nHYBbABwD4EsAc0TkKWPMQlezZQDOAnBxXqQsYurrNVLn00+B885Tl84RRwAjRwKnnlpo6QghpUYQ\nS38kgMXGmGXGmC0ApgMY625gjPncGDMfADPIeKivB66+WmPyAY3P/5//AaqrgT33LKhohJASJIjS\n3xnActfyisQ6EoD6ei1o3ru3s47x+YSQQhHEq+yX2a3FFn11dfW381VVVaiqqmrpoYqCujpV+vff\nD1xxBfDYY4zaIYRkpqamBjU1NXk5dtbUyiJyEIBqY8yYxPKlAIwx5lqftvcCeMYY83iaY5VcauWD\nDwauvx445BDgmWeAk04C5s8Hhg4ttGSEkGIh6tTKcwAMEJH+ItIJwDgAT2eSLwzB2gKHHQbMmgV0\n7arLvXrp1JZIJISQqAlUREVExgD4E/Qhcbcx5vciMg3AHGPMP0XkAABPAOgGoBHAV8aYvXyOU1KW\nvi2Ssm2bzm/bBnzyCfDd7xZWLkJIcRGmpc/KWXmioUF9+QCrYhFCWgcrZ8WcOXOcCJ199y2sLIQQ\n4oZKPw989RUwZoxa+HPnFloaQghxYCKAEPnmGx15W1vLWHxCSDyhpR8iU6YABx4IvP02Y/EJIfGE\nSj9E1q7V6eef09InhMQTKv2QeOgh4F//0syZ8+ZR6RNC4glDNkPCxuTfcouWQxw/XjNpEkJIa2Gc\nfszYvBno3Fnn7UAsQggJC8bpx4zbb3fmqfAJIXGGSj8EnnwSmDABeOWVQktCCCGZYZx+CNTVAWef\nDRx0UKElIYSQzNDSbyVbt2rahR12KLQkhBCSHSr9VrJ+vXbi7rFHoSUhhJDsUOm3gs2bdSDWTjsV\nWhJCCAkGlX4rOP104Mgjgb33LrQkhBASDHbktoKVK4FnnwWOOKLQkhBCSDCo9HPk9dc1iyYAfPEF\nSx8SQooLKv0c2LABOPpo/QDAsGFA//6FlYkQQnIhkNJP1Mi9CU6N3Gs92zsBuA/A/gBWA/iRMebz\nkGUtOOvWAX36AM8/X2hJCCGkZWTtyBWRdgBuAXAsgKEAxovIYE+zcwGsNcZ8F/pwuC7d8a66yv/z\n5pst/xJhU1NTk7Lu8ceB66+PlzvHT844UgxyFoOMAOUMm2KRM0yCRO+MBLDYGLPMGLMFwHQAYz1t\nxgL4W2L+UQDHpDvY5s2pn7lzk/PXFBq/P8Jvf6sDsa64Inp50lEsf9hikLMYZAQoZ9gUi5xhEsS9\nszOA5a7lFdAHgW8bY0yziKwXkR7GmLXeg111VeoJnnoKuPvuoCIXhg0btDJW376FloQQQlpOEKXv\nlzfSmx/Z20Z82qSle3d175x4YtA98svHHwPvvJO8btWqeLl2CCGkJWTNpy8iBwGoNsaMSSxfCsC4\nO3NF5LlEm9ki0h7ASmNMb59jtb1k+oQQEgFh5dMPYunPATBARPoDWAlgHIDxnjbPADgLwGwAPwTw\nst+BwhKaEEJIy8iq9BM++okAZsAJ2VwgItMAzDHG/BPA3QDuF5HFANZAHwyEEEJiRqTlEgkhhBSW\nyBKuicgYEVkoIotEZHJU500jy2ci8p6IzBOR/yTWdReRGSLysYi8ICLbu9rfLCKLReRdERmeR7nu\nFpGvReR917qc5RKRsxLX+WMROTMiOaeKyAoRmZv4jHFtuywh5wIRGe1an9f/hIj0FZGXReQjEflA\nRC5MrI/NNfWR8YLE+lhdTxHpLCKzE/fMByIyNbF+VxGZlbguD4pIh8T6TiIyPSHnWyKySzb58yzn\nvSKyNLF+rojs7dqnIPdR4hztEvI8nVjO//U0xuT9A324LAHQH0BHAO8CGBzFudPIsxRAd8+6awH8\nJjE/GcDvE/PHAXg2MX8ggFl5lOswAMMBvN9SuQB0B/AJgO0BdLPzEcg5FcAkn7ZDAMyDuhJ3TfwP\nJIr/BIA+AIYn5isBfAxgcJyuaQYZ43g9yxPT9gBmJa7RQwB+mFh/O4AJifn/AXBbYv5HAKYn5vf0\nkz8COe8FcLJP24LdR4nz/ArAAwCeTizn/XpGZekHGeAVJfYmceMeYPY3OPKNhaaYgDFmNoDtRWTH\nfAhljHkdwLpWynUsgBnGmA3GmPXQvpgxCJE0cgL+4b1joX/QrcaYzwAshv4f8v6fMMZ8ZYx5NzFf\nC2ABgL6I0TVNI+POic1xu571idnOUCVjABwF4LHE+r8B+L5LTveAzUTGKpyURv58yrktsZzuehbk\nPhKRvgCOB3CXa/XRyPP1jErp+w3w2jlN2ygwAF4QkTki8tPEuh2NMV8DeiMCsCGnXtm/QLSy9w4o\nl72mhZT3F4lX5LtcLpN08kT6nxCRXaFvJ7MQ/LeO9Jq6ZJydWBWr65lwRcwD8BWAmVDrd70xxipV\n9zmTBmwC2CAiPTLInzc5jTFzEpv+X+J6Xi8iHb1yer5DFL/5jQAuQWJMk4j0BLAu39czKqUfZIBX\nlBxijDkA+pT9hYgcnkGeuMluSTcgrlDy3gZgD2PMcOjNdr1LLj95IpNTRCqh1tFFCWs66G8d2TX1\nkTF219MYs80Ysy/0bWkk1NWU7pyxkVNE9gRwqTFmCIARAHpC3Xp+ckbym4vIfwH4OvGWZ88lPucN\n/XpGpfRXANjFtdwXwJcRnTuFhHUHY8w3AJ6E/oG/tm4bEekDYFWi+QoA/Vy7Ry17rnIV5FobY74x\nCScjgDvhvGIWVM5ER9ijAO43xjyVWB2ra+onY1yvZ0K2jQD+DeAgAN1EkzJ6z/mtnKIDNrc3xqzL\nIH8+5RzjerPbAvXvF/p6HgrgJBFZCuBBqLvmJqh7Kb/XM+yOiTSdFe3hdDJ1gnYyDYni3D6ylAOo\nTMxXAHgDwGho597kxPpL4XTuHQ+no+cg5LEjN3GOXQF84FrOSS4kd0DZ+W4RyNnHNf8rAP8wyR1N\nnQDsBqfjMZL/BNRfe4NnXayuaRoZY3U9AfRCoiMTQBmAVxPX6yFoKnVAOx7PS8yfD6fjcRxSOx6T\n5I9Azj6JdQJ1q1wdh/soca4jkdyRm9frGfoXyPDFxkAjExZDX7UiO7dHjt0SN8Q8AB9YWQD0APBi\nQsaZ7h8Ymlp6CYD3AOyXR9n+AX1KNwH4HMA5iT9cTnIBODtxnRcBODMiOe8D8H7i2j4J9Zvb9pcl\n5FwAYHRU/wmoNdXs+r3nJs6Z82+dr2uaQcZYXU8AeyVkezch1xTX/TQ7cV0eAtAxsb4zgIcTsswC\nsGs2+fMs50uJ3/T9xLUtL/R95DqPW+nn/XpycBYhhJQQkQ3OIoQQUnio9AkhpISg0ieEkBKCSp8Q\nQkoIKn1CCCkhqPQJIaSEoNInhJASgkqfEEJKiP8PCon1VmMovHYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1196787d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Average of rewards of latest 100 episode vs episodes\n",
    "plt.plot(get_consecutive_100_episode_average_list(result['episode_rewards']))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best 100-episode average reward was 0.73\n",
      "The average reward was 0.383906327853\n"
     ]
    }
   ],
   "source": [
    "print \"Best 100-episode average reward was {}\".format( \\\n",
    "                max(get_consecutive_100_episode_average_list(result['episode_rewards'])))\n",
    "\n",
    "print \"The average reward was {}\".format(sum(result['episode_rewards'])/len(result['episode_rewards']))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}