16 - Project - Capstone Project - Reinforcement Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Capstone Project - Reinforcement Learning\n",
    "### Goal\n",
    "- Learn Object Oriented Programming techniques\n",
    "- Build your own Machine Learning model from scratch - Reinforcement Learning\n",
    "- Teach it to pickup and deliver items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Reinforcement Learning?\n",
    "- Reinforcement learning teaches the machine to think for itself based on past action rewards.\n",
    "![Reinforcement Learning](img/ReinforcementLearning.png)\n",
    "- Basically, the Reinforcement Learning algorithm tries to predict actions that gives rewards and avoids punishment.\n",
    "- It is like training a dog. You and the dog do not talk the same language, but the dogs learns how to act based on rewards (and punishment, which I do not advise or advocate).\n",
    "- Hence, if a dog is rewarded for a certain action in a given situation, then next time it is exposed to a similar situation it will act the same.\n",
    "- Translate that to Reinforcement Learning.\n",
    "    - The **agent** is the dog that is exposed to the **environment**.\n",
    "    - Then the **agent** encounters a **state**.\n",
    "    - The **agent** performs an **action** to transition to a **new state**.\n",
    "    - Then after the transition the **agent** receives a **reward** or **penalty (punishment)**.\n",
    "    - This forms a **policy** to create a strategy to choose actions in a given **state**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What algorithms are used for Reinforcement Learning?\n",
    "- The most common algorithm for Reinforcement Learning are.\n",
    "    - **Q-Learning**: is a model-free reinforcement learning algorithm to learn a policy telling an agent what action to take under what circumstances.\n",
    "    - **Temporal Difference**: refers to a class of model-free reinforcement learning methods which learn by bootstrapping from the current estimate of the value function.\n",
    "    - **Deep Adversarial Network**: is a technique employed in the field of machine learning which attempts to fool models through malicious input.\n",
    "- We will focus on the **Q-learning** algorithm as it is easy to understand as well as powerful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How does the Q-learning algorithm work?\n",
    "- As already noted, I just love this algorithm. It is “easy” to understand and powerful as you will see.\n",
    "![Q-Learning algorithm](img/QLearning.png)\n",
    "- The Q-Learning algorithm has a Q-table (a Matrix of dimension state x actions – don’t worry if you do not understand what a Matrix is, you will not need the mathematical aspects of it – it is just an indexed “container” with numbers).\n",
    "- The agent (or Q-Learning algorithm) will be in a state.\n",
    "- Then in each iteration the agent needs take an action.\n",
    "- The agent will continuously update the reward in the Q-table.\n",
    "- The learning can come from either exploiting or exploring.\n",
    "- This translates into the following pseudo algorithm for the Q-Learning.\n",
    "- The agent is in a given state and needs to choose an action.\n",
    "\n",
    "#### Algorithm\n",
    "- Initialise the **Q-table** to all zeros\n",
    "- Iterate\n",
    "    - Agent is in state **state**.\n",
    "    - With probability **epsilon** choose to **explore**, else **exploit**.\n",
    "        - If **explore**, then choose a *random* **action**.\n",
    "        - If **exploit**, then choose the *best* **action** based on the current **Q-table**.\n",
    "    - Update the **Q-table** from the new **reward** to the previous state.\n",
    "    - Q[**state, action**] = (1 – **alpha**) * Q[**state, action**] + **alpha** * (**reward + gamma** * max(Q[**new_state**]) — Q[**state, action**])\n",
    "    \n",
    "#### Variables\n",
    "As you can se, we have introduced the following variables.\n",
    "\n",
    "- **epsilon**: the probability to take a random action, which is done to explore new territory.\n",
    "- **alpha**: is the learning rate that the algorithm should make in each iteration and should be in the interval from 0 to 1.\n",
    "- **gamma**: is the discount factor used to balance the immediate and future reward. This value is usually between 0.8 and 0.99\n",
    "- **reward**: is the feedback on the action and can be any number. Negative is penalty (or punishment) and positive is a reward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Description of task we want to solve\n",
    "- To keep it simple, we create a field of size 10×10 positions. In that field there is an item that needs to be picked up and moved to a drop-off point.\n",
    "![Field](img/Field-ReinforcementLearning.png)\n",
    "- At each position there are 6 different actions that can be taken.\n",
    "    - **Action 0**: Go south if on field.\n",
    "    - **Action 1**: Go north if on field.\n",
    "    - **Action 2**: Go east if on field.\n",
    "    - **Action 3**: Go west if on field.\n",
    "    - **Action 4**: Pickup item (it can try even if it is not there)\n",
    "    - **Action 5**: Drop-off item (it can try even if it does not have it)\n",
    "- Based on these actions we will make a reward system.\n",
    "    - If the agent tries to go off the field, punish with -10 in reward.\n",
    "    - If the agent makes a (legal) move, punish with -1 in reward, as we do not want to encourage endless walking around.\n",
    "    - If the agent tries to pick up item, but it is not there or it has it already, punish with -10.\n",
    "    - If the agent picks up the item correct place, reward with 20.\n",
    "    - If agent tries to drop-off item in wrong place or does not have the item, punish with -10.\n",
    "    - If the agent drops-off item in correct place, reward with 20.\n",
    "- That translates into the following code. I prefer to implement this code, as I think the standard libraries that provide similar frameworks hide some important details. As an example, and shown later, how do you map this into a state in the Q-table?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}