밑시딥 5장 - 오차역전파법.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![밑시딥 교재](http://www.hanbit.co.kr/data/books/B8475831198_l.jpg)\n",
    "\n",
    "> 작성자 : [Minsuk Sung](https://github.com/mssung94)  \n",
    "> 연락처 : mssung94@gmail.com  \n",
    "> GitHub : https://github.com/mssung94\n",
    "\n",
    "> 본 내용은 **대학생 빅데이터 동아리 `BOAZ`의 딥러닝 스터디 자료**입니다. **상업적인 목적으로 사용하는 것은 절대로 금합니다.**\n",
    "\n",
    "> 모든 내용은 `밑바닥부터 시작하는 딥러닝(Deep Learning from Scratch),한빛미디어`을 참고하여 만들어졌으며, 책의 내용에 대한 모든 저작권은 `한빛미디어`에게 있음을 미리 밝힙니다. 또한 여기에 사용된 코드의 대부분은 한빛미디어에서 제공해준 GitHub을 바탕으로 재구성하였습니다.\n",
    "\n",
    "한빛미디어 홈페이지 : http://www.hanbit.co.kr/store/books/look.php?p_code=B8475831198  \n",
    "밑바닥부터 시작하는 딥러닝 GitHub : https://github.com/WegraLee/deep-learning-from-scratch  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>5장 오차역전파법<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#계산-그래프-(p148)\" data-toc-modified-id=\"계산-그래프-(p148)-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>계산 그래프 (p148)</a></span><ul class=\"toc-item\"><li><span><a href=\"#계산-그래프로-풀다\" data-toc-modified-id=\"계산-그래프로-풀다-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>계산 그래프로 풀다</a></span><ul class=\"toc-item\"><li><span><a href=\"#문제1\" data-toc-modified-id=\"문제1-1.1.1\"><span class=\"toc-item-num\">1.1.1&nbsp;&nbsp;</span>문제1</a></span></li><li><span><a href=\"#해답1\" data-toc-modified-id=\"해답1-1.1.2\"><span class=\"toc-item-num\">1.1.2&nbsp;&nbsp;</span>해답1</a></span></li><li><span><a href=\"#문제2\" data-toc-modified-id=\"문제2-1.1.3\"><span class=\"toc-item-num\">1.1.3&nbsp;&nbsp;</span>문제2</a></span></li><li><span><a href=\"#해답2\" data-toc-modified-id=\"해답2-1.1.4\"><span class=\"toc-item-num\">1.1.4&nbsp;&nbsp;</span>해답2</a></span></li></ul></li><li><span><a href=\"#국소적-계산\" data-toc-modified-id=\"국소적-계산-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>국소적 계산</a></span></li><li><span><a href=\"#왜-계산-그래프로-푸는가?\" data-toc-modified-id=\"왜-계산-그래프로-푸는가?-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>왜 계산 그래프로 푸는가?</a></span></li></ul></li><li><span><a href=\"#연쇄법칙-(p152)\" data-toc-modified-id=\"연쇄법칙-(p152)-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>연쇄법칙 (p152)</a></span><ul class=\"toc-item\"><li><span><a href=\"#계산-그래프의-역전파\" data-toc-modified-id=\"계산-그래프의-역전파-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>계산 그래프의 역전파</a></span></li><li><span><a href=\"#연쇄법칙이란?\" data-toc-modified-id=\"연쇄법칙이란?-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>연쇄법칙이란?</a></span></li><li><span><a href=\"#연쇄법칙과-계산-그래프\" data-toc-modified-id=\"연쇄법칙과-계산-그래프-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>연쇄법칙과 계산 그래프</a></span></li></ul></li><li><span><a href=\"#역전파-(p155)\" data-toc-modified-id=\"역전파-(p155)-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>역전파 (p155)</a></span><ul class=\"toc-item\"><li><span><a href=\"#덧셈-노드의-역전파\" data-toc-modified-id=\"덧셈-노드의-역전파-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>덧셈 노드의 역전파</a></span></li><li><span><a href=\"#곱셈-노드의-역전파\" data-toc-modified-id=\"곱셈-노드의-역전파-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>곱셈 노드의 역전파</a></span></li><li><span><a href=\"#사과-쇼핑의-예\" data-toc-modified-id=\"사과-쇼핑의-예-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>사과 쇼핑의 예</a></span><ul class=\"toc-item\"><li><span><a href=\"#문제\" data-toc-modified-id=\"문제-3.3.1\"><span class=\"toc-item-num\">3.3.1&nbsp;&nbsp;</span>문제</a></span></li><li><span><a href=\"#정답\" data-toc-modified-id=\"정답-3.3.2\"><span class=\"toc-item-num\">3.3.2&nbsp;&nbsp;</span>정답</a></span></li></ul></li></ul></li><li><span><a href=\"#단순한-계층-구현하기-(p160)\" data-toc-modified-id=\"단순한-계층-구현하기-(p160)-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>단순한 계층 구현하기 (p160)</a></span><ul class=\"toc-item\"><li><span><a href=\"#곱셈-계층\" data-toc-modified-id=\"곱셈-계층-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>곱셈 계층</a></span></li><li><span><a href=\"#덧셈-계층\" data-toc-modified-id=\"덧셈-계층-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>덧셈 계층</a></span></li></ul></li><li><span><a href=\"#활성화-함수-계층-구현하기-(p165)\" data-toc-modified-id=\"활성화-함수-계층-구현하기-(p165)-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>활성화 함수 계층 구현하기 (p165)</a></span><ul class=\"toc-item\"><li><span><a href=\"#ReLU-계층\" data-toc-modified-id=\"ReLU-계층-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>ReLU 계층</a></span></li><li><span><a href=\"#Sigmoid-계층\" data-toc-modified-id=\"Sigmoid-계층-5.2\"><span class=\"toc-item-num\">5.2&nbsp;&nbsp;</span>Sigmoid 계층</a></span><ul class=\"toc-item\"><li><span><a href=\"#계산-그래프-정리-전\" data-toc-modified-id=\"계산-그래프-정리-전-5.2.1\"><span class=\"toc-item-num\">5.2.1&nbsp;&nbsp;</span>계산 그래프 정리 전</a></span></li><li><span><a href=\"#수식-정리\" data-toc-modified-id=\"수식-정리-5.2.2\"><span class=\"toc-item-num\">5.2.2&nbsp;&nbsp;</span>수식 정리</a></span></li><li><span><a href=\"#계산-그래프-정리-후\" data-toc-modified-id=\"계산-그래프-정리-후-5.2.3\"><span class=\"toc-item-num\">5.2.3&nbsp;&nbsp;</span>계산 그래프 정리 후</a></span></li></ul></li></ul></li><li><span><a href=\"#Affine/Softmax-계층-구현하기-(p170)\" data-toc-modified-id=\"Affine/Softmax-계층-구현하기-(p170)-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Affine/Softmax 계층 구현하기 (p170)</a></span><ul class=\"toc-item\"><li><span><a href=\"#Affine-계층\" data-toc-modified-id=\"Affine-계층-6.1\"><span class=\"toc-item-num\">6.1&nbsp;&nbsp;</span>Affine 계층</a></span><ul class=\"toc-item\"><li><span><a href=\"#Affine-계층의-순전파\" data-toc-modified-id=\"Affine-계층의-순전파-6.1.1\"><span class=\"toc-item-num\">6.1.1&nbsp;&nbsp;</span>Affine 계층의 순전파</a></span></li><li><span><a href=\"#Affine-계층의-역전파\" data-toc-modified-id=\"Affine-계층의-역전파-6.1.2\"><span class=\"toc-item-num\">6.1.2&nbsp;&nbsp;</span>Affine 계층의 역전파</a></span></li></ul></li><li><span><a href=\"#배치용-Affine-계층\" data-toc-modified-id=\"배치용-Affine-계층-6.2\"><span class=\"toc-item-num\">6.2&nbsp;&nbsp;</span>배치용 Affine 계층</a></span></li><li><span><a href=\"#Softmax-with-Loss-계층\" data-toc-modified-id=\"Softmax-with-Loss-계층-6.3\"><span class=\"toc-item-num\">6.3&nbsp;&nbsp;</span>Softmax-with-Loss 계층</a></span><ul class=\"toc-item\"><li><span><a href=\"#Softmax-with-Loss-계층-정리-전\" data-toc-modified-id=\"Softmax-with-Loss-계층-정리-전-6.3.1\"><span class=\"toc-item-num\">6.3.1&nbsp;&nbsp;</span>Softmax-with-Loss 계층 정리 전</a></span></li><li><span><a href=\"#Softmax-with-Loss-계층-정리--후\" data-toc-modified-id=\"Softmax-with-Loss-계층-정리--후-6.3.2\"><span class=\"toc-item-num\">6.3.2&nbsp;&nbsp;</span>Softmax-with-Loss 계층 정리  후</a></span></li></ul></li></ul></li><li><span><a href=\"#오차역전파법-구현하기-(p179)\" data-toc-modified-id=\"오차역전파법-구현하기-(p179)-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>오차역전파법 구현하기 (p179)</a></span><ul class=\"toc-item\"><li><span><a href=\"#신경망-학습의-전체-그림\" data-toc-modified-id=\"신경망-학습의-전체-그림-7.1\"><span class=\"toc-item-num\">7.1&nbsp;&nbsp;</span>신경망 학습의 전체 그림</a></span><ul class=\"toc-item\"><li><span><a href=\"#전제\" data-toc-modified-id=\"전제-7.1.1\"><span class=\"toc-item-num\">7.1.1&nbsp;&nbsp;</span>전제</a></span></li><li><span><a href=\"#1단계---미니배치\" data-toc-modified-id=\"1단계---미니배치-7.1.2\"><span class=\"toc-item-num\">7.1.2&nbsp;&nbsp;</span>1단계 - 미니배치</a></span></li><li><span><a href=\"#2단계---기울기-산출\" data-toc-modified-id=\"2단계---기울기-산출-7.1.3\"><span class=\"toc-item-num\">7.1.3&nbsp;&nbsp;</span>2단계 - 기울기 산출</a></span></li><li><span><a href=\"#3단계---매개변수-업데이트\" data-toc-modified-id=\"3단계---매개변수-업데이트-7.1.4\"><span class=\"toc-item-num\">7.1.4&nbsp;&nbsp;</span>3단계 - 매개변수 업데이트</a></span></li><li><span><a href=\"#4단계---반복\" data-toc-modified-id=\"4단계---반복-7.1.5\"><span class=\"toc-item-num\">7.1.5&nbsp;&nbsp;</span>4단계 - 반복</a></span></li></ul></li><li><span><a href=\"#오차역전파법을-적용한-신경망-구현하기\" data-toc-modified-id=\"오차역전파법을-적용한-신경망-구현하기-7.2\"><span class=\"toc-item-num\">7.2&nbsp;&nbsp;</span>오차역전파법을 적용한 신경망 구현하기</a></span></li><li><span><a href=\"#오차역전파법으로-구한-기울기-검증하기\" data-toc-modified-id=\"오차역전파법으로-구한-기울기-검증하기-7.3\"><span class=\"toc-item-num\">7.3&nbsp;&nbsp;</span>오차역전파법으로 구한 기울기 검증하기</a></span></li><li><span><a href=\"#오차역전파법을-사용한-학습-구현하기\" data-toc-modified-id=\"오차역전파법을-사용한-학습-구현하기-7.4\"><span class=\"toc-item-num\">7.4&nbsp;&nbsp;</span>오차역전파법을 사용한 학습 구현하기</a></span></li><li><span><a href=\"#여러가지-테스트를-해볼까?\" data-toc-modified-id=\"여러가지-테스트를-해볼까?-7.5\"><span class=\"toc-item-num\">7.5&nbsp;&nbsp;</span>여러가지 테스트를 해볼까?</a></span><ul class=\"toc-item\"><li><span><a href=\"#은닉층의-노드수를-늘려보자\" data-toc-modified-id=\"은닉층의-노드수를-늘려보자-7.5.1\"><span class=\"toc-item-num\">7.5.1&nbsp;&nbsp;</span>은닉층의 노드수를 늘려보자</a></span></li><li><span><a href=\"#은닉층의-수를-줄여보자\" data-toc-modified-id=\"은닉층의-수를-줄여보자-7.5.2\"><span class=\"toc-item-num\">7.5.2&nbsp;&nbsp;</span>은닉층의 수를 줄여보자</a></span></li><li><span><a href=\"#배치-사이즈를-늘려보자\" data-toc-modified-id=\"배치-사이즈를-늘려보자-7.5.3\"><span class=\"toc-item-num\">7.5.3&nbsp;&nbsp;</span>배치 사이즈를 늘려보자</a></span></li><li><span><a href=\"#배치-사이즈를-줄여보자\" data-toc-modified-id=\"배치-사이즈를-줄여보자-7.5.4\"><span class=\"toc-item-num\">7.5.4&nbsp;&nbsp;</span>배치 사이즈를 줄여보자</a></span></li><li><span><a href=\"#학습률을-올려보자\" data-toc-modified-id=\"학습률을-올려보자-7.5.5\"><span class=\"toc-item-num\">7.5.5&nbsp;&nbsp;</span>학습률을 올려보자</a></span></li><li><span><a href=\"#학습률을-낮춰보자\" data-toc-modified-id=\"학습률을-낮춰보자-7.5.6\"><span class=\"toc-item-num\">7.5.6&nbsp;&nbsp;</span>학습률을 낮춰보자</a></span></li></ul></li></ul></li><li><span><a href=\"#5장-정리-(p187)\" data-toc-modified-id=\"5장-정리-(p187)-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>5장 정리 (p187)</a></span></li><li><span><a href=\"#참고\" data-toc-modified-id=\"참고-9\"><span class=\"toc-item-num\">9&nbsp;&nbsp;</span>참고</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 내용 출처 : [WegraLee GitHub](https://github.com/WegraLee/deep-learning-from-scratch/blob/master/ch05/README.md)\n",
    "\n",
    "- 앞 장에서는 신경망 학습에 대해서 설명했습니다.\n",
    "- 그때 신경망의 가중치 매개변수의 기울기(정확히는 가중치 매개변수에 대한 손실 함수의 기울기)는 수치 미분을 사용해 구했습니다.\n",
    "- **`수치 미분`**은 단순하고 구현하기도 쉽지만 계산 시간이 오래 걸린다는 게 단점입니다.\n",
    "\n",
    "- 이번 장에서는 가중치 매개변수의 기울기를 효율적으로 계산하는 **`오차역전파법backpropagation`**을 배워보겠습니다.\n",
    "- 오차역전파법을 제대로 이해하는 방법은 두 가지가 있을 것입니다.\n",
    "    - 하나는 수식을 통한 것이고, 다른 하나는 계산 그래프를 통한 것입니다.\n",
    "    - 전자 쪽이 일반적인 방법으로, 특히 기계학습을 다루는 책 대부분은 수식을 중심으로 이야기를 전개합니다.\n",
    "- 확실히 수식을 사용한 설명은 정확하고 간결하므로 올바른 방법이라 할 수 있겠죠.\n",
    "- 하지만 졸업 후 너무 오랜만에 수식을 중심으로 생각하다 보면 본질을 놓치거나, 수많은 수식에 당황하는 일이 벌어지기도 합니다.\n",
    "- 그래서 이번 장에서는 계산 그래프를 사용해서 ‘시각적’으로 이해시켜드리겠습니다.\n",
    "- 그런 다음 실제로 코드를 작성해보면 ‘과연!’이란 탄성과 함께 더 깊이 이해하게 될 것입니다.\n",
    "\n",
    "*옮긴이_ 오차역전파법을 풀어쓰면 ‘오차를 역(반대 방향)으로 전파하는 방법(backward propagation of errors )’입니다. 너무 길고 쓸데없이 어려운 느낌이라 줄여서 ‘역전파법’ 혹은 그냥 ‘역전파’라고 쓰기도 합니다.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:49:18.977949Z",
     "start_time": "2019-08-03T16:49:18.959917Z"
    }
   },
   "outputs": [],
   "source": [
    "# coding: utf-8\n",
    "import os\n",
    "import sys\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "mpl.rcParams['font.family'] = 'AppleGothic' # mac에서는 한글이 나오게 하기 위해서\n",
    "mpl.rcParams['axes.unicode_minus'] = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 계산 그래프 (p148)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **`계산 그래프(computational graph)`** : 계산 과정을 그래프로 나타낸 것"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 계산 그래프로 풀다\n",
    "- 계산 그래프를 이용한 문제풀이는 항상 다음과 같은 흐름으로 진행\n",
    "    1. 계산 그래프를 구성한다.\n",
    "    2. 그래프에서 계산을 왼쪽에서 오른쪽으로 진행한다.\n",
    "- 순전파(forward propagation) : 왼쪽에서 오른쪽으로\n",
    "- 역전파(backward propagation) : 오른쪽에서 왼쪽으로"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 문제1\n",
    "> Q1. 현빈 군은 슈퍼에서 1개에 100원인 사과를 2개 샀습니다. 이때 지불 금액을 구하세요. 단, 소비세가 10% 부과됩니다.\n",
    "\n",
    "#### 해답1\n",
    "![](https://t1.daumcdn.net/cfile/tistory/99E90F395AA4FD0D12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 문제2\n",
    "> Q2. 현빈 군은 슈퍼에서 사과를 2개, 귤을 3개 샀습니다. 사과는 1개에 100원, 귤은 1개 150원입니다. 소비세가 10%일 때, 지불 금액을 구하세요.\n",
    "\n",
    "#### 해답2\n",
    "![](https://t1.daumcdn.net/cfile/tistory/997ED34B5B98F5F235)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 국소적 계산"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 계산 그래프의 특징은 `국소적 계산`을 전파함으로써 최종 결과를 얻는다는 점\n",
    "- `국소적`이란 뜻은 자신과 `직접적으로` 관계된 작은 범위\n",
    "- 각 노드는 자신과 관련된 계산 외에는 전혀 신경 쓸 게 없다\n",
    "- 계산 그래프는 국소적 계산에 집중하면 된다. 그러므로 **전체 계산이 아무리 복잡하더라도 각 단계에서 하는 일은 해당 노드의 국소적 계산뿐**임"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 왜 계산 그래프로 푸는가?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://t1.daumcdn.net/cfile/tistory/991F25415AA504B626)\n",
    "\n",
    "> 오른쪽에서 왼족으로 : `1 → 1.1 → 2.2`   \n",
    "> 사과 가격에 대한 지불 금액의 미분값 : `2.2`  \n",
    "> 즉, 사과 가격이 변할때 최종 금액이 얼마나 변하는지 : 사과 1원 오를 때, 최종 금액은 2.2원  \n",
    "\n",
    "> $$ \\frac{\\partial \\mbox{ Total Price}}{\\partial\\mbox{ Apple Price}} = 2.2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **계산 그래프의 이점**은 무엇인가?\n",
    "   - 국소적 계산\n",
    "   - 중간 결과 보관\n",
    "    \n",
    "    \n",
    "- 계산 그래프를 사용하는 이유\n",
    "   - 작은 문제에 집중하여 복잡한 문제를 단순화\n",
    "   - **역전파를 통해서 `미분`을 효율적으로 계산**\n",
    "\n",
    "\n",
    "- 결국 이것들을 종합하면, 각 노드별로 중간까지 구한 **미분 결과를 공유**할 수 있으며, 다수의 미분을 효율적으로 계산가능\n",
    "- 계산 그래프의 이점은 **순전파와 역전파를 활용해서 각 변수의 미분을 효율적으로 구할 수 있다**는 것"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 연쇄법칙 (p152)\n",
    "- 순전파는 왼쪽에서 오른쪽으로 전달\n",
    "- 역전파는 오른쪽에서 왼쪽으로 전달\n",
    "- 사실 `국소적 미분`을 전달하는 원리는 **`연쇄법칙(chain rule)`**을 기반으로 함!\n",
    "\n",
    "> **`역전파` == `연쇄법칙`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 계산 그래프의 역전파\n",
    "\n",
    "> 역전파의 계산 절차 : `신호 E`에 노드의 `국소적 미분`($\\frac{\\partial y}{\\partial x}$)을 `곱한 후 다음 노드로 전달`하는 것 \n",
    "\n",
    "![](https://t1.daumcdn.net/cfile/tistory/999FD3425B98F63F1A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 연쇄법칙이란?\n",
    "> *`합성 함수의 미분`은 **`합성 함수를 구성하는 각 함수의 미분의 곱`**으로 나타낼 수 있다.* ( 이하 자세한 내용은 생략 )\n",
    "\n",
    "$$ \\frac{\\mathrm{d} f(g(x))}{\\mathrm{d} x} = \\frac{\\mathrm{d} f(g(x))}{\\mathrm{d} g(x)} \\cdot  \\frac{\\mathrm{d} g(x)}{\\mathrm{d} x}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 연쇄법칙과 계산 그래프\n",
    "\n",
    "> **`역전파가 하는 일`**은 **`연쇄 법칙의 원리`**와 같다!!  \n",
    "\n",
    "\n",
    "- t는 z에 얼마나 영향을 주는지 : $\\frac{\\partial z}{\\partial z} \\cdot \\frac{\\partial z}{\\partial t}$  \n",
    "\n",
    "\n",
    "- x는 z에 얼마나 영향을 주는지 : $\\frac{\\partial z}{\\partial z} \\cdot \\frac{\\partial z}{\\partial t} \\cdot \\frac{\\partial t}{\\partial x}$  \n",
    "\n",
    "![](https://image.slidesharecdn.com/ch05-170403044028/95/ch05-13-638.jpg?cb=1491194513)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 역전파 (p155)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 덧셈 노드의 역전파\n",
    "\n",
    "> **입력된 값을 그대로 다음 노드로**\n",
    "\n",
    "![](https://t1.daumcdn.net/cfile/tistory/99771C465AA4FFEA13)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 곱셈 노드의 역전파\n",
    "\n",
    "> **상류의 값에 순전파 때의 입력 신호들을 `서로 바꾼 값`을 곱해서 하류로**\n",
    "\n",
    "![](https://t1.daumcdn.net/cfile/tistory/99A6DB435AA502DF0A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 사과 쇼핑의 예\n",
    "\n",
    "> 앞서 설명한 사과 쇼핑의 예시를 통해서 역전파를 연습해보자\n",
    "\n",
    "#### 문제\n",
    "![](https://t1.daumcdn.net/cfile/tistory/9960E8375AA5060D01)\n",
    "\n",
    "#### 정답\n",
    "![](https://t1.daumcdn.net/cfile/tistory/9964A5465AA506AA22)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  \n",
    "<center> <font size=\"5\">계산 그래프에서 집중해야할건 각 요소가<br> 결국 <font color=\"red\"><b>최종 결과에 어떤 영향을 주는지</b></font>를 확인하는 것이다 </font></center>\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 단순한 계층 구현하기 (p160)\n",
    "\n",
    "- 위에서 설명한 사과 쇼핑을 예시로 구현할 것\n",
    "- 기본적으로 모든 계층은 **순전파**를 해주는 함수인 **`forward()`**와 **역전파**를 해주는 함수인 **`backward()`**라는 공통의 메서드(인터페이스)를 갖도록 구현할 것"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 곱셈 계층"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T02:39:41.846657Z",
     "start_time": "2019-08-03T02:39:41.832179Z"
    }
   },
   "outputs": [],
   "source": [
    "class MulLayer:\n",
    "    \n",
    "    ##########\n",
    "    # 초기화 함수\n",
    "    ##########\n",
    "    # 순전파 시의 입력 값을 유지하기 위해서 사용, 그래야 역전파때도 저장한걸 이용하고\n",
    "    def __init__(self):\n",
    "        self.x = None\n",
    "        self.y = None\n",
    "\n",
    "    ##########\n",
    "    # 순전파 함수 : x와 y를 인수로 받고 두 값을 곱해서 반환\n",
    "    ##########\n",
    "    def forward(self, x, y):\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        out = x * y # 곱셈을 해서 순전파\n",
    "\n",
    "        return out\n",
    "\n",
    "    ##########\n",
    "    # 역전파 함수 : 상류에서 넘어온 미분(dout)에 순전파 때의 값을 '서로 바꿔' 곱한 후 하류로 흘림\n",
    "    ##########\n",
    "    def backward(self, dout):\n",
    "        dx = dout * self.y  # x와 y를 바꾼다.\n",
    "        dy = dout * self.x  # x와 y를 바꾼다.\n",
    "\n",
    "        return dx, dy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://t1.daumcdn.net/cfile/tistory/991F25415AA504B626)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T05:40:03.829098Z",
     "start_time": "2019-08-04T05:40:03.822204Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "price           : 220\n",
      "dApple          : 2.2\n",
      "dApple_num      : 110\n",
      "dTax            : 200\n"
     ]
    }
   ],
   "source": [
    "# 입력\n",
    "apple = 100\n",
    "apple_num = 2\n",
    "tax = 1.1\n",
    "\n",
    "# 곱셈 레이어 생성\n",
    "mul_apple_layer = MulLayer()\n",
    "mul_tax_layer = MulLayer()\n",
    "\n",
    "# 순전파\n",
    "apple_price = mul_apple_layer.forward(apple, apple_num)\n",
    "price = mul_tax_layer.forward(apple_price, tax)\n",
    "\n",
    "# 역전파\n",
    "# 레이어 호출 순서가 순전파랑 반대임에 주의하자!!\n",
    "dprice = 1\n",
    "dapple_price, dtax = mul_tax_layer.backward(dprice) # 파라미터가 '순전파의 출력에 대한 미분'임을 주의하자!!\n",
    "dapple, dapple_num = mul_apple_layer.backward(dapple_price) # 파라미터가 '순전파의 출력에 대한 미분'임을 주의하자!!\n",
    "\n",
    "# 결과 출력 : 가격에 영향을 얼마큼 주는지 역전파를 통해서 확인\n",
    "print(\"{:<15} : {}\".format(\"price\", int(price)))\n",
    "print(\"{:<15} : {}\".format(\"dApple\", dapple))\n",
    "print(\"{:<15} : {}\".format(\"dApple_num\", int(dapple_num)))\n",
    "print(\"{:<15} : {}\".format(\"dTax\", dtax))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 덧셈 계층"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T02:44:07.420174Z",
     "start_time": "2019-08-03T02:44:07.415565Z"
    }
   },
   "outputs": [],
   "source": [
    "class AddLayer:\n",
    "    \n",
    "    ##########\n",
    "    # 초기화 함수 : 덧셈계층에서는 초기화가 필요없음 => WHY? forward함수를 확인해보자 \n",
    "    ##########\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    ##########\n",
    "    # 순전파 함수 : 입력받은 두 인수 x,y를 더해서 반환\n",
    "    ##########\n",
    "    def forward(self, x, y):\n",
    "        out = x + y\n",
    "        \n",
    "        return out # 덧셈을 해서 순전파\n",
    "    \n",
    "    ##########\n",
    "    # 역전파 함수 : 상류에서 내려온 미분(dout)을 '그대로' 하류로 흘림\n",
    "    ##########\n",
    "    def backward(self, dout):\n",
    "        dx = dout * 1 # 그대로 훌려준다.\n",
    "        dy = dout * 1 # 그대로 흘려준다.\n",
    "\n",
    "        return dx, dy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://t1.daumcdn.net/cfile/tistory/9964A5465AA506AA22)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T05:39:48.773070Z",
     "start_time": "2019-08-04T05:39:48.758961Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "price           : 715\n",
      "dApple          : 2.2\n",
      "dApple_num      : 110\n",
      "dOrange         : 3.3\n",
      "dOrange_num     : 165\n",
      "dTax            : 650\n"
     ]
    }
   ],
   "source": [
    "# 입력\n",
    "apple = 100\n",
    "apple_num = 2\n",
    "orange = 150\n",
    "orange_num = 3\n",
    "tax = 1.1\n",
    "\n",
    "# 덧셈과 곱셈 레이어 생성\n",
    "mul_apple_layer = MulLayer()\n",
    "mul_orange_layer = MulLayer()\n",
    "add_apple_orange_layer = AddLayer()\n",
    "mul_tax_layer = MulLayer()\n",
    "\n",
    "# 순전파\n",
    "apple_price = mul_apple_layer.forward(apple, apple_num)  # (1)\n",
    "orange_price = mul_orange_layer.forward(orange, orange_num)  # (2)\n",
    "all_price = add_apple_orange_layer.forward(apple_price, orange_price)  # (3)\n",
    "price = mul_tax_layer.forward(all_price, tax)  # (4)\n",
    "\n",
    "# 역전파\n",
    "# 레이어 호출 순서가 순전파랑 반대임에 주의하자!!\n",
    "# 파라미터가 '순전파의 출력에 대한 미분'임을 주의하자!!\n",
    "dprice = 1\n",
    "dall_price, dtax = mul_tax_layer.backward(dprice)  # (4) \n",
    "dapple_price, dorange_price = add_apple_orange_layer.backward(dall_price)  # (3)\n",
    "dorange, dorange_num = mul_orange_layer.backward(dorange_price)  # (2)\n",
    "dapple, dapple_num = mul_apple_layer.backward(dapple_price)  # (1)\n",
    "\n",
    "# 결과 출력 : 가격에 영향을 얼마큼 주는지 역전파를 통해서 확인\n",
    "print(\"{:<15} : {}\".format(\"price\", int(price)))\n",
    "print(\"{:<15} : {}\".format(\"dApple\", dapple))\n",
    "print(\"{:<15} : {}\".format(\"dApple_num\", int(dapple_num)))\n",
    "print(\"{:<15} : {}\".format(\"dOrange\", round(dorange,2)))\n",
    "print(\"{:<15} : {}\".format(\"dOrange_num\", int(dorange_num)))\n",
    "print(\"{:<15} : {}\".format(\"dTax\", dtax))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 활성화 함수 계층 구현하기 (p165)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ReLU 계층\n",
    "\n",
    "$$ h(x) = \\left\\{\\begin{matrix}\n",
    "x ( x > 0 ))\n",
    "\\\\ \n",
    "0 (x \\leq 0))\n",
    "\\end{matrix}\\right. $$\n",
    "\n",
    "> 순전파 때의 입력인 x가 0보다 크면 역전파는 상류의 값을 그대로 하류로 흘려보냄  \n",
    "> 하지만, 순전파 때의 입력인 x가 **0보다 작으면 역전파 때는 하류로 신호를 보내지 않음(즉, 0을 보냄)**\n",
    "\n",
    "![](http://mblogthumb1.phinf.naver.net/MjAxNzA3MjZfMjIz/MDAxNTAxMDUxNjU4NTYx.-vuv9Yyt7NVFusV7lkeYUg1ndzQYTadYAwAuhNLNyPAg.R2uYubWcuijOa_iqgZSWR1jfuENh8O9azkICvJckptkg.PNG.cjswo9207/fig_5-18.png?type=w2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T03:07:08.924395Z",
     "start_time": "2019-08-03T03:07:08.909653Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1. , -0.5],\n",
       "       [-2. ,  3. ]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([[1.0, -0.5],\n",
    "              [-2.0, 3.0]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T03:07:23.249101Z",
     "start_time": "2019-08-03T03:07:23.244756Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[False,  True],\n",
       "       [ True, False]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask = (x<=0) # 조건을 만족시키는 애들을 True, 그렇지 않은 애들은 False로 반환해서\n",
    "mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T03:08:04.961815Z",
     "start_time": "2019-08-03T03:08:04.956162Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.5, -2. ])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T05:43:03.721396Z",
     "start_time": "2019-08-04T05:43:03.717815Z"
    }
   },
   "outputs": [],
   "source": [
    "from common.functions import relu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T03:05:56.742950Z",
     "start_time": "2019-08-03T03:05:56.727873Z"
    }
   },
   "outputs": [],
   "source": [
    "class Relu:\n",
    "    def __init__(self):\n",
    "        self.mask = None\n",
    "\n",
    "    #######\n",
    "    # 순전파 : 입력 값이 0 이하면 역전파 때의 값은 0이 돼야 함\n",
    "    #######\n",
    "    def forward(self, x):\n",
    "        self.mask = (x <= 0)\n",
    "        out = x.copy()\n",
    "        out[self.mask] = 0 # 0이하면 다 0으로 만들어버림\n",
    "\n",
    "        return out\n",
    "    \n",
    "    #######\n",
    "    # 역전파 : 순전파 때 만들어둔 mask를 이용해서 mask의 원소가 True인 곳에는 상류에서 전파된 dout을 0으로 설정\n",
    "    #######\n",
    "    def backward(self, dout):\n",
    "        dout[self.mask] = 0\n",
    "        dx = dout\n",
    "\n",
    "        return dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoid 계층\n",
    "\n",
    "$$ h(x) = \\frac{1}{1 + exp(-x)} $$\n",
    "\n",
    "점점 복잡해지더라도 최초 `x`가 입력되는 식부터 차근차근 생각해보면 됨!\n",
    "\n",
    "> `exp` 노드 : $y = e^x$를 수행  \n",
    "> `/` 노드 : $y = \\frac{1}{x}$를 수행 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 계산 그래프 정리 전\n",
    "![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTcdkJb52ltHvy6lqGVQIczw7yCrjCDB0uIO8Nosk_SSfc8IVW9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 수식 정리\n",
    "\n",
    "<center>$\\frac{\\partial L}{\\partial y} y^2 exp(-x) \\\\\\\\\n",
    "= \\frac{\\partial L}{\\partial y} \\cdot \\frac{1}{(1+exp(-x))^2} \\cdot exp(-x) \\\\\\\\\n",
    "= \\frac{\\partial L}{\\partial y} \\cdot \\frac{1}{(1+exp(-x))} \\cdot \\frac{exp(-x)}{(1+exp(-x))} \\\\\\\\\n",
    "= \\frac{\\partial L}{\\partial y} \\cdot y(1-y)$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 계산 그래프 정리 후"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://mblogthumb2.phinf.naver.net/MjAxNzA3MjZfMjYx/MDAxNTAxMDU0ODU3MjA5.zh-JigNWlXHD5sV2t4rXwHvg0XXXLczmH7oz_fAQFcEg.zxZDopfppTyE7wrQyOzgJ63-Z6qQIQLyWQ8244oHq5gg.PNG.cjswo9207/fig_5-22.png?type=w2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:54:16.027511Z",
     "start_time": "2019-08-03T16:54:16.024712Z"
    }
   },
   "outputs": [],
   "source": [
    "from common.functions import sigmoid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T15:57:40.720391Z",
     "start_time": "2019-08-03T15:57:40.704089Z"
    }
   },
   "outputs": [],
   "source": [
    "class Sigmoid:\n",
    "    def __init__(self):\n",
    "        self.out = None\n",
    "\n",
    "    #######\n",
    "    # 순전파 : sigmoid 함수를 통해서 순전파, 순전파의 출력을 self.out에 보관해둠\n",
    "    #######\n",
    "    def forward(self, x):\n",
    "        out = sigmoid(x)\n",
    "        self.out = out\n",
    "        return out\n",
    "\n",
    "    #######\n",
    "    # 역전파 : 순전파 때 보관해둔 out값을 이용해서 역전파에 쓰임, 순전파의 출력(y)만으로 계산가능\n",
    "    #######\n",
    "    def backward(self, dout):\n",
    "        dx = dout * (1.0 - self.out) * self.out # 간소화한 수식을 이용\n",
    "\n",
    "        return dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Affine/Softmax 계층 구현하기 (p170)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Affine 계층\n",
    "- 신경망의 순전파 때 수행하는 행렬의 곱을 해주는 레이어\n",
    "- 전까지는 `스칼라`만 처리했는데, 이제는 `행렬`이 흐르게 할 것"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:15:55.897332Z",
     "start_time": "2019-08-03T16:15:55.881898Z"
    }
   },
   "outputs": [],
   "source": [
    "X = np.random.rand(2) # 입력\n",
    "W = np.random.rand(2,3) # 가중치\n",
    "B = np.random.rand(2,3) # 편향"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:16:16.671542Z",
     "start_time": "2019-08-03T16:16:16.653573Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.8343816  0.72454461]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(2,)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(X)\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:16:23.571249Z",
     "start_time": "2019-08-03T16:16:23.566513Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.22004554 0.0729576  0.04395276]\n",
      " [0.07686227 0.69515846 0.09504852]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(2, 3)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(W)\n",
    "W.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:16:28.089119Z",
     "start_time": "2019-08-03T16:16:28.083178Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.89624141 0.33028769 0.29728268]\n",
      " [0.32236211 0.69824953 0.79176663]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(2, 3)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(B)\n",
    "B.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:16:40.572239Z",
     "start_time": "2019-08-03T16:16:40.507752Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.1355335 , 0.89483548, 0.40282295],\n",
       "       [0.5616542 , 1.26279732, 0.89730691]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y = np.dot(X,W) + B # 이게 사실상 Affine이 될듯\n",
    "Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Affine 계층의 순전파\n",
    "![](http://mblogthumb1.phinf.naver.net/MjAxNzA3MjZfMTg4/MDAxNTAxMDU2NjQzMDkz.5m_up0L9OiIwXFPmCtJ_AaiorDntuvFpH8KBeFBOsa8g.KuSP8t6dQBJ3gXg72HddolfT-hYz9UrWHDr3RY1GAoQg.PNG.cjswo9207/fig_5-24.png?type=w2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Affine 계층의 역전파\n",
    "\n",
    "$ \\frac{\\partial L}{\\partial X} = \\frac{\\partial L}{\\partial Y} \\cdot W^T $  \n",
    "$ \\frac{\\partial L}{\\partial W} = X^T \\cdot \\frac{\\partial L}{\\partial Y}  $\n",
    "\n",
    "위와 같이 행렬을 사용한 역전파도 `행렬의 원소마다 전개`해보면  \n",
    "스칼라값을 사용한 지금까지의 계산 그래프와 같은 순서로 생각가능함 \n",
    "\n",
    "![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAATIAAAClCAMAAADoDIG4AAAAilBMVEX////5+fng4ODv7+8AAADS0tL8/Pzm5uZlZWXZ2dn39/f09PRfX1/q6urW1tba2tqEhIS6urq0tLSamprNzc2MjIzBwcGsrKyfn5+3t7fGxsaRkZF6enqcnJyoqKiAgIBwcHBVVVV0dHRLS0tqampHR0cjIyNAQEAwMDAVFRUTExM5OTkrKysiIiIeM6YEAAAUlklEQVR4nO1diZaiuhbNAIkQEqYwI4pa3bfv7ff/v/cSQAUFC6zJ6navXtWoCOEkOdlnigA88cTfBTsFXh2qg9j86qZ8FxQiy93/BABkw7+6Ld8DqAYMgIOrDmX01Y35Hgi0nMRGH+J/v7gt3wSJEhkOCVGH2PjqxnwPiBWAq8AP/Byg1Vc35nsAHmwziiKa7UCRf3Vjvgloo8eARHn1xS35Pmj0mIL9tc14WED61S34fuAuGb5BKOccM/Xnaxr0DcAEG7yGq18/atP975/0ixr0HSAuxtMPowbgn+eMvQXqDl4iwyg2T4Ixgp4Ow2KwPIaG4X12a74FqvisxKDVn5ykNvaf355vAFxvk7OgEDp/wpxfRvIFLXp8mI5TRxJ2r7g4fXCQqWE8OcYYMme12p1mJzRxe1AoerE3nHuuSCmvYvU/f9t6G0vxUy/ZrCCvn/ypIIXjePj8EukHhZ6xY6I2jM3y5vKtHeUvUl/kLYNURCAAlXamm+EbLvMhYIegPJwnpGIbRAsOQYwopctFtqWEgVILHr/FceQgfYFGua4ejh5yYBd1dn7NxFuMcfxLja/QaJ7yX/ja2dP4pUa+vy/04fYh+aH/0tMY5NIUWILWjZs2IYPfbxgeWmSAN8p0F99/mQ8EXa16DOPCFFgCPcoAkIV2j7xllO0piHGQ2hsIHOv+y3wkWFT751dY3P20W8qcsCDyJ7bvWnA7yAKkRQ7sGtHHtUKCOjxPTijwjVNvgZcQq1XDXrMIvX72NKom6gwDUixTE7bQ6oCMrlvIgtpIfD/WIpyy1zp0ryJymy8yxt84n7o+gwsXIyzzHLAqHfuaFBUBeXG/2rkC29Xy/Ip/2+SCgoFior9kDuT6XW9W7XtORSjZ9JkXsCGEmGDC4Pkd9YJByGyi/i1vigyyLCNuEojXzx1AGR/U3o7ekSFRgUKOfXQ/5KrsKTFrthrJf/z6X8Qi43Aa9NmPX3uU/fNrb+LVf3ewFr43DAZIVN+YmOSM03tmyEvsji4ZduQmCdhOa2kk86UdpFtarnoTkqK548M3jADEZe/00tgCoIQIwDqb/NoNuIahOOzqlrSDjdfhPDkKCkqS+JqiXyIPQCCpRwCe4ANcBHfYO4q618H5FTbnso2D8Zu+NCOCV81Qk4YBQWoYNtheT3DCKFKgt66+MfYgLm7dM6zyPFfqPs/OoyqkKCU7yXL/6nQlsggmIaOTF3WvvzQL8T46PwkxZ7IN/D/DaFgF/NUewN+GT5wXI+ZXtiZ0BcVqsVd2rJhemLEauKubnR4GQsg0FmZ8FpntZ2rlkmY+MqHzjALTNPPBPUlLSZq5Lea6CGV0QjOa6bbuUSp0Sa+Uph97kNwwGstGTaj2oDBKsU4ML7k0eFyr70A3J6deaBi3V20tMtPzhZm/ge66ZYllGWEUYcDnUkl/53eoW/nDcD8wBdip5ZC6rp5RLuKXYov/M341M7A0nOZ/y/h5QNj4rx7KxL6IAE6bZ67x7+1HUCIz3WggMn5GMz0IatDcuPdZXyGENQHeDuAlk9LfuEL4mSWEcxyyWR2eH4zFHa+FyD1qTptfLKfugf5uIyzkqKF+a93/YpSD85h1pb/4hMzQHJFZQ5Gl4RFFo5Jh0kD3rxudPov6bZd7aXsOypdwd38jTHMdWuZZZEAcdqdjKqxcCxAPrSg+MERfkF41B5wn1EteMnyPiJFJPTEf1FS/veaPiEx16/AG7ajqHmRUL8NVKqXjJ0vMai0ykawHIgPY25+e1aZxRkaUmnkaieZPI4OJMZSZ9Vt9zn8M2nk5K1ugMX1m1auBf+Uajci8bKD+RV5RarqEtzrSXqcK60aMOI59FueIyKEHrlpVIFwtcjGNikyxnp/rc4dhTkcGrnlcljBmnFDG2EA+zYQbPDSeEMF9xlkYuK5MMssdqv/QLgogk7E1KqGpWtouLAr3xQTyZZEfYkJkQKzK3qgbVTjmQuY3JRp8lxMg3BRFobRWEW3Ob3IYQbYTYxLD2Dfham0HF+9vIbDL69NvoBVZeCUywD3nONHIuFZhyxw89uTpdxgqapE+6fMzn4orq4TA3Y8oAB6JQoBslV4OKA7GjIVbaFbMNJXiUmQAVvuut8Y1p5p28033G1dR15mpfMcdYWcwj0BlEgXFSNLSWoA1AuzwDvlM/japkiBQa/GlyNQK/OJpoZAp55c91/kE9WUGJ7t903NuKhty3StG2AeLgB+DXILDtQ5IEVamkr19h2xW6Z1w3XC8c8xbT+Rqv88MuIp1Dr2Oot/bMHetGXAlEllwq5uQ7wMzYqDaXJ2lbCn1GLIs7vU4z4W9rhOAThYbbnvvNI+4i2ZBPWsc9ygIdf2iR1vITKVILfq6KrAZAcye8BqpD94QxpmJuC5OBJSbG8X8AfSOU5jNXOu4yyHsySVPi0Ny1tpzRfaaLnsUoJO73NuuIuocEv/3tusquGDNJIO5YvV1ypLLfA90D2RGII6UwjvQU7bGomcdiMzs67L7iNk1cM6igzawR0xGZNJ1qdWp35vbOTXDzhPteisXv1tovnvWPFVSU0to6ZZDkbF5cYIBJWH9JaVTlnhBvGEMpII5Y40IkssrsRQggnQ/kXNnmbka+kGjY0lM0EYZXG9qwBldmjupi7qQTuGB6HByaTDkR4da4BkYM7paCG1MBZuVg+ZcZ1Ku0tcuw0K3Lb98dm0msbTp4eQ0NTzVVThqr0cAD4F90w+8AO5RQEg1lyreeeLxPD3UjrOah+uFvwNDfL2q517GmWRWiV7PZbOoo8vgm44h86TJFolPNrgORbnR0VWWqOE5g4Jgn3u7TgDFToJRX+4N2k5FogZHHZsz4E7xO0EIN6tSyT6fc53Jwapj7LwdXugyL60NuzcaJT65E5voHe6ka2oxvC4y4pBSJk3yCchM+Q+Ix2Q2aRy2ZgGWc9MAxoOIR4cZz99IM2UG6CYPROICeenNCUwlExTAlPcmZmSTHPsCa6dNXuQVhq/XOa8zgAE7Fvf6SjHWY0/vThDDkWEj0sAGzL2s8mlhj9nfo28ClC8Pf5IUUlOaRMmrGlH/wleaIbDIec4KZV3FFEBtqlumtPQbr+Gg+z3ozD13VQOwuf4S4WxcC405OAjBSvBThRYjTiR7PM8I8QABvFkW3D7OSiivJ6/VNtaCca//236Bx1bNIRnavWOdF92VBbxLdxJDGR13nAJ3fO0qANtMrWr40pU9nZnFKkBuh+WucSI/I5/Bi3NGTpxztzIHaINiN2CNb3Vrg+1w5NjCbcLuY/mgo6QBJzIi1uRaTfkgSxJe+cdPEDwk4r3W/HeEKEFY7kqwCnTQd60GqHMhG4laS9y8Gjfj4TQPkR2opkJcWitT6+gew5Y1ybHSWEagesRE2e7hTmT5MpPOokdGbg0/IROR7pLwCOwwsEeelpmxHmJ53fh6hHsjBQKXmlrtGICPV7HRtujYdjJc5W01tk6PpabU6Rgia2IRzWRFkefHI7zDlkgrLlprIvOKU8L2QGUhh6L04ZLYb4KfdTXJiVo5XcR1HIm67jSB4mpkkHGvlJB6pO68OR4rGguAhDCtb+LvadFTzXbU8jVbh+f4VLrRK8BICzIZZX7fClWahmydpgU3w7QnC9LLMKFbZ+ZzRme3+KUpCJu12Po5R6UzeTKP5OP50pBhJID+a6gndHqcjMnzjMjr6O3eX9JEPbEzqygJB76ftQgeMGv3oFNNtsYOkO1ZpfcZRFUvqM2MkxwwaF9rs5b3RtuZwqfjJtdjIDYMAPfqj3mKOhP3PA9Z5CxhRwxKGW/XkG+GS13LewNnrpffpAmOgxiI4KMjQncA/2PIvKgNGR4Nuv42EHS3tFYNFmBTEjq0VVnj80D1gsRZEbv7BKCHrD4ujfUOR0bhdP3ZK/8Fst4sq1bgifSAVV86kdreWC3YQ86lKUgdEjzkNjqm8W+pV4Huedze3ElflnbyjnEPgHU9nICt7R06s337xDODFLDae7z1UsNWMxOQvdHMHbuXeceiemllAS2AVOtuPiQlrSMkf5lfDCPWjfN0PXe1+GxEP9Sf9JdeongvMxPtDsutlTXS/rqq7mcTtG5XOu20v4YyMLWLNn1UkTUxxSYfnvYj2o53R4zM1qE3lssem2o5LPQWJXFJ7ULlkfeArKyPQcVv9fJeO76120pU9fcysGehX0zCvP179W/rE5P7R/R9vRH9kiW0PdxpPFPIfT27+XFStztm2PXCVDgISOsIWPat6fjhJO62Mojbmze5E90Z6kcx86nW2bBLC285LIm2y54d+yRZa81gLUoK5XEsAImTBf2NktgGEOPFkutzC6V2dGNHavdeR6TPbmqu2yKqLtM/3i/jV6RinJGGKAYLVWBhg2yJFoaAKu659ZeKrOdMBDDS26mAuFTY7fTf+YNd51bCtM1yabwfremF6su41SvQ+SYkaIROF61CNqv4lTf+NShatFpKZvqEH61WjYiCMggC31d/FvgE27h+W+pVsiOHhdulm23HmXY/NSUubAk3QYW14Wx76xQsxMDTa1ey4Es3ShhsWBDXRavGAs8SeZGaQiwQGY60U98XumjCg6ALGYeHpUpZNqMy0XqMbSbOyU4VfudxGCJQQlMNdTY5zyhNUV+vFgJ7th7QC2bmIPha1ccqg8AT5lpGqbVEZEqX4ULGII8ACgBseUpeL+YryiSnOU9IkAM0NTHDKE2rSjuVz0INKQ/B2ic8mBaAPbgg3OkkqZIDNr+RfcLfNyq1yKTIwmUiQz7QI4zlILE7Yxwv8VAekXAWWzZwAxBM3V0X5PhRgKxeQQ4NEoaDzE+urVmGGYdccR82pDuVSBFy/Hi+A6BP+OmhZ01rkZnuOl80MdU1mkFva17WZSVsdvdUbGfd1cSkKR8GIq9kmYl4liJi230FQqXpuRy2B7Fl1KxfHxnvo953tciswLcWiuyMLgeq2n+QW1WJTJpWqgR3EtlFuqNiw73iQr4q+Tvv+l3VAwWgRCb8Kg/knSKzW1+GWX+UodQUSpuF2avHrMJ1eEZC7Page6zYKa+GEwmSIyazIsaB2JVvLPBcEYZFZVn3iawto8OHD9vmTtdjiiQf1pbfpFfedckRQa6r/mlYi9icIk7lpYs/KLNAV5wH2V0i6xKpot3bkqxvQInMSgIZ9HWZ9JXZYAkmXMRMDOyWhHRjy8xWIxxPJnfV6a0dp7zsH/+c0HuHMuoUmb9/x12JLhAGliyKyLcGJaxkzZRgMnXfyzwGZRfl+2srBIJc+quCoXLJKJP1ylnOnW6hy1akP+/crWMOwrDdeCFLemSXKWaYOMSqfXhxa14kLPa8kaI/xTq83WVI7Db4ylmF7zoaOmPcXmwoLYEfFR3O66BMzRLSWviFZ14MGgIhgYRcKTscKGsB7XN/gavFLg9LnCVz0MVB0/kRpfeB7dnQI2QTSXM1c9ctUnKhOjZyloyZ3H9v4tRx2HxxfOqtUCautk6zPWSHmUFm7AFRaQ/hlwblO0WGnc//jQWzcZVxZZdXczMZIqpDWcFCh977ojPGwe6O3Y/ffvPj39GdiUa/obUIi2P5hWkznWOkuuaNfwk44U2C8eznR61SsL42ovSFQ0amtIy0l1fOtGi7LdjZxxlKUyAxSY/VKFkUg7gnNmbb8ZEVQA5OedIQd4EsBVMC9g4rPHQgJWClKUsxi+seC5IK57P7maTQx/jQ3BZRoOjGuZNxgNO420GIFT5gnQMCVpAVnUGQ56EPgrfLLFOUhe0bQmrO2lq962W//vQFyAqADU6VbqYSRHBqQ8gwBkGnKXR6jNVS/URNibgTGQNSgnfwDlWFrvr6R7MGq55xfle34zoLI0rvgKbe2e2sHhZscBdA0K+aup7jzjVaZHZrlmrn51FkwN1ainK8OW2mLZPdujYEYobIaKsX4Db6fPWbSL3h1vmlCeTRgd7EvcSRV2uRwV1zuO2LzKYlAdGb7SC0A7KMC+AVwH9dnx9LB9c3Nzf9IEilokIhXa3mGQcVB/FxzdbRVa5mXRvE1nXSsPUCNaMsO62vESH3Zgn04EkilSDcNRnZ7PQCRw5r/vyKDCc7VAosCEigRg1fx2oWhqdJVths5+0yedCtTNM2QK5QUbVqpJBqaylNfA7we6zzWXNfNNhdYgLddma8/prf1UTtrDylRvf2ykA+gRAC1viD1IHdZeCyBOqXJNbmgk0A8T+Xfnd1TcTbfeptzzhu3jB42aLdKggeh90pP6gNpZDjqfhzFcqxqH5kT6+/DMk6XYczyhVgx2FF/Zg/NvKJQP8ZpfhtvGYwkk6RQecBa5s/GzsjBZHx8spZx4pn71FTqD8TW6MCpTGVftPhWJnvf6mn7lGwNQ4b45W0iuM+GotqlP5cbA1P1L9u+6Q76mqXrwzGvwRbpcssw7i1Zh4T1NKvjTk8DLT6R90PIIzjuFXLHal3fyRwbXi4NG4w+iOHxV9kKD0c0l258zb5jTzKY8h+s7By8+/FUZEtSdz+3pi3+/AkThzW3f81htK83YcncSzjwaunobQQxeseyCcG8B/1lz8fFnRpjdIT28f9rdSPArq5if6rSOu/SpERKsOdU2f3L5o0frea6m8BmBzqZqtl526sviD17muBrWSzqn1O78ffaCjx/OlMfeKLcc8PHP/dQIFPgS0md2Z+YgRsDYD3gbU1fyISDhZs/fAEC+OCoo8sR/rjELnEg1nwSn3oEz3sII5AifpplU/cRh4niG+yPHzy3dnAf6NF9MRnghRnrJ9mwCxgTM+bdTwxCzKJSVuX/WQZs+FmwikY3v3ticMLQAqwXmH+5GVzQas8Atipsqfyn4sS843+zZG7Nnr5O1ECN9VbtjzJ/2ykMtTbIrz8gdsPfxh0KTYzhfnUZU888TigjDeF259ceveNwROW0VKL6+rHwJ8YR+P0T7Xqt/+25Is7YSsaS7J2y+by6WucA6zrkki7NcLH7UX5RwFudGhJ5vrnUJ+jbB5CyDw/BmYJ8VOXzUO3c7ktSfKk/zOBGw1GyHtsVfXEE0/8wfg/eM0a+W+0pe4AAAAASUVORK5CYII=)\n",
    "\n",
    "> 그리고 계산그래프에서는 항상 각 변수의 형상(shape)을 주의하자!!  \n",
    "\n",
    "특히, X와 $\\frac{\\partial L}{\\partial X}$은 같은 형상이고,   \n",
    "$ X=(x_0, x_1, x_2, ... , x_n) $  \n",
    "$ \\frac{\\partial L}{\\partial X} = ( \\frac{\\partial L}{\\partial x_0},\\frac{\\partial L}{\\partial x_1},\\frac{\\partial L}{\\partial x_2}, ... , \\frac{\\partial L}{\\partial x_n} ) $\n",
    "\n",
    "W와 $\\frac{\\partial L}{\\partial W}$도 같은 형상임을 기억하자!! \n",
    "$ W=(w_0, w_1, w_2, ... , w_m) $  \n",
    "$ \\frac{\\partial L}{\\partial W} = ( \\frac{\\partial L}{\\partial w_0},\\frac{\\partial L}{\\partial w_1},\\frac{\\partial L}{\\partial w_2}, ... , \\frac{\\partial L}{\\partial w_m} ) $\n",
    "![](http://mblogthumb4.phinf.naver.net/MjAxNzA3MjZfMjA3/MDAxNTAxMDU4NTg1NDI0.HDfJDox2xTBOmwEERulJw0AO_k5G05dXrzZnSRdmfX8g.MtGGa3V2dvLZvKUqE9kCp2VJWdky8KnfPrzMiQ59wxwg.PNG.cjswo9207/fig_5-26.png?type=w2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 배치용 Affine 계층\n",
    "\n",
    "- 데이터 N개를 묶어서(이게 사실상 배치) 순전파를 한다고 생각하면 됨\n",
    "\n",
    "![](http://mblogthumb1.phinf.naver.net/MjAxNzA3MjZfMTYg/MDAxNTAxMDU3NTM3MzE3.BUonxDiNBpQiy3JoXD-nj2AcvjFh5qtRBXSpks4-NVwg.yzkqjMB4GfgeFPIUErifh4FHFPbNc5aQtrX0ijjXJcEg.PNG.cjswo9207/fig_5-27.png?type=w2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:26:30.239223Z",
     "start_time": "2019-08-03T16:26:30.232633Z"
    }
   },
   "outputs": [],
   "source": [
    "class Affine:\n",
    "    def __init__(self, W, b):\n",
    "        self.W = W\n",
    "        self.b = b\n",
    "        \n",
    "        self.x = None\n",
    "        self.original_x_shape = None\n",
    "        # 가중치와 편향 매개변수의 미분\n",
    "        self.dW = None\n",
    "        self.db = None\n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        # 4차원 데이터인 텐서를 대응해서 구현함\n",
    "        ######################################################################\n",
    "        self.original_x_shape = x.shape # 책에서는 이 부분이 조금 다를 것이라고 말해줌\n",
    "        x = x.reshape(x.shape[0], -1) # 책에서는 이 부분이 조금 다를 것이라고 말해줌\n",
    "        ######################################################################\n",
    "        \n",
    "        self.x = x # 역전파때 다시 이용하려고 보관해둠\n",
    "        out = np.dot(self.x, self.W) + self.b\n",
    "\n",
    "        return out\n",
    "\n",
    "    def backward(self, dout):\n",
    "        dx = np.dot(dout, self.W.T)\n",
    "        self.dW = np.dot(self.x.T, dout)\n",
    "        self.db = np.sum(dout, axis=0)\n",
    "        \n",
    "        # 4차원 데이터인 텐서를 대응해서 구현함\n",
    "        ######################################################################\n",
    "        dx = dx.reshape(*self.original_x_shape)  # 입력 데이터 모양 변경(텐서 대응), 책과 다름\n",
    "        ######################################################################\n",
    "        \n",
    "        return dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Softmax-with-Loss 계층"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://t1.daumcdn.net/cfile/tistory/995A16395B98F76820)\n",
    "\n",
    "> Softmax 계층은 **입력 값을 정규화(출력의 합이 1이 되도록 변형)하여 출력**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Softmax-with-Loss 계층 정리 전\n",
    "- 정리할 때, 정답 레이블만 1이고 나머지는 0이므로 $t_1 + t_2 + t_3 = 1$임을 까먹지 말고 수식 전개\n",
    "- 항상 역전파할 때, 연쇄법칙이 적용된다고 생각하자! 즉, 누적이 계속된다는걸 생각!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://t1.daumcdn.net/cfile/tistory/99EBF5395B98F7792B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Softmax-with-Loss 계층 정리  후"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://mblogthumb2.phinf.naver.net/MjAxNzA3MjZfMTE1/MDAxNTAxMDYwMDMxMTE5.WaUKfNuRPqV3QncvdhQd8PQdDjT8DKTLqnd32dJyATQg.vMyLqXNdA8Uf2aLjG16ryeFmSXJVmB_3dVxld0ISHE8g.PNG.cjswo9207/fig_5-30.png?type=w2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:53:49.278696Z",
     "start_time": "2019-08-03T16:53:49.274591Z"
    }
   },
   "outputs": [],
   "source": [
    "from common.functions import softmax\n",
    "from common.functions import cross_entropy_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:31:06.755506Z",
     "start_time": "2019-08-03T16:31:06.746041Z"
    }
   },
   "outputs": [],
   "source": [
    "class SoftmaxWithLoss:\n",
    "    def __init__(self):\n",
    "        self.loss = None # 손실함수\n",
    "        self.y = None    # softmax의 출력\n",
    "        self.t = None    # 정답 레이블(원-핫 인코딩 형태)\n",
    "        \n",
    "    def forward(self, x, t):\n",
    "        self.t = t\n",
    "        self.y = softmax(x)\n",
    "        self.loss = cross_entropy_error(self.y, self.t)\n",
    "        \n",
    "        return self.loss\n",
    "\n",
    "    def backward(self, dout=1):\n",
    "        batch_size = self.t.shape[0]\n",
    "        if self.t.size == self.y.size: # 정답 레이블이 원-핫 인코딩 형태일 때\n",
    "            dx = (self.y - self.t) / batch_size\n",
    "        else:\n",
    "            dx = self.y.copy()\n",
    "            dx[np.arange(batch_size), self.t] -= 1\n",
    "            dx = dx / batch_size\n",
    "        \n",
    "        return dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 오차역전파법 구현하기 (p179)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 신경망 학습의 전체 그림\n",
    "\n",
    "#### 전제\n",
    "- 신경망에는 적응 가능한 가중치와 편향이 있고, 이 가중치와 편향을 훈련 데이터에 적응하도록 조정하는 과정을 `학습`이라고 함\n",
    "- 신경망 학습은 아래 4단계로 수행됨\n",
    "\n",
    "#### 1단계 - 미니배치\n",
    "- 훈련 데이터 중 일부를 무작위로 가져옴\n",
    "- 이 미니배치의 손실함수 값을 줄이는 것이 목표!\n",
    "\n",
    "#### 2단계 - 기울기 산출\n",
    "- 미니배치의 손실 함수 값을 줄이기 위해 각 가중치 매개변수의 기울기를 구함\n",
    "- 기울기는 손실함수의 값을 가장 작게 하는 방향으로 제시\n",
    "\n",
    "#### 3단계 - 매개변수 업데이트\n",
    "- 가중치 매개변수를 기울기 방향으로 아주 조금씩 업데이트\n",
    "\n",
    "#### 4단계 - 반복\n",
    "- 1~3단계를 반복"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 오차역전파법을 적용한 신경망 구현하기\n",
    "\n",
    "- 신경망을 구성하는 요소를 계층으로 구현한 덕분에 신경망을 쉽게 구축할 수 있음\n",
    "- 더 깊은 신경망을 만들고 싶다면, 단순히 필요한 만큼 계층을 더 추가하면 됨 ex) 레고 블럭"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> <font size=\"5\">\n",
    "    여기서 헷갈리면 안되는 것은<br>\n",
    "    <font color=\"red\"><b>신경망(Net)과 계층(Layer)</b></font>의 개념이다<br>\n",
    "    신경망을 이루는 요소들이 계층이라고 생각하자<br>\n",
    "    즉, 신경망 = A계층 + B계층 + C계층\n",
    "</font></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T16:52:31.725826Z",
     "start_time": "2019-08-03T16:52:31.694985Z"
    }
   },
   "outputs": [],
   "source": [
    "from common.layers import *\n",
    "from common.gradient import numerical_gradient\n",
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T17:03:11.449651Z",
     "start_time": "2019-08-03T17:03:11.434380Z"
    }
   },
   "outputs": [],
   "source": [
    "class TwoLayerNet:\n",
    "\n",
    "    ############################################\n",
    "    # __init__ : 초기화 수행\n",
    "    ############################################\n",
    "    def __init__(self, input_size, hidden_size, output_size, weight_init_std = 0.01):\n",
    "        # 가중치 초기화\n",
    "        self.params = {}\n",
    "        self.params['W1'] = weight_init_std * np.random.randn(input_size, hidden_size)\n",
    "        self.params['b1'] = np.zeros(hidden_size)\n",
    "        self.params['W2'] = weight_init_std * np.random.randn(hidden_size, output_size) \n",
    "        self.params['b2'] = np.zeros(output_size)\n",
    "\n",
    "        # 계층 생성\n",
    "        self.layers = OrderedDict()\n",
    "        \n",
    "        # 1층 \n",
    "        self.layers['Affine1'] = Affine(self.params['W1'], self.params['b1'])\n",
    "        self.layers['Relu1'] = Relu()\n",
    "        \n",
    "        # 2층\n",
    "        self.layers['Affine2'] = Affine(self.params['W2'], self.params['b2'])\n",
    "        self.lastLayer = SoftmaxWithLoss()\n",
    "    \n",
    "    ############################################\n",
    "    # predict : 계층을 사용함으로써 인식 결과를 얻는 처리\n",
    "    # 여기서는 각 레이어의 forward함수를 통해서 처리됨\n",
    "    # TwoLayerNet을 구성하는 모든 레이어에 대해서 순전파 수행\n",
    "    ############################################\n",
    "    def predict(self, x):\n",
    "        for layer in self.layers.values():\n",
    "            x = layer.forward(x)\n",
    "        \n",
    "        return x\n",
    "    \n",
    "    ############################################\n",
    "    # loss : 손실함수의 값을 구함\n",
    "    # x : 입력 데이터, t : 정답 레이블\n",
    "    ############################################\n",
    "    def loss(self, x, t):\n",
    "        y = self.predict(x)\n",
    "        return self.lastLayer.forward(y, t)\n",
    "    \n",
    "    ############################################\n",
    "    # accuracy : 정확도를 구함\n",
    "    ############################################\n",
    "    def accuracy(self, x, t):\n",
    "        y = self.predict(x)\n",
    "        y = np.argmax(y, axis=1)\n",
    "        if t.ndim != 1 : t = np.argmax(t, axis=1)\n",
    "        \n",
    "        accuracy = np.sum(y == t) / float(x.shape[0])\n",
    "        return accuracy\n",
    "        \n",
    "    ############################################\n",
    "    # numerical_gradient : 가중치 매개변수의 기울기를 수치 미분 방식으로 구함\n",
    "    # x : 입력 데이터, t : 정답 레이블\n",
    "    ############################################\n",
    "    def numerical_gradient(self, x, t):\n",
    "        loss_W = lambda W: self.loss(x, t)\n",
    "        \n",
    "        grads = {}\n",
    "        grads['W1'] = numerical_gradient(loss_W, self.params['W1'])\n",
    "        grads['b1'] = numerical_gradient(loss_W, self.params['b1'])\n",
    "        grads['W2'] = numerical_gradient(loss_W, self.params['W2'])\n",
    "        grads['b2'] = numerical_gradient(loss_W, self.params['b2'])\n",
    "        \n",
    "        return grads\n",
    "    \n",
    "    ############################################\n",
    "    # gradient : 계층을 사용해서 기울기를 구하는 처리\n",
    "    # 가중치 매개변수의 기울기를 오차역전파법으로 구함\n",
    "    # TwoLayerNet을 구성하는 모든 레이어에 대해서 역전파 수행\n",
    "    ############################################\n",
    "    def gradient(self, x, t):\n",
    "        # forward\n",
    "        self.loss(x, t)\n",
    "\n",
    "        # backward\n",
    "        dout = 1\n",
    "        dout = self.lastLayer.backward(dout)\n",
    "        \n",
    "        layers = list(self.layers.values())\n",
    "        layers.reverse()\n",
    "        for layer in layers:\n",
    "            dout = layer.backward(dout)\n",
    "\n",
    "        # 결과 저장\n",
    "        grads = {}\n",
    "        grads['W1'], grads['b1'] = self.layers['Affine1'].dW, self.layers['Affine1'].db\n",
    "        grads['W2'], grads['b2'] = self.layers['Affine2'].dW, self.layers['Affine2'].db\n",
    "\n",
    "        return grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Q. **`forward`** 함수와 **`predict`** 함수와의 차이점?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Q. **`backward`** 함수와 **`gradient`** 함수와의 차이점?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 오차역전파법으로 구한 기울기 검증하기\n",
    "\n",
    "- 지금까지 우리가 기울기를 구하는 방법\n",
    "    1. 수치 미분을 통해서 구하는 방법\n",
    "    2. 해석적으로 수식을 풀어서 구하는 방법 → 오차역전파법\n",
    "- 수치 미분의 장/단점\n",
    "    - 장점 : 구현하기 쉬움\n",
    "    - 단점 : 느림\n",
    "- 오차역전파법의 장/단점\n",
    "    - 장점 : 매개변수가 많아도 효율적으로 계산 가능\n",
    "    - 단점 : 구현하기 복잡함 (계산그래프 생각해보자)\n",
    "\n",
    "\n",
    "두 방법의 장점을 섞어서 기울기를 검증하자!\n",
    "\n",
    "> **수치미분 결과** vs **오차역전파법 결과**  \n",
    "> 두 결과를 비교해서 제대로 구현했는지 검증"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-03T17:10:00.422634Z",
     "start_time": "2019-08-03T17:09:55.300565Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W1:4.3706850869906725e-10\n",
      "b1:2.586230168630851e-09\n",
      "W2:6.260090353269149e-09\n",
      "b2:1.4031562908106698e-07\n"
     ]
    }
   ],
   "source": [
    "from dataset.mnist import load_mnist\n",
    "\n",
    "# 데이터 읽기\n",
    "(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True, one_hot_label=True)\n",
    "\n",
    "network = TwoLayerNet(input_size=784, hidden_size=50, output_size=10)\n",
    "\n",
    "x_batch = x_train[:3]\n",
    "t_batch = t_train[:3]\n",
    "\n",
    "grad_numerical = network.numerical_gradient(x_batch, t_batch)\n",
    "grad_backprop = network.gradient(x_batch, t_batch)\n",
    "\n",
    "# 각 가중치의 절대 오차의 평균을 구한다.\n",
    "for key in grad_numerical.keys():\n",
    "    diff = np.average( np.abs(grad_backprop[key] - grad_numerical[key]) )\n",
    "    print(key + \":\" + str(diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 오차역전파법을 사용한 학습 구현하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T05:49:30.406178Z",
     "start_time": "2019-08-04T05:49:14.004727Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1epoch => train acc, test acc | 9.675000%, 8.950000%\n",
      "  2epoch => train acc, test acc | 90.370000%, 90.590000%\n",
      "  3epoch => train acc, test acc | 92.401667%, 92.790000%\n",
      "  4epoch => train acc, test acc | 93.758333%, 93.700000%\n",
      "  5epoch => train acc, test acc | 94.575000%, 94.520000%\n",
      "  6epoch => train acc, test acc | 95.090000%, 94.910000%\n",
      "  7epoch => train acc, test acc | 95.546667%, 95.300000%\n",
      "  8epoch => train acc, test acc | 96.078333%, 95.700000%\n",
      "  9epoch => train acc, test acc | 96.286667%, 95.850000%\n",
      " 10epoch => train acc, test acc | 96.606667%, 96.170000%\n",
      " 11epoch => train acc, test acc | 96.755000%, 96.310000%\n",
      " 12epoch => train acc, test acc | 96.966667%, 96.500000%\n",
      " 13epoch => train acc, test acc | 97.325000%, 96.740000%\n",
      " 14epoch => train acc, test acc | 97.390000%, 96.830000%\n",
      " 15epoch => train acc, test acc | 97.518333%, 96.870000%\n",
      " 16epoch => train acc, test acc | 97.610000%, 96.890000%\n",
      " 17epoch => train acc, test acc | 97.815000%, 97.090000%\n"
     ]
    }
   ],
   "source": [
    "# 데이터 읽기\n",
    "(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True, one_hot_label=True)\n",
    "\n",
    "network = TwoLayerNet(input_size=784, hidden_size=50, output_size=10)\n",
    "\n",
    "iters_num = 10000\n",
    "train_size = x_train.shape[0]\n",
    "batch_size = 100\n",
    "learning_rate = 0.1\n",
    "\n",
    "train_loss_list = []\n",
    "train_acc_list = []\n",
    "test_acc_list = []\n",
    "\n",
    "iter_per_epoch = max(train_size / batch_size, 1)\n",
    "\n",
    "for i in range(iters_num):\n",
    "    batch_mask = np.random.choice(train_size, batch_size)\n",
    "    x_batch = x_train[batch_mask]\n",
    "    t_batch = t_train[batch_mask]\n",
    "    \n",
    "    # 기울기 계산\n",
    "    # 오차역전파법으로 기울기를 구함\n",
    "    #grad = network.numerical_gradient(x_batch, t_batch) # 수치 미분 방식\n",
    "    grad = network.gradient(x_batch, t_batch) # 오차역전파법 방식(훨씬 빠르다)\n",
    "    \n",
    "    # 갱신\n",
    "    for key in ('W1', 'b1', 'W2', 'b2'):\n",
    "        network.params[key] -= learning_rate * grad[key]\n",
    "    \n",
    "    loss = network.loss(x_batch, t_batch)\n",
    "    train_loss_list.append(loss)\n",
    "    \n",
    "    if i % iter_per_epoch == 0:\n",
    "        train_acc = network.accuracy(x_train, t_train)\n",
    "        test_acc = network.accuracy(x_test, t_test)\n",
    "        train_acc_list.append(train_acc)\n",
    "        test_acc_list.append(test_acc)\n",
    "        print(\"{:3d}epoch => train acc, test acc | {:.6f}%, {:.6f}%\".format(int(i // iter_per_epoch)+1,train_acc*100,test_acc*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 여러가지 테스트를 해볼까?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T06:07:24.992360Z",
     "start_time": "2019-08-04T06:07:24.971351Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def mnist_test(input_size=784,hidden_size=100,iters_num=10000,batch_size=100,learning_rate=0.1):\n",
    "    # 데이터 읽기\n",
    "    (x_train, t_train), (x_test, t_test) = load_mnist(normalize=True, one_hot_label=True)\n",
    "\n",
    "    network = TwoLayerNet(input_size=input_size, hidden_size=hidden_size, output_size=10)\n",
    "\n",
    "    train_size = x_train.shape[0]\n",
    "\n",
    "    train_loss_list = []\n",
    "    train_acc_list = []\n",
    "    test_acc_list = []\n",
    "\n",
    "    iter_per_epoch = max(train_size / batch_size, 1)\n",
    "\n",
    "    for i in range(iters_num):\n",
    "        batch_mask = np.random.choice(train_size, batch_size)\n",
    "        x_batch = x_train[batch_mask]\n",
    "        t_batch = t_train[batch_mask]\n",
    "\n",
    "        # 기울기 계산\n",
    "        # 오차역전파법으로 기울기를 구함\n",
    "        #grad = network.numerical_gradient(x_batch, t_batch) # 수치 미분 방식\n",
    "        grad = network.gradient(x_batch, t_batch) # 오차역전파법 방식(훨씬 빠르다)\n",
    "\n",
    "        # 갱신\n",
    "        for key in ('W1', 'b1', 'W2', 'b2'):\n",
    "            network.params[key] -= learning_rate * grad[key]\n",
    "\n",
    "        loss = network.loss(x_batch, t_batch)\n",
    "        train_loss_list.append(loss)\n",
    "\n",
    "        if i % iter_per_epoch == 0:\n",
    "            train_acc = network.accuracy(x_train, t_train)\n",
    "            test_acc = network.accuracy(x_test, t_test)\n",
    "            train_acc_list.append(train_acc)\n",
    "            test_acc_list.append(test_acc)\n",
    "            print(\"{:3d}epoch => train acc, test acc | {:.6f}%, {:.6f}%\".format(int(i // iter_per_epoch)+1,train_acc*100,test_acc*100))\n",
    "            \n",
    "    # 그래프 그리기\n",
    "    x = np.arange(len(train_acc_list))\n",
    "    plt.plot(x, train_acc_list, label='train acc')\n",
    "    plt.plot(x, test_acc_list, label='test acc', linestyle='--')\n",
    "    plt.xlabel(\"epochs\")\n",
    "    plt.ylabel(\"accuracy\")\n",
    "    plt.ylim(0, 1.0)\n",
    "    plt.legend(loc='lower right')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 은닉층의 노드수를 늘려보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T06:07:47.914253Z",
     "start_time": "2019-08-04T06:07:27.216928Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1epoch => train acc, test acc | 9.638333%, 9.260000%\n",
      "  2epoch => train acc, test acc | 90.455000%, 90.850000%\n",
      "  3epoch => train acc, test acc | 92.508333%, 92.690000%\n",
      "  4epoch => train acc, test acc | 93.910000%, 93.980000%\n",
      "  5epoch => train acc, test acc | 94.945000%, 94.770000%\n",
      "  6epoch => train acc, test acc | 95.576667%, 95.270000%\n",
      "  7epoch => train acc, test acc | 96.250000%, 95.830000%\n",
      "  8epoch => train acc, test acc | 96.045000%, 95.690000%\n",
      "  9epoch => train acc, test acc | 96.843333%, 96.310000%\n",
      " 10epoch => train acc, test acc | 97.098333%, 96.600000%\n",
      " 11epoch => train acc, test acc | 97.346667%, 96.790000%\n",
      " 12epoch => train acc, test acc | 97.575000%, 96.770000%\n",
      " 13epoch => train acc, test acc | 97.850000%, 97.020000%\n",
      " 14epoch => train acc, test acc | 97.921667%, 97.270000%\n",
      " 15epoch => train acc, test acc | 97.911667%, 97.090000%\n",
      " 16epoch => train acc, test acc | 98.170000%, 97.280000%\n",
      " 17epoch => train acc, test acc | 98.256667%, 97.290000%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAENCAYAAADkNanAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcVOWd7/HPr5ZeoDe6gQYEWQdFoxkNioJEVBRxiYoTl8Q96k2cm0SjUTMTryaTUUYlhknuTTRxNPfqJHFiosa4EFFcSETAEB00KhGEZumN7oZea3vuH+eARXeDXdDVp6C+79erXlXnOU9Vfaugz6/O8pxjzjlERETShYIOICIiuUfFQUREelBxEBGRHlQcRESkBxUHERHpQcVBRER6UHEQEZEeBqw4mNm3zOzLe5h/u5mtMLM3zez8gcolIiI9Zb04mNlBZvY68I099JkLVDnnpgLTgevMbGi2s4mISO+yXhyccxudc8cB39xDt6uB+X7/TuCnwMXZziYiIr2LBB3AV+2c25Q2/QFwfPdOZnYtcC3A4MGDP3PooYcOUDwRkQPDypUrG5xzwz6pX64UB+ulrcdJn5xzDwAPAEydOtWtWLEi27lERA4oZvZRX/rlytFK9WY2Im16MrAmqDAiIvkukDUHMxsO/LNz7ut+08/x9kncaGYFwGXAJUFkExHZk0QyRSyZoiuefp+kM56iK5EilkjRlUj696md94lUimTKkXKQSjlSzpF0Dufw253fDknXbTrlcH7/ZArGVBZz3axJWf2cQW1WKsVbOwDAOfdbMzvGzJbhbU660znXGFA2EekmlXLE/IViLJEinvx4odcV9xaGnf59l79w7PIXlj0epz2ny3+NeDKFc3gLROctEJ1/n76Q3HXex207nptM9bwEgaVttLa0Ldi7tqf3/3gqkUr1WMj39h79KRwyQgYhM0JmhEOG2Y5273bk6PKsZoABLA7OuYfTHv8NmNtt/j8NVBaRXJdI7rpAiiW8X6dd6dPp85PJXaY/fs7Hv2xjaW2xhL+wTySJJ93OefEd75v8uADEEikS/bBADBkURcMURkIURsIURkMURkIUREJEwyF/wQdhUhQTp8DipEIFxMLFREkyJvERBcQpIE7Uv68rGEtD4WgGp1o5ZvsLRF0cDJIWIUmEDwcdSW3heIqSrUxqe5OUhUlahARhkhalrmAM2yNDiKY6GRKvJUnEe65FiFuEZKSUSLSQwaEYValGikL4tySFoQTt5ZOx4grK43UM3baaAksSJbHzvmvSmURKhzNo62qKP1pMCK/4mHnFKXX0lYQGDyW0cTnhtUu8wpR+jZ3jvgJFZbDuNe+2w8yb9vnf45Pkyg5pkf1SeyxB/fYuGlq7qN8eo6HVe9zWlejxizP9l/LH85I9ikBXIklvy+JiOikiRhRvwRO1BADr3EgAJtsGqmwbkR3zSRILFbI8fDQFkRCn2huMsRYiYSMUChEJGa2RKlaVzKCsKMK0ztcoc62EwyEi4RDhkNFeOJyaqhkUREIcuvVFil0nkbARDoWIWopEySi2H/RZCqMhxr37AAWJVqKui4hLEEl1kRp9LHzmcgojIaL/+Q8Q74BkFyT825EXwIk3Q6wd7pnotbnkxx965k1wym3QWg/3zu35pZxyO8z8HGxdC/++sOf8MxfAMefB5r/A/bf1nH/e/fDpk+GjP8JDl/ecf9Ev4NAz4L3n4BcX9px/2VMw4VPw9uvw3P/sOf/Q46BqHKxbDa/d1XP+p86D8uGwcTksubPn/KMv/bg4LEl7/ozrIZzdxbftr1eC09FK+ck5x7bOBE1tMZraYzggGgoRjRjRcIiCsPcrNBo2opGPp8Oh3g6I6117LEHD9hj1rV07F/wNuzyO7XzcEYvjMMA4iHomhjZRQRvlkRiDwwkGh+I8XnAOkWgBJyX/xGdSf6GIOEUWp5AYEVI8NPbfKIiEOKPup3yq+UWiLkYk1UXExUiEivjt7FcoiISYvvIGRm1atEvWWMlBrL1kGQWRECOf+gJF65fs+mGGHQr/uMx7/OBpsGHZrvNHHwNXv+A9/j/ToW71rvMnzILLnvQe/+AIaF6/6/xDz4KLHvUe3z0RurZBuBAiBRApgsPPgzn/6s1/+CzvPlL4cZ9Jp8JRX4RUEl643W8vhLD//IOOhjHHQiIGHzz/8fwdfcpHQ8lwSCago8l7TfCmU3EoKIHCEq8oNa6BZBxSCf8+7n0/pSO84rP2ZX9ezJufjMPfnQpVE2HbJlj7qrdADhdAKArhKIw6CgZVQvtWaKnx2sIF3n0oCoOHeo9TSe9m3f4fhiL+KkSKXQ/Q9PvtWMXobTnd/bX6yMxW+gOO99xPxUGC0n1B39QeY2tbvNt0jKa2OE3tMba1tRPtqKfEtVFOG2XWRgjHX1IT2EIVZbRyWGg9CRciSdjbdECI9W447TaI0nCc6nArFo4QCkewcIRwOEoiPIhQJEK8s41w6xYK4i1UWBvltFJhrTyRnME2Sjin+C98IbyYSmujjFZK3HaKE9v4/WkvUVo1isPe+xHD3+zl1+u3aqCwFBZ/F1b+3FvoRYu8+0iRt3A2g+UPer9g0+cVlXu/rAHeXwRN63ZdQBWWer9swft13LnNXzj5faKDvIUbeAuwVCJtQeP8BViVN9la5y0QdyyknPMWwiXDvemWjd7zcd68UMRb8BYP8eanUhDKlQMgZXdUHGRAdcaTNLd7C/Hm9jjN7TGaO7zpFr+9pa2LVGs9yfZmUp3NhDpbKHFt/NWN4X03hmE0843If1FmbZRbO0NCHVSE2ni87HLeqZrDEbaGf1xzbY/3fuu4+6g5aC7lW/7IjKVX9Zj/5GE/4IPy6YxtWMLnP7i5x/x7R93HOwVHcFLnC1y6ZX6P+Y2XvEDZ+M8Qfec38McfegvDHbdBlXDcdd590zrYvgWKKryFZqTYW7gWDN7rX3ki/U3FQfZJVyLJlpZONjZ1sLGpna3tcZo74hQ2vkuivYVkRzOucxvW1co7sWG8HD+MCAnujj5AKR2UWjultFNCB792J/NY8ecZW9TBY9t6HqG8+pCvUnfU1xhOI4c89TmsuJxQ8RCsqByKK+DvvwgTT/J++b77O6+tqNy7WRgqxngL6o4m2PLf3q/bVNK/T3ibTkqroXnDx5sOdvRJxr1NH+UHedutNyzbdeG/4xYKB/CvINL/VBxkt5xzNLd2ULt5A1vrNlLXGmN18mA2NXcyc+NPKe+ooSSxlaHWwjBr4eXUkdwYv45o2Phz9CpK6Njl9VYNPZulh9/BkOIo57x6NhQMgqIywsXlRIrLiUw5Az41z9sOvPIh75f1zgV8hbfgLsr+oXkiouKQv1JJUk3radzwLtvq1tPRtJmtXWGeLTmXjc2dXLPpNg6Pr6aCVkLm/du/nprCFe52RlUU82DnjZTTRqyoitSgYYRKhxMZexzFx17GoIIw9sEib1t2YZl3FEVhqbdgjxYH/MFFpC/6Whx0KOv+yDlorSVe9wFb179Dc2Mti4ZczJr6Vi79241Mja9kGLDjzFp/TY3hDwXHM6qimJbSyayNjMBKhlNYMYKSyoM4dNRE3p0w1R/8s3LP7z15TpY/nIjkAhWHXNbRBI0f0lX3Pu8Pm8uahlaGr7yPz2x8lCLXQRSoBkpcIfd1HcXIisGMLz+bTYNPo3jEZCqqxzFs5BjGDa1kRXTHNvMTAvxAIrK/UHHIBZ0tEB1Mc1eK+jcep2LVTxjUuo7BiWYACoGrO39ELZV8LjKI2uJT6CofT2TYJMpGT2HUmL/j7eoyBhdGgJMD/SgicmBQcRhoiRjUvo2rWUnb2jdI1aykrPVDvjrobn63dTSnhd7jynCMDXY020vGkRoykUEjJ/Pdg6cwcUQFB1fOpSCiY8lFJLtUHLIplfJGZW5cSWL44fzVjWX9m4s4Y+XVGNDhylmVmshfwxdSNGQU3zzmEA4bdQyjh93MtIpiQhmM6hUR6U8qDv0t1g6v3E2yZiVu45tE4q0APODO5+6u8ykmxNLSm4kePJVJkw7hmPFVnDK8RIVARHKKisO+Wv4g/O1F2son8eqYL7N8bSNff/NBPkpW8ZfUNP7iJrKt8kiqJxzJv08YxtSxQxhVcX7QqUVE9kjFYR+41jrs99+g1obyq3iE7yfepCASYvXoX/CZ8UOZOq6Ssw8eQnlxNOioIiIZUXHYB42b1jIU+I+yL1N59DweH1fJpw4qozCiUy2IyP5NxWEftNR9xFBg9rF/zzEzJgYdR0Sk36g47IOtHbAtNYmy6vFBRxER6Vc6YH4fvF18DOfFvsuwkQcHHUVEpF+pOOyD2m2dFERCDBmkHc4icmDRZqV9cPo7t3BUUQizXq5tKyKyH9Oawz4Y3vE3yiPxoGOIiPQ7FYe95RxDkg10FVcHnUREpN+pOOwl19nCIDpJlo4KOoqISL9TcdhLLbXrAYiUqTiIyIFHO6T3Ul1bkqXJY6kYcWjQUURE+p3WHPbShtAo/jF+PYPHHh10FBGRfqfisJc2N3cAMLK8KOAkIiL9T5uV9tKnVt3BC4XLGFryTtBRRET6ndYc9lJB2xZSoULCukiPiByAVBz20qCuOrZFhwYdQ0QkK1Qc9lJ5ooGOIg2AE5EDk4rD3kh0McS1kCgZGXQSEZGs0A7pvbC9rY3HEnMZXj016CgiIlmhNYe9sKUzyr8kLsWN+2zQUUREsiLrxcHMqs3sWTNbZmZ/MLNezzdhZnea2Wt+vx+bWc6u1dQ2bqWQGCPKNMZBRA5MA7HmsAC4wzk3DfgWcG/3DmZ2OjDUOXeC3y8CnDMA2fZK6ds/572iKxhVFAs6iohIVmS1OJhZBVDlnFsG4JxbAZT77enCQE3adAOwMZvZ9kWqZRNtrpBhQ3Uoq4gcmLK95jAeWNOt7UO/Pd0zQKmZ3WdmC4E659zr3V/MzK41sxVmtqK+vj47ifsg0raZequiMJqzW75ERPZJtouDAa6X9u5txwFRYCHwL8BoM7uix5Oce8A5N9U5N3XYsGH9nbXPijvraI5orUFEDlzZ/um7DpjUrW2i357uLuA051wMwMxuApYCD2c33t4pjddTN+jTQccQEcmarK45OOe2Au1mdjSAmR2Jtz+hwN98tEMb3trDDhcAf81mtn3xsDubNcNOCzqGiEjWDMRG8+uBB82sBGgBrgRKgclpfb4M/NDMvou3yel94GsDkC1jnfEkP+6YzU1jJn9yZxGR/VTWi4NzrgaY0625Fpib1mcDcG62s/SHuoYGxtlmRpYeFnQUEZGs0QjpDHW89xJLCm9kUmpt0FFERLJGxSFDnY0bACivHhdsEBGRLFJxyFCyZSNxF2bYiNFBRxERyRoVhwyFWzdTb0MYXFQQdBQRkaxRcchQYUctTWENgBORA5vO/5ChX0bPo3QwHB50EBGRLNKaQ4ae7Tyc2hGzgo4hIpJVKg4ZiHe1M6H1TcYN6go6iohIVqk4ZKCp5j1+UfCvfDr256CjiIhklYpDBlrq1gNQXKnDWEXkwKbikIGOBm8AXFn12ICTiIhkl4pDBuJN3sXpho5UcRCRA5uKQwZs+ya2ulLKS0uCjiIiklUa55CBZwafR3PRp7nXLOgoIiJZpeKQgb90DidcVR10DBGRrNNmpQwc3vgHPl1UF3QMEZGsU3Hoo1Ssgzvi32dGbGnQUUREsk7FoY+a/TEO4fJRAScREck+FYc+at7yEQCFGgAnInlAxaGP2vwBcCXDDg44iYhI9qk49FGsqQaAqpHjgg0iIjIAdChrH/2x9HTuilfwq0pd6EdEDnxac+ijte2FbC49gnBYX5mIHPi05tBHh2x+ivKiIcDJQUcREck6/Qzuo3Obf86c5CtBxxARGRAqDn3gkgmqUltJDB4ZdBQRkQGh4tAH27duJmIpKNMAOBHJDyoOfdDkD4ArGHJQwElERAaGikMfbPdPnaEBcCKSL1Qc+uCdkuOY1vkjysZ9OugoIiIDQsWhDzZvT1JLJcMryoKOIiIyIFQc+mDUut/w5UEvURDR1yUi+UGD4Prg8PpnODzUFXQMEZEBo5/CfVAWr6etcHjQMUREBkzWi4OZVZvZs2a2zMz+YGa9DhYws1PN7FUzW2xmz5jZ4Gxn6xPnqEo1EBs0IugkIiIDZiA2Ky0A7nDOLTOzqcC9wBfSO5jZeOCfgbOdc81mNgJIDUC2T9SxvYliuqBUo6NFJH9kdc3BzCqAKufcMgDn3Aqg3G9P903gNudcs99vi3OuI5vZ+qqxdgMJFyKqAXAikkeyvVlpPLCmW9uHfnu6I4AuM3vSzF4xszvNrEc2M7vWzFaY2Yr6+vosRd7VhtBoJnf9X5KHnD0g7ycikguyXRwMcL20d28bBHwRuBQ40Z++qseTnHvAOTfVOTd12LBh/Z21V1u2dZAiRPWQ0gF5PxGRXJDt4rAOmNStbaLfnm49cLdzbptzzgGPAH+f5Wx9Mvj9J/le5EFGlBUGHUVEZMD0uTiY2ehMX9w5txVoN7Oj/dc4EmgACsxsYVrX/w3cZWZhf/ocYFmm75cNlXWvc0ZkOYMKo0FHEREZMJkcrbTAzMqB54BfO+dq+vi864EHzawEaAGuBEqByTs6OOdeMLNJwCtmlgD+iLf2ELjC9loaQ0OpDDqIiMgA6nNxcM5daGYFeNfJ/CczGw4sxSsUG/bwvBpgTrfmWmBut34/AX7S1zwDZXCsjpaC6qBjiIgMqIz2OTjnYsBLwMt44xCOAL5nZvdlIVtOqEw20DVIxUFE8kuf1xzMbBZwCTAS+DVwlXOu1Z/3x6ykC1g81kWbKyBRmvHuFhGR/Vom+xzmAPOdc93HLQCc2U95ckpde4oZXT9k/uFHBB1FRGRAZbJZ6b+BMTsmzGyOmV0A4Jxr6u9guWBLizdIe0R5UcBJREQGVibF4SpgSdr0IuCr/ZomxyTfe4EHo/dwUHRb0FFERAZURife8weo7XxsZsn+j5Q7XN1qTgn/mZYhOpBVRPJLJmsOfzaz/2mekJl9A3gzW8FywrZNtLkiyiqGBJ1ERGRAZVIcbsU759HLeJuXonhnUz1gFbRvoSE0FDMLOoqIyIDKZBBcArjbvwHg75B+LAu5csKgrjq2RYcGHUNEZMBlMs5hCnAHMBrvbKvgrUkcsMVhS7KcVMW4oGOIiAy4TDYr/QzvKm6r8U598f+An2cjVC5IpRxf6rye5ZO/EXQUEZEBl0lxaHfOLQfeBsY6534MfC47sYLX2BYjkXIa4yAieSmT4rDBzP4OeAr4jpldTo5c5zkbmj9cybMFt3BI/K9BRxERGXCZFIergXXOuY+Au4AK4KKspMoBbbV/Y0poA5VlJUFHEREZcJkMgnveOXcqgHPuDeCN7ETKDV1bvctVDBkxNuAkIiIDL5M1h+1mdnLWkuQYt20TMRemctiooKOIiAy4TNYchgIvmNmHQL3f5pxz0/s/VvAirZtptEpGhsOf3FlE5ACTySC4z2YzSK750I2ksTjKyKCDiIgEIJNBcH8Cdpx4bygwHviDc+6MbAQL2k/c+UwZU9bj+qYiIvkgkzWH49OnzWwecEq/J8oBzjk2t3Ry8qHDg44iIhKIjK4hnc459xu8a0gfcLY1NfCn0FXMbF8UdBQRkUBksllpcvokMA3o7PdEOaBxy1omWBulJaVBRxERCUQmRyt9C2+fg/n3a4GLsxEqaK216wEYNHTMJ/QUETkwZVIcXgA2O+deBO8a0nj7HA64s7J2+gPgKqo1AE5E8lOm15B+KW36gL2GdKJ5IwBVI1UcRCQ/ZbRDuvs1pIED8hrSa200T9hsogU6I6uI5CddQ7oXizie/6i6IegYIiKB0TWke9HYvJ0RZVprEJH8lUlxmAX8yTn3Wf9UGm8AB+QpNR7ZdhlXbH8g6BgiIoHJdM0h/TTdfwRu7984wWtv204FrYRLqoKOIiISmEx3SHf19vhA0rD5IwAiFTpVt4jkr0yKwzozO3vHhH9upQ/7P1KwttV6xaG46uCAk4iIBCeT4vB1YK6ZvWZmS4FTgcezEys47Y0bACgbrjEOIpK/MikOpcB2YL1/GwIszEaoIK3jIO5PnEnVqPFBRxERCUwmxeFRYIX/nDuBdcC/ZSFToN5OjefHBVdQXFIedBQRkcBkUhycc+6/8ApEzDl3K/DFT3qSmVWb2bNmtszM/mBmu93Ta2aFZva2mZ2eQa5+1bF1I2N1MlYRyXOZnHiv0cyqgWeAW8zs5318/gLgDufcMjObCtwLfGE3fb+Ht0YSmKs33UYiMhg4LcgYIiKBymTN4X8AcefcO8BTwLl4J+PbLTOrAKqcc8sAnHMrgHK/vXvf04CtwMoMMvW7IYl6Oourg4wgIhK4PhcH51yzc26r//gJ59wNzrn3P+Fp44E13do+9Nt3MrMq4Grg7j29mJlda2YrzGxFfX19X6P3WSwWp8o1kyoZ2e+vLSKyP9nry4T20Y4LA3XXve37wC3OuT2e5dU594BzbqpzbuqwYcP6K+NODbUbiFiKULkGwIlIfstkn8PeWAdM6tY2kbT9Cv4O6qnAQ2YGMA6YZ2afcc79a5bz7aJlyzpGAUWVugKciOS3rBYH59xWM2s3s6Odc2+a2ZFAA1BgZgudc193zm0CDt/xHDO7A3jdOfdcNrP1piY5hMfil3Lp2KMG+q1FRHJKttccAK4HHjSzEqAFuBJvQN3kAXjvjHwUL+eh5FyuHzUh6CgiIoHKenFwztUAc7o11wJzd9P/jmxn2p3O2jVMKailrGggaqaISO7K9g7p/cqxH93Pf4Tvwt/3ISKSt1Qc0hR31rEt2v9HQYmI7G9UHNKUJ+ppLxoedAwRkcCpOPiSyRTDUo0kBmsAnIiIioNva2MdxRbDyjQATkRExcG3pR3+R+x6uiacGnQUEZHAqTj4NrXB86ljKR89JegoIiKBU3HwtW/6KyeE3mZESTjoKCIigVNx8A1d9zSPFNxF5aBo0FFERAKn4uCLtG2mkQpC0cKgo4iIBE7FwVfUUUtLZGjQMUREcoKKg680Xk9boQbAiYiAigMAzjmGJhuIDRoRdBQRkZyg4gC0dMS5LHYLayddFnQUEZGcoOIAbG7p5C03kUGjDg06iohITlBxAJo3reHz4SUcVNQZdBQRkZyg4gC49cu4J/oAI8Pbg44iIpITVByARPNGAIaMGBtwEhGR3KDiAIRaN9HKIKKDyoOOIiKSE1QcgML2WprCVUHHEBHJGSoOQEmsju0FujyoiMgOkaAD5IKvJG/i3AlVHBZ0EBGRHJH3aw5tXQnWdZZQNHx80FFERHJG3heHurrNfDX8GybZpqCjiIjkjLwvDts2vs+N0V8zms1BRxERyRl5XxzaGjYAUDb84ICTiIjkjrwvDonmGgAqR4wLNoiISA7J++LAtk3EiFBUpms5iIjskPfFIdJey9ZQFYTy/qsQEdkp78c53FXwNcYOSfKjoIOIiOSQvP+5vHlbjNIhGh0tIpIur4tDVzzB9Z0/ZlpqVdBRRERySl4Xh/r6Oi6JLGZcan3QUUREckpeF4fm2nUAFFSODjaIiEiOyXpxMLNqM3vWzJaZ2R/MbFQvfSrM7H+b2Utm9iczW2Bmlu1s2+u9AXClwzQATkQk3UCsOSwA7nDOTQO+BdzbS5/BwAPOuZOcc8cDpcDcbAeLbfWKg64AJyKyq6weympmFUCVc24ZgHNuhZmVm1mFc655Rz/n3EZgY9pTPwIKspkNoKu1hS4XpaRKm5VERNJle81hPLCmW9uHfnuvzGwmcDzwdC/zrjWzFWa2or6+fp/DPVF8LmeWPgaRrNchEZH9SraLgwGul/Yebeb5BnA+8A/OuUSPJzn3gHNuqnNu6rBh+z42YXNLJyMqBu/z64iIHGiyPUJ6HTCpW9tEv727/wCeds59P8uZdrqkYSE2fAowbaDeUkRkv5DVNQfn3Fag3cyOBjCzI4EGoMDMFu7oZ2afA1Y45x7PZp50yZTj5ORSxruagXpLEZH9xkCcW+l64EEzKwFagCvxjkaanNbnBGCWmX0+re3XzrmsnfKooamZamtlXdnIbL2FiGRBPB6npqaGzs7OoKPktKKiIkaPHk00Gt2r52e9ODjnaoA53ZprSTtU1Tl3c7ZzdNe45SOqgYIhBw30W4vIPqipqaG0tJRx48YxAMOh9kvOORobG6mpqWH8+N0e/7NHeTtCurX2IwAGawCcyH6ls7OTqqoqFYY9MDOqqqr2ae0qb4tDc2srG1LDKK/eu6oqIsFRYfhk+/od5W1xWBk9mlOSP6RizJSgo4iI5Jy8LQ61LZ2MKC/SLxARydjDDz+cUf9bb701O0GyKG+Lw+wNC7nNPRB0DBHZD2VaHObPn5+dIFmUt8VhQsd/M8bqgo4hIvuZefPmsWrVKmbNmsUrr7zCvHnzOOWUU7jkkktoamriggsu4OSTT+akk05i06ZNABx33HGsW7eO2267jYsuuoiZM2dyzjnn0N7e3uP1b7rpJmbPns3UqVN55ZVXdrbPnz+fk046iZNPPpl77rkH8IrUiSeeyOzZs7nhhhv69XPm5TWknXNUJhupHXRI0FFEZB9853ereWfTtn59zcNGlXH72Yfvdv5vfvMbZs2axZIlS1i3bh1PP/00q1at4rDDDqOhoYGFCxcycuRIHnvsMX7wgx9w991373zuz372M5YuXcqECROYP38+Dz/8MNddd90ur3/55Zdz7733UlNTw4UXXsjSpUv51a9+RV1dHS+99BIAixYt4vXXX+eZZ57hxRdfJBwO8/zzz/fr95CXxaG5tYNhNLGlVAPgRGTfnHDCCRx22GEADB06lNWrV/PII4+wfPlywuHwLn3PO+88JkyYAHhrE70t0IcPH87999/PW2+9xZYtWwB4/PHH+f73Pz6z0GmnncbNN9/M1772tZ3vMWdO9+Fk+yYvi0P9lg0MMUekQgPgRPZne/qFP1CGDx++8/Fvf/tbnnzySW655RbmzJnDnXfeudu+4G3FSLdu3TquueYavvvd73LZZZfx6quv9tpvd239KS/3OTQ2b2NZ6lAKR0z+5M4iIt0kEj1OGg3Ayy+/zOWXX86UKVN49NFHM16Ar1q1iulj5STHAAALuklEQVTTp3P88cezePFiGhsbAW8/R/rmqWeeeYbzzjuP++67b2eW3//+93v5aXqXl8VhbaqaC2P/i9JDTwk6iojsh6qrq5k5cybr16/fpf0rX/kK3/72t/nsZz/LhAkTWL58OUuWLOnz686ZM4c///nPnHjiiSxZsoQjjjiCBQsWcPHFFzNixAhmzpzJ9OnT+eCDD5g+fTpnnnkms2bNYvr06TvXMvqLZXvVJFumTp3qVqxYsVfP/f6i9/jRS2t4/3tziYTzsj6K7LfeffddpkzR4NW+6O27MrOVzrmpn/TcvFwyHv7B/TxT9G0iIQ2AExHpTV4Wh5K2tVRYO2h0tIhIr/KzOHTV0Vqw75cZFRE5UOVlcahINNBRXB10DBGRnJV3xaG1M041jaRKNABORGR38m4QXCLWwYeVsxg0flrQUUREclbeFYeKsjIqvv7roGOIiOS0vNusJCKyrzI9ZTdAc3MzTzzxRP+HyRIVBxGRDKk4iIjILtKv57B+/Xpee+01Zs+ezYknnsh3vvMdAJYtW8Zxxx3HtGnTOOOMM2hpaeGiiy7iueeeY9asWcRisV1e89FHH915DYeFCxfubH/22WeZNWsWp556KpdeeinxeJzly5cze/ZsTj31VObNm0dTU1NWPmfe7XMQkQPMQ2f2bDv8XDj2Goi1w6Of7zn/778AR30R2hrhsct2nXflnk9gl349h+bmZm688UYWL15MSUkJX/rSl3j99deZP38+jzzyCJMmTeLdd98lmUzyy1/+kjvuuKPXtY4jjzySF154gUQiwbRp07jiiitobm5mwYIFPPvssxQXF/Pqq6/S0dHBDTfcwFNPPUVlZSWrV6+mra2NIUOGZPCF9Y2Kg4jIXnrttdeor6/nrLPOAqCtrY3Vq1fz+c9/nquvvpqrr76auXPnUllZybZtu78o0aRJk3jsscdYtmwZTU1NbNmyhcWLF3PFFVdQXFwMwMyZM3nmmWc4/fTTqaysBODww7N3ynIVBxHZv+3pl37BoD3PH1z1iWsKe2JmXHnlldx222095p111lk8//zzzJs3j3vuuafHtRzSfe5zn+MrX/kKt912Gxs2bMA5F8g1HNJpn4OISIZ2XENhxowZLFq0aOd2//fff5/Fixfz8ssvU1ZWxgUXXMBpp53GihUrCIVCPfY1gLejOhaLMW/ePLZv386LL76Ic44zzjiDBx98kLa2NgCWLFnCCSecwNNPP01DQwMAb731Fhs2bMjKZ9Sag4hIhnZcz+HRRx/lnnvu4fzzzyeVSjF27FgWLFjAk08+ye23345zjhEjRvDwww8TjUZZv349s2bNYtGiRRQUFABQUVHBjBkzmD59OtXV1VxzzTXceuutPPnkk9x0003MnTuXVCrFoYceyowZM1i4cCEXXHAB8XickSNHcv/992flM+bl9RxEZP+l6zn0na7nICIi/UrFQUREelBxEBGRHlQcRGS/s7/uKx1I+/odqTiIyH6lqKiIxsZGFYg9cM7R2NhIUVHRXr+GDmUVkf3K6NGjqampob6+PugoOa2oqIjRo0fv9fNVHERkvxKNRhk/fnzQMQ54Wd+sZGbVZvasmS0zsz+Y2ajd9LvdzFaY2Ztmdn62c4mIyO4NxD6HBcAdzrlpwLeAe7t3MLO5QJU/MGM6cJ2ZDR2AbCIi0ousFgczq8Bb6C8DcM6tAMr99nRXA/P9Pp3AT4GLs5lNRER2L9v7HMYDa7q1fei3/zmtrdo5tylt+gPg+O4vZmbXAtf6k61m9t4+ZBsKNOzD87NFuTKjXJlRrswciLnG9qVTtouDAb0db9a9zfrQB+fcA8AD/ZALM1vRl/OLDDTlyoxyZUa5MpPPubK9z2EdMKlb20S/PV29mY1Im55MzzUOEREZIFktDs65rUC7mR0NYGZH4q0KFZjZwrSuPwe+6fcpAC4DfpHNbCIisnsDMc7heuBBMysBWoArgVK8tQMAnHO/NbNjzGwZ3uakO51zjVnO1S+bp7JAuTKjXJlRrszkba799noOIiKSPTq3koiI9KDiICIiPeRdcejr6TwGONN0M/udmb1kZn80szlBZ+rOzGaa2cagc+xgZiEzm29mr5nZy2Z2S9CZAMzsYDP7vZktMbM3zOy8HMj0LTP7ctr0If539oaZPe7vD8yFXGeZ2XP+38HLZvaZXMiV1v4FM3s9iEz++3f/vorN7Kdm9op/u6xf39A5l1c34BFgmv94KvCfOZBpJlDqPx4CvBV0pm75yoHHgBVBZ0nL9E/ALWnT44PO5Of4JTDdf1wFvI+/by+ALAcBrwP1wJfT2p8DJviPzwHuyZFcpwCF/uNxwGu5kMufNxZ4CHg9h/4d7wcuTpse15/vm1drDhmczmNAOededc5t9yebgQ4z621gYFDuAf4ZSAQdBHYe7jwPuHtHm3NubXCJdhEGavzHrcCHzv/LHWjOuY3OuePwDxMHMLMjgPXOuQ/9Pk/inc8s0Fx++2LnXJc/uYXeB9AOeC4zC+H9X/tmr08MIJe/xWOcc+4Xaf3W9ef75lVxYM+n8wic/5/wXuDBoBYo3ZnZxcBK59wHQWdJMx54C/hHM3vRv80OOpTvZuA+M7sB+DXwjYDzdDcJ6H7ama1mNiSIML0xs0LgZ8BdQWfx3QI85JzLpdNoHAEsM7M7/E2Yz+4YT9Zf8q049PV0HgPOzIYDjwKvOO80IYEzs4OBM5xz9wedpZsS4FSg1jl3MnARcE+OLOC+gLfZ5sfAd4C7zOygYCPtImf/BgDMbCLeANgfO+eeyYE8U4Fhzrnngs7STQlwAfCyc24W8FXgof7c4pBvxWEdfTudx4Ayswl42zO/6a/m54pzgcn+L5MlwGH+40B2FKb5G/Cuc+6/AJxzdcBSvH/LwJjZFGCic+6nzrlOf7PlL4F/CDJXNx+SNgDVV+Wcaw4iTDozmwb8G/Al59zSoPP4LgCm9fI3EHTBX4P3Q/IlAOfcGmAD3n6ufpFXxcHt5nQeOfCHcQdwlXOu5pM6DiTn3L8756Y552b5v07e8R+vDDhXM7DFzE4H8I+2OZqem0sG2jbgCDMrhZ2bRy4AVgWaKo1zbhVewR8L4H+HrwabaqdvAZc655qCDrKDc+5m59yMXv4GAj1yzzn3F+BgMzsKwD83XVV/bvrKx8uE9nY6j6BNB37VbY3wav/XgPTua8APzexWf/rbaTv1A+Gc22hmdwLPmVkM78fXz5xzLweZqxdfBx7xN0FsBK4KOA9mNgjv7+DZbn8HZznnWoNJlfOuBn7kH1ATB3ocfrsvdPoMERHpIa82K4mISN+oOIiISA8qDiIi0oOKg4iI9KDiICIiPag4iAwQ/1QHpwedQ6QvVBxERKQHFQcREelBxUGkF+b5rpkt9i+kcqx/IZqfmNmrZvZfZlbm9/2q37bUzL7gt0XNbIF/xtiXzOwS/6WPNO8iU2+a2RfNrMjMfmVmr5vZCsvBCz1JfsrH02eI9MUX8S48c4qZVQJPAscCM51zb5jZRcBtZvY0cAJwIt7f0+/MbBVwNvCRc+5GMwsDs/BO+jgHmAsUAm8AbcAHzrkLzawYmDKgn1JkN3T6DJFemNmvgVFAzG+qBjqdc0el9XkN7wpdTzjnXvPb/gHvqmHnArOcc8m0/nfgFYJH/emXgcuAXwG/91/n7Sx/NJE+0ZqDSO8ccKFzbsOOBv+UzTseR/w+O27pz3N4103o7dz6jd361uKdcO5Y4Doz2+6cu7mfPoPIXtM+B5He/RbvAioAmNlVeKdILvGbrgOeBZ7AW6ibf/nSa/z23+Od/RQzC+3hENa/A4Y4514Hvoe3eUokcFpzEOmFc+4/zWyCmb0KJIHH8a6h/Z/+jui/Al9zzsX8i/y8BBQB/+6ce9fM1gB3+5uOIsAPd/NWCeBhMxsMRPELikjQtM9BpI/MbIl/wReRA542K4mISA9acxARkR605iAiIj2oOIiISA8qDiIi0oOKg4iI9KDiICIiPag4iIhID/8f6RNp9yYnnd0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mnist_test(784,100,10000,100,0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 은닉층의 수를 줄여보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T06:07:59.349769Z",
     "start_time": "2019-08-04T06:07:47.916099Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1epoch => train acc, test acc | 13.030000%, 13.340000%\n",
      "  2epoch => train acc, test acc | 89.260000%, 89.380000%\n",
      "  3epoch => train acc, test acc | 91.126667%, 91.330000%\n",
      "  4epoch => train acc, test acc | 91.925000%, 91.920000%\n",
      "  5epoch => train acc, test acc | 92.133333%, 92.150000%\n",
      "  6epoch => train acc, test acc | 92.518333%, 92.120000%\n",
      "  7epoch => train acc, test acc | 92.731667%, 92.410000%\n",
      "  8epoch => train acc, test acc | 92.991667%, 92.850000%\n",
      "  9epoch => train acc, test acc | 93.363333%, 92.950000%\n",
      " 10epoch => train acc, test acc | 93.090000%, 92.590000%\n",
      " 11epoch => train acc, test acc | 92.830000%, 92.380000%\n",
      " 12epoch => train acc, test acc | 93.643333%, 93.130000%\n",
      " 13epoch => train acc, test acc | 93.435000%, 92.980000%\n",
      " 14epoch => train acc, test acc | 93.896667%, 93.150000%\n",
      " 15epoch => train acc, test acc | 93.666667%, 93.270000%\n",
      " 16epoch => train acc, test acc | 93.950000%, 93.400000%\n",
      " 17epoch => train acc, test acc | 94.091667%, 93.270000%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAENCAYAAADkNanAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmclXXd//HX5yyzMAsDM+yogEguaWUkAqK4omaaaKZZpqaWttqi1p0/qd99G6lUtNyp5Z3+bmkxNTUVs1QULRBcsszcAGFYh4EBZj3b5/fHuYBhZsA5MGeugfN+Ph7Hc67vdZ1z3jM41+dc13W+36+5OyIiIu1Fwg4gIiJ9j4qDiIh0ouIgIiKdqDiIiEgnKg4iItKJioOIiHSi4iAiIp30WnEws2+a2ed2sf4GM1tsZi+a2Tm9lUtERDrLe3EwsxFmtgD46i62OQ2odvfxwCTgKjOryXc2ERHpWt6Lg7uvdPejgW/sYrPLgJnB9q3AL4AL8p1NRES6Fgs7QGCIu69qt/wmMLHjRmZ2BXAFQFlZ2QcPPvjgXoonIrJveOGFF9a7+6B3266vFAfroq3ToE/ufjtwO8D48eN98eLF+c4lIrJPMbN3urNdX/m2Up2ZDW23PA54K6wwIiKFLpTiYGaDzWx2u6a7CK5JmFkRcBHwmzCyiYhIeEcOFWSPDgBw9z8AbWa2EHgG+Lm714eUTUSk4PXaNQd3v7Pd47eB0zqs/1ZvZRERkV3rK9ccRESkD1FxEBGRTlQcRESkExUHERHpRMVBREQ6UXEQEZFO+srwGSIiu83daU6kaWxLkXGnKBohHotQFM3eIpGuRujp+QyJdIa2VIa2ZIbWZDr7OJWmNZkhmc6QzjipjJPOZEilnXTGSXZYzq7P3qfSmR2W05ns8rDKEi6ePDqvP4+Kg4jslLuzqSVJ3Za27K1x+30q7cSjEYqiRjzYGe+wvG0HnV0uCtbHgx12PGbbHifTGRrbUtlba4qmRPZ+S1uKpqCtsS1NY1uSprb0Du1NbSkaEym802hs28WjtmPBiAW36I738XbriqMRHLbt3NtS6exOP7hvv+NvS2ULQecMTow0UTIAJImR2YMTNrGIEY0Y79+vSsVBpKe1/4SXSG2/b06kaE2maU5kby2JrY9TtCTStCTbtSfTtCRSNLelaE0kSSSStCaTtCWSbEkaaYtTHI9QHItSvHVnE88+zt6iwfrt27TfvrjT9h1eKxYN7rPPK4pmty+KRohHDbNdf1JuakvtuLPf0sb6do+3tq9vbCOVTlNFI9W2mUqaedHHURSNMDX2CvtlVtKYiZAiSooozV7Mo5mjATjC3magbSYdrEt5lGZKeNVHATCCOootSYooFuxEU0R5x7PDrB1my6i0JopIUVEE5TGnorg/68o+QGVJjDMiC6gu20JpNEO/qFMSzZCo2I/aEaeTSGU49K3biSU24ek0nknhmRS1JeNYUHUGiVSGc1Z+n6JkMyTSkEljmRQvxj/A/fEzSKTSzGr+FjEyxCyz7f6vZSfwxIDzqYi08X9WXEGUDJFommg0Q7Q4zUv7fYpXx1xGVXoD0+edTCQoClst/+B11B3xOfo1vsN77jsZIlGwKB6JQSTKlmNvIPHej1O84Q3KH7gIi8QgEsMi0ez9CdfDQSf1/B9FF1QcpFds3fk2J9IkUhkS6ewOOZnOkEz7tseJdHC/bdlJBtsn27VtWx98esskW7FEE5FkI83pGGu8irZkmqPbniWeaqYo00xxppmSTDOLUmP5c2Y8pbRye/wHlFsrZbTQjxQVZLgrPY070qdTwyYeK742uwMgQxQnYhlui36CB0vOYmx0Lb/e/NlOP+vDo67hhUEfZcCW1/nsm1eSSJbQZiW0WTEtFPPrfp9kQeRIBre9w4fb7qXZi2jMZG9b0kU8kDmK5T6EQTRweGQJbcRJeowEcZLEWOpDaaaEEtqooIUEMZLbblHMLFtE2hWMrQWkOZFmfWMbzYk01Wxif1tHtW2m2jYzyDbxvqIm5lReQXVlPz5v9zI58yD9UpuIeBoAjxSx+WsrqewXx+6/H/5xD0S3/+yZfoOov/LbJNMZqh68k35LH9/hd9Navj8Lz3ySZCrD+5+6iJq6BTusbxpwKEvOeYzykhgjf3868bUvb1+ZAoZPhEu/kF3+6Rdg7Rs7/vLHngxHBRNOLnoEGtdBJAYWgUiM8fv356MfOSK7/tYVkGzJrg+2GX9IDVccd2x2/Z3V255HsPMe/Z73c+GRH4JUGzw0dVv71m0mjp3KxHFjoG0wZL62fb1lf0n7j5rC/vsNgKY0TPo8ZFKQSQe3FFXDD4KKEkhWwMjxwfp228RLuv03t6fMd3Us1odpyO7d55kMKSe7Y27aTKplM6lkK6m2VlLJNlLpJFsGvJdkOkPR2r8T3bKSZDo4fE47zZk4b1RMoDmRZsDGV4i31tOWclqDHXVDqpjFfghNiRSjE69TnNxCysExHKPFi3nZxwIwLfI8g2wTJSQoJkmxJVjpg/hd+ngAZsTuZJhtyK4jSYkleIGD+ZFdRDxqPJz5PIPZQJzUtp9vXukp/GrQNyiOGj9fOo0o6W3r0hbjpWHns/CgqymJpDnzxctIx8tIx8uwWDHRaIwto6bRetAZ9KOZIQtvJBaNE4vFiEaj2Z3FuFNh9BRo3gALbwv++CPb7w88AYYeDg0rYOGtkGzO7oQSTdn7yV/OPn/5QrjvM9n1iWZItQCQPP93tIw6Ef/XH+n/0CWd/v3+fvJvqBvwQWqWPsD7F13Taf3/HvG/1JYcxCFrH+LElbeSIkbS4qSIUeKt/L9Df0lp9QiOWXUHh73+sx2fXFwJX/479BsIr/wels2HskFQPhjKarKPDzgGIpEgc2t255VOZu9xGDAq+1r1b0NLA2SS27eJFsGoydn1S+dD49psu1l2R1o6AMaemF2/8oXse0Tj2VskDsUVMDA4ndK4LnsfiW1fH41n/x1kp8zshWDWzV1vp+Kwl8hkoG0TtGwk3dzAxopx1DVnaFv6N2Ir/kq6aQPeshFr3UQ8sYlrS2fQmIpwYetvOCv1J+IkiXuKOEmiZDiwbQ4A34/dzsdj83Z4qy1eyuFtdwDw0/iPOSO646e71T6QiW0/pTgW4Y7Y9zmGl3ZYvyq+PzeOvot+RVE+v/SLHND09x3Wb6w6jOdOuJd4NMKkP3+UiobXtq1zi9A88ljqzvo18ViEQfefR7RlPcSKsXgpFi/B9p8ExwUTCz7xXfAMFJVnb8XlUPMe2O9D2fXrXoN4v+3rYsV7+i+RP5lMtkBEi7I7uZYG2PB29lNqOpHdiaYTsP/E7M57/Vuw7BlIJYL1wTYf+kx2Z750Pvzzvu3rUm1QVAYn3gCVw2D9m7Bx2fadfr+aXv1kKuFQcdibNG+A2sW0vvM8LRtXk2rcwAsHf5UVqWpGLLmHqct/QkmmiUi7+Y8mtf6YVdTw+egDfCN+D20ep4FymiPlNEcruXXEf2ElVXyo8Sne0/ICHinCo0V4pAiiRbww+rPEY1H22/QCA9pWEIkVYbHi7C1eyuaRxxGPRqhorqUo3URxPE5J3CiNRygpLqZ4+HuJRiz76bC1IRvKg//EirOfnAHWvgqtm7Pt7tn7kv7b129Zk/20HSuBeGlweJ//b5aIFCoVh74qk4a612krreFfDXE2Lv49J7yS/RScdmMDFWz2Mq5MfoU3fD8mxV5nevEiUkX9obSKSL8BxMqqaRkxiaqqAQwuSVNTUUz1gCoqimPveiFSRApbd4uDLkjnW7KVzNL5bHrjORLvLKR//d8pyTTx3dRlzEmdwHCK+WTZhbQMOZLyMRMYMbiamvJifl5RTE15MZUlp2uHLyK9TsWhJ2UyUPdvGt/+K2+1VPBk+v28uewdfr7qY1S68brvz5M2iQ0D3sfQ0cdx64EH8b79qhjW/9NhJxcR2YGKwx5qTaZZP/d7sGw+1Q3/oDTTRDmwPD2Rn6a+yHuGVnLrgT9j0NgPctjoEZw3uCJ7rl5EpA9TcdgDrck0k2c+yX8n/0glzSyKTWbzkA9QMvpoRh10BP/cr4p+RTFgSthRRURyouKwB2o3tlDflGD+1P/hokkHcnalvgYoIvsGjcq6BxrfeIZb4z/kxOFpBqswiMg+RMVhD6RX/4NTo4sYPKAy7CgiIj1KxWEPpBtqSXiUwcNGhh1FRKRHqTjsgXjjKtZZDfGYLt2IyL5FxWEP9GtZQ0N8cNgxRER6nD7y7oH16X4kykeEHUNEpMfpyGE3uTufSXyV5w7qPGSyiMjeTsVhNzU0J2lNZhhWVRp2FBGRHqfisJvq31zIg0Xf5uDM22FHERHpcSoOu6l5zZu8L7KEmv7lYUcREelxKg67KbFhBQADR4wJOYmISM9Tcdhdm1bQ6KVUDxwUdhIRkR6n4rCb4k2rqYvUENHw2yKyD1I/h920LDOEotKBjA47iIhIHqg47KabMp9kwuiBnBZ2EBGRPNBppd2QzjhrNrcyrErDdIvIvinvxcHMhpjZXDNbaGZ/NrPhO9nuRjN7Ntju52bWZ49q6le8xt/iVzK+bVHYUURE8qI3jhxmATPcfQLwTeCWjhuY2alAjbsfE2wXA87qhWy7pWHVUgZbA1X9+4cdRUQkL/JaHMysCqh294UA7r4Y6B+0txcFatstrwdW5jPbnmiqWwZA5dBRoeYQEcmXfB85jAbe6tC2JGhv71Ggwsx+aGazgXXuvqDji5nZFWa22MwW19XV5SdxN6Q2ZjvADVIHOBHZR+W7OBjgXbR3bDsaiAOzgf8LjDSzizs9yf12dx/v7uMHDQqv81lkcy313p/Kcg2dISL7pnxf9F0GjO3QdmDQ3t73gFPcPQFgZl8HngPuzG+83fNvG8PS4gjnhh1ERCRP8nrk4O4bgGYzOxLAzI4gez2hKDh9tFUT2aOHrc4D/p3PbHvit34Kfxz6hbBjiIjkTW98XfQrwB1mVg5sAi4BKoBx7bb5HPATM/su2VNObwBf6oVsuXNn3cbNvHfEyLCTiIjkTd6Lg7vXAtM6NK+F7Z2L3X0F8NF8Z+kJrVs28Lf0J3im6RrgiLDjiIjkhXpI56h+1RIi5hRXDQk7iohI3qg45GjzmmUAlNXsH24QEZE8UnHIUUv9OwAMGKY+DiKy71JxyFGmoZakRxk0bL+wo4iI5I2KQ47+GT2YOyNnU1JcFHYUEZG86bMjn/ZVT2WOZMPA93J52EFERPJIRw45Sm5YzsjKaNgxRETySsUhF5kMd225gk80zwk7iYhIXqk45GDLhpXELY31HxF2FBGRvFJxyEH9yiUAxAeqj4OI7NtUHHLQuG4ZABVDRoWaQ0Qk31QcctC2fjkAA4d3nKtIRGTfouKQg38WHc6NqQsZVDM07CgiInml4pCDv6dG8Uj5uUSj+rWJyL5Ne7kcxOv+wSHlzWHHEBHJO/WQzsE167/N6/0nA2eEHUVEJK905NBNmUQr1TSQKh8edhQRkbxTceimDWuXARCp0misIrLvU3HopobVSwEo1SQ/IlIAVBy6qWlddpKfisGjwg0iItILVBy66bWSI/hS4gsMGjk27CgiInmn4tBNb7VW8Xj0GKoqy8OOIiKSd/oqazeVr/4rx5ZHMLOwo4iI5J2OHLrp7DU/5gq/J+wYIiK9QsWhm6rTdbT2Ux8HESkMKg7dkGjcSDnNZCo0yY+IFAYVh27YEPRxiA1UBzgRKQwqDt3QsCZbHPrVHBByEhGR3qHi0A1vFb+Xc9v+DxWj3h92FBGRXqHi0A3Lm6Ms9oMZNqg67CgiIr1CxaEbKt75C2eWvEy/InULEZHCoL1dN3xo9a85MpoE/iPsKCIivUJHDt1QmVhHY8mQsGOIiPQaFYd3k8lQk6mjrWxY2ElERHpN3ouDmQ0xs7lmttDM/mxmXXYzNrOTzWy+mT1hZo+aWVm+s3VHU8MaikhB5ciwo4iI9JreuOYwC5jh7gvNbDxwC/CJ9huY2WiyJ/Q/4u4NZjYUyPRCtne1YeXblAFF6gAnIgUkr0cOZlYFVLv7QgB3Xwz0D9rb+wZwvbs3BNutcfeWfGbrrqXxsUxunU1s7PFhRxER6TX5Pq00GnirQ9uSoL29w4E2M3vQzJ4xsxvNrFM2M7vCzBab2eK6uro8Rd7R6i1JVjKIITXq4yAihSPfxcEA76K9Y1s/4ELgU8BxwfKlnZ7kfru7j3f38YMGDerprF0qfftRLo3NZWj/kl55PxGRviDf1xyWAR3n1TwwaG9vOXCTu28GMLO7gYvznK1bDlg1l/fF3iAe1Re7RKRwdHuPZ2Y5f13H3TcAzWZ2ZPAaRwDrgSIzm91u058B3zOzaLB8FrAw1/fLh7LWNWyMDw47hohIr8rlyGGWmfUHHgPudffabj7vK8AdZlYObAIuASqAcVs3cPe/mNlY4BkzSwF/Be7OIVveVCXXsa7iqLBjiIj0qm4XB3f/uJkVAScA3zKzwcBzZAvFil08rxaY1qF5LXBah+1uBW7tbp7e4Kk2BmY2kirTJD8iUlhyuubg7gkzewroD5xD9ltG7zezDe5+dT4ChmlT3SoqAatSBzgRKSzdLg5mNhX4JDAMuBe41N0bg3V/zUu6kNVmBnJ221387OD3hR1FRKRX5XLkMA2Y6e4d+y0AfLiH8vQpqze1kiTGkIEd++yJiOzbcvl+5j+BbWNImNk0MzsPwN039nSwviD2+h/5TuxXDOtfFHYUEZFelUtxuBSY1275ceCLPZqmj+m/+jnOjP6NmvLSsKOIiPSqnHp2ubt3eJzu8UR9SLxpNesjNUQiFnYUEZFelUtxeMnMvmBZETP7KvBivoL1BRVta9hUpEl+RKTw5FIcriM75tHTZE8vxcmOprrPGpiqo7WfJvkRkcKTSye4FHBTcAMguCB9Tx5yhS6daKHFY6Qr1MdBRApPLv0cDgFmACPJjrYK2SOJfbI4rGuBiW3/zX8deljYUUREel0up5V+SXYWt1fJDn3xv8Bd+QjVF6xqyM41NLyqX8hJRER6Xy7FodndFwH/AA5w958DZ+YnVvgy/3qY2+OzGFHaFnYUEZFel0sP6RVmdhDwEPAjM3uAPjLPcz5E1r7CiZEXaRpUE3YUEZFel0txuAyIunvSzL4HTATOz0+s8EU2r6SOAQztpw5wIlJ4cjmt9Cd3TwK4+/PuPtvde2ci5xCUNK+mPqZJfkSkMOVSHLaY2Ql5S9LHVCbW0lisDnAiUphyOa1UA/zFzJYAW48Y3N0n9Xys8K3OVNFQMe7dNxQR2Qfl0gnu2HwG6Utak2nObf02Xxs3rtMUdiIihSCXTnB/A7YOvFcDjAb+7O6n5yNYmFZvagVgWJUuRotIYcrlyGFi+2Uzmw6c2OOJ+oCWfz3GH4u+SyLyP2Q7hIuIFJachuxuz93vJzuH9D4nsfZNDo8sY1C1+jiISGHK5bRS+6uzBkwAWns8UR+QbqilzeMMHjoi7CgiIqHI5dtK3yR7zcGC+6XABfkIFbZY40rWWjX7F+Xy6xER2Xfksvf7C7Da3Z+E7BzSZK857HOjsvZrWcPG2BD2DzuIiEhIcp1D+ql2y/vsHNKv+QEsq/hA2DFEREKjOaQ7cHe+2XYxL42+IuwoIiKh0RzSHWxuTdGUSDO8qiTsKCIiodEc0h00vD6fxcWf47D0v8OOIiISmlyKw1Tgb+5+bDCUxvPAPjekRuPapdTYZqoGDgo7iohIaHI9cni+3fJfgRt6Nk742uqXA1A9fHTISUREwpPrBem2rh7vUzbVssnL1DtaRApaLsVhmZl9ZOtCMLbSkp6PFK5402rqIjVEIxZ2FBGR0OTSCe7LwM1mdi3ZXtKvAPflJVWIXrJDiZWNZWzYQUREQpTLkUMFsAVYHtwGALPzESpMv0x/mAUjLgk7hohIqHIpDnOAxcFzbgSWAd/PQ6bQZNJp6jdtYpj6OIhIgculOLi7/55sgUi4+3XAhe/2JDMbYmZzzWyhmf3ZzIbvYttiM/uHmZ2aQ64es7H2NV6Nf5rJLU+H8fYiIn1GLsWh3syGAI8C15rZcXTvmsUsYIa7TyA7sustu9j2P8kekYSiYfVSAPoN1FDdIlLYcikOnwWS7v4v4CHgo2QH49spM6sCqt19IYC7Lwb6B+0dtz0F2AC8kEOmHtVc9w4AFUNGhRVBRKRP6HZxcPcGd98QPH7A3a929zfe5Wmjgbc6tC0J2rcxs2rgMuCmXb2YmV1hZovNbHFdXV13o3dbcuMKMm4MHqEOcCJS2HZ7mtBu2joxUEcd234AXOvuuxzl1d1vd/fx7j5+0KCeH97CNteyniqqKsp6/LVFRPYm+Z7qbBl06jJwIO2uKwQXqMcDvzIzgFHAdDP7oLv/V57z7WBRbDx/Kx7MVaYOcCJS2PJaHNx9g5k1m9mR7v6imR0BrAeKzGy2u3/Z3VcBh219jpnNABa4+2P5zNaVR9NHUT54Elf19huLiPQxvTFJ8leAO8ysHNgEXEK2Q924Xnjv7nMnsnEJIw86OOwkIiKhy3txcPdaYFqH5rXAaTvZfka+M3UlsaWe+1NfYF7r1cCHwoggItJn5PuC9F5jw6q3AYgP2C/kJCIi4VNxCGxauwyAfjUHhBtERKQPUHEItK7PTvJTNXxMyElERMKn4hBIN6wg4VGGDBsZdhQRkdD1xreV9grPl0zkkUgx1xcXhR1FRCR0Kg6BRcmxrKrSUYOICOi00jaV61/k4LLmsGOIiPQJKg4AmTQ3N17HWclHwk4iItInqDgAzfW1xMhAf83jICICKg4AbAgm+SkeqA5wIiKg4gDAlqADXPlgzeMgIgIqDgC01mc7wA0YpuIgIgL6KisAL5Ydwy+SKWYPHhx2FBGRPkFHDsC/W6t5oexY4lH9OkREQEcOAAxZ8xQTy6rDjiEi0mfoozLwmQ0/4Nx0r088JyLSZxV8cfBEMwN8E6ny4WFHERHpMwq+OGxa9w4AVqU+DiIiWxV8cdi4KtsBrrRGxUFEZKuCLw5NdcsAqBiiSX5ERLYq+OLwSvkxTG+bQc2IA8OOIiLSZxR8cXinKc4/IwdTXVkWdhQRkT6j4IvD0BWPcFb5q0QiFnYUEZE+o+CLw0nr7uJcezLsGCIifUphFwd3BqbX0VY6LOwkIiJ9SkEXh3RLA2W0kqnUJD8iIu0VdHHYsGoJANEB6uMgItJeQReHhjXZDnD9avYPOYmISN9S0MXh9bKjOLr1J1SM/mDYUURE+pSCLg4rNydZQzVDq/uHHUVEpE8p6OIwZMm9XFr8JJUl8bCjiIj0KQU92c+hax9mTCwddgwRkT6noI8cKhNraSweGnYMEZE+p3CLQyZDdaaeRJk6wImIdJT34mBmQ8xsrpktNLM/m1mnKdfMrMrMfmZmT5nZ38xslpnldbCj1obVxEnhlSPz+TYiInul3jhymAXMcPcJwDeBW7rYpgy43d2Pd/eJQAVwWj5D1a9ZTsojFFWrA5yISEd5LQ5mVgVUu/tCAHdfDPQP2rdx95Xu/vd2Te8ARfnM9k7RQbyn7S4iB52Yz7cREdkr5fvIYTTwVoe2JUF7l8xsCjAReLiLdVeY2WIzW1xXV7dHwVY2tJAmyrABlXv0OiIi+6J8FwcDvIv2Tm2W9VXgHOBcd091epL77e4+3t3HDxo0aI+CDXz9t3wrNoeh/Uv26HVERPZF+e7nsAwY26HtwKC9o/8BHnb3H+Q5EwBD1s5ndOxtSuLR3ng7EZG9Sl6PHNx9A9BsZkcCmNkRwHqgyMxmb93OzM4EFrv7ffnM016/1tU0xAf31tuJiOxVeqOH9FeAO8ysHNgEXEL220jj2m1zDDDVzD7Wru1ed/9pvkJVJdexsrzjQY2I9HXJZJLa2lpaW1vDjtKnlZSUMHLkSOLx3RseKO/Fwd1rgWkdmtfS7quq7n5NvnPskCnZykBvIFmuSX5E9ja1tbVUVFQwatQo8twdaq/l7tTX11NbW8vo0Tv9/s8uFWQP6S0b17HSq7Eq9XEQ2du0trZSXV2twrALZkZ1dfUeHV0VZHFYlalicttPaDrkY+++sYj0OSoM725Pf0eFWRwaWgAY1r805CQiIn1TQRaH0n/+hl/Eb2F4peZxEJHc3XnnnTltf9111+UnSB4VZnGoe5kjI28yuH9Z2FFEZC+Ua3GYOXNmfoLkUUEWh6LGVdRFBhGN6LyliORm+vTpvPzyy0ydOpVnnnmG6dOnc+KJJ/LJT36SjRs3ct5553HCCSdw/PHHs2rVKgCOPvpoli1bxvXXX8/555/PlClTOOuss2hubu70+l//+tc56aSTGD9+PM8888y29pkzZ3L88cdzwgkncPPNNwPZInXcccdx0kkncfXVV/foz1mQM8GVt61lTZHmcRDZ233nj6/yr1Wbe/Q1Dx1eyQ0fOWyn6++//36mTp3KvHnzWLZsGQ8//DAvv/wyhx56KOvXr2f27NkMGzaMe+65hx/96EfcdNNN2577y1/+kueee44xY8Ywc+ZM7rzzTq666qodXv/Tn/40t9xyC7W1tXz84x/nueee43e/+x3r1q3jqaeeAuDxxx9nwYIFPProozz55JNEo1H+9Kc/9ejvoSCLw8DUOpaWHxl2DBHZBxxzzDEceuihANTU1PDqq69y9913s2jRIqLRHYfnOfvssxkzZgyQPZroaoc+ePBgbrvtNl555RXWrFkDwH333ccPfrB9ZKFTTjmFa665hi996Uvb3mPatI7dyfZMwRWHTKKFNzIjaB4w7t03FpE+bVef8HvL4MHbh+H5wx/+wIMPPsi1117LtGnTuPHGG3e6LWQ7q7W3bNkyLr/8cr773e9y0UUXMX/+/C6321lbTyq4aw7r24zpie9QN+6CsKOIyF4qleo0aDQATz/9NJ/+9Kc55JBDmDNnTs478JdffplJkyYxceJEnnjiCerr64HsdY72p6ceffRRzj77bH74wx9uy/LII4/s5k/TtYIrDlWlRTz0hcmcetjQsKOIyF5qyJAhTJkyheXLl+/QfuWVV/Ltb3+bY489ljFjxrBo0SLmzZvX7dedNm0aL730Escddxzz5s3j8MMPZ9asWVxwwQUMHTqUKVOmMGnSJN58801NzyLMAAAKxUlEQVQmTZrEhz/8YaZOncqkSZO2HWX0FMv3oUm+jB8/3hcvXhx2DBHpZa+99hqHHHJI2DH2Cl39rszsBXcf/27PLbgjBxEReXcqDiIi0omKg4iIdKLiICIinag4iIhIJyoOIiLSiYqDiIh0ouIgIpKjXIfsBmhoaOCBBx7o+TB5ouIgIpIjFQcREdlB+/kcli9fzrPPPstJJ53Ecccdx3e+8x0AFi5cyNFHH82ECRM4/fTT2bRpE+effz6PPfYYU6dOJZFI7PCac+bM2TaHw+zZs7e1z507l6lTp3LyySfzqU99imQyyaJFizjppJM4+eSTmT59Ohs3bszLz1lwo7KKyD7mVx/u3HbYR+GoyyHRDHM+1nn9+z8BH7gQmurhnot2XHfJrgewaz+fQ0NDA1/72td44oknKC8v5zOf+QwLFixg5syZ3H333YwdO5bXXnuNdDrNb3/7W2bMmNHlUccRRxzBX/7yF1KpFBMmTODiiy+moaGBWbNmMXfuXEpLS5k/fz4tLS1cffXVPPTQQwwcOJBXX32VpqYmBgwYkMMvrHtUHEREdtOzzz5LXV0dZ5xxBgBNTU28+uqrfOxjH+Oyyy7jsssu47TTTmPgwIFs3rzzSYnGjh3LPffcw8KFC9m4cSNr1qzhiSee4OKLL6a0tBSAKVOm8Oijj3LqqacycOBAAA47LH9Dlqs4iMjebVef9Iv67Xp9WfW7HinsiplxySWXcP3113dad8YZZ/CnP/2J6dOnc/PNN3eay6G9M888kyuvvJLrr7+eFStW4O6hzOHQnq45iIjkaOscCpMnT+bxxx/fdt7/jTfe4IknnuDpp5+msrKS8847j1NOOYXFixcTiUQ6XWuA7IXqRCLB9OnT2bJlC08++STuzumnn84dd9xBU1MTAPPmzeOYY47h4YcfZv369QC88sorrFixIi8/o44cRERytHU+hzlz5nDzzTdzzjnnkMlkOOCAA5g1axYPPvggN9xwA+7O0KFDufPOO4nH4yxfvpypU6fy+OOPU1RUBEBVVRWTJ09m0qRJDBkyhMsvv5zrrruOBx98kK9//eucdtppZDIZDj74YCZPnszs2bM577zzSCaTDBs2jNtuuy0vP6PmcxCRvYrmc+g+zecgIiI9SsVBREQ6UXEQEZFOVBxEZK+zt14r7U17+jtScRCRvUpJSQn19fUqELvg7tTX11NSUrLbr6GvsorIXmXkyJHU1tZSV1cXdpQ+raSkhJEjR+7281UcRGSvEo/HGT16dNgx9nl5P61kZkPMbK6ZLTSzP5vZ8J1sd4OZLTazF83snHznEhGRneuNaw6zgBnuPgH4JnBLxw3M7DSgOuiYMQm4ysxqeiGbiIh0Ia/FwcyqyO70FwK4+2Kgf9De3mXAzGCbVuAXwAX5zCYiIjuX72sOo4G3OrQtCdpfatc2xN1XtVt+E5jY8cXM7ArgimCx0cxe34NsNcD6PXh+vihXbpQrN8qVm30x1wHd2SjfxcGArr5v1rHNurEN7n47cHsP5MLMFndnfJHeply5Ua7cKFduCjlXvq85LAPGdmg7MGhvr87MhrZbHkfnIw4REekleS0O7r4BaDazIwHM7Aiyh0JFZja73aZ3Ad8ItikCLgJ+k89sIiKyc73Rz+ErwB1mVg5sAi4BKsgeHQDg7n8wsw+Z2UKyp5NudPf6POfqkdNTeaBcuVGu3ChXbgo21147n4OIiOSPxlYSEZFOVBxERKSTgisO3R3Oo5czTTKzP5rZU2b2VzObFnamjsxsipmtDDvHVmYWMbOZZvasmT1tZteGnQnAzPY3s0fMbJ6ZPW9mZ/eBTN80s8+1W35P8Dt73szuC64H9oVcZ5jZY8HfwdNm9sG+kKtd+yfMbEEYmYL37/j7KjWzX5jZM8Htoh59Q3cvqBtwNzAheDwe+HUfyDQFqAgeDwBeCTtTh3z9gXuAxWFnaZfpW8C17ZZHh50pyPFbYFLwuBp4g+DaXghZRgALgDrgc+3aHwPGBI/PAm7uI7lOBIqDx6OAZ/tCrmDdAcCvgAV96N/xNuCCdsujevJ9C+rIIYfhPHqVu8939y3BYgPQYmZddQwMy83AfwCpsIPAtq87Twdu2trm7kvDS7SDKFAbPG4Elnjwl9vb3H2lux9N8DVxADM7HFju7kuCbR4kO55ZqLmC9ifcvS1YXEPXHWh7PZeZRcj+v/aNLp8YQq7gjMcod/9Nu+2W9eT7FlRxYNfDeYQu+J/wFuCOsHYoHZnZBcAL7v5m2FnaGQ28AnzezJ4MbieFHSpwDfBDM7sauBf4ash5OhoLdBx2ZoOZDQgjTFfMrBj4JfC9sLMErgV+5e59aRiNw4GFZjYjOIU5d2t/sp5SaMWhu8N59DozGwzMAZ7x7DAhoTOz/YHT3f22sLN0UA6cDKx19xOA84Gb+8gO7hNkT9v8HPgO8D0zGxFupB302b8BADM7kGwH2J+7+6N9IM94YJC7PxZ2lg7KgfOAp919KvBF4Fc9ecah0IrDMro3nEevMrMxZM9nfiM4zO8rPgqMCz6ZzAMODR6HcqGwnbeB19z99wDuvg54juy/ZWjM7BDgQHf/hbu3BqctfwucG2auDpbQrgNqoNrdG8II056ZTQC+D3zG3Z8LO0/gPGBCF38DYRf8t8h+kHwKwN3fAlaQvc7VIwqqOPhOhvPoA38YM4BL3b323TbsTe7+Y3ef4O5Tg08n/woevxByrgZgjZmdChB82+ZIOp8u6W2bgcPNrAK2nR45D3g51FTtuPvLZAv+AQDB73B+uKm2+SbwKXffGHaQrdz9Gnef3MXfQKjf3HP3vwP7m9kHAIKx6ap78tRXIU4T2tVwHmGbBPyuwxHhZcGnAenal4CfmNl1wfK3213UD4W7rzSzG4HHzCxB9sPXL9396TBzdeHLwN3BKYiVwKUh58HM+pH9O5jb4e/gDHdvDCdVn3cZ8NPgCzVJoNPXb/eEhs8QEZFOCuq0koiIdI+Kg4iIdKLiICIinag4iIhIJyoOIiLSiYqDSC8Jhjo4NewcIt2h4iAiIp2oOIiISCcqDiJdsKzvmtkTwUQqRwUT0dxqZvPN7PdmVhls+8Wg7Tkz+0TQFjezWcGIsU+Z2SeDlz7CspNMvWhmF5pZiZn9zswWmNli64MTPUlhKsThM0S640KyE8+caGYDgQeBo4Ap7v68mZ0PXG9mDwPHAMeR/Xv6o5m9DHwEeMfdv2ZmUWAq2UEfpwGnAcXA80AT8Ka7f9zMSoFDevWnFNkJDZ8h0gUzuxcYDiSCpiFAq7t/oN02z5KdoesBd382aDuX7KxhHwWmunu63fYzyBaCOcHy08BFwO+AR4LX+UeefzSRbtGRg0jXHPi4u6/Y2hAM2bz1cSzYZuut/fOc7LwJXY2tX99h27VkB5w7CrjKzLa4+zU99DOI7DZdcxDp2h/ITqACgJldSnaI5PKg6SpgLvAA2Z26BdOXXh60P0J29FPMLLKLr7AeBAxw9wXAf5I9PSUSOh05iHTB3X9tZmPMbD6QBu4jO4f2r4ML0f8GvuTuiWCSn6eAEuDH7v6amb0F3BScOooBP9nJW6WAO82sDIgTFBSRsOmag0g3mdm8YMIXkX2eTiuJiEgnOnIQEZFOdOQgIiKdqDiIiEgnKg4iItKJioOIiHSi4iAiIp2oOIiISCf/H3Fs0b6PZRhbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mnist_test(784,10,10000,100,0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 배치 사이즈를 늘려보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T06:08:19.960173Z",
     "start_time": "2019-08-04T06:07:59.351703Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1epoch => train acc, test acc | 11.098333%, 10.850000%\n",
      "  2epoch => train acc, test acc | 86.456667%, 87.170000%\n",
      "  3epoch => train acc, test acc | 89.378333%, 89.630000%\n",
      "  4epoch => train acc, test acc | 90.591667%, 90.450000%\n",
      "  5epoch => train acc, test acc | 91.446667%, 91.490000%\n",
      "  6epoch => train acc, test acc | 91.773333%, 91.880000%\n",
      "  7epoch => train acc, test acc | 92.098333%, 92.150000%\n",
      "  8epoch => train acc, test acc | 92.143333%, 91.950000%\n",
      "  9epoch => train acc, test acc | 92.243333%, 91.900000%\n",
      " 10epoch => train acc, test acc | 92.518333%, 92.520000%\n",
      " 11epoch => train acc, test acc | 92.681667%, 92.550000%\n",
      " 12epoch => train acc, test acc | 92.800000%, 92.640000%\n",
      " 13epoch => train acc, test acc | 92.908333%, 92.460000%\n",
      " 14epoch => train acc, test acc | 93.245000%, 92.810000%\n",
      " 15epoch => train acc, test acc | 93.320000%, 92.930000%\n",
      " 16epoch => train acc, test acc | 93.555000%, 93.260000%\n",
      " 17epoch => train acc, test acc | 93.623333%, 93.220000%\n",
      " 18epoch => train acc, test acc | 93.613333%, 93.200000%\n",
      " 19epoch => train acc, test acc | 93.978333%, 93.480000%\n",
      " 20epoch => train acc, test acc | 94.011667%, 93.640000%\n",
      " 21epoch => train acc, test acc | 94.055000%, 93.670000%\n",
      " 22epoch => train acc, test acc | 93.883333%, 93.600000%\n",
      " 23epoch => train acc, test acc | 94.151667%, 93.800000%\n",
      " 24epoch => train acc, test acc | 93.915000%, 93.750000%\n",
      " 25epoch => train acc, test acc | 94.178333%, 93.540000%\n",
      " 26epoch => train acc, test acc | 94.205000%, 93.630000%\n",
      " 27epoch => train acc, test acc | 94.418333%, 93.540000%\n",
      " 28epoch => train acc, test acc | 94.333333%, 93.630000%\n",
      " 29epoch => train acc, test acc | 94.428333%, 93.880000%\n",
      " 30epoch => train acc, test acc | 94.495000%, 93.760000%\n",
      " 31epoch => train acc, test acc | 94.638333%, 93.920000%\n",
      " 32epoch => train acc, test acc | 94.551667%, 93.950000%\n",
      " 33epoch => train acc, test acc | 94.670000%, 93.970000%\n",
      " 34epoch => train acc, test acc | 94.363333%, 93.830000%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAENCAYAAADkNanAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmYnFWZ/vHvU3v1lqWzE5MQAsMiCBoNBGICBEJYFIKETdkEFEYFFRRmRIIzYoZFzcz8ZhRFo8KMMqDEwbBIIITFBAJEFBDBTEhCIEtn6b1re35/VCUU3Z2QDl39VlP357rqqq5Tp6ruejt5n363c8zdERERKRYKOoCIiJQfFQcREelCxUFERLpQcRARkS5UHEREpAsVBxER6ULFQUREuuiz4mBm15rZ53fx/PVmttzMnjOz0/sql4iIdFXy4mBme5nZUuAru+gzE6h394nAZOByMxtS6mwiItK9khcHd3/D3Q8Hrt5Ft4uBuYX+7cCPgLNLnU1ERLoXCTpAwXB3X1f0+FXgiM6dzOxS4FKA6urqj+y///59FE9E5P3h2Wef3eTuQ9+tX7kUB+umrcugT+5+G3AbwMSJE3358uWlziUi8r5iZq/vTr9yOVtpo5mNKHq8H/BaUGFERCpdIMXBzIaZ2byipp9ROCZhZjHgPOC/g8gmIiLBbTnUkt86AMDdfwN0mNkyYAnwn+7eEFA2EZGK12fHHNx9ftHPfwNmdnr+H/oqi4iI7Fq5HHMQEZEyouIgIiJdqDiIiEgXKg4iItKFioOIiHRRLldIi4h0kc05LakMLR35W3NHlrZUlnDIiISNWDhEJGxEQiGiYSMSDhEN5e+zOSedzRVu+Z8zWSdVaMtknXQuRzbrZHJONudkcrnCve+4T2dytGeydKTfvu/IZGlP52hPZ+nI5N8vHHo7R7SQKxoq3IdDREJGJud0ZPKvSW2/ZXN0pHM7ciWiYQYmowyqijGwKsrAwv2gqigDkjEGVUcZXB1jWG2ipMtexUEkQO3pLJtbUjQ0p2ho6SCddXLuuDs5h1zhPv/YyeW2tznZHGQLfbOFlZl7vi1kkIyGScYiJKNhqmJhEoX7ZCxMMhomEjaa2zNsa0vT2J6msS1TuE/n29oyNKcyJCJhahMRahMRauIRagr3dYkoNYkI1bEI6WyOxvY0Te0ZGrt5v8b2DO3pLF74Tg47vmPxfSbnO4pAS0eGtnS2JMvdyFFLG4bTSoL0bq4K45EQ8YhRG3EGRDPUhdNkYnVkI0limRaGpN8kl8vSkcvSks3huSx/y41iSy5JTW4bfxdaRyxsRENGPAw1YeP1+L5kYnWMDG1mnL9Oa0eILU0h1rUbL7SH+FtmCCmiRMlgOCkifGj0QBZ84aiSLJvtVByk4qWzOZrbMzR3ZGgq3Dd35Fd07ens239FZr3or8ocmUJbzt8eBswMrDBUmBVGDDPyK+zNLWkamjtoaEnl75tTtHSkqKOFbVTjhBhj6xlvbxInRZIUSesgSYqfZmfghDgh9DSTQi9jOCGcEDnAuCFzEQBnhBczKfQX3CFDmHZCbCLJ3Mw5AJwSeooJoTfIeBiAKuug3WPMy+anULk2cieHh16lxtqpsQ6qrINVNpoLct+kuSPDv0e+zwH2OjlCZAu3l3wcX01fBsA1kf+inkZiJKgiQSxaQzY6htdqjyIeCTOl/TGStBIjQ4QsEU+zPvoBnq06kpAZp2/7GfE4hGtihCMxwtEYLQMPYOuoo6iKhZnw5u+IRiOQzZDLpshlUmyr3Y8Ngw4jl+pg31d/hGfTkO0gnm0lnm1h3ajj2PCBE6jr2MARi88ikm4ikmnZ8Tt7c9J1bPnQpSQbVzL27hPwcBwicQjH8EicjqnfJHzgycTfeg77+Sch0waZHGQKbzD753DgdHhtEdzxpa7/wD59D0yYDi8tgLuuhyz5W7rw/JkPwZhJ8PydsKDT5V4RaP/8E2ypnQDLfsTIp67DMR6btmJP/qn3iIqD9Cu5nNPUnmFza4rNLR1sbknvuN/Smv8LfGtrilQ2t2OFnilakW//OZfJEE41EUttpiqzjZU+ki3UsY+9wezwYgbTxCBrYjBpHONfMmfzoo9jov2FSyO/I0uIHEaOEGB8Pzeb130EH7WXmB1aTLZQIHIYjvH97Bmkq0YwO/Yk/5i+lxpvoTrcTCKRX0kt/sQfqK0fwQdWfI9hz/9rl+/9mb//JharZtAfnqL2z0vBQoWbQSjCyZcdRyhkJBc/ReTllYDjuSxk02SjNRx+xpG0pjLss/gXDF9z/473zYaitNeO5ehPfZe6RISRS5cQb2gmFK+BWDVEkwyu24sXps3A3UkvWkF246tkMhmy2QyZTIbDqsZy+2ETqUtGOWDRj0hs/RvhdDN0NGOehbEz4Nxr8x/43Yuh8Y13frkDT4XZhelebj4HGhvAc28//+Hz4GPngDvc283I/4dfDoeeDKkWWPgfYGEIRyFWA4k6xg47Bg7dC9pr4c1jIV4HiQEQr4VQmJFjjmDkqDqoHQEf/SxkU5DpyN+yHUSqayAazj8/8UKIJvO3SOF+xCH5HCMOgTPvfPv3sv13NPLQ/PNjJsN5CwB7uw8Gww7IP7/v8XDRQ5BLQ6Yd0u2QaScxZAwjE0k46ChIXIdlOph24Ad2/z/NHjL3LoOf9gsalTVY7k5LR5rmhnW0t7bQTpSm6BDaUllCm18jleoglc7Qkc6QSmdpsio2RkaSyuQYvWUZZDrIZdOQSZHLpllrI3klsh+pdJZD2p+hJRejORejMRujKRdlY7aabdk4qUyGQblt1Fsjg62RehoZbE08l9uXv4YncHDVZr6Z+w8SpIiRIUqamKe4e9DFPFt7DBPSf+HqtV8kTO4d3+eRQ26mYcxMxm5bzkee+hzZRD2eHAzRJEaO1mO/g436MLHXHyX+6BzMc+C5HffM/jmM+CD8+R54eE5+TGHPAZ5fqV1wH9TvAy//Lzx/ByQGQnLg2/eHnguJOti6GprWv70C2n5LDHx7U+S9//Ly2TyXX4mWint+BZtL51fEANveyK8Yw1EIRSAcK9w6/Z2ay+Vfl03l+8eq8++35f8gly28NgqhaP65eE3hezmEdJ7NrpjZs4WJ1XbdT8WhH3OHdBu0byPVuo2tVePY0pomt/4lQm2byISrSEeqyYSSpCJVpCK1+X3SOSfrbx+syx8Uyx9429GWzR84a2xNk2vZSK51C69kRtDYluZLLf/GAblXGWdvkbQUAA9mJ/K5dP6vv2fjn6Pemt4R9dfZo/ha9u+JR0I8F/o08R3b1HkLEycxf+AXSYaz/OyNk7p81ceHncviMV+g3jdz+bNdn09N/Uei067Gmt6Cuy+CWBWE4xCJ5e8P+zTsPQUa18EzP863JQZAVT1UDc7/1VczNL9Me2slLFKGVBz6sY4tb9Dyxsu0bX2T1Na3yDaux1o28MC4a9nUnuOjq2/nyIZ7qMo1Ey3s+My6MaHjFzgh5kZu46zI4ne8Z5MnObjjdgBujPyYaeEV+f3Gnt9vvN4Hc3b6GwDMicznsNBrRC3HGNtADa38LTKBb436T+qSUS5Z/09U0UFrzVg6BowjHK8hW7sX7aMnk4yGGfrGw8RDWaKRCLFIhFgkRHTwGEJ7HZYPs+bpwqZ/4S/HUBSSg6C6Pv9X4brn87sI0q35W6oVhh0Ioz8C2TQ89zOoGgLVQ96+Tw6CULjPfkci/ZWKQ5lLpbO8/srzbH15MbG1TzGi6c9cHvsnXmkbyGcyv+br0V++3dfDbGIAJ3fcSCo+mNPiTzM59CLZ2AA8XoclBxKpGsimsTMZVJ1kcMdaku3riWbbiGSaiWTbCJOjYf9zCRkMfeVOqjf+kTA5QoUbyUGkZtxENBwi8fhcQm+twCwEA8dA/QQYtj+MnxbY8hKR3qHiUEbS2RyvvtXIi2sb+OObrYRWLuaKbTdRb40ArGcQKxMH8+DoLxIaMJqxoQ2MYiOxgSNJDhpB3aChDK6OM7AqRiyi/akisud2tzjobKUScHdef/3/+Mvzj9O06jkGb/kTH7G/MD9zDvdHjuOY4XuxLnoka8ZMZugHj2HU3gcyPBQqmjT7wADTi4ioOLx37rB1NR1rnuPPW8Lcu2U8z77yNxa2nce4QpeG5AfYNnwGV048kbkfPJpQyIBZAYYWEdk1FYc91byRtoduIPTSAuKZRuLApuxE7uZqjpwwiqVV32T8/ocybN+J1CcGUB90XhGRHlBx2EObfno2Axqe47fZyayu+iB1e3+EAz50BCv2HUk8EgY+GnREEZE9puLQE68+zLb6Q/jHB9ayct1p7D3yEq4480ROH14bdDIRkV6l4rA7Nr0GD/4DvPogvwqdyQPtp/Ll42fwuY+PJxLW2UMi8v6j4rAr7dvgsZvwZT+kgyi3pM/lqfpZ3PvZj/LBvQYEnU5EpGRUHHbl/mvwP/43CyPHMKflU5w25TB+fdx+JKK6EldE3t9UHHYilcnxEzuD+1MH0pA8iH+/5ENMGq9zjkSkMqg47MQ3fvEQC15p49SJU/nGyQdQmyjh6JUiImVGxaE77tz4+tlM3+scjv/UaUGnERHpczrVpjvpViLkCCd10FlEKpOKQzfSrdsAsLiuXxCRyqTi0I22pq0AhJJ1AScREQmGikM32pq2ABBWcRCRCqXi0I1t0SF8O30O2fr9g44iIhIIFYdubAkP4UfZk4nU7x10FBGRQKg4dKO9cSPj7E1qYppoXkQqk4pDNwa9di+L419lQKg16CgiIoFQcehGrj0/t3N1zcCAk4iIBEPFoTsdjbR5jJrqZNBJREQCUfLiYGbDzex+M1tmZr83s1E76XejmT1R6PefZhbc0B4dTTSTJKnRV0WkQvXFlsOtwBx3nwRcC9zSuYOZnQAMcfejCv0iwCf7IFu3wqkmWq0KMx2QFpHKVNLiYGYDgXp3Xwbg7suBAYX2YmFgbdHjTcAbpcy2K0vqTub26LlBfbyISOBKvetmb+C1Tm0rC+3PF7UtBKaZ2ffIF6xV7r6085uZ2aXApQBjxowpSWCA58OHsLZ6QsneX0Sk3JV6t5IB3k1757bDgSgwD/gnYLSZXdDlRe63uftEd584dOjQ3s66w9Cml9gn0lCy9xcRKXel3nJYBXT+E3yfQnux7wDHu3sKwMyuAp4E5pc2Xveu3PxPvJo8FNBcDiJSmUq65eDum4FWM/swgJkdQv54QszM5hV1bSG/9bDdbOAvpcy2K1W5FrLRmqA+XkQkcH1xuuiVwO1mVgNsAy4EaoH9ivp8Hvg3M/sW+V1OfwW+1AfZunKnijZyMRUHEalcJS8O7r4WmNGpeT0ws6jPGuDUUmfZLelWwuRwTfQjIhVMV0h30tFSmOgnobkcRKRyqTh00kQVn0tdScPwo4KOIiISGBWHTpqzMR7MfQwGjQs6iohIYFQcOmnb/CYfD/2RgeH2oKOIiARGxaETe2MZP4/9C/XpN4OOIiISGBWHTjKFA9JJzeUgIhVMxaGTbFt+oh8VBxGpZCoOnWyfBa6qblDASUREgqPi0Il3NNHmMWo1C5yIVDAVh06erv8En89eTTyiWeBEpHIFNxVnmVrDSP4c1wxwIlLZtOXQyciGZRwVfTnoGCIigdKWQyczNv6EKdkwQQ0KKyJSDrTl0Eks20JHRMN1i0hlU3HoJJFtIROpDjqGiEigVBw6qfJWzQInIhVPxaGYO1W0kotpLgcRqWwqDkXcnVPT3+blvU4POoqISKBUHIq0ZXL8KTsOrxsddBQRkUCpOBRp2bKec8MPMzy3PugoIiKBUnEo0rFxJd+O/oRRqVVBRxERCZSKQ5H25m0ARKt0QFpEKpuKQ5F0YaKfWLXmchCRyqbiUCTdmi8OCRUHEalwKg5Fts8Cp4l+RKTSqTgU+fOwUzim4xaqB9QHHUVEJFAqDkU2ZxKs9FFUJ2JBRxERCZSKQ5GR6x/hvNijRMNaLCJS2bQWLPJ3Gx7gs6HfBR1DRCRwKg5FIukW2kJVQccQEQmcikORaLaZjrDmchARUXEoEs80k1JxEBFRcSiWyLWSidYGHUNEJHCRoAOUk3Oj3+XwvQZzRNBBREQCVvItBzMbbmb3m9kyM/u9mY3aSb/jzOxxM1tkZgvNrM/376zviBOt1tXRIiJ9seVwKzDH3ZeZ2UTgFuCc4g5mtjfwj8Ap7r7VzEYAuT7ItkMu1cYXsj+nruNU4KC+/GgRkbJT0i0HMxsI1Lv7MgB3Xw4MKLQXuxq4zt23Fvq95e5tpczWWeu2jXwuch+j0yv78mNFRMpSqXcr7Q281qltZaG92MFAh5ktMLMlZnajmXXJZmaXmtlyM1u+cePGXg3a2pwfkTWcHNCr7ysi0h+VujgY4N20d26rAs4FPgNMLTy+qMuL3G9z94nuPnHo0KG9GrS9aXtx0EQ/IiKlLg6rgAmd2vYptBdbDdzk7o3u7sAdwKElzvYO7S35WeBiVdpyEBHZ7eJgZqN7+ubuvhloNbMPF97jEGATEDOzeUVd/x/wHTMLFx5/EljW0897L1KF4hCvUXEQEenJ2Uq3mtkA4AHgbndfu5uvuxK43cxqgG3AhUAtsN/2Du7+sJlNAJaYWQZ4ivzWQ59ZNewYTmv/OfcN/2BffqyISFna7eLg7meaWQw4BvgHMxsGPEm+UKzZxevWAjM6Na8HZnbq9wPgB7ubp7c1t2dIE6G2Kh5UBBGRstGjYw7ungIeBR4jfx3CwcA/m9n3SpCtTw1e83tuiPyUmnj43TuLiLzP7faWg5lNAz4NjATuBi5y9+bCc0+VJF0fGtzwHFPCjxGPR4OOIiISuJ4cc5gBzHX3ztctAJzUS3kCY6kmmqkiGbKgo4iIBK4nu5X+DHxg+wMzm2FmswHcfUtvB+tr4VQTrZroR0QE6FlxuAhYXPT4IeCLvZomQJFMM+2m4iAiAj0ceK9wgdqOn80s2/uRgpHN5khHdHW0iAj0rDg8b2ZfIH/BmpG/fuG5kqQKwDfrvsWAZJSfBx1ERKQM9GS30jXkxzx6jPzupSj50VTfF5ra09TGNfeRiAj07CK4DHBT4QZA4YD0XSXI1ee+0vw9mto/Dnw46CgiIoHryXUOBwBzgNHkdytBfkui/xcHd07ILWFpZlzQSUREykJPdiv9mPwsbi+SH/riF8DPShGqr2Xam4lYDo/XBh1FRKQs9KQ4tLr7M8CfgLHu/p/AJ0oTq2+1FuZysIRGZBURgZ6drbTGzPYFfgt838zupY/neS6VlqYt1AHhhE5lFRGBnhWHi4Gwu6fN7DvAEcBZpYnVt9ra2njTBxOpGRx0FBGRstCT3UoPunsawN2fdvd57t67EzkHpKFmX47o+Hfax04LOoqISFnoSXFoMrNjSpYkQE3taQBqdJ2DiAjQs91KQ4CHzWwlsH2Lwd19cu/H6lvJ1x/hJ9EfMMB/DgwKOo6ISOB6chHcx0sZJEiRLf/HMeEVbEjEgo4iIlIWenIR3B+A7QPvDQH2Bn7v7ieWIlhf8o5GAGrqBgacRESkPPRky+GI4sdmNgs4ttcTBcDam2j1OMlEIugoIiJloUdzSBdz91+Tn0O637NUEy2WxEyzwImIQM92K+1X/BCYBLT3eqIAbKWGbGgcQ4MOIiJSJnpyttK15I85WOH+/4CzSxGqr/1qwEWsybbyQNBBRETKRE+Kw8PAm+7+COTnkCZ/zKHfj8ra3J6hNqFrHEREtuvJGvEiYHrR44eAJbwPisNlm77NluQHgH5/yYaISK/o0QHpznNIA++LOaT3S7/MCN8UdAwRkbLRk+LwvJl9wfJCZvYV3idzSFd5K9mY5nIQEdlOc0i7U+2teKwm6CQiImWjJ8VhGvAHd/94YSiNp4F+P6RGR1sTYXOIay4HEZHterrl8HTR46eA63s3Tt9rbWtnWW5/UrUfCDqKiEjZ6OkB6Y7ufu7PGqnmzNQ32Ty23w8RJSLSa3pSHFaZ2SnbHxTGVlrZ+5H6VlN7BoAaXecgIrJDT4rDFcBMM3vCzJ4EjgPuKU2sPrT6Dzwcu4rhLX8NOomISNnoSXGoBZqA1YXbIGBeKUL1pWzTBiaE1lEd11wOIiLb9aQ43AksL7zmRmAV8C8lyNSnMq3bAEjUai4HEZHtelIc3N3/h3yBSLn7NcC57/YiMxtuZveb2TIz+72ZjdpF37iZ/cnMTuhBrvck256f6KdKxUFEZIeeFIcGMxsOLAS+bmZT2b2xmW4F5rj7JPIju96yi77/TH6LpM94oThU12nuaBGR7XpSHD4HpN39JeC3wKnkB+PbKTMbCNS7+zIAd18ODCi0d+57PLAZeLYHmd6zDaHhLMp9hEQ83pcfKyJS1noyTejWop/vBe7djZftDbzWqW1lof357Q1mVg9cTH5+iOt29mZmdilwKcCYMWN2N/ouLas7joXRg98fg0SJiPSSPZ4mdDdtnxios85t3wW+7u67HOXV3W9z94nuPnHo0N6Zt625PUNNXNc4iIgUK/VacRUwoVPbPhQdVygcoJ4I/LQwh/M4YJaZfcTdv13ifJyzeg5np1uBo0v9USIi/UZJi4O7bzazVjP7sLs/Z2aHAJuAmJnNc/cr3H0dcND215jZHGCpu/fJrJ116Q2kTdc4iIgU64v9KVcCt5tZDbANuJD8BXX79cFnv6t4rpXW+OCgY4iIlJWSFwd3XwvM6NS8Hpi5k/5zSp2pWDLXwsao5nIQESlW6gPSZa/KW8mqOIiIvENFn6bj7jyY/SiJgYcEHUVEpKxU9JZDRybH1elLWTP6pKCjiIiUlYouDo3taQBqE9GAk4iIlJeKLg4db/6FV+Lns1/Do0FHEREpKxVdHNqatxK3NLFEMugoIiJlpaKLQ6olP5dDrGpAwElERMpLRReHdEt+LMF4jeZyEBEpVtHFIdOW33KoqtVcDiIixSq6OGyIjuaOzLEkB/TOCK8iIu8XFV0c/pY8mG9kPktNnXYriYgUq+ji0NbWQlUUouGKXgwiIl1U9Frx6JW3sCR8edAxRETKTkUXh3C6mTarCjqGiEjZqejiEEk30xbWiKwiIp1VdHGIZVtIhbXlICLSWUUXh0S2hXREWw4iIp1V9HwOC0LHMmTgaA4LOoiISJmp6C2H2zMn8NrwE4KOISJSdiq2OHguR7JjIwNj2aCjiIiUnYotDi0tjTwd/3sO33h30FFERMpOxRaH1sb8iKyhRF3ASUREyk/FFoe25i0AhJOay0FEpLMKLg75LYdolbYcREQ6q9jikCoUh1i1thxERDqr2OKwKbYX30mfTWTIhKCjiIiUnYotDhsjI/hh9hSS9XsFHUVEpOxUbHFIN25gjK2nNl6xi0BEZKcqds04/vW7WBL/MtXRil0EIiI7VbFrRutopNXjhCPRoKOIiJSdii0OoVQzrZYMOoaISFmq2OIQTjfTFqoOOoaISFmq2OIQzTTTHtJEPyIi3anY+RzujZ/CgOoMVwYdRESkDJW8OJjZcGA+MBhoBM5393Wd+gwEvg0cCCSAp4Cr3N1LletxP5T9BmsWOBGR7vTFbqVbgTnuPgm4Frilmz7VwG3ufrS7HwHUAjNLGWp068uMts2l/AgRkX6rpMWhsEVQ7+7LANx9OTCg0L6Du7/h7n8sanodiJUy27z0tzh2612l/AgRkX6r1FsOewOvdWpbWWjvlplNAY4A7uvmuUvNbLmZLd+4ceMeh8pmc9TQCnHtVhIR6U6pi4MB3R036NJmeV8BTgc+5e6ZLi9yv83dJ7r7xKFDh+5xqObmbYTNIa7hukVEulPq4rAK6Dzs6T6F9s5+Arzu7le6e3spQ7U25YfrDidqS/kxIiL9VkmLg7tvBlrN7MMAZnYIsAmImdm87f3M7BPAcne/p5R5tmtrKswCV6W5HEREutMX1zlcCdxuZjXANuBC8mcj7VfU5yhgmpmdUdR2t7v/eykCbQsP5rLUFVww8mOleHsRKaF0Os3atWtpby/pDoZ+L5FIMHr0aKLRPRs/ruTFwd3XAjM6Na+n6FRVd/9aqXMU2+pV3J+bxKX1Y/ryY0WkF6xdu5ba2lrGjRuHmQUdpyy5Ow0NDaxdu5a9997p+T+7VJHDZ6S3vMFRoT9RF04HHUVEeqi9vZ36+noVhl0wM+rr69/T1lVFFoe6N5ZwR+w71OW2Bh1FRPaACsO7e6/LqCKLQ7atEYCq2kEBJxERKU8VWRzoKBSHGp2tJCI9N3/+/B71v+aaa0oTpIQqsjhYRxNtxLFISUfoEJH3qZ4Wh7lz55YmSAlVZnFINdOC5nIQkZ6bNWsWK1asYNq0aSxZsoRZs2Zx7LHH8ulPf5otW7Ywe/ZsjjnmGI4++mjWrcsPQH344YezatUqrrvuOs466yymTJnCJz/5SVpbW7u8/1VXXcX06dOZOHEiS5Ys2dE+d+5cjj76aI455hhuvvlmIF+kpk6dyvTp0/nyl7/cq9+zIudzuL92Fr9JH8FNQQcRkffkhv99kZfWNfbqex44qo7rTzlop8//+te/Ztq0aSxevJhVq1Zx3333sWLFCg488EA2bdrEvHnzGDlyJHfddRff//73uemmt9c0P/7xj3nyyScZP348c+fOZf78+Vx++eXveP/zzz+fW265hbVr13LmmWfy5JNP8qtf/YoNGzbw6KOPAvDQQw+xdOlSFi5cyCOPPEI4HObBBx/s1eVQkcXhtdwoUjUjgo4hIu8DRx11FAceeCAAQ4YM4cUXX+SOO+7gmWeeIRwOv6Pvaaedxvjx44H81kR3K/Rhw4bxwx/+kBdeeIG33noLgHvuuYfvfve7O/ocf/zxfO1rX+NLX/rSjs+YMaPz5WTvTUUWh/2alhJJDgAmBx1FRN6DXf2F31eGDRu24+ff/OY3LFiwgK9//evMmDGDG2+8cad9IX+xWrFVq1ZxySWX8K1vfYvzzjuPxx9/vNt+O2vrTRV5zOG85p9wSkufDOMkIu9DmUyXQaMBeOyxxzj//PM54IADuPPOO3u8Al+xYgWTJ0/miCOOYNGiRTQ0NAD54xzFu6cWLlzIaaedxve+970dWX73u9/t4bfpXkUWh2SuhUxUczmIyJ4ZPnw4U6ZMYfXq1e9ov+yyy/jGN77Bxz/JsmPkAAAKcElEQVT+ccaPH88zzzzD4sWLd/t9Z8yYwfPPP8/UqVNZvHgxBx98MLfeeitnn302I0aMYMqUKUyePJlXX32VyZMnc9JJJzFt2jQmT568Yyujt1ipN01KZeLEib58+fI9em3j9SP564iTmHjZj3s5lYiU2ssvv8wBBxwQdIx+obtlZWbPuvvEd3ttxW05pDNZamjDY5rLQURkZyquODQ3bSNkDgnNAicisjMVd7ZSPFnFwiN+yX4T9gk6iohI2aq44lCVSHDijJnv3lFEpIJV3G4lERF5dyoOIiLShYqDiIh0oeIgItJDPR2yG2Dr1q3ce++9vR+mRFQcRER6SMVBRETeoXg+h9WrV/PEE08wffp0pk6dyg033ADAsmXLOPzww5k0aRInnngi27Zt46yzzuKBBx5g2rRppFKpd7znnXfeuWMOh3nz5u1ov//++5k2bRrHHXccn/nMZ0in0zzzzDNMnz6d4447jlmzZrFly5aSfM+KO5VVRN5nfnpS17aDToWPXQKpVrjzjK7PH3oOHHYutDTAXee987kLdz2AXfF8Dlu3buWrX/0qixYtoqamhs9+9rMsXbqUuXPncscddzBhwgRefvllstksv/zlL5kzZ063Wx2HHHIIDz/8MJlMhkmTJnHBBRewdetWbr31Vu6//36SySSPP/44bW1tfPnLX+a3v/0tgwcP5sUXX6SlpYVBgwb1YIHtHhUHEZE99MQTT7Bx40ZOPvlkAFpaWnjxxRc544wzuPjii7n44ouZOXMmgwcPprFx55MSTZgwgbvuuotly5axZcsW3nrrLRYtWsQFF1xAMpkEYMqUKSxcuJATTjiBwYMHA3DQQaUbslzFQUT6t139pR+r2vXz1fXvuqWwK2bGhRdeyHXXXdfluZNPPpkHH3yQWbNmcfPNN3eZy6HYJz7xCS677DKuu+461qxZg7sHModDMR1zEBHpoe1zKBx55JE89NBDO/b7//Wvf2XRokU89thj1NXVMXv2bI4//niWL19OKBTqcqwB8geqU6kUs2bNoqmpiUceeQR358QTT+T222+npaUFgMWLF3PUUUdx3333sWnTJgBeeOEF1qxZU5LvqC0HEZEe2j6fw5133snNN9/M6aefTi6XY+zYsdx6660sWLCA66+/HndnxIgRzJ8/n2g0yurVq5k2bRoPPfQQsVgMgIEDB3LkkUcyefJkhg8fziWXXMI111zDggULuOqqq5g5cya5XI7999+fI488knnz5jF79mzS6TQjR47khz/8YUm+Y0XO5yAi/Zfmc9h9ms9BRER6lYqDiIh0oeIgIiJdqDiISL/TX4+V9qX3uoxUHESkX0kkEjQ0NKhA7IK709DQQCKR2OP30KmsItKvjB49mrVr17Jx48ago5S1RCLB6NGj9/j1Kg4i0q9Eo1H23nvvoGO875V8t5KZDTez+81smZn93sxG7aTf9Wa23MyeM7PTS51LRER2ri+OOdwKzHH3ScC1wC2dO5jZTKC+cGHGZOByMxvSB9lERKQbJS0OZjaQ/Ep/GYC7LwcGFNqLXQzMLfRpB34EnF3KbCIisnOlPuawN/Bap7aVhfbni9qGu/u6osevAkd0fjMzuxS4tPCw2cxeeQ/ZhgCb3sPrg6LcfUu5+5Zyl97Y3elU6uJgQHfnm3Vus93og7vfBtzWC7kws+W7M75IuVHuvqXcfUu5y0epjzmsAiZ0atun0F5so5mNKHq8H123OEREpI+UtDi4+2ag1cw+DGBmh5Df9IqZ2byirj8Dri70iQHnAf9dymwiIrJzfXGdw5XA7WZWA2wDLgRqyW8dAODuvzGzj5rZMvK7k25094YS5+qV3VMBUO6+pdx9S7nLRL+dz0FEREpHYyuJiEgXKg4iItJFxY2tZGbDgfnAYKAROL/TNRZlp3A9x5tFTbe7+y+CyrO7zOxaYIu7/6Dw+O/I75tNAmvIL/vmACN2qzi3mY0GHgSKR3n7trv/Pph072Rmk8mPPFADxIEb3P3Bcl/W3eUGXqSMlzWAmVUD3yZ/FuYAYANwobs3lvsy7zF3r6gbcAcwqfDzROC/gs60G5mXBp2hh3n3ApaS/0/++aL2B4DxhZ8/CdwcdNZ3yw2MA34ZdLZdZJ4C1BZ+HgS80E+WdZfc5b6sC1nrgQ8VPf574Kr+sMx7equo3Uo9GM5D3gN3f8PdD6dwejKAmR0MrHb3lYU+C8iPo1U2ustd7tz9cXdvKjzcCrT1k2XdJTfdXwxbVty9wd3/CGBmUfLXbb3cH5Z5T1VUcWDXw3mUsyFmdruZLTKzu81sXNCB9sAEoPNwJ5vNbFAQYXpoHzO7w8weNbP55TgopJmFyA9qeTv9aFl3yu30j2V9mpk9BvyN/Cn5T9KPlvnuqrTisLvDeZSbG4Cr3f1Y8v+Rfhpwnj3RX5f9JvIjC3/O3Y8GFgDzdv2SvmVmw4A7gSWeH2KmXyzrbnKX/bKG/HVZ7j7V3ccAPwF+QD9Z5j1RacVhFbs3nEdZcfdfeP5qc9x9KfkDXv3NSooufCyod/etQYTZXe7e7O6/dPeWwuPfkP83UxbMbDz5PxauLuzKgH6wrLvLXe7Lujvu/mvyx0rKfpn3VEUVB9/JcB7l/gssnNmx/eepwOoA4+wRd18B7GdmYwHM7ATg8WBTvTszqzOzg4oenwP8IcBInc0BLnL3tdsb+smynkOn3P1gWWNm9WY2vejxZ4An+8ky75GKO5WV7ofzKHdnmtn1QBRYB1wWcJ49dQVwh5kZ8AZwUcB5dkcW+KqZ7QXEgJeAq4KN9A6TgV/lF+kOF1P+y7q73FcAV5TxsgZoBWab2Y1AC/njDF8pPFfuy7xHNHyGiIh0UVG7lUREZPeoOIiISBcqDiIi0oWKg4iIdKHiICIiXag4iPQRM5tTOP9dpOypOIiISBcqDiIi0oWKg0g3LO9bhZFwl5jZx8zsATP7gZk9bmb/Y2Z1hb5fLLQ9WRjyATOLmtmtZvZIYYTRTxfe+hAz+72ZPWdm55pZwsx+ZWZLzWy5mc0I7EuLFKnE4TNEdse5QNzdjzWzweRHCP0YMMXdnzazs4DrzOw+4ChgKvn/T/9rZiuAU4DX3f2rZhYGppEf9HEGMJP87GdPkx+C4VV3P9PMksABffotRXZCw2eIdMPM7gZGAalC03Cg3d0PK+rzBPmZ4+519ycKbZ8CxgKnAtPcPVvUfw75QnBn4fFjwHnAr4DfFd7nTyX+aiK7RVsOIt1z4Ex3X7O9wcwWF/0cKfTZfit+nZMf37+7mc0aOvVdT34Quo8Bl5tZk7t/rZe+g8ge0zEHke79Bvji9gdmdhEwpjCaL8DlwP3AveRX6mZmMeCSQvvvyI/SiZmFdnEK677AoMI8Hf9MfveUSOC05SDSDXf/LzMbb2aPkx+2+x4gA/xX4UD0X4AvuXvKzA4AHgUSwL+6+8tm9hpwU2HXUQT4t518VAaYb2bV5Idkv6K030xk9+iYg8huMrPF7j4t6BwifUG7lUREpAttOYiISBfachARkS5UHEREpAsVBxER6ULFQUREulBxEBGRLlQcRESki/8PfF4mYpqcGcoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mnist_test(784,10,10000,200,0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 배치 사이즈를 줄여보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T06:08:27.076087Z",
     "start_time": "2019-08-04T06:08:19.962109Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1epoch => train acc, test acc | 11.236667%, 11.350000%\n",
      "  2epoch => train acc, test acc | 91.010000%, 91.130000%\n",
      "  3epoch => train acc, test acc | 92.051667%, 92.260000%\n",
      "  4epoch => train acc, test acc | 92.983333%, 92.780000%\n",
      "  5epoch => train acc, test acc | 93.233333%, 93.040000%\n",
      "  6epoch => train acc, test acc | 93.681667%, 93.530000%\n",
      "  7epoch => train acc, test acc | 93.898333%, 93.390000%\n",
      "  8epoch => train acc, test acc | 93.686667%, 93.400000%\n",
      "  9epoch => train acc, test acc | 93.793333%, 93.130000%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAENCAYAAADkNanAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8lOW99/HPb2aysCTs+76UqixuVBBFcEW0tRVbtVq3upxTn9NWW7XaUx/Rp8dSldNyTntarbS2lR7rqXU5iltRRFSQAC4giIgsAYEASYCELDPze/6YAYdMgAxmcifk+3698srMdV/3zDd5wfxyL9d1mbsjIiKSKhR0ABERaX5UHEREJI2Kg4iIpFFxEBGRNCoOIiKSRsVBRETSqDiIiEiaJisOZnaHmf3zQbbfZWZFZrbEzC5qqlwiIpIu68XBzPqY2QLgBwfpMxno4u6jgXHAjWbWNdvZRESkflkvDu6+0d3HArcepNt1wLRk/yrgd8A3s51NRETqFwk6QFIPd9+U8vwj4OS6nczsBuAGgHbt2p141FFHNVE8EZEjw+LFi7e5e7dD9WsuxcHqaUub9MndHwIeAhg9erQXFRVlO5eIyBHFzNY1pF9zuVupxMx6pjwfBqwOKoyISGsXSHEws+5mNiOl6Y8kr0mYWS5wJfDfQWQTEZHgjhwKSBwdAODuTwLVZrYQmAf8xt23B5RNRKTVa7JrDu7+SMrjj4HJdbb/uKmyiIjIwTWXaw4iItKMqDiIiEgaFQcREUmj4iAiImlUHEREJI2Kg4iIpGku02eIyBEoHndi7sTiyS/3RNu+xxCNxzEzwmaEQhA2IxwyQiH77PG+72BW32w70thUHESagXjcqYnFqY3FqYnGqY05NdE4Ncnnqdtq9vWJU1MbJVZTRay2ilhNNfFoNZWex65QIbXRGrqXL8Oj1RCthlg18bizKdKPjTn9CUWrGVa1lFjciDrE3Ii5sTHUi62hboRjexhQu4ZY3IhhROOJPpvoQqkXEIlV0cO3Eo1DNLlvTRy2xQvZ7XmE4zV0oII4lvwK4RiV5BElQog4OUST7Z+JEcIJYcSJEE/7XXkoTMjCREJxci3+WRGxREFxixAKh8mxGLkWx0JGJKXAxEO5hEIhci1GrsUIGfsVoFgon1A4RJ5FMXdihIm64Wa4O+4Qd8eBuLN/W8q2+vriTrxOX6hv38T+qX193+s4I3p3YObVX8rqv0kVB2lxorE4FdUxdlXXUhtzYvE40fhnf51G44m/TlPbUrfF9v01Gycac+Ke0h6LE4/HqPUQsbgTqq1IfLjGosTjUTxWS9TDlEW6Eo07narWE6mtwD0G8Rgei7LH8vkkMoTqaJzhFQtpU1tGKF6LxWoIxWvY6F14ibHUxuLcEHuMDvFyIh4l12rJo5Yl8S/wcOx8AP6S81M6WAXtqSWXWnItyjOxcdwbvRwjzif530r7/fwueh6/iV9Bx3AVi8Pp62v9Oe9SFrS5nG7s4Idl/5q2fVaHG3ixwzfoG9vCvTvSZ9p/vNcPWdT5q/SvXsV3V9/02QYDwvD0sHtY1X0SA3Yu4eLlN6bt//KxM9jUYyL9t73G6Uu+l7b9hdEPs6njaAZ9OpvTl6ePjf3TyEfY1HYoo7Y8yXlrpyUaPfkVh/u+MIstOf04bdtjfHXrf6Xtf1Of/2Z7qAtfK/sLF+16NG371zs9zm5vw3WVD/P1mqf3tccIESfElM5/BwtzXcXvmFA9l5iFiRMmZiH2WDt+1O3XhAwu2fVHRlYvJW5h4hYmRpjd4Y482P0nGHBe+X8zoHZ1Yjth4qEw5ZGuPNftWsyM08qeokvt5uT+EdxC7MzpRlHnr9C/S9u03I3N3NMmP20RNCtrC+EO0WritVVU7qmgsrKCPXv2sCO3F7trjfiOtbBjDbXVexJfNXuIVVcxv+2ZlNeGGLhzEUMq38Wj1Vi0GovVEPEabq29ASfEt8Ivc3ZoMYn/ek7YYsQJcWnNnQDcGnmMc5LbI8QIW5xd3pZza34OwIycX+3bP7HdWRfvzoSaXwLwaM6/cWp4+X4/0kofwCWhB4iEjD/G72CEf7Tf9mXho7mt8H5yIyF+VfrP9I2u33//difxxyHTyQ2H+JcVl9MuWkYslEs8lEM8nMfGruN5f/gt5IRDjFt8MxGvxSP5EM7FInlU9h7L7i9+nbxIiK5LZhCO5BLKySOck08kN49Qz5GE+50I8RismQuRPAjnQSQXLATtukFBT4jWwOb3wOP7f3UaCB36Qk0FrH8r+Wdscls8Bj1HJPpU7oA1r362PR5LfB8wDjoPgp2b4MPnk/umvMYXJye2b/8YPngaPLb/v5lRl0DH/rB1Bax8Lv3f1PHfSuT/9F1Y/Y/07SdeA207Q/Fi+OS19O0nXQ95BbDuLdiwMH372BsTv6tP5kFxUeLnikcTXx6Ds6Ym+r33eOL3E49+1ieUA1/7dWL7/F/A2vn7b8/vCJc9ltj+7A9Stif7dOwP334+sf3PFyYy7n1fj0Pv4+GGuemZM2Bmi5MLqx28n4qDUFsFVWVQVQ57kt97H4+360rNpmXElsyitroicfqipop4bTXvDvsXNuf2p8umVzlhzYOEYtWE4tWEY7WEvZofF05jRaw351c+zQ9jv097y1OrZ1Ds3bgx/BS35Tyetv0rbf5ELL8z19f8mQsr/kqt5RC1PGLhXOKhXJ445RnatmnD8E8eoc+nL+OhMFgEQmE8kscHp88kbEbvD/9EYckiCEWwUARCETyvgB3j7yESMgpX/pXc0lVYKIKFI4RCEWjbmfhJNxAOGZEPnyW061MsnNgXC0PbLnDUeYmgn7wO1TsT20LhxPf8jtD7uMT27R8nvqd+QEfyE89FMhFPFtfw5zvho+LQmsTjiQ+oqvLPPuQ7DaKmfR92b9tAqGgm0cpSvLIMqsqwqnIWDbyBle1OouvWt7jio/RD+5tCt/Nc9bGM98X8Kuc/qSSPGiJUew7V5HJL7T+xzAczLrSM6yOziYdyiYUSH34ezuOFTpdT0743R8VWM7xqMaGcNkRy88nJa0NOXj67B5xNm4KOdKzZQmHNZtq0aUebNm3JyWuT+OAs6JX4sI3HEn/t6iKkSKNQcWiJqnclDsWryonmd2Z3u/7s2rmT3KLfEqssJb6nDPaUEarZybKu57GowyQiO9dz64eX1rmkBz+NXcXDtZP4ghXzQu6P2Ek7yr0dO2nLTm/Lb2MXMD8+kiF55Xw9Mp9oTiHRvELieR2wNh2pKBxMbrtOFOSFKWiTS2F+hHa5EdrnR2ifl/xKPs6LhHQHiUgL0dDioAvSzUR0zy6q7zuKdr4bgD9Fz+We6JXkUcOH+fezx3Mppx07vS3ltOOlLZ/yXGg9PfJr6ZV7CdHcAmK5hXh+R8jvQGHhYG4p7Elh/jE8k38eBfm5FORHKGyTw8D8CL/Oz6F9XoRwyIDLgv3hRaTZUXFoJjYsf5NBvpunCi+nsseJhAoHcVfnQRTk5/BSzvu0b9eWwvwcCvNz6JYf4Wf5ER4I7x3D+JVAs4vIkUfFoZkoXfUmg4AvXXIHffr0CzqOiLRyKg7NxHN+Co+FIvy8d9+go4iIaG6l5uLVzbnsGDBZF3ZFpFlQcWgGykuKGbvjGcb1TJ8qQEQkCCoOzUDx0pe5N2cmJ3aqDDqKiAig4tAsVH2ykD2ey9ARY4KOIiICqDg0C4Xb32F1zhdo17ZN0FFERAAVh8DFa6oYUPMRpZ2ODTqKiMg+Kg4BK/5oKblECfc/KegoIiL7qDgEbMGevoyqeoieJ3w56CgiIvuoOARs6fpSyO/IoF5dg44iIrKPikPATl85leu6LiMU0uA3EWk+NH1GgCq2F3NOzRzyCkYFHUVEZD86cgjQhvfmAVA49OSAk4iI7E/FIUCVaxZQ42EGjxwXdBQRkf2oOASoXclSVoeH0KGwIOgoIiL7UXEIiMfjbK8KsbnzIVfrExFpcioOAVlfuofLqm7j0y/dHnQUEZE0Kg4BWbquFIDj+3UKOImISDrdyhqQfm/czh/yNvLFnucFHUVEJE3WjxzMrIeZPW9mC83sZTPrfYB+95rZ/GS/35jZEV24epQtoX2bPMIa/CYizVBTnFaaDkx19zHAHcADdTuY2blAV3c/NdkvAny1CbIFoqq8hL6xjVR2Oz7oKCIi9cpqcTCzjkAXd18I4O5FQIdke6owUJzyfBuwMZvZgrTh/cTgt3ZDNPhNRJqnbB85DAJW12lbk2xPNRsoMLNfmNkMYKu7L6j7YmZ2g5kVmVlRSUlJdhI3gZ2r3yLmxoBRpwYdRUSkXtkuDgZ4Pe1128YCOcAM4P8Bfc3s6rSd3B9y99HuPrpbt26NnbXJLKnqxRM5X6Zb585BRxERqVe2L/quBYbWaRuSbE/1M+Acd68BMLNbgDeAR7IbLxgzdxzHmKFncHHQQUREDiCrRw7uvgOoNLMTAMxsFInrCbnJ00d7VZA4etjrYmBlNrMFZfPWrdTu3MLx/epedhERaT6a4nbRm4CZZtYeKAeuAQqAYSl9/hn4TzO7h8Qpp1XA95ogW5PbuvBxFuf/Kys6vhZ0FBGRA8p6cXD3YmBSneYtwOSUPhuAr2U7S3MQW/82Zd6OIcNGBB1FROSANH1GE+tc+i5r8o4mN+eIHuMnIi2cikMTqqkoo1/tOnZ1PS7oKCIiB6Xi0IQ2LJtPyJz8QWMP3VlEJEAqDk1oUUVPbq29gQHHTgg6iojIQak4NKG3toZ5vd259OzePegoIiIHpeLQVNzpsebvnNGrJugkIiKHpOLQRHZs/JAf1/wH5+a/H3QUEZFDUnFoIpuSM7F2/uIpAScRETk0FYcmUrN2Ibs9nyHHfCnoKCIih6Ti0EQ67niXj3O+SJv83KCjiIgckopDE4jV7KFfzRrKuxwbdBQRkQZRcWgCq7bXMrb6V1Qcd13QUUREGkTFoQksXV/GdjpwzLC6S1uIiDRPKg5NoHDJf/HtNvPo37lt0FFERBpEU4Nmmztjt/6Vrm2Pw8yCTiMi0iA6csiynVvW0dV3UN3zxKCjiIg0mIpDlm1YlljxreOwcQEnERFpOBWHLKtas4Aqz2HISE3TLSIth4pDlu0qL2NFztG0b6uL0SLScuiCdBbF4873K6/hvBE9OD7oMCIiGdCRQxZ9sr2C8j21HN+/c9BRREQyouKQRaXzHuLx3Ls5oVdO0FFERDKi00pZlLPhDXqGSujWu0fQUUREMqIjhyzqvvN91rUZTiikwW8i0rKoOGRJxY5N9IpvYU+PE4KOIiKSMRWHLNnwXmLlt4KhJwecREQkc7rmkCUrS50NsRMYPUIjo0Wk5VFxyJJnd36BNR3u4pWOHYKOIiKSMZ1WygKPRfl4/QaO698x6CgiIodFxSELtqxeyquxq7kgb2nQUUREDouKQxZs/uB1AHp/UdN0i0jLpOKQBb5hEdu9kMFDhwcdRUTksKg4ZEG38vf4JP9oIpFw0FFERA6LikMjq9q5jb6xYiq6afCbiLRcWS8OZtbDzJ43s4Vm9rKZ9T5Av7PN7HUzm2Nms82sXbazZcOKLRX8uPZawsecH3QUEZHD1hTjHKYDU919oZmNBh4ALkvtYGaDgH8FvuLuZWbWE4g3QbZGt3hzjL/EzuSmkV8KOoqIyGHL6pGDmXUEurj7QgB3LwI6JNtT3Qrc6e5lyX6b3X1PNrNlS+XKOYzusIvuBflBRxEROWzZPq00CFhdp21Nsj3VSKDazJ42s3lmdq+ZpWUzsxvMrMjMikpKSrIU+XOIx7l6413clP9s0ElERD6XbBcHA7ye9rptbYHLgSuACcnn307byf0hdx/t7qO7devW2Fk/t5J1yymkAu+j8Q0i0rJluzisBYbWaRuSbE+1HrjP3Xe6uwOPAsdlOVuj27w8MRNrl6NODTiJiMjn0+DiYGZ9M31xd98BVJrZCcnXGAVsA3LNbEZK118DPzOzvQMDvgoszPT9ghZd9zbl3o4hR7e4uiYisp9M7laabmYdgBeAv7l7cQP3uwmYaWbtgXLgGqAAGLa3g7v/w8yGAvPMLAq8SeLooUXpVPouH+cdxQk5WjNaRFq2BhcHd7/EzHKBM4Afm1l34A0ShWLDQfYrBibVad4CTK7T77fAbxuap7mpjcW5oupWvjmqExr+JiItXUbjHNy9xsxeBToAF5G4y+g4M9vh7jdnI2BLsfLTXWyIdqT/UTqlJCItX4OLg5lNBL4F9AL+Bnzb3Xcnt72ZlXQtyLaiJ7g2vJjj+50edBQRkc8tkyOHScA0d687bgGg1c8V0XX137gi5xN6d2wTdBQRkc8tk1tZlwH99j4xs0lmdjGAu5c2drAWxZ0+u5exqWAEZhZ0GhGRzy2T4vBtYG7K85eA7zZqmhaqdOMqOlNOtNfooKOIiDSKjAbBJQeopT6ONXqiFmjjssTKb52GjQs4iYhI48ikOCw1s3+xhJCZ/QBYkq1gLcmOTWvY6W0ZOuKkoKOIiDSKTIrD7STmPHqNxOmlHBKzqbZ6D8Yv4Mous2iTnxd0FBGRRtHg4uDuUXe/z91PS379nMRYh1YtFnfe3VDOyP7dg44iItJoMhnncDQwFehLYrZVSBxJPN74sVqO4vdf53fxqezuNC3oKCIijSaTcQ4Pk5gn6XoSp5MuA1r9ijY7VrzGyeEPWD+wf9BRREQaTSbXHCrdfRHwPjDA3X8DXJCdWC1HeFMRG+lGv/4Dg44iItJoMikOG8zsC8AzwN1mdhUtdJ3nxtRz1zI2tB2uwW8ickTJpDhcB6x193XAz4COwKVZSdVC7Nyyju6+jeqeGvwmIkeWTK45vOjuZwO4+9vA29mJ1HKs2rCJythICr84PugoIiKNKpMjh11mdkbWkrRAb5Z35aroHQw59pSgo4iINKpMjhy6Av8wszVASbLN3b3VzhmxbO1mvtC9PYX5WvlNRI4smawEd1o2g7Q0Hq3hlxsu4rUeVwMTgo4jItKoMhkE9xawd+K9rsAg4GV3Py8bwZq7jR8uoi/VFPYeGnQUEZFGl8mRw8mpz81sCnBmoydqIbatmE9foOcxOqASkSNPRlN2p3L3v5NYQ7p12ljEFu/EoMHDgk4iItLoMjmtlPopaMAYoKrRE7UQ3cvfZ13b4fQIH3Z9FRFptjK5W+kOEtccLPn9E+Cb2QjV3FVW1/JgzbkcO2wkWsFBRI5EmRSHfwCfuvsrkFhDmsQ1h1Y3K+v7G3fyx+jZ/P44jYwWkSNTpmtIv5ryvNWuIb32g0X0oYTj+nUKOoqISFZkcuSQtoa0mbXKNaRHLZvG79uW07nd1UFHERHJCq0hnSGPRelftYKSDqOCjiIikjVaQzpDW9a8SzuqoO+Xgo4iIpI1mRSHicBbe9eQJjEra6sbAbZl+esAdD9aM7GKyJEr0yOH1Gm63wTuatw4zV98w9vs8AIGDRsRdBQRkazJ9IJ0derj1rj62a+5mD5dJnJ3JBx0FBGRrMnkyGGtmX1l75Pk3EprGj9S81UdjTFvcx75Q3VKSUSObJkUh+8Dk81svpm9AZwNPJGdWM3TmmULuZTnGd0rowMuEZEWJ5PiUADsAtYnvzoBM7IRqrmqfO8Zpkb+xLF9OwQdRUQkqzIpDrOAouQ+9wJrgZ9nIVOzlb95CWtDfenerXvQUUREsiqT4uDu/j8kCkSNu98OXH6oncysh5k9b2YLzexlM+t9kL55Zva+mZ2bQa6m4U6/yuV8WtB6ZykXkdYjk+Kw3cx6ALOBH5nZBBp2t9N0YKq7jyExs+sDB+n7UxJHJM3O9vUfUMhuYn002Z6IHPkyKQ7/BNS6+wfAM8DXSEzGd0Bm1hHo4u4LAdy9COiQbK/b9xxgB7A4g0xNZt3KJcTd6PLFU4OOIiKSdQ0uDu5e5u47ko+fcveb3X3VIXYbBKyu07Ym2b6PmXUBrgPuO9iLmdkNZlZkZkUlJSUNjd4oXoyP5sTowww55oQmfV8RkSBkexmzvQsD1VW37d+BH7n7QWd5dfeH3H20u4/u1q1bY2VskKXry+jfuxf5uTlN+r4iIkHI9g37a4GhddqGkHJdIXmBejTwh+SI64HAFDM70d3/Lcv5GiS6Zxff3fQj1gy7Fjgl6DgiIlmX1eLg7jvMrNLMTnD3JWY2CtgG5JrZDHf/vrtvAobv3cfMpgIL3P2FbGbLxPrlbzLe3iWnq44aRKR1aIqhvjcBM82sPVAOXENiQN2wJnjvRlG26k0A+o3QtBki0jpkvTi4ezEwqU7zFmDyAfpPzXamTOV8WsQ6etG/d9+go4iINIlsX5Bu+dzps3sZm9qPoDXOQisirZNmkDuE0rJSVkZ7U9VnXNBRRESajI4cDuGdLVEuq/0JbcdcFXQUEZEmo+JwCEvXbSdkMEozsYpIK6LTSodw/uJrGV7Qh7a55wcdRUSkyejI4SDiNVUMqv6QNh26Bh1FRKRJqTgcxMYVC8i1KJEBY4KOIiLSpFQcDmLbyvkA9B6hmVhFpHVRcTiI0KbFbKIr/fsPCTqKiEiTUnE4iJdrRjK30zcIhTT4TURaFxWHA9hVVcuvy8ZQMvzaoKOIiDQ5FYcDWLlqFd28lOP7py1aJyJyxNM4hwMIvf1b5ufNYk+v9UFHERFpcjpyOID2Je/wcWQIHQraBx1FRKTJqTjUw6M19K9eybYOo4KOIiISCBWHeny6ajFtqCHU76Sgo4iIBELFoR5bk4Pfeg7X4DcRaZ1UHOrxcmw0t8a/x8AhRwcdRUQkELpbqR7zNudQ0O98wmHVThFpnfTpV8ee8u0cu+XvnNozGnQUEZHAqDjUsf69ufw0MpOTCkqDjiIiEhgVhzp2r36LmBsDR50SdBQRkcCoONTRdusS1oQH0rVz56CjiIgERsUhhcei9NvzAVsLRwYdRUQkUCoOKbauX0l79uB9vhR0FBGRQKk4pCja1YVjqx6i44lTgo4iIhIoFYcUS9eXUhUpZFj/3kFHEREJlAbBpThx+U9p22UkuZHJQUcREQmUikNSdUUpkypn07ZTr6CjiIgETqeVkja8/wYhc9oOHht0FBGRwKk4JO386E0A+o88LeAkIiLBU3FIyt28mE+sLz169Ag6iohI4FQckkqr4qwvPDHoGCIizYKKA1Cyq5orKm5i1YlTg44iItIsZL04mFkPM3vezBaa2ctmljaIwMw6mtmvzexVM3vLzKabmWU7215L1ydmYD2+f8emeksRkWatKY4cpgNT3X0McAfwQD192gEPufvp7n4yUAA02WCDjvPv5tHcexnRp0NTvaWISLOW1eJgZh2BLu6+EMDdi4AOyfZ93H2ju7+b0rQOyM1mtlRdt71NQV6Y/JxwU72liEizlu0jh0HA6jpta5Lt9TKz8cDJwLP1bLvBzIrMrKikpKRRAkardtO/Zg3lnY9tlNcTETkSZLs4GOD1tKe1WcIPgIuAr7t72jqd7v6Qu49299HdunVrlIDFy98iYnFyB45plNcTETkSZLs4rAWG1mkbkmyv6/fAOne/yd2rspxrnx2r3gCg7wgNfhMR2SurxcHddwCVZnYCgJmNArYBuWY2Y28/M7sAKHL3J7KZpz7vVnTlCTubPn37NfVbi4g0W00x8d5NwEwzaw+UA9eQuBtpWEqfU4GJZvaNlLa/ufuvsh3uz+UjGDxwDBc13Z2zIvI51NbWUlxcTFVVk51gaJHy8/Pp27cvOTk5h7V/1ouDuxcDk+o0byHlVlV3vy3bOepTVlZKeckmjj/hpCDeXkQOQ3FxMQUFBQwcOJAmHA7Vorg727dvp7i4mEGDDnj/z0G16hHSGxc9w+L873BK+01BRxGRBqqqqqJLly4qDAdhZnTp0uVzHV216uJQ9ckCqjyHocNHBx1FRDKgwnBon/d31KqLQ8G2d/g4MpT2bdsGHUVEpFlptcUhXlvNgJqP2N5Jg99EJDOPPPJIRv1vv/327ATJolZbHDauXEgetYQHaPCbiGQm0+Iwbdq07ATJolZbHJbs7MCttTfQa9QZQUcRkRZkypQpvPPOO0ycOJF58+YxZcoUzjzzTL71rW9RWlrKxRdfzBlnnMHpp5/Opk2Jm13Gjh3L2rVrufPOO7n00ksZP348X/3qV6msrEx7/VtuuYWzzjqL0aNHM2/evH3t06ZN4/TTT+eMM87g/vvvBxJFasKECZx11lncfPPNjfpzNsU4h2ZpwZYQL+acxc/7DQg6iogcprv/dzkfbNrZqK95TO9C7vrK8ANu//vf/87EiROZO3cua9eu5dlnn+Wdd97hmGOOYdu2bcyYMYNevXrx+OOP88tf/pL77rtv374PP/wwb7zxBoMHD2batGk88sgj3Hjjjfu9/lVXXcUDDzxAcXExl1xyCW+88QZ//etf2bp1K6+++ioAL730EgsWLGD27Nm88sorhMNhXnzxxUb9PbTa4tBh9VOc2fs4QiHd9SAih+/UU0/lmGOOAaBr164sX76cRx99lEWLFhEO7z/T84UXXsjgwYOBxNFEfR/o3bt358EHH+S9995j8+bNADzxxBP8+7//+74+55xzDrfddhvf+9739r3HpEl1h5N9Pq2yOFTs2MTtlQ/wWvfvA18JOo6IHKaD/YXfVLp3777v8ZNPPsnTTz/Nj370IyZNmsS99957wL6QGKyWau3atVx//fXcc889XHnllbz++uv19jtQW2Nqldcc1r+XOI9XMPTkgJOISEsUjaZNGg3Aa6+9xlVXXcXRRx/NrFmzMv4Af+eddxg3bhwnn3wyc+bMYfv27UDiOkfq6anZs2dz4YUX8otf/GJflueee+4wf5r6tcriUPnxW9R4mMEjxwUdRURaoB49ejB+/HjWr1+/X/t3vvMdfvKTn3DaaacxePBgFi1axNy5cxv8upMmTWLp0qVMmDCBuXPnMnLkSKZPn843v/lNevbsyfjx4xk3bhwfffQR48aN4/zzz2fixImMGzdu31FGY7FsH5pky+jRo72oqOiw9l057TSoqeSo/3t4+4tIcFasWMHRRx8ddIwWob7flZktdvdDTgvR6o4cPBalf9WHlHQcFXQUEZFmq9UVh+LyGsZX/YKSY/856CgiIs1Wq7tbqV/ntsz+8RTyI+H+rL/YAAAKWklEQVRDdxYRaaVaXXEA6FGYH3QEEZFmrdWdVhIRkUNTcRARkTQqDiIikkbFQUQkQ5lO2Q1QVlbGU0891fhhskTFQUQkQyoOIiKyn9T1HNavX8/8+fM566yzmDBhAnfffTcACxcuZOzYsYwZM4bzzjuP8vJyLr30Ul544QUmTpxITU3Nfq85a9asfWs4zJgxY1/7888/z8SJEzn77LO54oorqK2tZdGiRZx11lmcffbZTJkyhdLS0qz8nK3yVlYROYL84fz0tuFfg5Ouh5pKmPWN9O3HXQbHXw4V2+HxK/ffds3BJ7BLXc+hrKyMH/7wh8yZM4f27dtz7bXXsmDBAqZNm8ajjz7K0KFDWbFiBbFYjMcee4ypU6fWe9QxatQo/vGPfxCNRhkzZgxXX301ZWVlTJ8+neeff542bdrw+uuvs2fPHm6++WaeeeYZOnfuzPLly6moqKBTp04Z/MIaRsVBROQwzZ8/n5KSEr785S8DUFFRwfLly/nGN77Bddddx3XXXcfkyZPp3LkzO3ceeFGioUOH8vjjj7Nw4UJKS0vZvHkzc+bM4eqrr6ZNmzYAjB8/ntmzZ3PuuefSuXNnAIYPz96U5SoOItKyHewv/dy2B9/ersshjxQOxsy45ppruPPOO9O2ffnLX+bFF19kypQp3H///WlrOaS64IIL+M53vsOdd97Jhg0bcPdA1nBIpWsOIiIZ2ruGwimnnMJLL72077z/qlWrmDNnDq+99hqFhYVcfPHFnHPOORQVFREKhdKuNUDiQnVNTQ1Tpkxh165dvPLKK7g75513HjNnzqSiogKAuXPncuqpp/Lss8+ybds2AN577z02bNiQlZ9RRw4iIhnau57DrFmzuP/++7nooouIx+MMGDCA6dOn8/TTT3PXXXfh7vTs2ZNHHnmEnJwc1q9fz8SJE3nppZfIzc0FoGPHjpxyyimMGzeOHj16cP3113P77bfz9NNPc8sttzB58mTi8ThHHXUUp5xyCjNmzODiiy+mtraWXr168eCDD2blZ2yV6zmISMul9RwaTus5iIhIo1JxEBGRNCoOIiKSRsVBRFqclnqttCl93t+RioOItCj5+fls375dBeIg3J3t27eTn3/4C5vpVlYRaVH69u1LcXExJSUlQUdp1vLz8+nbt+9h76/iICItSk5ODoMGDQo6xhEv66eVzKyHmT1vZgvN7GUz632AfneZWZGZLTGzi7KdS0REDqwprjlMB6a6+xjgDuCBuh3MbDLQJTkwYxxwo5l1bYJsIiJSj6wWBzPrSOJDfyGAuxcBHZLtqa4DpiX7VAG/A76ZzWwiInJg2b7mMAhYXadtTbJ9aUpbD3fflPL8I+Dkui9mZjcANySf7jazDz9Htq7Ats+xf7YoV2aUKzPKlZkjMdeAhnTKdnEwoL77zeq2WQP64O4PAQ81Qi7MrKgh84s0NeXKjHJlRrky05pzZfuaw1pgaJ22Icn2VCVm1jPl+TDSjzhERKSJZLU4uPsOoNLMTgAws1EkDoVyzWxGStc/Arcm++QCVwL/nc1sIiJyYE0xzuEmYKaZtQfKgWuAAhJHBwC4+5Nm9iUzW0jidNK97r49y7ka5fRUFihXZpQrM8qVmVabq8Wu5yAiItmjuZVERCSNioOIiKRpdXMrmVkP4BGgM7ATuKrOGItAmdkdQKm7/7YZZBlHYlR7eyAPuNvdXww2FZhZO+DfSNwJ1wHYClzj7jsDDZbCzMYDj7l7n2aQ5UPg05Smme7+56DypDKzEHAvcCoQA2a7+8+DTQVmFgbm1Gnu4e6Brk9qZv2B3wDtgLbAz9z9yWy8V6srDnw2ncdCMxtNYjqPywLOhJn1AZ4gcavvnQHH2SsMXObuu8ysE/AaEHhxAPKBP7j7uwBm9n9IDI5Mm5olCGbWAfgu+38gB6nU3ScGHeIAbieR71QAM2sWM+q5ewyYuPe5mfUF7gss0GfuA/7N3d80sy7AW2b2lGfh4nGrOq2UwXQeTc7dN7r7WJK39DYH7v66u+9KPi0D9phZfQMWm5S7b08pDDkkCuqKYFPt537gX4Fo0EGas+Rt61NI+dB190+CS3RQ3wf+M+gQJP5gK04+3g2syUZhgFZWHDj4dB5yAMlD/wdInI5oFre3mdmFZvYa8DGJ26LfCDgSAGb2TWCxu38UdJYUXc1sppnNMbO/mdnAoAMlDQLeA/6Pmb2S/Dor6FB1mVkBMMLd3wo6C3Ab8Aszuxn4G/CDbL1RaysODZ3OQ5LMrDswC5iXnL6kWXD3J919grv3B34PNIdrNP2B89z9waCz1HE3cKu7n0miyP8h4Dx7tQfOBra4+xnApcD9yVOYzcl1wMygQyRdBrxA4rrD3cDPkqekG11rKw5radh0HgKY2WASHyS3uvvTQec5EHf/OzAw6BzA14BhZjbXzOYCxyQfnxhkKHf/c3K2Atx9AdAmyDwpPgZWuPv/ALj7VhJHgEMCTZUieWH6AiArF30zzHI0MMTdf+fuVcnT4o8BX8/G+7Wq4nCg6TzcvSzYZM3WVODb7l58qI5Nycy6pJ5+MLMraAanldz9P9x9jLtPTF4A/iD5eHGQuZJ3ne19PAFYH2CcfZL/7zab2bkAyVkUTgA+z2zLje3rwFPJC9RB2wmMTJ7mwszygIuBd7LxZq1uhHTyroOZJA5py0ncArkl2FSfMbOrgfxmcivraj67+LXXde4e6KSIZtYGmAEcB1SQ+DD5gbtXBpmrLjNbkLzJIOgcM4CjgBxgE/D9JpiepkGSN4P8J9Av2XSPu78SYKT9mNlLwBR33x10FkhcawNuAWpI/HH/cLZuS251xUFERA6tVZ1WEhGRhlFxEBGRNCoOIiKSRsVBRETSqDiIiEgaFQeRJmJmU/fe0y/S3Kk4iIhIGhUHERFJo+IgUg9LuCc5k+k8MzvJzF4ws9+a2etm9j9mVpjs+91k2xtmdlmyLcfMpidnGn3VzL6VfOlRZvaymS0xs8vNLN/M/mpmC8ysyMwmBfZDi6RojYv9iDTE5UCeu59pZp2Bp4GTgPHu/raZXQrcaWbPkljFbAKJ/0//a2bvAF8B1rn7D5OTt00kMenjJGAyiZX13iYx/cdH7n5JclqQQFcaE9lL02eI1MPM/gb0JjGHDUAPoMrdj0/pMx9YQGJitvnJtq8DA0jM0DoxdcI2M5tKohDMSj5/DbgS+CvwXPJ13s/yjybSIDpyEKmfA5e4+4a9DclpuPc+jiT77P1K3c9JrB1S36p52+v03QKMI3FUcqOZ7XL32xrpZxA5bLrmIFK/J0msAw2AmX0b6J+cVhrgRuB54CkSH+qWXPby+mT7cySWlsTMQge5hfULQKfkOgs/JXF6SiRwOnIQqYe7/8XMBpvZ60AMeILEmtB/SV6IXgl8z91rkouwvArkA//h7iuS053flzx1FOHA6w9HgUfMrB2JKbW/n92fTKRhdM1BpIHMbG5yER+RI55OK4mISBodOYiISBodOYiISBoVBxERSaPiICIiaVQcREQkjYqDiIikUXEQEZE0/x9C/ZGnSipIIQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mnist_test(784,10,10000,50,0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 학습률을 올려보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T06:08:38.003344Z",
     "start_time": "2019-08-04T06:08:27.078042Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1epoch => train acc, test acc | 10.218333%, 10.100000%\n",
      "  2epoch => train acc, test acc | 81.871667%, 82.320000%\n",
      "  3epoch => train acc, test acc | 88.335000%, 87.930000%\n",
      "  4epoch => train acc, test acc | 84.733333%, 85.080000%\n",
      "  5epoch => train acc, test acc | 86.261667%, 86.230000%\n",
      "  6epoch => train acc, test acc | 87.463333%, 87.640000%\n",
      "  7epoch => train acc, test acc | 87.490000%, 86.990000%\n",
      "  8epoch => train acc, test acc | 84.063333%, 83.290000%\n",
      "  9epoch => train acc, test acc | 80.165000%, 79.740000%\n",
      " 10epoch => train acc, test acc | 89.321667%, 88.880000%\n",
      " 11epoch => train acc, test acc | 86.428333%, 85.570000%\n",
      " 12epoch => train acc, test acc | 88.171667%, 88.030000%\n",
      " 13epoch => train acc, test acc | 89.263333%, 88.940000%\n",
      " 14epoch => train acc, test acc | 89.435000%, 89.190000%\n",
      " 15epoch => train acc, test acc | 87.765000%, 87.080000%\n",
      " 16epoch => train acc, test acc | 88.206667%, 87.770000%\n",
      " 17epoch => train acc, test acc | 65.833333%, 65.380000%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAENCAYAAADkNanAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd4lFXax/Hvncmk9w4EqSJF0NUoCEYiIIhdVBB7wbK+q2sv66q4RVkRXdRd++ruim2xYAFEQZrUqCwKKCACKYQU0kifmfP+MQMbkoAMZOaZMPfnunKZOXNmnt9EZu55yjlHjDEopZRSzYVYHUAppVTg0eKglFKqFS0OSimlWtHioJRSqhUtDkoppVrR4qCUUqoVLQ5KKaVa8VtxEJEHROTmA9z/iIjkisg3InKRv3IppZRqzefFQUS6iMgK4M4D9BkLJBtjsoChwC0ikuLrbEoppdrm8+JgjCkwxgwB7jlAt0nAFE//euBlYKKvsymllGpbqNUBPNKNMYXNbm8CTmnZSURuBG4EiI6OPrFv375+iqeUUkeGr7/+utQYk/pL/QKlOEgbba0mfTLGvAS8BJCVlWVyc3N9nUsppY4oIrLtYPoFytVKJSKS0ex2H2CzVWGUUirYWVIcRCRNRKY3a/onnnMSIhIGXAW8ZUU2pZRS1u05xOLeOwDAGPMB0CAiK4HFwPPGmDKLsimlVNDz2zkHY8zrzX7/CRjb4v7f+SuLUkqpAwuUcw5KKaUCiBYHpZRSrWhxUEop1YoWB6WUUq1ocVBKKdWKFgellFKtaHFQSinVihYHpZRSrWhxUEop1YoWB6WUUq1ocVBKKdWKFgellFKtaHFQSinVihYHpZRSrWhxUEop1YoWB6WUUq34bbEfpZT3KmubsIcKUWFH7lvV4XRR2+TE4TQ4nC6aXJ7/Og0OlwuH09DkdOFwef7raW9y/u/2nvsdThfJMeH07xTHUUlRhISI1S+vwzpy/8Up1cH9WFTNLS/OodJhI6tPd84a1ImRfdOIDu9Yb1uH08WOynryy+vIL68lv7yOgl27qSrbgaOiEFtNES5jmO86EYBzQ5bRVUoIlybCaCIMB8UmgRed5wLwYOgb9JQdRNNImDgIo4n1rm78znEDAH8KfZW5RPKdbQC16Vl0y+zCgM5xDOgcT++0GMJC9YDJwehY/8qUChI/l9bwu1c+4ENzHxGhDhZsOYk3NmRzX8ggsvukc/agTozom0ZshN3qqK0+/HeWllFTso3G8gKo3kFE3U7CaeApx3gAnrb/nVttywjF5X4CO1RGduXD7KsItQlnrHyatF2rAXCF2HHawtmdOIDsMwYTahP6LHmLiKomCI0AWziEhtMzdQCnnzaCUBEi336OyB2LsJlPcBULG3d25V8rz+Ae50jCbCEcnR6zt1gM6BxHv05xHa7g+oMYY6zOcEiysrJMbm6u1THazdzvi3j043V0SYjkiiHdGDswg/BQm9WxlAXyy2sZ/8JynE2NzB24gES7E/P9TKS+kkp7GpP4PaurUwgLDeG0o1M5e1AGI/ulE+fjQtHgcPLDjmrWb99J3eYlSNkm7DU7iW4sIYUKrmq6H0MIU+wvc6nty30eWx+WxNeXrCIzKYouP79PaOXPENsJ4jpDbAZEpUBiN3fnxloIsYEtDOQQDws11UHB17BtGWbbMso6n86y1EvYsi2PCWuvY6XzGBY3HM0q048CUuiRHEO/znH7FI2UmPDWT+t0UdPgYHeDg5oGJzWNDmoaHJ42Z7P79m2raXQwom8a1w7rcWivpx2JyNfGmKxf7KfFwVr1TU6e+GQNCV8/Q2HCSawwA6gsK2JYVB5HnTiWiaf0omtSlNUxD48xIIKrqZHaRU/T5BJcEQk4w5NwRiTSGJuJM7YLxtMVDHv+We5pM83bPLcBeqXGEGE/corozqp67nj+fXbU2XjuxjMZ0DnefUdTPfw4GzZ8jOuCF/mmYDf5C1/lu7xy3qo5EYctiuyjUzhrYCdG9U8nPvLwCkV9k5Mfiqr5rqCS7/Mq2JW3nq9Ko6h1hXJX6LvcGvohAA6xUxOWQlNUOpvPeJ1O6Wl0rlqLvTrf/aG/58M/LPow/zLtZNcW+OxBzLZlSH0FANVh6bySfDfvVfSmqLwaBzZASI8LJzEqzFMAnOxucNDocLV4QkMkDdQRAcAg+YneUkByyG7SQ3eTFrIbbHYeD7mBitomVj040vK9PS0OHcCWkt38/V9vcHPldHqHFOIcdicy8mF+mvssR696iEoTxReuE8nPGMVxOePI7tcVW4CfYGsqXEvFz/+ltnA9lPxIVNVPbAw9hkdt/8f2XTWsCJlEgtTs85h/Oc7gYce1hOLgm/CbqDQxlBNDhYmhnFhmOwfzmeskwmjizJBVVBBDuYmlnFiqwtM5//iuTDipK8d2ibfoVbePst0N3PbCRzxZfR+x6T2I+fUXB/7m/K/zYctCnKFRfBd/Oi9VDWF2dU/sthCyj05l7LEZjO6fQXzUgT+M6pucbNhRxfcFlXxXUMl3BVUU7izmZL5neMh/OT10LV0o4Z2+zxA3YDTHR5eR4dyBdP4VRCUf+rd7K7lcULIBti1z/5z+IKT0pnbVv7HPf5iCuONZE9KfYhNPsuzm24yLiY6wM7hsFn13LSCqqZyIpgrCGssxIaH8eN1GYsJDSZ9/K+HrZ7q3ERLq/vsk9uDbM95m3N+X8tj5/Zl4Si9LX7oWhwD38aof2P3p75kon1MXnUnkhc9A75HuO5vqYcuX1P73fUJ+nEOEs5oqE8klES9z3pB+TMjKJCU2wrLstZWlFG9ZS3X+OlzFP1JR7+Ll8KvZtquGl3ffRt+QPBwmhG0mnZ+lCxsiT+S/ncbTLTmK7vE2YsNCCGuqILypgvCmShrCU9gdfzQ2Rx39NkwnrLFin59tPS9ja59riKwt5PQ5I/bJUmrvxAP1V/F503EM7BLPhJO6ct7xnX1+iKW9VdY18ZsXP+XP5ffQObye0OtmQ8bAAz/IGMhbBWvegO8/gMZqSvpezkuxv2H2d0UUVNRhtwnDeqdw1rGdGD0gnQi7jfV7CkG+uxhsKt6N0+UikgYio+MYnVbBn4tuwmacuOzRSM/hSO9R0O9ciEnzzx/EKtuWwTf/hm1fQcW2/7Xftw0iE2DZs7DhY/eHflSS578pcMpvICQEKraDs8ndHhG/t3CamlJ2TDuVjyMv4KZ7plj04ty0OASo2kYHD89aR+iaf/GY/R/UnnADMWc+sv/dbkcjji2L+GnNEiZXnMXyLWU8F/YsmXF2Yk+4iJ5DL0Qi2v8bs6uhlh3bN1K07QfKdxYw2z6S7WW1TCr+M2eapXv7NRg730pfHk+ZwlHJ0Qyx/0RycjLJXftyVFoiabHhSHt9s3Q2QflWqC2D2l2wuwjW/ofdpz3Me8Wd+HL5KnaWlvJzaA/OHtiZiSd35cRuie23fR/Z3eDg/16ex0Mld9PdXkHoNR9B5i++d/fVWAMbPoGkHtD1ZEzpJna/fzsLIs7gmR3H8FO5i9AQwQBOl/s93yOqgYsTNzM85L/0rl6J45hzib7gKcS4YOHj0GM4dB0MoWHt/6I7gsoCaKj2FIBk94f/YSibNpidlXVw0xL6W7iXq8UhAG38aTMvzJzDBxU9uTWnJ7cNbCK08yCvnmNz8W5KZt5F7+K5pFJBI3Z2pg4lJedmIgecdfBP5HJB9Q6o2EbVjk18lzSGH4rrSP/+RYYWv0OSa9ferk4jjAh/i4zkBM6zLadHWAWh6ccQ3/VYMrodQ3y0dXsxzZmP70C+/gc/Rx/Hs9Wn81HjCXRLjefSk45i3AldSG7jBKPV6pucXPPaKi7Pe5Szwr7FduX70H3Y4T/xloUw61ao3I4Jj2NXj3OYGzqSHTHHcmxmAsO/+S0RP89zF4KIBOg1Ao69CPqdc/jbVm2q/eoFoj6/jxf7/oObLr3IshxaHAKIcblY8f4z9P/uCRoljM0Tv+KUY7oc1nPWNjSyfOFsqr99n5PqlvKBGcGOX/2WK7PS6Fs8F/qeDTY7lG9zf9vuOZwaiaZ4+QySVz9FVG0hoaZx7/Od2jCdfJPKpVG5nBnxPa6EbkSm9SI5sw+ZPfoSldQ58I8t15XDt2/AqpehYhu14Wm8Yz+PR0tHYLcJo/tnMOGkrpzaOyUgBkc1OJzc9O+vWbSxhL9fcBRj06vapzDs4XLBtqXw7QxYPwsiE+GOde5vwIumgssBvUdBlxPcVwcp36qroPGJo/nQDOe8B9+x7EIKLQ4BorpwI4X/vpFj6r7lh/CBpF/2Aondjm235zfG8N+8Ct5atokPvy/jFNfXvB42tVW//wv/M59W9iA7ZC2X2hawIySDxtijCEvuQXyXo+nc/RiO6ZzY5uV7HY7LCZu/gJUvQnIvNp74MO+s2s76b5awvC6TLglRTDipK5dkZdIpPtKSiA6nizveWE63ja9x1LkPMH6Ij09S1lfBD59C//MhrINf/daBFf/zaiK2fMbic5dwTtbRlmTQ4hAA1q//jp7vjqDJ2PjmmDvInnA3ITbffVsor2lkZm4eK5cvpG/1choJpYB0XAndiOjUj56dUjgmI5a+GXFkJkYGxLdnn3O53N+Ut34Fr59FRcIA3pGxPLXjWJokjOF9Uplw0lGM7JeG3eafkbMul+Get1dx/oa7yLZ9j1z9EfQ4zS/bVtZyFa7l969/yrbk05hxUzvuJXpBi4OFXLtLefWbKv4y9wdui5pHzribGNS/n/+27zJ8m1dBpN1Gr7RoHUwH7hOLa99xH3Iq+QFnRCK5yefxQPFItlSHkhITzsSTu3LlKd1I8+GVYMYYfv/+GoavuYvRtq/hvOfghCt9tj0VeP725WamfvYjC+/OoXuK/8d/aHGwQmMtdV88Rsjqlzm7/g/06n8iT1x03C9eZ678yBjYugRWvQR5q3DcuoZFW6qZu+xrZm52YQ+xcf7xnbk+uwd9M+LaedOGP338PcetvofzbMth7FQYfGO7bkMFvuKdhbz17INEnjiRGy8c7ffta3Hwty2LqP/gViKqt/Ef1+k4R/2BCdkDA/4yyqDWWOO+hNjlhGd+RROhfBF9No9uP46ipkiyj07h+lN7MLxParv8f5w270c+/XIxs6MmE55zN5J9Rzu8CNXh7C7G8WRf3go5m0sf/KffDmfucbDFQacnPFzG4Prot/Cv8yiqqueuiD/S/+Z/culpg7QwBLo9Y0uMgZwHsMckM7bgGZaH/x9zu72JY8c6rnltNaOfXszbq7ZT3+Q85E39feFmnl2wmZOzBhP+21wtDMEsJo2yLiMZ61zIwvX5VqfZLy0Oh2lndQOzNtbzvONcnu/7bx6945b/zYejOgZbKBw/ESZ9DjcvRY6/nL7lC/nXuDSennAciVLLn95fybApC3j6842U7m7w6ulfX7oF+WIy07su4s8XDkTiOvnohaiOIuW0SaRIFZsW/8fqKPulh5UOQ4PDSc7UhVTUNvHHC47l4hMzLc2j2lHDbs+U0KGY+X/EtfzvLI08nSdKh7HJ1pMLj+/C9dk96JMee8CneXd1HvkfTuZO+0xcJ15HyDlPBf54EeV7LidVU/ryTX0n+t41j4x4/w0k1cNKflD8wwrurn2aqWckamE40oTHuPcoAOl3LrZjxzG8bgGfhv+O+XGPYta+xeinF3P1P1axZFMJbX3JmrWmgE2zHudO+0ycgyYScvY0LQzKLcSG64RrqDHhvJe71eo0bfJ5cRCRdBGZIyIrReRzEem8n36PichST7/nRSTgV9+o3fYNF9mW0DXJmoFUyk86Hw8X/A3u2gBn/oWuUS7+1DePu0f3Yf2OKh74x6ec+dclvJubR4PDfV5i3roicmc+yYOhM3D0PR/b+c8d9tw86siScOaDzDjqj7z9dSEuV+AdwfHHB/A0YLIxZqWIZAFPApc17yAiZwIpxphTPbdfBs4H3vNDvkPWtGsbDhNCamfrF/BQfhCZCENuhsE3EdZQzW8i4rixv4OwFy7iu90DePn9HJ6ek82Y447izZXbuT0xGkenMwm9+JW9eyFKNTfhpK489c5clm8ewLA+6VbH2YdPv8qISAKQbIxZCWCMyQXiPe3N2YDmp+1LgQJfZmsPUpnHTpJIT4ixOoryJxGIcI+BCItLhzP+wLFxtTwT9jc+MzcRtXI6R6dGcfktjxB62dvBO6up+kVjo35kUfidrFn8odVRWvH1fm4PYHOLti2e9uZmA7Ei8rSITAeKjTErWj6ZiNwoIrkikltSUuKbxF4IrymkxJYW8AvwKB+KSoJhv0Vu/Qau/IC4Y07j7ujZzDzX7h78qOcY1AGE9RxGTWgCPbe/R3lN4y8/wI98XRwEaOtgWsu2IYAdmA78EcgUkWtaPciYl4wxWcaYrNTU1PbO6rVaZwhlEd2sjqECQUiIe9rrCW8QcvtaIrt5uR6DCk6h4dT3v4SRksvsld9ZnWYfvi4OW4HeLdp6edqbexy41xiz1RhTCtwNBPy8AjfyMLO7P2B1DBVoopL0UJI6aMmnTiJMnOxe9e82r3qzik+LgzFmF1ArIicAiMgg3OcTwjyHj/aowb33sMd44AdfZjtcjQ4XO6vryUzUK5WUUochrS8liceTXbuAb/MqrE6zlz8uobgdeFVEYoBK4FogFujTrM/NwLMi8gfch5w2Arf5Idsh27VxBTNC/0RV6GPs+1KUUso7URf/jetf3MBpq/I44ahEq+MAfigOxph8YEyL5p3A2GZ98oALfJ2lPVXnr2OobT3fxPl/yl2l1JElusuxZA9y8vHaQh46tz8x4dZf+qyjcg5Rfdk2AFIye1qcRCl1JLi+WzGvmEf5LPdHq6MAWhwOXcV2Skw8GclJVidRSh0B+nRKYqhtPcXLZ1gdBdDicMjCdhdQHJJGWKj+CZVSh0+6nEBZTB+GVc3mh6Iqq+NocThUhSaJLZEDrI6hlDpSiBAx+BoGhfzMokVfWJ1Gi8Ohephf83nX262OoZQ6gkSfOJEmsRO/4e3DWlyqPWhxOAROl2FHhY5xUEq1s6gkCo+9hZWNPflsXZGlUbQ4HIJdm1fyeejtHMdGq6MopY4wXS98lNXxo3lndZ6lObQ4HILKwk30CNlJUkLLyWWVUurwhIQIVx6fQOzPc9hWVmNdDsu23IHVl2wFILlLL2uDKKWOSBNDvuDFsL8yb8lXlmXQ4nAIXOV5VJkoOqVnWB1FKXUEihtyFU5CsK99E4fTZUkGLQ6HwF6dR5GkEhlmszqKUupIFJtBWeccznYuYNGGQksiaHE4BBukJ99GnmJ1DKXUESwpexKpUsmGxTMt2b4Wh0PwnLmEhV0CfrkJpVQHFtpnDNX2VMJ3rGZnVb3ft6/FwUvG5aSwolbHOCilfMsWyq6rF/HnpsuY+XW+3zevxcFLFVvXsCbkKk5uyrU6ilLqCNctswtDeiYxc9VWXC7/rhKnxcFL5YVbiJAm4pL1SiWllO89HD+HV2p+w4qfSvy6XS0OXqor+RmAhE46xkEp5Xu9jzmWXiE7+GbRLL9uV4uDlxy7tlNv7GR07mp1FKVUEAgbcB61tjh6bH+P8ppGv21Xi4OXQqvz2SEpxEeFWR1FKRUM7BHU9buEUbKaOau+99tmtTh4Kdd2PPMjWi6JrZRSvpOcfT3h4qBq5QyM8c+JaS0OXnrLOYIVna6wOoZSKpikD2B1/weYUTmA/+ZX+mWTWhy84XJSU76TzAQd46CU8q++591FaWhn3lm93S/b0+LgheodG1kikxhev8DqKEqpIBMbYefOnnlErXmNmgaHz7enxcELZQWbAYhI7W5tEKVUULowbBV3ygzmfrPJ59vS4uCFmp1bAYjP6GltEKVUUErOnkS0NFC07E2fbyvU51s4gjTt2obDhJCe2cPqKEqpICRdT6akyyhGdOuJMQYR8dm2tDh4IaQqj50k0Tk2yuooSqlgJELqDe+R6odNaXHwwmL7aTRE9OIuH1ZrpZQKBFocvPBZ03Ekpp9kdQyllPI5PSF9sFxOYsvX0TPOv9PmKqWUFbQ4HKS6sjzedN3L8MZFVkdRSimf0+JwkErz3WMcwlO6WxtEKaX8QIvDQare6V7HIS5DL2NVSh35tDgcpMayrQCkdOltbRCllPIDnxcHEUkXkTkislJEPheRzvvpd4aILBGR+SIyW0SifZ3NKxV5lJo40hITrE6ilFI+549LWacBk40xK0UkC3gSuKx5BxHpATwInGuMqRCRDMDlh2wHbV7UWKrDj+GPITrGQSl15PNpcRCRBCDZGLMSwBiTKyLxIpJgjKlo1vUe4KE9bcaYIl/mOhQr6o4iIlXPNyilgoOvDyv1ADa3aNviaW9uINAgIrNEZLGIPCYirbKJyI0ikisiuSUlJT6K3AZj6LFrMf2id/tvm0opZSFfFwcB2ho11rItCrgcuBIY7rl9XasHGfOSMSbLGJOVmuqP2UXcGiqLmOaYwqmO5X7bplJKWcnXxWEr0PLynl6e9ua2A08YY6qMe4HUN4DjfZztoJUV/ASAPambxUmUUso/Dro4iEimt09ujNkF1IrICZ7nGASUAmEiMr1Z178Bj4uIzXP7fGClt9vzlaqiLQDEpus6Dkqp4ODNCelpIhIPzAVmGmPyD/JxtwOvikgMUAlcC8QCffZ0MMZ8ISK9gcUi4gCW4d57CAgNJVsBSM7sZW0QpZTyk4MuDsaYCSISBowAficiacBXuAtF3gEelw+MadG8Exjbot8LwAsHm8efXBXbqTaRpKelWx1FKaX8wqtLWY0xjSLyJRAPXIT7KqPjRWSXMeYOXwQMBJ/EXExh2CCet+mAcqVUcDjo4iAiOcAVQCdgJnCdMWa3575lPkkXIL6viceVnGV1DKWU8htv9hzGAFOMMS3HLQCc3U55AlJW6Ye4upxsdQyllPIbb46TfA903XNDRMaIyHgAY0x5ewcLFI6acu5peoEhrm+sjqKUUn7jTXG4DljY7PY84NZ2TROAygrdl7HadIyDUiqIeHWG1TNArfnvznZPFGAqd7gHwEWn6bxKSqng4U1x+FZEfiNuISJyJ3DEH2upK3Ev8pPYWcc4KKWChzfF4X7ccx4twn14yY57NtUjmnNXHg3GTkbnrr/cWSmljhAHXRyMMQ5jzBPGmNM8P3/BPdbhiPZh/BVcFvokEWF2q6MopZTfeDPOoR8wGcjEPdsquPck3m3/WIHj52rBkXS01TGUUsqvvDms9AruVdzW4Z764t/AP30RKpCcvvN1Tg/70eoYSinlV94Uh1pjzGrgO6CbMeZ54DzfxAoMrsY6rmt8k1+xweooSinlV96MkM4TkaOBj4C/isiHBNg6z+2tvOhnkoGQxKOsjqKUUn7lzZ7DJGCrMWYb8DiQAFzqk1QBotyzyE9UandrgyillJ95s+fwmTHmDABjzCpglW8iBY6aYvfo6IROusiPUiq4eLPnUC0iI3yWJAA1lhfiNEJaFx0drZQKLt7sOaQAX4jIFqDE02aMMUPbP1Zg+CjhSm6TISyPjrI6ilJK+ZU3K8Gd5ssggaigoo7ExBSrYyillN95MwhuObBn4r0UoAfwuTHmLF8ECwTnFk6nOOF4INvqKEop5Vfe7Dmc0vy2iIwDRrZ7ogBhnE2c2/ApyyTR6ihKKeV3h7wosjHmfdxrSB+RqkryCBUXomMclFJByJvDSn2a3wQGA/XtnihAlOVvJh6ITNFFfpRSwcebq5UewH3OQTz//RmY6ItQgWB3sXsdh7gMXcdBKRV8vCkOXwA7jDELwL2GNO5zDkfkrKxVVVVUm0hSMrU4KKWCj7drSH/Z7PYRvYb0guizOdn1Gonx8VZHUUopv9M1pPejoKKWzMQoROSXOyul1BFG15Dej0vz/sjVts+sjqGUUpbQNaTbYgxDGpfRLbTM6iRKKWUJb4pDDrB8zxrSuGdlPSKn1KgpLyKSRojXMQ5KqeDk7Z5D82m6lwGPtG+cwFBasBmAMB3joJQKUt6ekG5o6/cjTXWRex2H2HSdqlspFZy8KQ5bReTcPTc8cyttaf9I1ivb3cQmVxdSuvS2OopSSlnCm0FwvwWmish9uEdJrwXe80kqiy2LGMZrzi78kJJmdRSllLKEN3sOsUA1sN3zkwhM90UoqxWU19E5IYKQEB3joJQKTt4UhxlArucxjwFbgb/4IJPlrt56L3eGvGV1DKWUsow3xcEYY/6Du0A0GmPuBy7/pQeJSLqIzBGRlSLyuYh0PkDfcBH5TkTO9CJXuzumcT0p9iP2fLtSSv0ib4pDmYikA7OB+0RkOAd3zmIaMNkYMxj3zK5PHqDvn3DvkVimfnc5cdTgiutqZQyllLKUN8XhJqDJGLMe+Ai4APdkfPslIglAsjFmJYAxJheI97S37Dsa2AV87UWmdlea/xMA9iQd46CUCl4HXRyMMRXGmF2e3z80xtxhjNn4Cw/rAWxu0bbF076XiCQDk4AnDvRkInKjiOSKSG5JScnBRvdK5Q533Ggd46CUCmKHvEzoQdqzMFBLLdueAu4zxhxwlldjzEvGmCxjTFZqamp7ZdzHzvpQFjsHktjlaJ88v1JKdQTejHM4FFuBliPJetHsvILnBHUW8JpneuzuwDgROdEY82cf52vlW9tA/u78HT920nMOSqng5dPiYIzZJSK1InKCMeYbERkElAJhIjLdGPNbY0whMGDPY0RkMrDCGDPXl9n2p2BXLRlxEYTafL1TpZRSgcvXew4AtwOvikgMUAlci3tAXR8/bNtrk366lQtsqcAIq6MopZRlfF4cjDH5wJgWzTuBsfvpP9nXmQ4krSmfqig9pKSUCm567KQZR0MtKZTjjMu0OopSSllKi0MzpQXuSWZtibrIj1IquGlxaGZXoXsAXFRad2uDKKWUxbQ4NFPYGMXbjhziM/tZHUUppSylxaGZ9aY79ztuJL2Ljo5WSgU3LQ7NlJSWkRZjJ8JuszqKUkpZyh/jHDqMiT/dw3gBGG11FKWUspTuOTST2FREXYQuDaqUUlocPFwOBymuMppiu1gdRSmlLKfFwWPXzm3YxYktQcc4KKWUFgeP0gL3GIeI1O7WBlFKqQCgxcEj3xHP000XEdt1kNUVuFmWAAARvklEQVRRlFLKclocPDY1pjDdeRHpXXtaHUUppSynxcGjpngLPSJriQnXq3uVUko/CT3GbJnCeSGVwCVWR1FKKcvpnoNHfGMR1REZVsdQSqmAoMUBMC4Xqc5iGqN1HQellAItDgBUlu4gUhohQVeAU0op0OIA/G+MQ3hKN4uTKKVUYNDiAGx3JnFP041EdT/J6ihKKRUQtDgAW+qi+I8zh4xMXcdBKaVAL2UFwFG4lhPDioiPtFsdRSmlAoIWB2DIthc5MzQfkZutjqKUUgFBDysBsQ07qAzXMQ5KKbWHFgcg1VlMfZSu46CUUnsEfXGortxFHDWYeB3joJRSewR9cSjN3wSAPVkX+VFKqT2Cvjhsc6ZwReMDhPUebnUUpZQKGEFfHLbX2FjqGkh6Z91zUEqpPYK+OMj25Yy2f0tKdLjVUZRSKmAEfXHol/cOD4e+QUiIWB1FKaUCRtAXh5j6HVSE6RgHpZRqLuiLQ7JjJ3VRna2OoZRSASWoi0N9XS2plOOM1UV+lFKqOZ8XBxFJF5E5IrJSRD4XkVZf00UkQUT+JiJfishyEZkmIj4/CbAz372OQ2iSXqmklFLN+WPPYRow2RgzGHgAeLKNPtHAS8aY040xpwCxwFhfB9vuTGZkw1Sk71m+3pRSSnUoPp2VVUQSgGRjzEoAY0yuiMSLSIIxpmJPP2NMAVDQ7KHbgDBfZgPIr3Lwk+lCRkYnX29KKaU6FF/vOfQANrdo2+Jpb5OIZAOnAJ+0cd+NIpIrIrklJSWHHS705wVcHfo56bE6xkEppZrzdXEQwLTR3qpN3O4ELgIuNsY4Wj3ImJeMMVnGmKzU1NTDDtetYDa3hH5EqC2oz8srpVQrvv5U3Ar0btHWy9Pe0j+AbcaY240x9T7OBUBU/Q7K7en+2JRSSnUoPi0OxphdQK2InAAgIoOAUiBMRKbv6Sci5wG5xpj3fJmnpaTGImoidYyDUkq15I9lQm8HXhWRGKASuBb31Uh9mvU5FcgRkUuatc00xjznq1BNTU2kmjLydIyDUh1KU1MT+fn51Nf75QBDhxUREUFmZiZ2u/2QHu/z4mCMyQfGtGjeSbNLVY0x9/o6R0slO7bTWZzYEnWRH6U6kvz8fGJjY+nevTt+GA7VIRljKCsrIz8/nx499nv9zwEF7ZnYbU0J9K//B03Hjrc6ilLKC/X19SQnJ2thOAARITk5+bD2roK2OBRU1FFLBJ1Skq2OopTykhaGX3a4f6OgLQ7hm2dzf+ibdIr3+Vg7pZTqcIK2OKQWLWZC6GLCD/FkjVIqeL3++ute9b///vt9E8SHgrY4RNYWUhaaZnUMpVQH5G1xmDJlim+C+FDQFoeExp3sjtAxDkop74wbN441a9aQk5PD4sWLGTduHCNHjuSKK66gvLyc8ePHM2LECE4//XQKCwsBGDJkCFu3buWhhx7i0ksvJTs7m/PPP5/a2tpWz3/33XczatQosrKyWLx48d72KVOmcPrppzNixAimTp0KuIvU8OHDGTVqFHfccUe7vk5/jHMIOC6nizRXMTtjsq2OopQ6DI9+vI71hVXt+pz9O8fxyLkD9nv/+++/T05ODgsXLmTr1q188sknrFmzhv79+1NaWsr06dPp1KkT7777Ln/961954okn9j72lVde4auvvqJnz55MmTKF119/nVtuuWWf57/66qt58sknyc/PZ8KECXz11Ve88847FBcX8+WXXwIwb948VqxYwezZs1mwYAE2m43PPvusXf8OQVkcSkpLCMeOJOgYB6XU4Tn11FPp378/ACkpKaxbt4433niD1atXY7PZ9ul74YUX0rNnT8C9N9HWB3paWhovvvgia9eupaioCID33nuPp556am+f0aNHc++993Lbbbft3caYMS2Hkx2eoCwO+fV2Lmp4mdeOP9HqKEqpw3Cgb/j+kpb2v3OXH3zwAbNmzeK+++5jzJgxPPbYY/vtC+7Bas1t3bqVG264gT/84Q9cddVVLFmypM1++2trT0F5ziG/vA6AzMRoi5MopToih6PVpNEALFq0iKuvvpp+/foxY8YMrz/A16xZw9ChQznllFOYP38+ZWVlgPs8R/PDU7Nnz+bCCy/k6aef3pvl008/PcRX07agLA4RP87iWfszdIkNypevlDpM6enpZGdns3379n3af/3rX/P73/+e0047jZ49e7J69WoWLlx40M87ZswYvv32W4YPH87ChQsZOHAg06ZNY+LEiWRkZJCdnc3QoUPZtGkTQ4cO5eyzzyYnJ4ehQ4fu3ctoL+LrXRNfycrKMrm5uYf02GXPXc9xpZ8Q/UgR6EhLpTqUDRs20K9fP6tjdAht/a1E5GtjTNYvPTYovzqH1xRSakvXwqCUUvsRlMUhvmEHVeEZVsdQSqmAFXTFwRhDirOYhuguVkdRSqmAFXSXsu6qrKbYJNGUeLTVUZRSKmAFXXGIiopm2/gv6JsRa3UUpZQKWEFXHCLDbJx5rJ5vUEqpAwm6cw5KKaV+mRYHpZTykrdTdgNUVFTw4Ycftn8YH9HioJRSXtLioJRSah/N13PYvn07S5cuZdSoUQwfPpxHH30UgJUrVzJkyBAGDx7MWWedRWVlJZdeeilz584lJyeHxsbGfZ5zxowZe9dwmD59+t72OXPmkJOTwxlnnMGVV15JU1MTq1evZtSoUZxxxhmMGzeO8vJyn7zOoDshrZQ6wrx2duu2ARfAyTdAYy3MuKT1/cdfBr+6HGrK4N2r9r3v2gNPYNd8PYeKigruuusu5s+fT0xMDNdffz0rVqxgypQpvPHGG/Tu3ZsNGzbgdDp5++23mTx5cpt7HYMGDeKLL77A4XAwePBgrrnmGioqKpg2bRpz5swhMjKSJUuWUFdXxx133MFHH31EUlIS69ato6amhsTERC/+YAdHi4NSSh2ipUuXUlJSwjnnnANATU0N69at45JLLmHSpElMmjSJsWPHkpSURFXV/hcl6t27N++++y4rV66kvLycoqIi5s+fzzXXXENkZCQA2dnZzJ49mzPPPJOkpCQABgzw3ZTlWhyUUh3bgb7ph0Ud+P7o5F/cUzgQEeHaa6/loYceanXfOeecw2effca4ceOYOnVqq7UcmjvvvPP49a9/zUMPPUReXh7GGEvWcGhOzzkopZSX9qyhMGzYMObNm7f3uP/GjRuZP38+ixYtIi4ujvHjxzN69Ghyc3MJCQlpda4B3CeqGxsbGTduHNXV1SxYsABjDGeddRavvvoqNTU1ACxcuJBTTz2VTz75hNLSUgDWrl1LXl6eT16j7jkopZSX9qznMGPGDKZOncpFF12Ey+WiW7duTJs2jVmzZvHII49gjCEjI4PXX38du93O9u3bycnJYd68eYSFhQGQkJDAsGHDGDp0KOnp6dxwww3cf//9zJo1i7vvvpuxY8ficrno27cvw4YNY/r06YwfP56mpiY6derEiy++6JPXGJTrOSilOi5dz+Hg6XoOSiml2pUWB6WUUq1ocVBKKdWKFgelVIfTUc+V+tPh/o20OCilOpSIiAjKysq0QByAMYaysjIiIiIO+Tn0UlalVIeSmZlJfn4+JSUlVkcJaBEREWRmZh7y47U4KKU6FLvdTo8ePayOccTz+WElEUkXkTkislJEPheRzvvp94iI5IrINyJyka9zKaWU2j9/nHOYBkw2xgwGHgCebNlBRMYCyZ6BGUOBW0QkxQ/ZlFJKtcGnxUFEEnB/6K8EMMbkAvGe9uYmAVM8feqBl4GJvsymlFJq/3x9zqEHsLlF2xZP+7fN2tKNMYXNbm8CTmn5ZCJyI3Cj5+ZuEfnxMLKlAKWH8Xhf0Vze0Vze0VzeORJzdTuYTr4uDgK0db1ZyzY5iD4YY14CXmqHXIhI7sHML+Jvmss7mss7mss7wZzL1+cctgK9W7T18rQ3VyIiGc1u96H1HodSSik/8WlxMMbsAmpF5AQAERmEe1coTESmN+v6T+AeT58w4CrgLV9mU0optX/+GOdwO/CqiMQAlcC1QCzuvQMAjDEfiMhJIrIS9+Gkx4wxZT7O1S6Hp3xAc3lHc3lHc3knaHN12PUclFJK+Y7OraSUUqoVLQ5KKaVaCbricLDTefg501AR+VhEvhSRZSIyxupMLYlItogUWJ1jDxEJEZEpIrJURBaJyH1WZwIQkaNE5FMRWSgiq0TkwgDI9ICI3Nzs9jGev9kqEXnPcz4wEHKdIyJzPe+DRSJyYiDkatZ+mYissCKTZ/st/16RIvKyiCz2/FzVrhs0xgTVD/AGMNjzexbwZgBkygZiPb8nAmutztQiXzzwLpBrdZZmmX4H3Nfsdg+rM3lyvA0M9fyeDGzEc27PgixdgBVACXBzs/a5QE/P7+cDUwMk10gg3PN7d2BpIOTy3NcNeA1YEUD/H18EJja73b09txtUew5eTOfhV8aYJcaYas/NCqBORNoaGGiVqcCDgMPqILD3cudxwBN72owxP1uXaB82IN/z+25gi/G8c/3NGFNgjBmC5zJxABEZCGw3xmzx9JmFez4zS3N52ucbYxo8N4toewCt33OJSAjuf2v3tPlAC3J5jnh0N8a81azf1vbcblAVBw48nYflPP8InwReteoDpSURmQh8bYzZZHWWZnoAa4H/E5EFnp9RVofyuBd4WkTuAGYCd1qcp6XeQMtpZ3aJSKIVYdoiIuHAK8DjVmfxuA94zRgTSNNoDARWishkzyHMOXvGk7WXYCsOBzudh9+JSBowA1hs3NOEWE5EjgLOMsa8aHWWFmKAM4CdxpgRwKXA1AD5gLsM92Gb54FHgcdFpIu1kfYRsO8BABHphXsA7PPGmNkBkCcLSDXGzLU6SwsxwHhgkTEmB7gVeK09jzgEW3HYysFN5+FXItIT9/HMezy7+YHiAqCP55vJQqC/53dLThQ28xOwwRjzHwBjTDHwFe7/l5YRkX5AL2PMy8aYes9hy7eBi63M1cIWmg1A9Ug2xlRYEaY5ERkM/AW43hjzldV5PMYDg9t4D1hd8Dfj/iL5JYAxZjOQh/s8V7sIquJg9jOdRwC8MSYD1xlj8n+poz8ZY54xxgw2xuR4vp2s9/z+tcW5KoAiETkTwHO1zQm0Plzib1XAQBGJhb2HR8YDayxN1YwxZg3ugt8NwPM3XGJtqr0eAK40xpRbHWQPY8y9xphhbbwHLL1yzxjzX+AoEfkVgGduuuT2PPQVjMuEtjWdh9WGAu+02COc5Pk2oNp2G/CsiNzvuf37Zif1LWGMKRCRx4C5ItKI+8vXK8aYRVbmasNvgTc8hyAKgOsszoOIROF+H8xp8T44xxiz25pUAW8S8JzngpomoNXlt4dDp89QSinVSlAdVlJKKXVwtDgopZRqRYuDUkqpVrQ4KKWUakWLg1JKqVa0OCjlJ56pDs60OodSB0OLg1JKqVa0OCillGpFi4NSbRC3P4jIfM9CKid7FqJ5QUSWiMh/RCTO0/dWT9tXInKZp80uItM8M8Z+KSJXeJ56kLgXmfpGRC4XkQgReUdEVohIrgTgQk8qOAXj9BlKHYzLcS88M1JEkoBZwMlAtjFmlYhcCjwkIp8ApwLDcb+fPhaRNcC5wDZjzF0iYgNycE/6OAYYC4QDq4AaYJMxZoKIRAL9/PoqldoPnT5DqTaIyEygM9DoaUoH6o0xv2rWZynuFbo+NMYs9bRdjHvVsAuAHGOMs1n/ybgLwQzP7UXAVcA7wKee5/nOxy9NqYOiew5Ktc0AE4wxeXsaPFM27/k91NNnz0/zxxnc6ya0Nbd+WYu+O3FPOHcycIuIVBtj7m2n16DUIdNzDkq17QPcC6gAICLX4Z4iOcbTdAswB/gQ94e6eJYvvcHT/inu2U8RkZADXMJ6NJBojFkB/An34SmlLKd7Dkq1wRjzpoj0FJElgBN4D/ca2m96TkT/ANxmjGn0LPLzJRABPGOM2SAim4EnPIeOQoFn97MpB/C6iEQDdjwFRSmr6TkHpQ6SiCz0LPii1BFPDysppZRqRfcclFJKtaJ7DkoppVrR4qCUUqoVLQ5KKaVa0eKglFKqFS0OSimlWtHioJRSqpX/B1KVDA4QUmvRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mnist_test(784,10,10000,100,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 학습률을 낮춰보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-04T06:08:48.929433Z",
     "start_time": "2019-08-04T06:08:38.004990Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1epoch => train acc, test acc | 9.185000%, 8.180000%\n",
      "  2epoch => train acc, test acc | 36.785000%, 36.760000%\n",
      "  3epoch => train acc, test acc | 54.825000%, 54.240000%\n",
      "  4epoch => train acc, test acc | 77.046667%, 77.300000%\n",
      "  5epoch => train acc, test acc | 82.781667%, 83.210000%\n",
      "  6epoch => train acc, test acc | 85.405000%, 85.670000%\n",
      "  7epoch => train acc, test acc | 86.808333%, 87.200000%\n",
      "  8epoch => train acc, test acc | 87.921667%, 88.180000%\n",
      "  9epoch => train acc, test acc | 88.318333%, 88.780000%\n",
      " 10epoch => train acc, test acc | 88.891667%, 89.150000%\n",
      " 11epoch => train acc, test acc | 89.253333%, 89.350000%\n",
      " 12epoch => train acc, test acc | 89.486667%, 89.760000%\n",
      " 13epoch => train acc, test acc | 89.728333%, 90.030000%\n",
      " 14epoch => train acc, test acc | 89.930000%, 90.010000%\n",
      " 15epoch => train acc, test acc | 90.183333%, 90.200000%\n",
      " 16epoch => train acc, test acc | 90.330000%, 90.470000%\n",
      " 17epoch => train acc, test acc | 90.481667%, 90.610000%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAENCAYAAADkNanAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl41OW5//H3PZOVJEgSIKgoqwiouKEgiwRBEJe2YkVtrVvRU+1qW5cuVmt7rFVp5bT9tdp6tD3SqqcuWAVBUUC0IBEQRUQQESJb2CEh28z9+2NGTkwCJJDJN8l8Xtc1V2aeeWbmMwOZO893eR5zd0RERGoKBR1ARERaHhUHERGpQ8VBRETqUHEQEZE6VBxERKQOFQcREalDxUFEROpotuJgZj8ys28c4P47zazIzBaZ2SXNlUtEROpKeHEws6PNbD7w/QP0GQfku/tAYAhwk5l1THQ2ERGpX8KLg7t/6u6DgVsO0G0icG+8fznwZ+CKRGcTEZH6pQQdIK7A3dfXuL0SOKt2JzO7AbgBICsr6/S+ffs2UzwRkbbh7bff3uLunQ7Wr6UUB6unrc6kT+7+MPAwwMCBA72oqCjRuURE2hQz+6Qh/VrK0UolZtalxu0+wKqgwoiIJLtAioOZdTazyTWa/kp8n4SZpQFXAf8IIpuIiAQ3csghNjoAwN2fBSrMbAEwF/iju28NKJuISNJrtn0O7v5YjesfAeNq3f/j5soiIiIH1lL2OYiISAui4iAiInWoOIiISB0qDiIiUoeKg4iI1KHiICIidag4iIhIHSoOIiJSh4qDiIjUoeIgIiJ1qDiIiEgdKg4iIlKHioOIiNSh4iAiInW0lGVCRURajOjeXVRWVlBduZdoVQXVleVUp7SjMutIIlEn9eNX8apyolXleFU5Xl1OWfvebO90BpGqCo5d+iBUl2PVFVgkdinuPJI1R46Dip0MX/xDzCMQrcY8gkWrWFxwKe90vICsvev5yorvYB4h5NXxS4SpHW9gXs44uuxdRcaRfbntwpMT+hmoOIhI45Vtg73bobo8fqmASCX0LIzdv3Y+lKyItX92vxmc/cPY/W/9GV/3VuwLtroCj1ZTnZHHtjG/o6o6Ss6/7yOt5F2iDu5OFGNv5pEsP/VnVEWi9Hp3Elm7Vsfuc4i4sSPzGOZ1+xaVkSiFq35Nh71rsWgVFq0mFK3i47Q+PJr7HSqro/ys5GY6V28khSpSvJoUIszlNG7271NZHWV+ykTybM/n3vLTkeH8oOpGAFakX0u6VX3u/r9Vn8vPqqOEifB++n9TQWr8kkaFpzJzTUf+O9KVHMp4LG0r1YSJeCj2kzRe3L6DmXxMp/Bueoa6EbUwUcK4hYmGwizc1YFVe/ew0zI5urQywf/AKg4irZ87RKtjX84pGRAKQ8VuKC2BSLw9UkmkupKKjidQaelENi7D1i0gWrEHryyFit14RSkfn3IrZeEsOq18iqM/epJQdRkp1WWEq/eSEinjkSGz2Es6w1ZNYuDGJ+pEub7HK1RGnKu2/JZRZS997r49tKNw7slUVEe5PTqTYbaECk+lklSqCbHVt/H1Ra8CcGfKcgaGPgLAAMNZ57v4xrtFAExK/YD+9gngWPz513kZkz78EDPok7qRcGgnEVKIhlKJWjobyGRHWSVpKSFWpZ/I5vTuEE7FQ7HLjqzeTOh0DOmpIRZv/iZpFsFS0vCUDDycTlbWsdyXfxIpIWP+zscJpaRjKWlYagaWmslx6dk8kZZFathYHvqIlJARDhmpYSMcCnFtyLg+HG8LfZFw2EgNhQiHjJSQMTJkNT6tCXU+2/GH+d+ksczdm/klm8bAgQO9qKgo6Bgi9YtGoGIXVJZBVRlUlsZ+5vWCnALYvQk+eCF+X7xPVRmc+jWqOp9I6eoFpM++m2hFKR6phOpKiFYy/6T/ZF3OAArWv8zo5T8j7FWEvXrfy97VeTLLQn0Ytns63y39rzqxRlfcxyrvyjXhl7gr9W/72ss8nVLSuajiP9lIPheHXufi8DxKyaCMDEo99vPB6vGUk84p4dX0CW8kEk4nGkrHw2lEw+l8mHYC6alhOoZ20S4UgZSM+JdnBqmpaaSlhEgLh0hPCe27nha/nhoOkRq2+M/YJS3l87dr3p8WDpGa8vnbKeHYF62Z1XnvEmNmb7v7wIP108hBZH8iVVC+M7b5ZO+O2M/cbtDp+NhmlTn3xdrKd/xfn6HfxU/5ChXr3yfjL8PqPOW8/ndRlHcB2SWLmbji+/vaK0mhnAx+PL89L1SuY4B9xE9Tt7LX06mgHZW0p4oUHpqzng88hX5WzfrwSDyciofSsJQ0CKextjqXlIwQGzuczl9zbieckk5Kahrh1NjPK/JOxdJzyKI7z3MtobRsQhntSEtJJS0lxIP7vrSH1fnyTksJcU38SzgU0pdvW6eRgyQnd1j+POz8FN9ZTMW2dURLt7C96yjW9rmWitKdFD5zap2HzS24ihc6TYS927hr9VfYE8pmNznsJIsdnsUz0bOZUXUKmZHdfDn8OmWkU+bplJNGGRmsiHalhFzy0p2uGeWkZmaTnplNTrsM2mekckRm/NIudd/trPQU2qWFyUwLk5kapl1amIzUMOkpIf2FLI3W0JGDioO0HdEoVJVCek7s9tt/ha2r8F2fUr19Hb5zPZvyBjKt189Yt72Mn7wzhkwvo9xT2eB5bOEIpkUG8WhkHOB8K/wcu2jHDs9mF1nsthy2pRZQnt6RzLQwWWmxL+12aWHapaeQlRamXVoKmWlhcjJS/u+LPjP1c1/8ORkppIR1FLkEQ5uVpG2KRiEU/2L99x9g/WLY+SnRnZ/C7g3syDuZZ05+mOLte7n+3Ul0qlrPRs/j02g+G+jOW9s78sSHH3BEZipb2k8iK7cLeR0L6JqXRW5WGsPSwoyJf+lnpY+g3b4CkEJair7QJXmoOEjLtWsDbHgHNi+DzcuJblxGZVUlMwqfZ9XmPYx+50UK9q6iOJpHceRoNvgAVqw/hueKl5OVFmZp7n3k5ebTNa8dx+S145jcTK7KbceP8zJpn5Ea9LsTadFUHCR4Zdtg8/uw6X3YsoKdhfewakspHV75Cb3WPQ3ARuvE+9VH84H35r4nFhMy44W8W+h2TBbH5LbjmLxMuuW2Y1huO+7MzaRDu1Rtjxc5DCoO0nzcY0UgtweemsmuBVPImHM36Xs37euyk2zGvH4Km8ijj51JfngAlR37clRBAb07ZdO7czYzOmfTvWM70lPCAb4ZkbZNxUESq2wbrH6N6g9fJvLhLNLLN/PjI37Nv3b04MSq9Xw53IcV0XNYm9qDqvy+5BV047qCHHp3zua4ziM5OjeTsA6bFGl2Kg7StKKR2Mlc6TmUrFxI/pQxhIiyx7N4PXoSC0KXsCWzJxf3PIrenY/nyE7XMbxzNp1y0rUZSKQFUXGQw7d7E3w0C1/5CpFVs3gnbxx3lH+VFRu2c1N4PCuzz+CoE4Ywqv9R/Kx7no76EWkFVByk8dxjk6gBkb9+kfDHswHYRgdeiwzghd3HknVsmFvGncDofiPp1SlbowKRVkbFQRpm+yew6hX46FUqt37Ck6dP4dXlmzhpdQFV0ctZmHIaXfqczqj+XfhNn87kZaUFnVhEDoOKgxzYp4vwF76HbXgHgM2hTrxcOYBfPreIgrwOdD/zu4zuV8DN2lwk0qaoOMgBvbfiQzptLObhqiuZ4yeTe8wJjOrfhRf7ddbmIpE2TMVB6tqwlLJPFvGzdafyz7dz6Nfxj1x/0Ql86/jO5GpzkUhSUHGQ/xOpgnm/JTr71+z0DkyrnMQ3R/bl2+ccR0aqTjgTSSYqDhKzeTnVT/8HKZve4V+RITye+02eumw4Jx59RNDJRCQACS8OZlYAPAbkAbuAq919fT397gHOBlKBRcC33WsscSUJ46VbiDx0DrsjKdxR/T36jLySKSN6aQezSBJrjpHDJOAud19gZgOBB4Cv1OxgZucBHd19WPz2n4EvAk83Q77kVbqFTZFsfvrcGjL2XsfOLmfxkwkjOL5LTtDJRCRgCS0OZtYByHf3BQDuXmRmR5hZB3ffUaNrGCiucXsL8GkisyW1aBRf8EciL9/Nz6I/YG71SfzgvOu4bmgPLUIjIkDiRw49gFW12lbH2xfXaJsGFJrZb4EQsMbd59d+MjO7AbgB4Nhjj01I4DZv28dUPP0N0j+dz5zIqdClP9MnDKdnp+ygk4lIC5Lo4mBAfeuQ1m4bTGxfw4PAHuBHZnaNuz/2uQe5Pww8DLFlQps8bRsXfftvRKbdSmU13O3f4Phx3+CPg7trsXgRqSPRxWEN0LtWW694e02/Asa4eyWAmf0QeIPYjmxpAmu2lPLKnBX0qezNP4++jVsmjOKYvHZBxxKRFiqhxcHdt5lZmZmd5u6LzGwAsf0JaWY22d2/G+9aSmz0MDd+ewLwQSKzJQV3oosfZ/ZHu7hpaU9SQ8P56UU3MPmMY3Vms4gcUHMcrfQ94BEzywZ2AtcCOUCfGn2+AfzOzO4mtsnpQ+A7zZCt7dq1gdKnv0nWJ7OojpzO0F6/5j8vPokuR2QEnUxEWoGEFwd3LwbG1mreBIyr0Wcd8KVEZ0kalaWU/n4YoYpd/Dp0LX3H/4C/nNJVowURaTCdId0GbZj/FEdWbuGBLr/muiuvpVNOetCRRKSVUXFogxauLqEg2o9rr7yGfBUGETkEOuOpjamojnDn2lP42/H/j/wc7V8QkUOj4tDGzFuwgF1l5Vx+5jFBRxGRVkybldqSaJQBr17NQ1l9GNrroqDTiEgrppFDG7LpnRl0imymuvc4nfUsIodFI4c2ZOu8R8nwdpwy5sqgo4hIK6eRQxtRVbqdnltfY1H7UXTJ7xB0HBFp5VQc2ogPZv+DDCrJGnR10FFEpA3QZqU24rebTiM15Rf8YfA5QUcRkTZAI4c2YP2OvcxeuZXjzhhLSko46Dgi0gaoOLQBHz/7C24N/4PLBnYNOoqItBHarNTKRaoq6fvJFDJy+nFMflbQcUSkjdDIoZV7//WnyWcH0ZO/GnQUEWlDVBxaueq3H2crRzBg5KVBRxGRNkTFoRXbsulTTtzzb1YWnE96uibZE5Gmo30OrdhLSz4mMzqEM0ZeH3QUEWljVBxaKXfnkXer6XT0T7ik7+lBxxGRNkablVqpxUuXkrl1GZedoam5RaTpaeTQSu2e/SDPpv+L6HFfCTqKiLRBGjm0Qjt27WbAtpl82OFsMtvnBh1HRNogFYdWaPErfyfX9tD+rGuCjiIibZSKQyvj7mS9/yQloY50O+OCoOOISBul4tDKvLO6mL5Vy9nU42IIaZI9EUkM7ZBuZf6+eBuv+h+ZfdGwoKOISBumkUMrsntvJf96Zz2jBvQku0PHoOOISBum4tCK/HvOSzxrt3DNcXuDjiIibZw2K7Umi/+HbqESMvr2DzqJiLRxGjm0Eu9/soEh5XMpPmoMlp4TdBwRaeNUHFqJ5bMeJ9vK6TLi60FHEZEkoOLQCuytjHDM2mcpST2KnD4jgo4jIklAxaEVmPbuBh6vLGT7oFvBLOg4IpIEtEO6FXhi4Vq25I3huFEaNYhI89DIoYVbtWkn/dY9wTUnt8M0ahCRZpLw4mBmBWY23cwWmNnLZnbUfvqda2avm9ksM5tmZlmJztYazJ/1HHen/pWLO6wOOoqIJJHm2Kw0CbjL3ReY2UDgAeBzixCYWQ/gJ8BF7r7DzLoA0WbI1qJVVEfIW/kUZaFs2p/ypaDjiEgSSejIwcw6APnuvgDA3YuAI+LtNd0C3OHuO+L9Nrp70p8G/NqSlZwTXcD2nl+A1Iyg44hIEkn0ZqUewKpabavj7TWdBFSY2VQzm2tm95hZnWxmdoOZFZlZUUlJSYIitxzr500hw6roMmJi0FFEJMkkujgY4PW0125rB3wV+BowIn77ujoPcn/Y3Qe6+8BOnTo1ddYWZe3WMiJbVlHS7jjCXU8LOo6IJJlEF4c1QO9abb3i7TWtBe5z913u7sDjwCkJztaiPVW0jl9FrqT666/o3AYRaXYNLg5m1rWxT+7u24AyMzst/hwDgC1AmplNrtH1D8CvzOyz1Wu+CCxo7Ou1FdWRKM8XraLw+M4cmV9794yISOI15milSWZ2BPAS8E93L27g474HPGJm2cBO4FogB+jzWQd3f8XMegNzzawaeJPY6CEpzV6+gScrv8WerK8BZwQdR0SSUIOLg7tfZmZpwDnAj82sM/AGsUKx7gCPKwbG1mreBIyr1e9PwJ8amqctWzb3GUbbNqpPODPoKCKSpBp1noO7V5rZa8ARwCXEjjI6xcy2ufvNiQiYbDbs3EvfDVMpzcgl6/jaNVVEpHk0uDiYWSFwJXAk8E/gOnffE7/vzYSkS0IvvPku14QWsfeEiRBODTqOiCSpxowcxgL3unvt8xYALmiiPEktGnXKFv2DVIuQetY1QccRkSTWmENZ3wOO+eyGmY01swkA7r69qYMlo3mrtjBl9+ksPvWXUKClQEUkOI0pDtcBs2vcngl8u0nTJLknFq6lql1n+l9wU9BRRCTJNeokuPgJajWvR5o8UZLasHMvHT94nB/3WEl6SvjgDxARSaDGFIfFZvYtiwmZ2feBRYkKlmz+e/b73Bx6ivPDbwUdRUSkUcXhdmJzHs0htnkpldhsqnKYSnZXECp6lFzbQ9awG4OOIyLSqJPgqoH74hcA4jukn0pArqTy1znLuD40lbKuw2l37OCg44iINOo8h37AXUBXYrOtQmwkoeJwGLaXVhJ96xE6hnbBuT8JOo6ICNC4zUp/IbaK2zJiU1/8D/DXRIRKJo++uYa11bns6HsZdDsr6DgiIkDjikOZuy8E3gW6ufsfgS8kJlZy2FVexWNvfExVvy/R4fKHg44jIrJPY4rDOjM7Dnge+LmZXY3WeT4s/5j3AV+smsa3hx9z8M4iIs2oMcVhIrDG3T8BfgV0AC5PSKokUFZZTdkbD/GL1Mc4MbQm6DgiIp/TmLmVZrj7uQDu/hagA/IPw1NvruBr0ansOnoY7Y8dFHQcEZHPaczIYbeZnZOwJEmkvCrCzrl/oqPtov15dwQdR0SkjsaMHDoCr5jZaqAk3ubuPqTpY7Vtz85fwVcjz7HjqGF00HkNItICNeYkuLMTGSRZVFZHeeb1JZyadjTHn/fToOOIiNSrMSfB/Rv4bOK9jkAP4GV3Pz8Rwdqq5xZ/ysJdR7Dhmqn07VYQdBwRkXo1ZuTwuTO0zGw8MKrJE7Vh1ZEob856lqFH9qDw+M5BxxER2a9GrSFdk7s/Y2bfa8owbd1Liz/ijr33EckdiNlFQccREdmvxmxW6lPzJjAIKG/yRG1UNOqsf/n35NtuoufdHnQcEZEDaszI4UfE9jlY/OfHwBWJCNUWvfLOai4pf5rNBUPp3E1HKIlIy9aY4vAKsMHdX4XYGtLE9jloVtaDcHfWzfwdY2w3kQt+FnQcEZGDauwa0q/VuK01pBvotRWbSdu9jo2dhhLWqEFEWoFG7ZCuvYa0mWkN6YNwd/5r1ipKsr/J5dcPDTqOiEiDaA3pBJu/Yh2lxe9xY2EvUtPSg44jItIgWkM6wT568UFmpN/GpT0qgo4iItJgjSkOhcC/3f3s+FQabwGaUuMAij4sZtyup1iffxbpBX0O/gARkRaisSOHmtN0vwnc2bRx2pZV0x4k33bTSUcoiUgr05jigLtX1Hdd6np39XrO3f4Ea3PPIr2n1oYWkdalMcVhjdWY8yE+t9Lqpo/UNsx4+UXaWQX5F2rUICKtT2OKw3eBcWY2z8zeAM4Fnk5MrNZt+YZd/P7jo3hs8HSyemm5CxFpfRpTHHKA3cDa+CUXmJyIUK3dlJlvkp0e5oqzBwQdRUTkkDSmOEwBiuKPuQdYA/w6AZlatY8+3cTNqyfyyFHP06FdWtBxREQOSWOKg7v7/xIrEJXufjvw1YM9yMwKzGy6mS0ws5fN7KgD9E03s3fN7LxG5GpR3p86iXzbzfEjD/rRiIi0WI0pDlvNrACYBtxmZiNo2PQbk4C73H0QsZldHzhA318SG5G0Sus2lDBk099Z1X4wHfpoqgwRab0aUxz+A6hy9/eB54EvEZuMb7/MrAOQ7+4LANy9CDgi3l677xhgG/B2IzK1KO/FRw15598RdBQRkcPS4OLg7jvcfVv8+nPufrO7f3iQh/UAVtVqWx1v38fM8oGJwH0HejIzu8HMisysqKSkpKHRm8WGHWX02PACH+YMIq/vsKDjiIgclkadBHcIPlsYqLbabb8BbnP3A87y6u4Pu/tAdx/YqVOnpsrYJB6a+zFfrvoF2V/+f0FHERE5bIe8hnQDrQF612rrRY39CvEd1AOBR80MoDsw3sxOd/f/THC+JlGyfSf/+9Zqzj+1J0d1q/12RURan4QWB3ffZmZlZnaauy8yswHAFiDNzCa7+3fdfT1wwmePMbO7gPnu/lIiszWl956+l2nhf+JnzQ46iohIk0j0yAHge8AjZpYN7ASuJXZCXZuYpnT79m2csu5/2JDdj/5djw46johIk0h4cXD3YmBsreZNwLj99L8r0Zma0nvPPsBw283uMT8JOoqISJNJ9A7pNm1vWSknrv0b72WeybEnFwYdR0Skyag4HIZlb7xALruxwf8RdBQRkSal4nAYntzakxu4gz6DLwg6iohIk1JxOERVkSgzPthGdv/RpKZnBh1HRKRJqTgconfffpObqv+HL/RKDTqKiEiTU3E4RLvefoqJ4RcZfFxB0FFERJqcisMhiEadoze9xurMk8ho3zHoOCIiTU7F4RC8v/xdjmMtFb3GBB1FRCQhVBwOwYa3ngGg+5AJAScREUkMFYdGcnc+2bCZlWn9yDm6TcwAIiJSh4pDI63cvIdf7rqA+SOfCDqKiEjCqDg00itLP8EMxp7QJegoIiIJ0xyzsrYpJy68neezd9K5vc6KFpG2SyOHRlhXsoNTKopI6dgz6CgiIgml4tAI7705nfa2l7zTvhR0FBGRhFJxaIwV0ygnjYKTay9PISLStqg4NNDW3eUMKH2T4tzBkNYu6DgiIgml4tBAs95fz/1VEwifdWPQUUREEk5HKzXQS8u38uER59L9jJFBRxERSTiNHBpgT0U1+aue4dLeYGZBxxERSTiNHBrg34uXcn/K/2Ndag5wTtBxREQSTiOHBti6aCoARw0aH3ASEZHmoeJwEBXVEbpueo2StK6EOx8fdBwRkWah4nAQC5av4UzeY2+PsaD9DSKSJFQcDmLFormEidJFm5REJImoOBxAJOr8aW1XftL7WdK6nxV0HBGRZqPicABFa7axtbSSYScfD6Fw0HFERJqNisMBLJ8/g7+n3cPIzmVBRxERaVY6z2E/3J2Mj6ZxRmgFqbla2EdEkotGDvux7NOdDK56i5JOgyE9O+g4IiLNSsVhP94uepPuoU20P+ULQUcREWl2Kg77EVk+HYDsky4KOImISPNTcajHx1tKWbK7PSuOHg/tjww6johIs1NxqMeMZRt5PjqErC//IegoIiKBSHhxMLMCM5tuZgvM7GUzO6qePh3M7A9m9pqZ/dvMJlmAc2MvXrqEM48M0zVXK76JSHJqjpHDJOAudx8E/Ah4oJ4+WcDD7j7S3c8CcoBxzZCtjk27yrl08+94qPyWIF5eRKRFSGhxMLMOQL67LwBw9yLgiHj7Pu7+qbu/U6PpEyAtkdn259WlHzMs9B70HhXEy4uItAiJHjn0AFbValsdb6+XmQ0HzgJeqOe+G8ysyMyKSkpKmjToZzYueYkMq6LDKV9KyPOLiLQGiS4OBng97XXaLOb7wCXAl929us6D3B9294HuPrBTp05NHnZnWRVdN82mPJyNdR/a5M8vItJaJLo4rAF612rrFW+v7b+BT9z9e+5enuBc9Zq1fD2FoUWUdTsHwqlBRBARaRESWhzcfRtQZmanAZjZAGALkGZmkz/rZ2ZfAIrc/elE5jmYGcs28620X9Bh7I+DjCEiErjmmHjve8AjZpYN7ASuJXY0Up8afYYBhWZ2aY22f7r775shHwB7KyPMWbmFS08fSKigX3O9rIg0UlVVFcXFxZSXB7KBodXIyMiga9eupKYe2laQhBcHdy8GxtZq3kSNQ1Xd/dZE5ziYuStLuNn/h7M6XgmcGHQcEdmP4uJicnJy6N69OwGeDtWiuTtbt26luLiYHj32e/zPAekM6bhFixbyHykvckJ4bdBRROQAysvLyc/PV2E4ADMjPz//sEZXKg5AVSRK2kczAQj3Oz/gNCJyMCoMB3e4n5GKA/DWx9sYFn2LXUf0hQ7HBh1HRCRwKg7A3CUfMNBWkKnpuUWkAR577LFG9b/99tsTEySBkr44RKPOBx+8x7bULqT2vyDoOCLSCjS2ONx7772JCZJASV8c3inewZw9x/D6uJfhyFOCjiMiLdz48eNZsmQJhYWFzJ07l/HjxzNq1CiuvPJKtm/fzoQJEzjnnHMYOXIk69evB2Dw4MGsWbOGO+64g8svv5zhw4fzxS9+kbKysjrP/8Mf/pDRo0czcOBA5s6du6/93nvvZeTIkZxzzjncf//9QKxIjRgxgtGjR3PzzTc36ftsjvMcWrSZ760nPRRlVL8uoJ1cIq3Kz/+1jPfX72rS5+x/VHvuvOiE/d7/zDPPUFhYyOzZs1mzZg0vvPACS5YsoX///mzZsoXJkydz5JFH8tRTT/Hggw9y33337XvsX/7yF9544w169uzJvffey2OPPcZNN930uee/+uqreeCBByguLuayyy7jjTfe4Mknn2Tz5s289tprAMycOZP58+czbdo0Xn31VcLhMDNmzGjSzyGpi4O7s2PpdIoyJpOz+yVot///ECIi9Rk2bBj9+/cHoGPHjixbtozHH3+chQsXEg6HP9f34osvpmfPnkBsNFHfF3rnzp156KGHWLp0KRs3bgTg6aef5je/+c2+PmPGjOHWW2/lO9/5zr7XGDu29ulkhyepi8OqzXs4cc+bZKRXQ37tKaBEpKU70F/4zaVz5877rj/77LNMnTqV2267jbFjx3LPPffsty/E/kCtac2aNVx//fXcfffdXHXVVbz++uv19ttfW1NK6n0OM95bz+gXxg/QAAAMFElEQVTw20R6joKU9KDjiEgrUV1dZ9JoAObMmcPVV19Nv379mDJlSqO/wJcsWcKQIUM466yzmDVrFlu3bgVi+zlqbp6aNm0aF198Mb/97W/3ZXnxxRcP8d3UL6mLw0dL51FgO8g48cKgo4hIK1JQUMDw4cNZu/bzMyrceOON/PSnP+Xss8+mZ8+eLFy4kNmzZzf4eceOHcvixYsZMWIEs2fP5qSTTmLSpElcccUVdOnSheHDhzNkyBBWrlzJkCFDuOCCCygsLGTIkCH7RhlNxRI9NEmUgQMHelFR0SE/vnh7Gc9MupFvpf6L0C2roF1eE6YTkURZvnw5/fppcsyGqO+zMrO33X3gwR6btCOHmcs28WrkNLYPu1OFQUSklqTdIT1j2UbKOp9C/qgRQUcREWlxknLksHVPBf7Jm1x3zCZopZvVREQSKSlHDrOWb+a74ac5vbgCbELQcUREWpykHDnMXbqSQeHlpJ+gifZEROqTdMVhT0U1aR/PIoUo1ldrN4iI1CfpikN5VYTrOi2nKqMjHH160HFERFqkpCsOHdulcmJkBan9xkEofPAHiIjU0tgpuwF27NjBc8891/RhEiTpigOhEHxnMYz+edBJRKSVUnFoq1LSICs/6BQi0grVXM9h7dq1zJs3j9GjRzNixAh+/vPYH50LFixg8ODBDBo0iPPPP5+dO3dy+eWX89JLL1FYWEhlZeXnnnPKlCn71nCYPHnyvvbp06dTWFjIueeey9e+9jWqqqpYuHAho0eP5txzz2X8+PFs3749Ie8zKQ9lFZE25NF6VnA84Utw5vVQWQZTLq17/ylfgVO/CqVb4amrPn/ftQeewK7meg47duzgBz/4AbNmzSI7O5uvf/3rzJ8/n3vvvZfHH3+c3r17s3z5ciKRCE888QR33XVXvaOOAQMG8Morr1BdXc2gQYO45ppr2LFjB5MmTWL69OlkZmby+uuvs3fvXm6++Waef/558vLyWLZsGaWlpeTm5jbiA2sYFQcRkUM0b948SkpKuPDC2OSdpaWlLFu2jEsvvZSJEycyceJExo0bR15eHrt27X9Rot69e/PUU0+xYMECtm/fzsaNG5k1axbXXHMNmZmZAAwfPpxp06Zx3nnnkZcXm/LnhBMSN2W5ioOItG4H+ks/rd2B78/KP+hI4UDMjGuvvZY77rijzn0XXnghM2bMYPz48dx///111nKo6Qtf+AI33ngjd9xxB+vWrcPdA1nDoabk3OcgInIYPltDYejQocycOXPfdv8PP/yQWbNmMWfOHNq3b8+ECRMYM2YMRUVFhEKhOvsaILajurKykvHjx7N7925effVV3J3zzz+fRx55hNLSUgBmz57NsGHDeOGFF9iyZQsAS5cuZd26dQl5jxo5iIg00mfrOUyZMoX777+fSy65hGg0Srdu3Zg0aRJTp07lzjvvxN3p0qULjz32GKmpqaxdu5bCwkJmzpxJWloaAB06dGDo0KEMGTKEgoICrr/+em6//XamTp3KD3/4Q8aNG0c0GqVv374MHTqUyZMnM2HCBKqqqjjyyCN56KGHEvIek3Y9BxFpnbSeQ8NpPQcREWlSKg4iIlKHioOIiNSh4iAirU5r3VfanA73M1JxEJFWJSMjg61bt6pAHIC7s3XrVjIyMg75OXQoq4i0Kl27dqW4uJiSkpKgo7RoGRkZdO3a9ZAfr+IgIq1KamoqPXr0CDpGm5fwzUpmVmBm081sgZm9bGZH7affnWZWZGaLzOySROcSEZH9a459DpOAu9x9EPAj4IHaHcxsHJAfPzFjCHCTmXVshmwiIlKPhBYHM+tA7Et/AYC7FwFHxNtrmgjcG+9TDvwZuCKR2UREZP8Svc+hB7CqVtvqePviGm0F7r6+xu2VwFm1n8zMbgBuiN/cY2YrDiNbR2DLYTw+UZSrcZSrcZSrcdpirm4N6ZTo4mBAfceb1W6zBvTB3R8GHm6CXJhZUUPmF2luytU4ytU4ytU4yZwr0fsc1gC9a7X1irfXVGJmXWrc7kPdEYeIiDSThBYHd98GlJnZaQBmNoDYUCjNzCbX6PpX4JZ4nzTgKuAficwmIiL71xznOXwPeMTMsoGdwLVADrHRAQDu/qyZnWFmC4htTrrH3bcmOFeTbJ5KAOVqHOVqHOVqnKTN1WrXcxARkcTR3EoiIlKHioOIiNSRdMWhodN5NHOmIWb2LzN7zczeNLOxQWeqzcyGm9mnQef4jJmFzOxeM5tnZnPM7LagMwGY2bFm9qKZzTazt8zs4haQ6Udm9o0at4+Pf2ZvmdnT8f2BLSHXhWb2Uvz3YI6Znd4SctVo/4qZzQ8iU/z1a39emWb2ZzObG79c1aQv6O5JdQEeBwbFrw8E/t4CMg0HcuLXc4GlQWeqle8I4CmgKOgsNTL9GLitxu0eQWeK53gCGBK/ng98SHzfXgBZjgbmAyXAN2q0vwT0jF//InB/C8k1CkiPX+8OzGsJueL3dQMeBea3oH/Hh4Aratzu3pSvm1Qjh0ZM59Gs3P11d98dv7kD2Gtm9Z0YGJT7gZ8A1UEHgX2HO48H7vuszd0/Di7R54SB4vj1PcBqj//mNjd3/9TdBxM/TBzAzE4C1rr76nifqcTmMws0V7x9lrtXxG9upP4TaJs9l5mFiP1fu6XeBwaQK77Fo7u7/6NGvzVN+bpJVRw48HQegYv/J3wAeCSoL5TazOwK4G13Xxl0lhp6AEuBb5rZq/HL6KBDxd0K/NbMbgb+CXw/4Dy19QZqTzuzzcxygwhTHzNLB/4C/CroLHG3AY+6e0uaRuMkYIGZ3RXfhDn9s/PJmkqyFYeGTufR7MysMzAFmOuxaUICZ2bHAue7+0NBZ6klGzgX2OTu5wCXA/e3kC+4rxDbbPNH4OfAr8zs6GAjfU6L/R0AMLNexE6A/aO7T2sBeQYCndz9paCz1JINTADmuHsh8G3g0abc4pBsxWENDZvOo1mZWU9i2zNviQ/zW4ovAX3if5nMBvrHrweyo7CGj4Dl7v6/AO6+GXiD2L9lYMysH9DL3f/s7uXxzZZPAF8OMlctq6lxAmpcvrvvCCJMTWY2CPg18HV3fyPoPHETgEH1/A4EXfBXEftD8jUAd18FrCO2n6tJJFVx8P1M59ECfjHuAq5z9+KDdWxO7v5f7j7I3Qvjf528H7/+dsC5dgAbzew8gPjRNqdRd3NJc9sFnGRmObBv88gEYEmgqWpw9yXECn43gPhn+Hqwqfb5EfA1d98edJDPuPut7j60nt+BQI/cc/d3gGPN7FSA+Nx0+U256SsZlwmtbzqPoA0Bnqw1IpwY/2tA6vcd4Hdmdnv89k9r7NQPhLt/amb3AC+ZWSWxP77+4u5zgsxVj+8Cj8c3QXwKXBdwHsysHbHfg+m1fg8udPc9waRq8SYCv48fUFMF1Dn89nBo+gwREakjqTYriYhIw6g4iIhIHSoOIiJSh4qDiIjUoeIgIiJ1qDiINJP4VAfnBZ1DpCFUHEREpA4VBxERqUPFQaQeFnO3mc2KL6RyZnwhmj+Z2etm9r9m1j7e99vxtjfM7CvxtlQzmxSfMfY1M7sy/tQDLLbI1CIz+6qZZZjZk2Y238yKrAUu9CTJKRmnzxBpiK8SW3hmlJnlAVOBM4Hh7v6WmV0O3GFmLwDDgBHEfp/+ZWZLgIuAT9z9B2YWBgqJTfo4FhgHpANvAaXASne/zMwygX7N+i5F9kPTZ4jUw8z+CRwFVMabCoBydz+1Rp95xFboes7d58Xbvkxs1bAvAYXuHqnR/y5ihWBK/PYc4CrgSeDF+PO8m+C3JtIgGjmI1M+By9x93WcN8SmbP7ueEu/z2aXm45zYugn1za2/tVbfTcQmnDsTuMnMdrv7rU30HkQOmfY5iNTvWWILqABgZtcRmyI5O950EzAdeI7Yl7rFly+9Pt7+IrHZTzGz0AEOYT0OyHX3+cAviW2eEgmcRg4i9XD3v5tZTzN7HYgATxNbQ/vv8R3RHwDfcffK+CI/rwEZwH+5+3IzWwXcF990lAL8bj8vVQ08ZmZZQCrxgiISNO1zEGkgM5sdX/BFpM3TZiUREalDIwcREalDIwcREalDxUFEROpQcRARkTpUHEREpA4VBxERqUPFQURE6vj/B3CDy9EcIOEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mnist_test(784,10,10000,100,0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> <font size=\"5\">\n",
    "    다음 6장에서는 어떻게 해야 학습이 잘 되는지에 관한<br><font color=\"red\"><b>학습 관련 기술들</b></font>에 대해서 논의할 것 </font></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5장 정리 (p187)\n",
    "\n",
    "> 참고 : [WegraLee GitHub](https://github.com/WegraLee/deep-learning-from-scratch/blob/master/ch05/README.md)\n",
    "\n",
    "* 계산 그래프를 이용하면 계산 과정을 시각적으로 파악할 수 있다.\n",
    "* 계산 그래프의 노드는 국소적 계산으로 구성된다. 국소적 계산을 조합해 전체 계산을 구성한다.\n",
    "* 계산 그래프의 순전파는 통상의 계산을 수행한다. 한편, 계산 그래프의 역전파로는 각 노드의 미분을 구할 수 있다.\n",
    "* 신경망의 구성 요소를 계층으로 구현하여 기울기를 효율적으로 계산할 수 있다(오차역전파법).\n",
    "* 수치 미분과 오차역전파법의 결과를 비교하면 오차역전파법의 구현에 잘못이 없는지 확인할 수 있다(기울기 확인)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 참고\n",
    "- 밑바닥부터 시작하는 딥러닝 GitHub : https://github.com/WegraLee/deep-learning-from-scratch  \n",
    "- Mathjax 웹 수식 : https://www.codecogs.com/latex/eqneditor.php\n",
    "- 책에 나온 사진 예제 : https://kolikim.tistory.com/46\n",
    "- 계산 그래프 참고 : https://www.youtube.com/watch?v=hCP1vGoCdYU"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "5장 오차역전파법",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "336px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}