1_introduction_and_interpretation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modern Methods of Data Analysis 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "The goal of Data analysis is always to extract Information from Data. Some pieces of Information will stand out immediately\n",
    "(for example if a Value appears very often in the data set that will be noticable without the aid of staistical tools)\n",
    "but most of the information has to be extracted via analytical procedures. In this context the information that can be\n",
    "extracted from the data will be a model of the process that generated the data.\n",
    "\n",
    "This Model will have a specific form (an exponential decay for example), that most often will have to be infered from the experience of the\n",
    "person analyzing the Data.\n",
    "\n",
    "To be more precise, a model is a function of some set of input parameters $\\vec{\\theta}$ and of a set\n",
    "of observables $\\vec{x}$ that produce the \"true\" value(s) $\\vec{y} = f(\\vec{x}, \\vec{\\theta})$ that when combined with uncertainties\n",
    "produce the observed data $\\vec{x}$.\n",
    "These fluctuations arise either from (ramdom) fluctuations (that sometimes by their very nature cannot be suppressed) of the\n",
    "quantities of interest ($\\vec{x}$), or of noise and other sideeffects that are part of the measuring process.\n",
    "\n",
    "Every Measurement will have some amount of uncertainty associated with it (remember that we are actually all just big interference phenomenons\n",
    "when viewed from a quantum mechanical perspective) as we are allways measuring processes that are by there very nature statistical.\n",
    "Furthermore the Instruments themselves are quantummechanical objects that are subject to the same random behaviour that causes fluctuations.\n",
    "\n",
    "There are models that by their very nature can approximate any underlying function (for example a fourier series), these methods are however\n",
    "mostly avoided, because the calculations involved are very time consuming and don't present the result in a way that is understandable for\n",
    "humans (Do you know the coeficients of the fourier series for an exponential decay? Well I definitely don't) even if the result can be made\n",
    "arbitrarily accurate and the situation where \"the model is wrong\" can always be avoided."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "#### Side note on Data vs Information\n",
    "As you may have noticed I am making the distiction between Data and information. I am doing so to avoid the seemingly common confusion of both. For this set of documents data allways means the raw measurements. There is a LOT of data out there. The problem with data is obvious when considering the following example: \n",
    "\n",
    "I give you 400,000 pages of rows and rows of numbers that describe the process that you are to become an expert on. Even though you have \"all the data\" you don't have an *understanding* (i.e Informationn), you can't *predict* anything based on the Data alone . You job is to *make sense* of all these numbers, to find the *patterns* in\n",
    "the Data, thus deriving information from all of these rows and columns.\n",
    "As you may see now, information is closely related to understanding and therefor to something we humans can comprehend and act upon or think about. Data on the other hand is only a vast quantity of numbers.\n",
    "\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The role of Statistics in Data Analysis\n",
    "\n",
    "As mentioned above, most often we are dealing with systems that are either to complex to be accurately modeled in detail (the stock market would come to mind)\n",
    "or that by their very nature are statistical (i.e involve some sort of randomness)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "#### Assumptions for complex systems\n",
    "The reason that we can not only model truly random processes (like quantum mechanics) but also highly complex models is the assumption, that in highly complex\n",
    "models small fluctuations only cause small effects, which seems to hold true for the most part (and for further information please consult your local chaos theorist).\n",
    "So if the small fluctuations are assumed to be random, the resulting behaviour is still mostly dependent on the main \"forces\" acting on the system because small random fluctuations are still small and will only have a small effect. These small effects then add up to the uncertainty of the prediction based on the more coarse model.\n",
    "\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As randomness is involved in the processes the mathematical methods of the field of statistics (the study of random quantities) can be applied to gain information from Data.\n",
    "\n",
    "Statistics offers a wide range of procedures and definitions that can be used to reproducably and repeatedly gain information from data.\n",
    "In this specific case we would like a model ($\\vec{y} = f(\\vec{x},\\vec{\\theta})$) that describes the data. The model therefor represents *information*.\n",
    "\n",
    "As we are dealing with statistical processes we can not make certain claims, but must use probabilities in stead.\n",
    "Another problem that can easily be overlooked when only taking a superficial look at statistical analysis, is that most statements are not\n",
    "clearly Yes or No but are far more often something inbetween. This thing inbetween is called the Probability.\n",
    "The concept of probability allows for a far more nuanced view of the data. It allows to weigh different options against each other using the probability as\n",
    "a measure of which statement is *less wrong*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretations of Probability\n",
    "Before I get to the more formal math of probability theory, I would like to present two common interpretations and an example to ease in to this (sometimes quite confusing) studt of Probability. Both interpretations of Probability will be used depending on the circumstances, so they don't exclude but complement themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The bayesian interpretation\n",
    "In the bayesian interpretation of Probability views statements as either true or false and interprets probability as a degree of belief in that statement.\n",
    "I think it is important to note that the type of statements that are normally used in combination with the bayesian interpretation are relatively basic.\n",
    "This is a consequence of the quickly declining probability of multiple events that are interdependent and have to occurr in a precice order, to make a complex\n",
    "statement true. The Probability that statement $E$ is correct is written as $P(E)$.\n",
    "\n",
    "If $P(E)$ is **true** we define $P(E) = 1$ and if it is **false** we define $P(E) = 0$\n",
    "\n",
    "If you are unsure that a statement $F$ is true, lets say you give it a 50% chance of being true, you would subsequently write $P(F) = 0.5$.\n",
    "\n",
    "The beauty of this way of formulating probability is that logical operations can be mapped to mathematical operations of the real numbers according to the following transformation\n",
    "\n",
    "| Logical operation  | mathematical operation |\n",
    "|---------|----------------------|\n",
    "|**AND**  | $P(E_1) \\cdot P(E_2)$|\n",
    "|**OR**   | $P(E_1) + P(E_2)$    |\n",
    "|**NOT**  |$1 - P(E)$            |\n",
    "\n",
    "Which this transformation the logical operations can be expanded to include not only the values of **true** and **false** but also all probabilities in between.\n",
    "So we now have a rule for calculating the probability (or truth content) of composit statements like **A AND B** or any other combinations that can be built with\n",
    "bolean logic.\n",
    "\n",
    "As we are able to calculate the possibility of a statement $E$ being **true** we can also calulate the probability (or degree of belief) of a $E$ being **false**. To make the distinction a bar is placed over the statement as so $\\bar{E}$ meaning **NOT E**. As a result of this we get:\n",
    "\n",
    "$$P(\\bar{E}) = 1 - P(E)$$\n",
    "\n",
    "(*FYI*: the computer you are reading this on is running entirely on boolean logic, so you can formulate quite a lot with it)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The frequentist interpretation\n",
    "In the frequentist interpretation is not concerned with the truth or falsehood of a statement, but rather interprets probabilities as relative frequencies.\n",
    "So given a process like an experiment or an event of some sort (this could be a Rock concert, or the flight of an airliner or the collision of subatomic particles in a collider ;) ) the probability is interpreted as the probability of something happening during that event (the formation of a moshpit, the failiure of one of the gas turbine engines, or the production of a Higgs-boson respectively).\n",
    "\n",
    "Given this interpretation even a very unlikely thing is bound to happen, if the surrounding circumstances (the concert, flight or bunch-crossing) is reproduces often enough.\n",
    "\n",
    "Probabilities in this model are defined as: number of events of interest $E$ divided by the total amount of times $N$ that the surrounding circumstances have been reproduced.\n",
    "\n",
    "$$P(E) = \\frac{E}{N}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Engineering is careful optimisation of probability\n",
    "----\n",
    "Lets consider a process normally occurring billions of time every day. When we get into vehicles we expect the engine to turn on and run until we switch it off.\n",
    "A gasoline engine at idle speed revs at about 500rpm. A four cylinder four stroke engine will fire one cylinder once every two revolutions making two firings per revolution for our four cylinder engine. The speed of the engine is geared down (or up, depending on gear) to drive the wheels which ultimately drive the vehicle.\n",
    "\n",
    "The following data is taken from https://en.wikipedia.org/wiki/Gear_train and is an example for a 2005 Corvette C5 Z06. So you have a feeling for what you are driving, here's a picture. Oh by the way it runs a V8 so we will have to take that into consideration.\n",
    "\n",
    "<img src=\"images/01-04_Chevrolet_Corvette_Z06.jpg\">\n",
    "\n",
    "Lets calculate the amount of cylinder firings per second."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Firings per second at a speed of 50.0 km/h in gear 4 = 22.329 \n"
     ]
    }
   ],
   "source": [
    "# ---- play around with the numbers here\n",
    "current_gear = 4\n",
    "current_speed = 50 # in km/h\n",
    "cylinder_count = 8\n",
    "# ----\n",
    "\n",
    "# we are assuming that we want to travel forward, which is why the reverse is being ignored\n",
    "# engine revolutions per wheel revolution \n",
    "gear_ratio = [2.97/1, 2.07/1, 1.43/1, 1.00/1, 0.84/1, 0.56/1]\n",
    "wheel_circumfrence = 2.09 # in meters\n",
    "\n",
    "# convert speed to m/s\n",
    "current_speed = current_speed/3.6\n",
    "\n",
    "# calculate revolutions per second (wheels) given the current speed\n",
    "wheel_rps = current_speed / wheel_circumfrence\n",
    "# do the gear reduction \n",
    "engine_rps = wheel_rps * gear_ratio[current_gear]\n",
    "# determin the average number of firings per revolurion based on cylinder count\n",
    "fpr = cylinder_count / 2 # we need two revolutions for a cylinder to start the cycle over\n",
    "# determin firings per second\n",
    "fps = fpr * engine_rps\n",
    "print(\"Firings per second at a speed of {} km/h in gear {} = {:.3f} \".format(current_speed*3.6, current_gear, fps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So for your (imaginary?) 20 min commute to work (at an average speed of `current_speed`) the amount of firings can be calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cylinder firings per Commute = 26794.258\n"
     ]
    }
   ],
   "source": [
    "firings_per_commute = fps * 20 * 60\n",
    "print(\"Cylinder firings per Commute = {:.3f}\".format(firings_per_commute))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nowadays misfires of cylinders are a rare occasion, I'd bet you have never had one outside of racing or pushing your car a bit too hard.\n",
    "Just as a reference here is a video of the kind of misfires I mean.\n",
    "\n",
    "In https://www.youtube.com/watch?v=-8XFjpwODIY there is a misfire around second 50 (so start at 0:48 or so).  \n",
    "\n",
    "For the sake of being able to formulate consice mathematical statements, let's give these two events a name we can reference in such statements. The event of a cylinder firing successfully we shall call $F$ and the event of a cylinder misfiring we call $M$.\n",
    "\n",
    "So assuming you want to get through your commute without a single misfire we can calculate the probability that a cylinder fire has to be successful\n",
    "for you to have even a chance to get through an **average** commute without a misfire (be aware of the average in the last sentence, it will get important later)\n",
    "\n",
    "For the next calculation to be any good we need to know how often you want to tolerate a misfire on a commute. So lets say we can tolerate a misfire in 10% of our commutes (the real value is probably as low as 0.01% maybe? this is just a guess). That means that on average, we have one misfire every week assuming a 5 day working week and one commute there and one back.\n",
    "\n",
    "So we have $P(M_{commute}) = 0.1$, where $M_{commute}$ is the chance that we get a misfire during our commute. From this we can calculate the probability of a cylinder firing without problem.\n",
    "\n",
    "A probabiliry of 10% means that in 1 of 10 commutes we can have a misfire. As we allready know a commute has `firing_per_commute` chances for a cylinder to either fire or misfire. There is only allowed to be a single misfire in $\\frac{1}{P(M_{commute})}$ commutes. As we know how many times a cylinder fires per commute we can calculate  the total count of cylinder firings that can on average contain a single misfire."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "267942.6\n"
     ]
    }
   ],
   "source": [
    "# lets say we want to get through 10% of our commutes without experiencing a misfire\n",
    "tolerable_ratio_of_commutes_with_misfire = 0.1\n",
    "\n",
    "# so we can calculate the amount of cylinder firings that ocurr where one of them is allowed to misfire\n",
    "firings_with_a_maximum_of_one_misfire = firings_per_commute/tolerable_ratio_of_commutes_with_misfire\n",
    "print(\"{:.1f}\".format(firings_with_a_maximum_of_one_misfire))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As there are `firings_with_a_maximum_of_one_misfire` total possibilities for a cylinder to misfire and we want it to only misfire once the probability of misfire is $\\frac{1}{N}$ where N is the total number of times the cylinder fires (not neccesarily succesfully).\n",
    "\n",
    "As this is the probability of a cylinder misfiring and we want the probability of a cylinder firing successfully we and these events are mutualy exclusive (that part is quite important but you will see that later on) the probability of firing successfully is $P(F) = P(\\bar{M})= 1 - P(M)$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9999962678571429\n"
     ]
    }
   ],
   "source": [
    "probability_of_cylinder_firing = 1-1/firings_with_a_maximum_of_one_misfire\n",
    "print(probability_of_cylinder_firing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the result with the following calculation.\n",
    "Every time a cylinder fires, it can either fire correctly or misfire (or at least we have asssumed those are the only two possibilities),\n",
    "so having a cylinder fire twice in a row (like tossing a coin tow times in a row and getting the same result twice) requires us to multiply the probabilities\n",
    "together.\n",
    "\n",
    "For a total of $n$ consecutive successful firings the probability is given as $P(F)^n$. We expect that the probability that `firings_per_commute` amount of consecutive successful cylinder firings would occurr in 90% of cases, which would be consistent with our tolerance threshhold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9048370803359409"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probability_of_cylinder_firing**firings_per_commute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which as you can see is the case.\n",
    "\n",
    "Lets step back for a moment and try and comprehend what we have just calculated. To achieve a for modern standards miserable reliability a cylinder allready needs to\n",
    "be able to produce the desired outcome (a successful firing) 99.9996% of the time. Now try plugging in more realistic numbers and see how quickly the number of 9s at the end of that probability grow. I think a pull of hat towards the engineers that designed these machines that make the improbable commonplace is in order. I think it is also important to note that these engineers not only have to design with reliability but with a host of other constraints (like cost of the car and production time of a vehicle) in mind. This shows what we humans have come to expect from our machines and what makes modern life and it's conveniences possible.\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}