coursera/machine-learning-standford-univerity/Anomaly Detection/Building an Anomaly Detection System.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Developing and Evaluating an Anomaly Detection System\n",
    "\n",
    "To evaluate our learning algorithm, we take some labeled data, categorized into anomalous and non-anomalous examples ( y = 0 if normal, y = 1 if anomalous).\n",
    "\n",
    "Among that data, take a large proportion of good, non-anomalous data for the training set on which to train p(x).\n",
    "\n",
    "Then, take a smaller proportion of mixed anomalous and non-anomalous examples (you will usually have many more non-anomalous examples) for your cross-validation and test sets.\n",
    "\n",
    "For example, we may have a set where 0.2% of the data is anomalous. We take 60% of those examples, all of which are good (y=0) for the training set. We then take 20% of the examples for the cross-validation set (with 0.1% of the anomalous examples) and another 20% from the test set (with another 0.1% of the anomalous).\n",
    "\n",
    "In other words, we split the data 60/20/20 training/CV/test and then split the anomalous examples 50/50 between the CV and test sets.\n",
    "\n",
    "### Algorithm evaluation:\n",
    "\n",
    "1. Fit model p(x) on training set \\\\(\\lbrace x^{(1)},\\dots,x^{(m)} \\rbrace\\\\)\n",
    "2. On a cross validation/test example x, predict:\n",
    "    * If p(x) < ϵ (anomaly), then y=1\n",
    "    * If p(x) ≥ ϵ (normal), then y=0\n",
    "3. Possible evaluation metrics (see \"Machine Learning System Design\" section):\n",
    "    * True positive, false positive, false negative, true negative.\n",
    "    * Precision/recall\n",
    "    * F1 score\n",
    "\n",
    "Note that we use the cross-validation set to choose parameter ϵ\n",
    "\n",
    "\n",
    "## Anomaly Detection vs. Supervised Learning\n",
    "\n",
    "When do we use anomaly detection and when do we use supervised learning?\n",
    "\n",
    "Use anomaly detection when...\n",
    "\n",
    "* We have a very small number of positive examples (y=1 ... 0-20 examples is common) and a large number of negative (y=0) examples.\n",
    "* We have many different \"types\" of anomalies and it is hard for any algorithm to learn from positive examples what the anomalies look like; future anomalies may look nothing like any of the anomalous examples we've seen so far.\n",
    "\n",
    "Use supervised learning when...\n",
    "\n",
    "* We have a large number of both positive and negative examples. In other words, the training set is more evenly divided into classes.\n",
    "* We have enough positive examples for the algorithm to get a sense of what new positives examples look like. The future positive examples are likely similar to the ones in the training set.\n",
    "\n",
    "\n",
    "## Choosing What Features to Use\n",
    "\n",
    "he features will greatly affect how well your anomaly detection algorithm works.\n",
    "\n",
    "We can check that our features are **gaussian** by plotting a histogram of our data and checking for the bell-shaped curve.\n",
    "\n",
    "Some **transforms** we can try on an example feature x that does not have the bell-shaped curve are:\n",
    "\n",
    "* log(x)\n",
    "* log(x+1)\n",
    "* log(x+c) for some constant\n",
    "* \\\\(\\sqrt{x}\\\\)\n",
    "* \\\\(x^{1/3}\\\\)\n",
    "\n",
    "We can play with each of these to try and achieve the gaussian shape in our data.\n",
    "\n",
    "There is an **error analysis procedure** for anomaly detection that is very similar to the one in supervised learning.\n",
    "\n",
    "Our goal is for p(x) to be large for normal examples and small for anomalous examples.\n",
    "\n",
    "One common problem is when p(x) is similar for both types of examples. In this case, you need to examine the anomalous examples that are giving high probability in detail and try to figure out new features that will better distinguish the data.\n",
    "\n",
    "In general, choose features that might take on unusually large or small values in the event of an anomaly.\n",
    "\n",
    "\n",
    "## Multivariate Gaussian Distribution (Optional)\n",
    "\n",
    "The multivariate gaussian distribution is an extension of anomaly detection and may (or may not) catch more anomalies.\n",
    "\n",
    "Instead of modeling \\\\(p(x_1),p(x_2),\\dots\\\\) separately, we will model p(x) all in one go. Our parameters will be: \\\\(\\mu \\in \\mathbb{R}^n\\\\)\n",
    "\n",
    "The important effect is that we can model gaussian contours, allowing us to better fit data that might not fit into the normal circular contours.\n",
    "\n",
    "Varying Σ changes the shape, width, and orientation of the contours. Changing μ will move the center of the distribution.\n",
    "\n",
    "\n",
    "## Anomaly Detection using the Multivariate Gaussian Distribution (Optional)\n",
    "\n",
    "When doing anomaly detection with multivariate gaussian distribution, we compute μ and Σ normally. We then compute p(x) using the new formula in the previous section and flag an anomaly if p(x) < ϵ.\n",
    "\n",
    "The original model for p(x) corresponds to a multivariate Gaussian where the contours of \\\\(p(x;\\mu,\\Sigma)\\\\) are axis-aligned.\n",
    "\n",
    "The **multivariate Gaussian model** can **automatically** capture correlations between different features of x, but computationally **expensive**. Also, this model required at m>n, where m is training set and n is number of features. If not, then Σ not invertible.\n",
    "\n",
    "However, the **original model** need to **manually** create features, but maintains some advantages: it is computationally **cheaper** (no matrix to invert, which is costly for large number of features) and it performs well even with small training set size (in multivariate Gaussian model, it should be greater than the number of features for Σ to be invertible)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}