plot_outlier_detection_housing.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Outlier detection on a real data set\n\n\nThis example illustrates the need for robust covariance estimation\non a real data set. It is useful both for outlier detection and for\na better understanding of the data structure.\n\nWe selected two sets of two variables from the Boston housing data set\nas an illustration of what kind of analysis can be done with several\noutlier detection tools. For the purpose of visualization, we are working\nwith two-dimensional examples, but one should be aware that things are\nnot so trivial in high-dimension, as it will be pointed out.\n\nIn both examples below, the main result is that the empirical covariance\nestimate, as a non-robust one, is highly influenced by the heterogeneous\nstructure of the observations. Although the robust covariance estimate is\nable to focus on the main mode of the data distribution, it sticks to the\nassumption that the data should be Gaussian distributed, yielding some biased\nestimation of the data structure, but yet accurate to some extent.\nThe One-Class SVM does not assume any parametric form of the data distribution\nand can therefore model the complex shape of the data much better.\n\nFirst example\n-------------\nThe first example illustrates how robust covariance estimation can help\nconcentrating on a relevant cluster when another one exists. Here, many\nobservations are confounded into one and break down the empirical covariance\nestimation.\nOf course, some screening tools would have pointed out the presence of two\nclusters (Support Vector Machines, Gaussian Mixture Models, univariate\noutlier detection, ...). But had it been a high-dimensional example, none\nof these could be applied that easily.\n\nSecond example\n--------------\nThe second example shows the ability of the Minimum Covariance Determinant\nrobust estimator of covariance to concentrate on the main mode of the data\ndistribution: the location seems to be well estimated, although the covariance\nis hard to estimate due to the banana-shaped distribution. Anyway, we can\nget rid of some outlying observations.\nThe One-Class SVM is able to capture the real data structure, but the\ndifficulty is to adjust its kernel bandwidth parameter so as to obtain\na good compromise between the shape of the data scatter matrix and the\nrisk of over-fitting the data.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(__doc__)\n\n# Author: Virgile Fritsch <virgile.fritsch@inria.fr>\n# License: BSD 3 clause\n\nimport numpy as np\nfrom sklearn.covariance import EllipticEnvelope\nfrom sklearn.svm import OneClassSVM\nimport matplotlib.pyplot as plt\nimport matplotlib.font_manager\nfrom sklearn.datasets import load_boston\n\n# Get data\nX1 = load_boston()['data'][:, [8, 10]]  # two clusters\nX2 = load_boston()['data'][:, [5, 12]]  # \"banana\"-shaped\n\n# Define \"classifiers\" to be used\nclassifiers = {\n    \"Empirical Covariance\": EllipticEnvelope(support_fraction=1.,\n                                             contamination=0.261),\n    \"Robust Covariance (Minimum Covariance Determinant)\":\n    EllipticEnvelope(contamination=0.261),\n    \"OCSVM\": OneClassSVM(nu=0.261, gamma=0.05)}\ncolors = ['m', 'g', 'b']\nlegend1 = {}\nlegend2 = {}\n\n# Learn a frontier for outlier detection with several classifiers\nxx1, yy1 = np.meshgrid(np.linspace(-8, 28, 500), np.linspace(3, 40, 500))\nxx2, yy2 = np.meshgrid(np.linspace(3, 10, 500), np.linspace(-5, 45, 500))\nfor i, (clf_name, clf) in enumerate(classifiers.items()):\n    plt.figure(1)\n    clf.fit(X1)\n    Z1 = clf.decision_function(np.c_[xx1.ravel(), yy1.ravel()])\n    Z1 = Z1.reshape(xx1.shape)\n    legend1[clf_name] = plt.contour(\n        xx1, yy1, Z1, levels=[0], linewidths=2, colors=colors[i])\n    plt.figure(2)\n    clf.fit(X2)\n    Z2 = clf.decision_function(np.c_[xx2.ravel(), yy2.ravel()])\n    Z2 = Z2.reshape(xx2.shape)\n    legend2[clf_name] = plt.contour(\n        xx2, yy2, Z2, levels=[0], linewidths=2, colors=colors[i])\n\nlegend1_values_list = list(legend1.values())\nlegend1_keys_list = list(legend1.keys())\n\n# Plot the results (= shape of the data points cloud)\nplt.figure(1)  # two clusters\nplt.title(\"Outlier detection on a real data set (boston housing)\")\nplt.scatter(X1[:, 0], X1[:, 1], color='black')\nbbox_args = dict(boxstyle=\"round\", fc=\"0.8\")\narrow_args = dict(arrowstyle=\"->\")\nplt.annotate(\"several confounded points\", xy=(24, 19),\n             xycoords=\"data\", textcoords=\"data\",\n             xytext=(13, 10), bbox=bbox_args, arrowprops=arrow_args)\nplt.xlim((xx1.min(), xx1.max()))\nplt.ylim((yy1.min(), yy1.max()))\nplt.legend((legend1_values_list[0].collections[0],\n            legend1_values_list[1].collections[0],\n            legend1_values_list[2].collections[0]),\n           (legend1_keys_list[0], legend1_keys_list[1], legend1_keys_list[2]),\n           loc=\"upper center\",\n           prop=matplotlib.font_manager.FontProperties(size=12))\nplt.ylabel(\"accessibility to radial highways\")\nplt.xlabel(\"pupil-teacher ratio by town\")\n\nlegend2_values_list = list(legend2.values())\nlegend2_keys_list = list(legend2.keys())\n\nplt.figure(2)  # \"banana\" shape\nplt.title(\"Outlier detection on a real data set (boston housing)\")\nplt.scatter(X2[:, 0], X2[:, 1], color='black')\nplt.xlim((xx2.min(), xx2.max()))\nplt.ylim((yy2.min(), yy2.max()))\nplt.legend((legend2_values_list[0].collections[0],\n            legend2_values_list[1].collections[0],\n            legend2_values_list[2].collections[0]),\n           (legend2_keys_list[0], legend2_keys_list[1], legend2_keys_list[2]),\n           loc=\"upper center\",\n           prop=matplotlib.font_manager.FontProperties(size=12))\nplt.ylabel(\"% lower status of the population\")\nplt.xlabel(\"average number of rooms per dwelling\")\n\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}