03_summary_statistics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/MIT-LCP/hack-aotearoa/blob/master/03_summary_statistics.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1G_TVh1ybQkl"
   },
   "source": [
    "# eICU Collaborative Research Database\n",
    "\n",
    "# Notebook 3: Summary statistics\n",
    "\n",
    "This notebook shows how summary statistics can be computed for a patient cohort using the `tableone` package. Usage instructions for tableone are at: https://pypi.org/project/tableone/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L9XF77F2bnee"
   },
   "source": [
    "## Load libraries and connect to the database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wXiSE558bn_w"
   },
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Make pandas dataframes prettier\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "# Access data using Google BigQuery.\n",
    "from google.colab import auth\n",
    "from google.cloud import bigquery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pLGnLAy-bsKb"
   },
   "outputs": [],
   "source": [
    "# authenticate\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PUjFDFdobszs"
   },
   "outputs": [],
   "source": [
    "# Set up environment variables\n",
    "project_id='hack-aotearoa'\n",
    "os.environ[\"GOOGLE_CLOUD_PROJECT\"]=project_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iWDUCA5Nb5BK"
   },
   "source": [
    "## Install and load the `tableone` package\n",
    "\n",
    "The tableone package can be used to compute summary statistics for a patient cohort. Unlike the previous packages, it is not installed by default in Colab, so will need to install it first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "F9doCgtscOJd"
   },
   "outputs": [],
   "source": [
    "!pip install tableone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SDI_Q7W0b4Le"
   },
   "outputs": [],
   "source": [
    "# Import the tableone class\n",
    "from tableone import TableOne"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "14TU4lcrdD7I"
   },
   "source": [
    "## Load the patient cohort\n",
    "\n",
    "In this example, we will load all data from the patient data, and link it to APACHE data to provide richer summary information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HF5WF5EObwfw"
   },
   "outputs": [],
   "source": [
    "# Link the patient and apachepatientresult tables on patientunitstayid\n",
    "# using an inner join.\n",
    "%%bigquery cohort\n",
    "\n",
    "SELECT p.unitadmitsource, p.gender, p.age, p.ethnicity, p.admissionweight, \n",
    "    p.unittype, p.unitstaytype, a.acutephysiologyscore,\n",
    "    a.apachescore, a.actualiculos, a.actualhospitalmortality,\n",
    "    a.unabridgedunitlos, a.unabridgedhosplos\n",
    "FROM `physionet-data.eicu_crd_demo.patient` p\n",
    "INNER JOIN `physionet-data.eicu_crd_demo.apachepatientresult` a\n",
    "ON p.patientunitstayid = a.patientunitstayid\n",
    "WHERE apacheversion LIKE 'IVa'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k3hURHFihHNA"
   },
   "outputs": [],
   "source": [
    "cohort.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qnG8dVb2iHSn"
   },
   "source": [
    "## Calculate summary statistics\n",
    "\n",
    "Before summarizing the data, we will need to convert the ages to numerical values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oKHpqwAPkx6U"
   },
   "outputs": [],
   "source": [
    "cohort['agenum'] = pd.to_numeric(cohort['age'], errors='coerce')\n",
    "\n",
    "# temporary bug fix: deal with data type issue!\n",
    "cohort = cohort.astype({\"apachescore\": \"int64\",\n",
    "                        \"actualiculos\": \"int64\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FQT-u8EXhXRG"
   },
   "outputs": [],
   "source": [
    "columns = ['unitadmitsource', 'gender', 'agenum', 'ethnicity',\n",
    "          'admissionweight','unittype','unitstaytype',\n",
    "          'acutephysiologyscore','apachescore','actualiculos',\n",
    "          'unabridgedunitlos','unabridgedhosplos']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3ETr3NCzielL"
   },
   "outputs": [],
   "source": [
    "TableOne(cohort, columns=columns, labels={'agenum': 'age'}, \n",
    "         groupby='actualhospitalmortality',\n",
    "         label_suffix=True, limit=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LCBcpJ9bZpDp"
   },
   "source": [
    "## Questions\n",
    "\n",
    "- Are the severity of illness measures higher in the survival or non-survival group?\n",
    "- What issues suggest that some of the summary statistics might be misleading?\n",
    "- How might you address these issues?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2_8z1CIVahWg"
   },
   "source": [
    "## Visualizing the data\n",
    "\n",
    "Plotting the distribution of each variable by group level via histograms, kernel density estimates and boxplots is a crucial component to data analysis pipelines. Vizualisation is often is the only way to detect problematic variables in many real-life scenarios. We'll review a couple of the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "81yp2bSUigzh"
   },
   "outputs": [],
   "source": [
    "# Plot distributions to review possible multimodality\n",
    "cohort[['acutephysiologyscore','agenum']].dropna().plot.kde(figsize=[12,8])\n",
    "plt.legend(['APS Score', 'Age (years)'])\n",
    "plt.xlim([-30,250])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kZDUZB5sdhhU"
   },
   "source": [
    "## Questions\n",
    "\n",
    "- Do the plots change your view on how these variable should be reported?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "03-summary-statistics",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}