dos-gdc-lambda-12-2017.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GA4GH DOS - GDC Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data stored in the signpost service are issued identifiers and made available for use in the NCI Genomic Data Commons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In an effort to maintain an interoperability layer that is inclusive of all implementations of data access services, we offer the Data Object Service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Design"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```                                                                                         \n",
    "+------------------+      +--------------+        +-------------------+\n",
    "| ga4gh-dos-client |------|dos-gdc-lambda|--------|api.gdc.cancer.gov |\n",
    "+--------|---------+      +--------------+        +-------------------+\n",
    "         |                        |                                                         \n",
    "         |                        |                                                         \n",
    "         |------------------swagger.json                                                    \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this pilot we have created a lambda that creates a lightweight layer that can be used to access data in signpost using GA4GH libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is an example of using Python to [GDC API here](https://docs.gdc.cancer.gov/API/Users_Guide/Getting_Started/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lambda accepts GA4GH requests and converts them into requests against requisite signpost endpoints. The results are then translated into GA4GH style messages before being returned to the client."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make it easy for developers to create clients against this API, the Open API description is made available, which we will see later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initializing the DOS GA4GH Client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Note, the URLs are subject to change!\n",
    "\n",
    "We begin by initializing the client, which will access the lambda to get the swagger description and may take a moment.\n",
    "\n",
    "To install this client use `pip install git+git://github.com/david4096/data-object-schemas@dos-minimal2 --process-dependency-links`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ga4gh.dos.client import Client\n",
    "local_client = Client('https://dos-gdc.ucsc-cgp-dev.org/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience, we then initialize a few objects that will make it easier to use the DOS endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = local_client.client\n",
    "models = local_client.models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing data from GDC via GA4GH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have initialized the DOS client against the DOS-GDC lambda, we can access data using GA4GH methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Data Objects: 100 \n"
     ]
    }
   ],
   "source": [
    "list_request = client.ListDataObjects(page_size=100)\n",
    "list_response = list_request.result()\n",
    "print(\"Number of Data Objects: {} \".format(len(list_response.data_objects)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[URL(system_metadata=SystemMetadata(access=u'open', acl=[u'open'], created_datetime=u'2016-05-29T10:51:15.714019-05:00', data_category=u'Transcriptome Profiling', data_format=u'TXT', data_type=u'Gene Expression Quantification', experimental_strategy=u'RNA-Seq', file_id=u'0934a0b3-858d-430c-a5c9-ea459104a392', file_name=u'81cab804-bfb1-44f0-ac2d-923e0766ac61.htseq.counts.gz', file_size=249298, file_state=u'submitted', md5sum=u'5cbfed753ee5073d3a8fcd68818ed0a1', state=u'live', submitter_id=u'81cab804-bfb1-44f0-ac2d-923e0766ac61_count', type=u'gene_expression', updated_datetime=u'2017-03-04T16:39:14.644494-06:00'), url=u'https://api.gdc.cancer.gov/data/0934a0b3-858d-430c-a5c9-ea459104a392', user_metadata=None)]\n"
     ]
    }
   ],
   "source": [
    "data_object = client.GetDataObject(\n",
    "    data_object_id=list_response.data_objects[1].id).result().data_object\n",
    "print(data_object.urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading data using DOS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For publicly available data, we can quickly download the files using the DOS client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py\n",
    "import requests\n",
    "def download_file(url, filename):\n",
    "    # NOTE the stream=True parameter\n",
    "    r = requests.get(url, stream=True)\n",
    "    with open(filename, 'wb') as f:\n",
    "        for chunk in r.iter_content(chunk_size=1024): \n",
    "            if chunk: # filter out keep-alive new chunks\n",
    "                f.write(chunk)\n",
    "                #f.flush() commented by recommendation from J.F.Sebastian\n",
    "    return filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'0934a0b3-858d-430c-a5c9-ea459104a392'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "download_file(data_object.urls[0].url, data_object.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have an ERA Commons account and DBGap access you can generate a token from the [GDC portal](https://portal.gdc.cancer.gov/). \n",
    "\n",
    "Simply log in, click your profile information, and go to download credentials.\n",
    "\n",
    "Then, using a URL returned from DOS, you can download controlled access data by adding your key to the header. In this way, the lambda does not handle your key at any time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !curl -O -J -H \"X-Auth-Token: $token\" 'https://api.gdc.cancer.gov/data/a1c1b23b-cc41-4e85-b1b7-62a42873c5af'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verifying a checksum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have downloaded a file we can verify the checksum on that file against what is in the DOS record."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hashlib\n",
    "# https://stackoverflow.com/questions/3431825/generating-an-md5-checksum-of-a-file\n",
    "def md5(fname):\n",
    "    hash_md5 = hashlib.md5()\n",
    "    with open(fname, \"rb\") as f:\n",
    "        for chunk in iter(lambda: f.read(4096), b\"\"):\n",
    "            hash_md5.update(chunk)\n",
    "    return hash_md5.hexdigest()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5cbfed753ee5073d3a8fcd68818ed0a1\n",
      "5cbfed753ee5073d3a8fcd68818ed0a1\n"
     ]
    }
   ],
   "source": [
    "print(md5(data_object.id))\n",
    "print(data_object.checksums[0].checksum)\n",
    "# If these don't match you probably tried to download a controlled access file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}