how-to-programmatically-generate-and-execute-an-ipython-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to programmatically generate and execute an IPython notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a response to [this question on StackOverflow](http://stackoverflow.com/questions/22328052/ipython-notebook-programatically-read-and-execute-cells). It illustrates how to generate an IPython notebook programmatically by generating markdown and code cells on the fly, and how to execute the resulting notebook.\n",
    "\n",
    "The automatically generated notebook which this code produces can be seen [here](http://nbviewer.ipython.org/github/maxalbert/auto-exec-notebook/blob/master/executed_notebook_with_freq_5.0.ipynb).\n",
    "\n",
    "This code has been tested with IPython version 3.1. It may not work with different versions, since the `nbconvert` machinery is in constant flux, and the `nbconvert` is currently being moved to a [separate Jupyter repository](https://github.com/jupyter/jupyter_nbconvert) anyway."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using IPython version 3.1.0\n"
     ]
    }
   ],
   "source": [
    "import IPython\n",
    "print(\"Using IPython version {}\".format(IPython.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating a new notebook with some code cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an empty notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we create a new IPython notebook using the current nbformat version 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython import nbformat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb = nbformat.v4.new_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notebook contains an attribute `nb.cells` which initially is an emtpy list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb.cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we could have loaded the notebook from an existing `.ipynb` file via:\n",
    "\n",
    "    nb = nbformat.read(filename, as_version=4)\n",
    "\n",
    "In this case `nb.cells` would be a list containing the existing cells. We could then append/insert new cells into this list, or even overwrite existing cells with new content by setting `nb.cells[i] = ...`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a markdown cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin filling the list of cells by generating a markdown cell and appending it to `nb.cells`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "markdown_cell = nbformat.v4.new_markdown_cell(\"This notebook was automatically generated.\")\n",
    "\n",
    "nb.cells.append(markdown_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'cell_type': 'markdown',\n",
       "  'metadata': {},\n",
       "  'source': 'This notebook was automatically generated.'}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb.cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a couple of code cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we generate a code cell which simply prints \"Hello world\", and also append it to the list of cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "source_1 = \"print('Hello world')\"\n",
    "code_cell_1 = nbformat.v4.new_code_cell(source=source_1)\n",
    "nb.cells.append(code_cell_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that when printing the list of cells, the `outputs` field of the code cell is currently empty because the cell has not been executed yet (this will happen below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'cell_type': 'markdown',\n",
       "  'metadata': {},\n",
       "  'source': 'This notebook was automatically generated.'},\n",
       " {'cell_type': 'code',\n",
       "  'execution_count': None,\n",
       "  'metadata': {},\n",
       "  'outputs': [],\n",
       "  'source': \"print('Hello world')\"}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb.cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating multi-line code cells and injecting values on-the-fly works pretty much the same way. The following creates two code cells which define a variable `frequency` and display a sine wave with this frequency using matplotlib.\n",
    "\n",
    "Note that we inject the frequency value into the source of the code cell on the fly. In a real-world example this would allow us to produce a parameter study with one automatically generated notebook documenting the outcome for each parameter value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import textwrap\n",
    "\n",
    "freq = 5.0  # frequency of the sine wave to be displayed by the code cells below\n",
    "\n",
    "source_2 = \"frequency = {:.1f}\".format(freq)\n",
    "\n",
    "source_3 = textwrap.dedent(\"\"\"\\\n",
    "    %matplotlib inline\n",
    "    import matplotlib.pyplot as plt\n",
    "    import numpy as np\n",
    "\n",
    "    x = np.linspace(0, 2*np.pi, 200)\n",
    "    y = np.sin(frequency*x)\n",
    "\n",
    "    plt.plot(x, y)\n",
    "    \"\"\")\n",
    "\n",
    "code_cell_2 = nbformat.v4.new_code_cell(source=source_2)\n",
    "code_cell_3 = nbformat.v4.new_code_cell(source=source_3)\n",
    "\n",
    "nb.cells.append(code_cell_2)\n",
    "nb.cells.append(code_cell_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'cell_type': 'markdown',\n",
       "  'metadata': {},\n",
       "  'source': 'This notebook was automatically generated.'},\n",
       " {'cell_type': 'code',\n",
       "  'execution_count': None,\n",
       "  'metadata': {},\n",
       "  'outputs': [],\n",
       "  'source': \"print('Hello world')\"},\n",
       " {'cell_type': 'code',\n",
       "  'execution_count': None,\n",
       "  'metadata': {},\n",
       "  'outputs': [],\n",
       "  'source': 'frequency = 5.0'},\n",
       " {'cell_type': 'code',\n",
       "  'execution_count': None,\n",
       "  'metadata': {},\n",
       "  'outputs': [],\n",
       "  'source': '%matplotlib inline\\nimport matplotlib.pyplot as plt\\nimport numpy as np\\n\\nx = np.linspace(0, 2*np.pi, 200)\\ny = np.sin(frequency*x)\\n\\nplt.plot(x, y)\\n'}]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb.cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look at the JSON code generated so far:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      " \"cells\": [\n",
      "  {\n",
      "   \"cell_type\": \"markdown\",\n",
      "   \"metadata\": {},\n",
      "   \"source\": [\n",
      "    \"This notebook was automatically generated.\"\n",
      "   ]\n",
      "  },\n",
      "  {\n",
      "   \"cell_type\": \"code\",\n",
      "   \"execution_count\": null,\n",
      "   \"metadata\": {},\n",
      "   \"outputs\": [],\n",
      "   \"source\": [\n",
      "    \"print('Hello world')\"\n",
      "   ]\n",
      "  },\n",
      "  {\n",
      "   \"cell_type\": \"code\",\n",
      "   \"execution_count\": null,\n",
      "   \"metadata\": {},\n",
      "   \"outputs\": [],\n",
      "   \"source\": [\n",
      "    \"frequency = 5.0\"\n",
      "   ]\n",
      "  },\n",
      "  {\n",
      "   \"cell_type\": \"code\",\n",
      "   \"execution_count\": null,\n",
      "   \"metadata\": {},\n",
      "   \"outputs\": [],\n",
      "   \"source\": [\n",
      "    \"%matplotlib inline\\n\",\n",
      "    \"import matplotlib.pyplot as plt\\n\",\n",
      "    \"import numpy as np\\n\",\n",
      "    \"\\n\",\n",
      "    \"x = np.linspace(0, 2*np.pi, 200)\\n\",\n",
      "    \"y = np.sin(frequency*x)\\n\",\n",
      "    \"\\n\",\n",
      "    \"plt.plot(x, y)\\n\"\n",
      "   ]\n",
      "  }\n",
      " ],\n",
      " \"metadata\": {},\n",
      " \"nbformat\": 4,\n",
      " \"nbformat_minor\": 0\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(nbformat.v4.writes_json(nb))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to share this notebook (or inspect it for debugging), we can write it out to a `.ipynb` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('automatically_generated_notebook.ipynb', 'w') as f:\n",
    "    f.write(nbformat.v4.writes_json(nb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "automatically_generated_notebook.ipynb\r\n"
     ]
    }
   ],
   "source": [
    "!ls automatically_generated_notebook.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Executing the newly generated notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to execute the notebook `nb` which we generated above we first need to create a new instance of `ExecutePreprocessor`. Here we set a timeout for execution of code cells of 30 seconds. If you have very long-running cells you may want to increase this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.nbconvert.preprocessors.execute import ExecutePreprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pp = ExecutePreprocessor()\n",
    "pp.timeout = 30  # seconds\n",
    "pp.interrupt_on_timeout = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can execute the notebook by calling `pp.preprocess()`. I'm not 100% sure what the role of the `resources` dictionary is, but it needs to be supplied or else `preprocess()` will produce an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nb_executed, resources = pp.preprocess(nb, resources={})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write the executed notebook to a file. Note that we include the value of `frequency` in the filename. This may be useful if we'd like to run the same \"template\" notebook for multiple values of `frequency`, e.g. for a parameter study with automatically generated reports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filename of executed notebook: 'executed_notebook_with_freq_5.0.ipynb'\n"
     ]
    }
   ],
   "source": [
    "filename = 'executed_notebook_with_freq_{:.1f}.ipynb'.format(freq)\n",
    "print(\"Filename of executed notebook: '{}'\".format(filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(filename, 'w') as f:\n",
    "    f.write(nbformat.v4.writes_json(nb_executed))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a look at the first few lines of the saved `.ipynb` file, we notice that the `outputs` field of the code cells have now been filled with the computed results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      " \"cells\": [\r\n",
      "  {\r\n",
      "   \"cell_type\": \"markdown\",\r\n",
      "   \"metadata\": {},\r\n",
      "   \"source\": [\r\n",
      "    \"This notebook was automatically generated.\"\r\n",
      "   ]\r\n",
      "  },\r\n",
      "  {\r\n",
      "   \"cell_type\": \"code\",\r\n",
      "   \"execution_count\": 1,\r\n",
      "   \"metadata\": {},\r\n",
      "   \"outputs\": [\r\n",
      "    {\r\n",
      "     \"name\": \"stdout\",\r\n",
      "     \"output_type\": \"stream\",\r\n",
      "     \"text\": [\r\n",
      "      \"Hello world\\n\"\r\n",
      "     ]\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 20 executed_notebook_with_freq_5.0.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the contents of the automatically executed notebook, see generated notebook, see [here](http://nbviewer.ipython.org/github/maxalbert/auto-exec-notebook/blob/master/executed_notebook_with_freq_5.0.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}