03_deg_dist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Trade network analysis\n",
    "**Brian Dew (brianwdew@gmail.com)**\n",
    "\n",
    "`03_deg_dist.ipynb`\n",
    "\n",
    "This notebook estimates an alpha value for the degree distribution for each product in each year.\n",
    "\n",
    "The distribution of weighted indegree and outdegree is estimated as $p(x) = Cx^{-\\alpha}$.\n",
    "\n",
    "Powerlaw follows the methods described in [Clauset et al (2009)](https://arxiv.org/abs/0706.1062). \n",
    "\n",
    "---\n",
    "\n",
    "METODO:\n",
    "\n",
    "1. Add description of weighted outdegree centrality and what it represents!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# uses panadas, networkx, powerlaw, and os packages\n",
    "import pandas as pd\n",
    "import networkx as nx\n",
    "import powerlaw\n",
    "import os\n",
    "os.chdir('C:/Working/trade_network/data/')        # Change to working directory\n",
    "if not os.path.exists( 'summary/.'):              # Creates folder in directory if missing.\n",
    "    os.makedirs('summary/.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define alpha estimation function\n",
    "\n",
    "This function uses a product ID, `prod` as an input. It builds a `networkx` network for the product, calculates the `out_degree` weighted to the USD value of trade `v` (in thousands), and finally estimates how well the weighted `out_degree` scores fit a power law distribution. The output is an alpha score for the product. \n",
    "\n",
    "Weighted outdegree:\n",
    "\n",
    "Each country's USD export volume for each product as a share of total exports of all countries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Define function called deg_dist which calculates alpha value of degree distribution.\n",
    "def deg_dist(prod):\n",
    "    \"Calculates the degree distribution for a product\"\n",
    "    try:\n",
    "        G = nx.from_pandas_dataframe(df.loc[prod], 'i', 'j', 'v', nx.DiGraph())  #build network\n",
    "        deg = G.out_degree(weight='v').values()         # calc weighted outdeg for each country\n",
    "        fit = powerlaw.Fit(deg)                          # est. distrib.\\\n",
    "        pl_alpha[prod] = fit.power_law.alpha                   # Estimated alpha\n",
    "        pl_sigma[prod] = fit.power_law.sigma                   # Estimated standard deviation\n",
    "    except Exception:                                   # some products don't converge\n",
    "        pass \n",
    "    return;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply function to all data\n",
    "\n",
    "The fucntion defined above is applied to data for each year `y` in the range. The power law distribution is recorded for each product in a dictionary called `pl_alpha` which is saved as a separate csv file for each year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%capture \n",
    "for y in map(str, range(2008,2015)):         # start year & end year + 1 \n",
    "    # read csv file for year\n",
    "    df = pd.read_csv('clean/baci07_' + y + '_clean.csv', index_col='hs6', header=0).sort_index()\n",
    "    df = df[['i','j','v']]         # take only relevant columns\n",
    "    pl_alpha = {}                        # blank dictionary\n",
    "    pl_sigma = {}\n",
    "    map(deg_dist,df.index.unique()) # This runs the program above\n",
    "    pl_alpha = pd.Series(pl_alpha)            \n",
    "    pl_sigma = pd.Series(pl_sigma)\n",
    "    combined = pd.concat([pl_alpha, pl_sigma], axis=1) # One series from all dictionaries\n",
    "    # Save as csv\n",
    "    combined.to_csv('summary/deg_dist_alpha_' + y + '.csv', index=True, float_format='%g')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dummy csv\n",
    "\n",
    "Creates a csv file that lists each product that has a fat-tailed centrality score (alpha greater than 2). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fat_tailed_prod = {}\n",
    "for y in range(2008,2015):\n",
    "    prod = pd.read_csv('summary/deg_dist_alpha_'+str(y)+'.csv').set_index('Unnamed: 0')\n",
    "    fat_tailed_prod[y] = prod[prod['0'] > 1.999]                              # fat tail is alpha above 2\n",
    "fat_tailed_prod = pd.concat(fat_tailed_prod, axis=0).reset_index()\n",
    "fat_tailed_prod['c'] = 1                                                # to merge and create dummy\n",
    "fat_tailed_prod.columns = ['year', 'hs6', 'alpha','sigma', 'c']\n",
    "fat_tailed_prod[['year','hs6','c']].to_csv('fat_tailed_prod.csv', index=None)    # save as csv file"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}