6_load_data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Make Your Own Dataset\n",
    "=====================\n",
    "\n",
    "This tutorial assumes that you already know [the basics of training a\n",
    "GNN for node classification](1_introduction.ipynb) and [how to\n",
    "create, load, and store a DGL graph](2_dglgraph.ipynb).\n",
    "\n",
    "By the end of this tutorial, you will be able to\n",
    "\n",
    "-  Create your own graph dataset for node classification, link\n",
    "   prediction, or graph classification.\n",
    "\n",
    "(Time estimate: 15 minutes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``DGLDataset`` Object Overview\n",
    "------------------------------\n",
    "\n",
    "Your custom graph dataset should inherit the ``dgl.data.DGLDataset``\n",
    "class and implement the following methods:\n",
    "\n",
    "-  ``__getitem__(self, i)``: retrieve the ``i``-th example of the\n",
    "   dataset. An example often contains a single DGL graph, and\n",
    "   occasionally its label.\n",
    "-  ``__len__(self)``: the number of examples in the dataset.\n",
    "-  ``process(self)``: load and process raw data from disk.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a Dataset for Node Classification or Link Prediction from CSV\n",
    "----------------------------------------------------------------------\n",
    "\n",
    "A node classification dataset often consists of a single graph, as well\n",
    "as its node and edge features.\n",
    "\n",
    "This tutorial takes a small dataset based on [Zachary’s Karate Club\n",
    "network](https://en.wikipedia.org/wiki/Zachary%27s_karate_club). It\n",
    "contains\n",
    "\n",
    "* A ``members.csv`` file containing the attributes of all\n",
    "  members, as well as their attributes.\n",
    "\n",
    "* An ``interactions.csv`` file\n",
    "  containing the pair-wise interactions between two club members.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "import pandas as pd\n",
    "urllib.request.urlretrieve(\n",
    "    'https://data.dgl.ai/tutorial/dataset/members.csv', './members.csv')\n",
    "urllib.request.urlretrieve(\n",
    "    'https://data.dgl.ai/tutorial/dataset/interactions.csv', './interactions.csv')\n",
    "\n",
    "members = pd.read_csv('./members.csv')\n",
    "members.head()\n",
    "\n",
    "interactions = pd.read_csv('./interactions.csv')\n",
    "interactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial treats the members as nodes and interactions as edges. It\n",
    "takes age as a numeric feature of the nodes, affiliated club as the label\n",
    "of the nodes, and edge weight as a numeric feature of the edges.\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "    \n",
    "**Note**: The original Zachary’s Karate Club network does not have\n",
    "   member ages. The ages in this tutorial are generated synthetically\n",
    "   for demonstrating how to add node features into the graph for dataset\n",
    "   creation.\n",
    "\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "    \n",
    "**Note**: In practice, taking age directly as a numeric feature may\n",
    "   not work well in machine learning; strategies like binning or\n",
    "   normalizing the feature would work better. This tutorial directly\n",
    "   takes the values as-is for simplicity.\n",
    "\n",
    "</div>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dgl\n",
    "from dgl.data import DGLDataset\n",
    "import torch\n",
    "import os\n",
    "\n",
    "class KarateClubDataset(DGLDataset):\n",
    "    def __init__(self):\n",
    "        super().__init__(name='karate_club')\n",
    "        \n",
    "    def process(self):\n",
    "        nodes_data = pd.read_csv('./members.csv')\n",
    "        edges_data = pd.read_csv('./interactions.csv')\n",
    "        node_features = torch.from_numpy(nodes_data['Age'].to_numpy())\n",
    "        node_labels = torch.from_numpy(nodes_data['Club'].astype('category').cat.codes.to_numpy())\n",
    "        edge_features = torch.from_numpy(edges_data['Weight'].to_numpy())\n",
    "        edges_src = torch.from_numpy(edges_data['Src'].to_numpy())\n",
    "        edges_dst = torch.from_numpy(edges_data['Dst'].to_numpy())\n",
    "        \n",
    "        self.graph = dgl.graph((edges_src, edges_dst), num_nodes=nodes_data.shape[0])\n",
    "        self.graph.ndata['feat'] = node_features\n",
    "        self.graph.ndata['label'] = node_labels\n",
    "        self.graph.edata['weight'] = edge_features\n",
    "        \n",
    "        # If your dataset is a node classification dataset, you will need to assign\n",
    "        # masks indicating whether a node belongs to training, validation, and test set.\n",
    "        n_nodes = nodes_data.shape[0]\n",
    "        n_train = int(n_nodes * 0.6)\n",
    "        n_val = int(n_nodes * 0.2)\n",
    "        train_mask = torch.zeros(n_nodes, dtype=torch.bool)\n",
    "        val_mask = torch.zeros(n_nodes, dtype=torch.bool)\n",
    "        test_mask = torch.zeros(n_nodes, dtype=torch.bool)\n",
    "        train_mask[:n_train] = True\n",
    "        val_mask[n_train:n_train + n_val] = True\n",
    "        test_mask[n_train + n_val:] = True\n",
    "        self.graph.ndata['train_mask'] = train_mask\n",
    "        self.graph.ndata['val_mask'] = val_mask\n",
    "        self.graph.ndata['test_mask'] = test_mask\n",
    "        \n",
    "    def __getitem__(self, i):\n",
    "        return self.graph\n",
    "    \n",
    "    def __len__(self):\n",
    "        return 1\n",
    "\n",
    "dataset = KarateClubDataset()\n",
    "graph = dataset[0]\n",
    "\n",
    "print(graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since a link prediction dataset only involves a single graph, preparing\n",
    "a link prediction dataset will have the same experience as preparing a\n",
    "node classification dataset.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a Dataset for Graph Classification from CSV\n",
    "----------------------------------------------------\n",
    "\n",
    "Creating a graph classification dataset involves implementing\n",
    "``__getitem__`` to return both the graph and its graph-level label.\n",
    "\n",
    "This tutorial demonstrates how to create a graph classification dataset\n",
    "with the following synthetic CSV data:\n",
    "\n",
    "-  ``graph_edges.csv``: containing three columns:\n",
    "\n",
    "   -  ``graph_id``: the ID of the graph.\n",
    "   -  ``src``: the source node of an edge of the given graph.\n",
    "   -  ``dst``: the destination node of an edge of the given graph.\n",
    "\n",
    "-  ``graph_properties.csv``: containing three columns:\n",
    "\n",
    "   -  ``graph_id``: the ID of the graph.\n",
    "   -  ``label``: the label of the graph.\n",
    "   -  ``num_nodes``: the number of nodes in the graph.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urllib.request.urlretrieve(\n",
    "    'https://data.dgl.ai/tutorial/dataset/graph_edges.csv', './graph_edges.csv')\n",
    "urllib.request.urlretrieve(\n",
    "    'https://data.dgl.ai/tutorial/dataset/graph_properties.csv', './graph_properties.csv')\n",
    "edges = pd.read_csv('./graph_edges.csv')\n",
    "properties = pd.read_csv('./graph_properties.csv')\n",
    "\n",
    "edges.head()\n",
    "\n",
    "properties.head()\n",
    "\n",
    "class SyntheticDataset(DGLDataset):\n",
    "    def __init__(self):\n",
    "        super().__init__(name='synthetic')\n",
    "        \n",
    "    def process(self):\n",
    "        edges = pd.read_csv('./graph_edges.csv')\n",
    "        properties = pd.read_csv('./graph_properties.csv')\n",
    "        self.graphs = []\n",
    "        self.labels = []\n",
    "        \n",
    "        # Create a graph for each graph ID from the edges table.\n",
    "        # First process the properties table into two dictionaries with graph IDs as keys.\n",
    "        # The label and number of nodes are values.\n",
    "        label_dict = {}\n",
    "        num_nodes_dict = {}\n",
    "        for _, row in properties.iterrows():\n",
    "            label_dict[row['graph_id']] = row['label']\n",
    "            num_nodes_dict[row['graph_id']] = row['num_nodes']\n",
    "            \n",
    "        # For the edges, first group the table by graph IDs.\n",
    "        edges_group = edges.groupby('graph_id')\n",
    "        \n",
    "        # For each graph ID...\n",
    "        for graph_id in edges_group.groups:\n",
    "            # Find the edges as well as the number of nodes and its label.\n",
    "            edges_of_id = edges_group.get_group(graph_id)\n",
    "            src = edges_of_id['src'].to_numpy()\n",
    "            dst = edges_of_id['dst'].to_numpy()\n",
    "            num_nodes = num_nodes_dict[graph_id]\n",
    "            label = label_dict[graph_id]\n",
    "            \n",
    "            # Create a graph and add it to the list of graphs and labels.\n",
    "            g = dgl.graph((src, dst), num_nodes=num_nodes)\n",
    "            self.graphs.append(g)\n",
    "            self.labels.append(label)\n",
    "            \n",
    "        # Convert the label list to tensor for saving.\n",
    "        self.labels = torch.LongTensor(self.labels)\n",
    "        \n",
    "    def __getitem__(self, i):\n",
    "        return self.graphs[i], self.labels[i]\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.graphs)\n",
    "\n",
    "dataset = SyntheticDataset()\n",
    "graph, label = dataset[0]\n",
    "print(graph, label)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}