refs #2: Add more contents in docs/gsg

docs/gsg/build.rst

@@ -7,8 +7,8 @@ Supported Platforms
 Currently NBA is only tested on Linux x86_64 3.10 or newer kernels,
 and the Ubuntu 14.04 LTS distribution.
 
-Step-by-step Guide for Ubuntu 14.04 LTS
----------------------------------------
+Step-by-step Guide
+------------------
 
 **Software packages to install**
 
@@ -19,9 +19,9 @@ Check out the latest DPDK source tree:
 
 .. code-block:: console
 
-   ~$ git clone git://dpdk.org/dpdk
-   ~$ cd dpdk
-   ~/dpdk$ make install T=x86_64-native-linuxapp-gcc
+   $ git clone git://dpdk.org/dpdk
+   $ cd dpdk
+   $ make install T=x86_64-native-linuxapp-gcc
 
 .. note::
 
@@ -50,14 +50,14 @@ Clone the project source code:
 
 .. code-block:: console
 
-   ~$ git clone https://github.com/anlab-kaist/NBA nba
+   $ git clone https://github.com/anlab-kaist/NBA nba
 
 Install our 3rd-party libraries, the Click configuration parser:
 
 .. code-block:: console
 
-   ~$ cd nba
-   ~/nba$ git submodule init && git submodule update
+   $ cd nba
+   $ git submodule init && git submodule update
 
 .. note::
 
@@ -66,14 +66,75 @@ Install our 3rd-party libraries, the Click configuration parser:
 
 **Compilation**
 
-* Set the environment variable as follows:
+Set the environment variable as follows:
 
 .. code-block:: console
 
-  ~/nba$ export NBA_DPDK_PATH=/home/userid/dpdk/x86_64-native-linuxapp-gcc
-  ~/nba$ snakemake
+  $ export NBA_DPDK_PATH=/home/userid/dpdk/x86_64-native-linuxapp-gcc
+  $ export USE_CUDA=0  # for testing CPU-only version without CUDA installation
+  $ snakemake -j
 
-* If all is well, the executable is located in :code:`bin/main`.
+If all is well, the executable is located in :code:`bin/main`.
+
+Network Cards
+-------------
+
+Intel X520 Series (82599 chipset)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There is no other work required to use these network cards as they are natively
+supported by DPDK.
+
+You just need to bind the PCI addresses of network cards to igb_uio using
+:code:`tools/dpdk_nic_bind.py` script provided by DPDK.
+
+Mellanox ConnectX Series
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+You need to install the OFED toolchain provided by Mellanox because DPDK's mlx4
+poll-mode driver uses Mellanox's kernel Infiniband driver to control the
+hardware and perform DMA.
+We recommend to use version 3.0 or later, as these new versions have much
+better performance and includes firmware updates.
+
+To use mlx4_pmd on DPDK, turn on it inside the DPDK compile configuration (edit
+:code:`configs/common_linuxapp` before :code:`make config` / :code:`make install`):
+
+.. code-block:: properties
+
+   CONFIG_RTE_LIBRTE_MLX4_PMD=y
+
+To increase throughputs, set the following in the same config:
+
+.. code-block:: properties
+
+   CONFIG_RTE_LIBRTE_MLX4_SGE_WR_N=1
+
+For maximum throughputs, turn off the followings:
+
+* blueflame: :code:`sudo ethtool --set-priv-flags ethXX blueflame off`
+* rx/tx auto-negotiation for flow control: :code:`sudo ethtool -A ethXX rx off tx off`
+
+Note that above settings must be done in packet generators as well.
+
+.. warning::
+   We recommend to turn off blueflame when loading the mlx4_core kernel module
+   as module parameters, instead of using ethtool afterwards.
+
+.. note::
+   "blueflame" is a Mellanox-specific feature that uses PCI BAR for tranferring
+   descriptors of small packets instead of using DMA on RX/TX rings.  It is
+   known to have lower latency, but causes throughput degradation with NBA.
+
+You do not need to explicitly bind the PCI addresses of Mellanox cards to
+igb_uio because mlx4_pmd automatically detects them using the kernel driver.
+
+To use mlx4 in NBA, set the following environment variable and rebuild:
+
+.. code-block:: console
+
+   $ export NBA_PMD=mlx4
+   $ snakemake clean && snakemake -j
 
 Optional Installations
 ----------------------
@@ -86,16 +147,23 @@ We recommend to download the latest version of :code:`.bin` package from `the NV
 
 .. note::
 
-  A small daemon is required to "pin" GPU's interrupts to specific cores.
-  See details in `our gist <https://gist.github.com/3404967>`_.
+   A small daemon is required to "pin" GPU's interrupts to specific cores.
+   See details in `our gist <https://gist.github.com/3404967>`_.
 
 Make CUDA binaries accessible from your shell:
 
 .. code-block:: console
 
-  $ echo 'export PATH="$PATH:/usr/local/cuda/bin"' >> ~/.profile
-  $ sudo sh -c 'echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda.conf'
-  $ sudo ldconfig
+   $ echo 'export PATH="$PATH:/usr/local/cuda/bin"' >> ~/.profile
+   $ sudo sh -c 'echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda.conf'
+   $ sudo ldconfig
+
+To use CUDA in NBA, do:
+
+.. code-block:: console
+
+   $ export USE_CUDA=1
+   $ snakemake clean && snakemake -j
 
 CPU statistics
 ~~~~~~~~~~~~~~
@@ -111,6 +179,7 @@ Our build script offers a few configurable parameters as environment variables:
 * :code:`NBA_DPDK_PATH`: specifies the path to Intel DPDK (required)
 * :code:`NBA_RANDOM_PORT_ACCESS`: randomizes the RX queue scanning order for each worker thread (default: :code:`false`)
 * :code:`NBA_OPENSSL_PATH`: specifies the path of OpenSSL library (default: :code:`/usr`)
+* :code:`DEBUG`: build without compiler optimization (default: 0)
 * :code:`USE_CUDA`: activates NVIDIA CUDA support (default: 1)
 * :code:`USE_PHI`: activates Intel Xeon Phi support (default: 0, not fully implemented yet)
 * :code:`USE_NVPROF`: activates nvprof API calls to track GPU-related timings (default: 0)

docs/gsg/running.rst

@@ -4,7 +4,7 @@ Running NBA
 Standalone Execution
 --------------------
 
-Execute `bin/main` with DPDK EAL arguments and NBA arguments.  
+Execute :code:`bin/main` with DPDK EAL arguments and NBA arguments.  
 For example,
 
 .. code-block:: console

docs/gsg/sys_reqs.rst

@@ -4,7 +4,18 @@ System Requirements
 Supported Hardware
 ------------------
 
+NBA can run on most Intel servers with supported NICs,
+but we recommend Sandy Bridge or newer generations of Intel CPUs
+since integrated memory controller and integrated PCIe controller is a
+important performance booster.
+
+For GPU acceleration, the current version only supports NVIDIA CUDA GPUs,
+either desktop class (GeForce) or server class (Tesla) models.
+
 BIOS Settings
 -------------
 
+* If available, turn off Intel EIST technology in BIOS.
 
+* You need to fix the clock speed of individual CPU cores on Haswell systems
+  (Xeon E5-XXXXv3) for accurate timing control and performance measurements.