Home

Exelixi is a distributed framework for running genetic algorithms at scale. The framework is based on Apache Mesos and the code is mostly implemented in Python.

Why build yet another framework for this purpose? Apache Hadoop would be quite a poor fit, due to requirements for in-memory iteration. Apache Spark could fit the problem more closely, in terms of iterative tasks. However, task overhead can become high in proportion to tasks being performed ("small file problem"), plus there is a considerable amount of configuration required at scale. Server-side operations and coprocessors in Apache Cassandra or Apache HBase might also provide a good fit for GA processing, but both of those also require lots of configuration. Moreover, many of the features for these more heavyweight frameworks are not needed, plus it helps to have some lightweight messaging available among the shards -- which these existing frameworks tend to lack.

On the one hand, Exelixi provides the basis for a tutorial for building distributed frameworks in Apache Mesos. On the other hand, it provides a general-purpose GA platform that emphasizes scalability and fault tolerance, while leveraging the wealth of available Python analytics packages.

Quick Start

More details are given below about customizing this framework for solving specific GA problems. The following instructions will help you get started quickly, running Exelixi either on Apache Mesos or in standalone mode.

Usage for Apache Mesos launch

First, launch an Apache Mesos cluster. The following instructions are based on using the Elastic Mesos service, which uses Ubuntu Linux servers running on Amazon AWS. However, the basic outline of steps should apply in the general case.

Once you have confirmation that your cluster is running -- Elastic Mesos sends you an email messages with a list of masters and slaves -- then use ssh to login on any of the masters:

ssh -A -l ubuntu <master-public-ip>

You must install the Python bindings for Apache Mesos, In this instance the Apache Mesos version is 0.14.0-rc4, so you must install the Python egg for that exact release. Also, you need to install the Exelixi source.

On the master, download the master branch of the Exelixi code repo on GitHub and install the required libraries:

wget https://github.com/ceteri/exelixi/archive/master.zip ; \
unzip master.zip ; \
cd exelixi-master ; \
./bin/local_install.sh

You can test the installation at any point simply by attempting to import the mesos package into Python:

python -c 'import mesos'

If there is no ImportError exception thrown, then your installation should be complete. Next, run the installation commands on each of the slaves:

python ./src/exelixi.py -n localhost:5050 | ./bin/install.sh

Great, ready to roll! Now launch the Framework, which in turn launches the Executors remotely on slave nodes. In the following case, it runs on two slave nodes:

python ./src/exelixi.py -m localhost:5050 -e 2

If everything runs successfully, the log should conclude with a final line:

all tasks done, and all messages received; exiting

See a GitHub gist for an example of a successful run.

Usage for Standalone Mode

To get started quickly on a single node (i.e., your laptop) in standalone mode simply follow two steps. First, launch one Executor locally:

nohup ./src/exelixi.py -p 9311 &

Then launch a Framework to run the default GA as an example:

./src/exelixi.py -s localhost:9311

Note that there are trade-offs regarding standalone mode. Pros: simple to test the customization of a GA quickly, without requiring an Apache Mesos cluster. Cons: difficult to configure and manage a GA in production at scale, since it lacks features for security and fault tolerance.