Experiments

Experiments

This guide describes how to run experiments with Linnea on the RWTH Aachen University cluster.

Installation

All necessary software can be installed with the setup_linnea_env.sh script that is located in the experiments directory. This script

• creates a suitable directory structure,
• installs Linnea in a virtual environment,
• installs Julia, linked against MKL,
• installs the MatrixGenerator package in Julia,
• installs Eigen,
• installs Armadillo,
• installs the MatrixGeneratorCpp library, and
• installs the MatrixGeneratorMatlab package.

In the first few lines of setup_linnea_env.sh, the paths are defined that determine the directory structure that gets created. In the following, we assume that the default values are used. If those paths are changed, the paths in the config file have to be changed accordingly.

Generation of Jobscripts

The jobscripts to run the experiments are generated as follows:

1. Go to the directory that contains the virtual environment (default is ~/Linnea) and activate it with source linnea.venv/bin/activate
2. Go to the experiments directory of the Linnea repository (by default, the full path is ~/Linnea/src/linnea/experiments).
3. Run python3 experiments.py jobscripts random and/or python3 experiments.py jobscripts application to generate the jobscripts for the random and/or application test cases, respectively.
4. By default, the jobscripts are written to ~/Linnea/output/jobscripts/random and ~/Linnea/output/jobscripts/application.

Several values in the generated jobscripts are determined by the Linnea config file. Examples of this config file for the Slurm and LSF job submission systems are located in the examples/experiments directory in the Linnea repository. The config file has to be in a location where Linnea can find it; either in the current directory, or the users home directory, and it needs to have the name linnea_config.json.

For each set of test cases ("random" and "application"), the following jobscripts are generated:

• generate_code.sh: Generates all code to time the execution (including the reference code, but excluding the "k best" experiments).
• generate_code_ref.sh: Only generates the reference code, that is, the naive and recommended implementations, as well as the runner and operand generator files. This script can be used if Linnea was updated and only the generation of the reference code was changed.
• generate_code_k_best.sh: Generates the code for the "k best" experiments.
• time_generation_{m,nm}.sh: Measures the execution time of the code generation with merging (m) or without (nm).
• time_{julia,cpp,matlab,k_best}_t{1,24}.sh: Measures the execution time of the Julia code (julia, excluding "k best"), the C++ code (cpp), the Matlab code (matlab), or the "k best" experiments (k_best), with 1 or 24 threads.

All generated jobscripts are array jobs, with one job per test case.

Processing the Generated Data

By default, the raw data generated by the experiments is stored in the ~/Linnea/output/results directory in several subdirectories. This data can be processed with the process_data.py script in the experiments directory of the Linnea repository. This script has to be run in the results directory and produces several CSV files. For this script to function correctly, it is necessary that all experiments have been run. If some files are missing, warning messages are printed. However, if entire experiments are missing, the script may fail to run.

Custom Experiments

Several jobscripts make use of the experiments.py script. This script does not only generate jobscripts, it is also used by the jobscripts to generate code with Linnea or measure the execution time of the code generation. It can also be used for custom experiments. The possible options can be displayed with python3 experiments.py -h. To generate code or time the generation only for a single test case, the -j flag can be used. As an example, to generate the code for the "k best" experiment with the second application test problem, one can use python3 experiments.py generate_code application -j 2 --k-best. If the -j flag is not used, all test problems from the respective test set are used.