Skip to content
This repository has been archived by the owner on Nov 9, 2023. It is now read-only.

Setup of Abel Computing Cluster

Jump to bottom
Håvard Heitlo Holm edited this page Aug 19, 2019 · 16 revisions

First time setup

# Login to login node (you might have to use <username>@abel.uio.no)
ssh -XY abel.uio.no

# Login to compute node with GPUs. This is necessary to compile pycuda.
# Use the following command to ensure enough memory for compilation:
# (The time is subtracted from our quota, so please logout when setup is done.)
qlogin --account=nn9550k --time=00:20:00 --nodes=1 --mem-per-cpu=4G \
--partition=accel --gres=gpu:1

# Load modules
module purge
#module load python2/2.7.10.gnu # uncomment for Python 2, use `pip2` below
module load Python/3.5.2-foss-2016b
module load openmpi.gnu/1.10.6
module load cuda/9.1

# Upgrade pip
pip3 install --upgrade --user pip
# Use the pip you had installed
export PATH=$HOME/.local/bin:$PATH

# Install necessary Python packages
pip3 install --user pycuda
pip3 install --upgrade --user mpi4py
pip3 install --user netcdf4

Verification of installed packages and troubleshooting

  • pycuda (and CUDA) installation can be verified in an interactive python shell (see e.g. https://documen.tician.de/pycuda/ for "cut and paste" code example)
  • mpi4py installation can be verified with mpiexec -n 1 python -m mpi4py.bench helloworld
  • Make sure you are using the new (local) version of pip: pip3 --version and which pip3
  • Fast sanity check:
wget https://raw.githubusercontent.com/inducer/pycuda/master/examples/hello_gpu.py
python hello_gpu.py # Check pycuda
mpiexec -n 1 python -m mpi4py.bench helloworld # Check MPI
mpiexec -n 1 python hello_gpu.py # Check pycuda in MPI

Running batch jobs

# Login to login node (you might have to use <username>@abel.uio.no)
ssh -XY abel.uio.no

# Launch jobs from the login nodes using the SLURM queue system:
sbatch <job script>

# Other useful commands:
squeue -A nn9550k                   # view all queued jobs in project
watch squeue -A nn9550k             # same as above, but with automatic update every 10 sec
scancel <job id returned by sbatch> # cancel job
cost -u                             # how much have I used of the quota
cost                                # prints project quota

Interactive shell setup (qlogin)

# Login to login node (you might have to use <username>@abel.uio.no)
ssh -XY abel.uio.no

# Login to compute node with GPUs.
# one hour, one GPU
# (The time is subtracted from our quota, so please logout when work is done.)
qlogin --account=nn9550k --time=01:00:00 --nodes=1 --partition=accel --gres=gpu:1

# Load modules
module purge
#module load python2/2.7.10.gnu # uncomment for Python 2
module load Python/3.5.2-foss-2016b
module load openmpi.gnu/1.10.6
module load cuda/9.1

export PATH=$HOME/.local/bin:$PATH

OpenCL installation

Installation of pyopencl: Follow "Interactive shell setup (qlogin)" then do:

# If pybind is required, uncomment lines below
#pip3 install --user pip==9.0.3 # downgrade pip to install pybind
#pip3 install --user pybind
#pip3 install --user --upgrade pip # upgrade pip again

pip3 install --user pybind11
pip3 install --user pyopencl

References

Clone this wiki locally