Can't restart properly if timed out in first slurm steps

[nylander]

If running out of time using one of the two initial slurm scripts (refdata_and_init_and_plast.slurm.sh, or init_and_plast.slurm.sh), we cannot simply rerun using make (as we would have wished/expected). The reason is that the first step, make refdata lacks the proper file checks in the Makefile. This has to be addressed.

One work-around

The current remedy if faced with the "timed out" situation in the first step (most probably due to the make plast is taking longer time than expected), is to first remove the "unfinished" .selected.plast.tab file inside birdscanner/run/plast/ (see example approach below), and then rerun the plast step on its own. Rerunning could possibly be done using the slurm script init_and_plast.slurm.sh or perhaps better, by creating a new script containing only the steps make -C .. plast and make -C .. (see example plast.slurm.sh below).

One approach:

bsdir='/proj/abc123456/nobackup/tmp/birdscanner'
slurmdir="${bsdir}/slurm"
lastfile=$(grep -A 4 ^plast "${slurmdir}/refdata_and_init_and_plast.out" | \
        perl -ne 'print"$1\n" if ( /run\/plast\/(\w+\.selected\.plast\.tab)/ )')
rm -v "${bsdir}/run/plast/${lastfile}"
cd "${bsdir}"
sbatch plast.slurm.sh

plast.slurm.sh

As an alternative to the init_and_plast.slurm.sh script - if the run was timed out during the plast step - is to use a "plast only" script. A minimal example is given here (to be placed in, and started from birsdcanner/slurm/):

#!/bin/bash -l
#SBATCH -A abc123456
#SBATCH -t 50:00:00
#SBATCH -p core
#SBATCH -n 10
#SBATCH -M rackham,snowy
#SBATCH --output=plast.out
#SBATCH --error=plast.err
module load bioinfo-tools hmmer/3.2.1-intel blast/2.9.0+ gnuparallel
make -C .. plast
>&2 echo "did make plast"
make -C .. parseplast
>&2 echo "did make parseplast"
>&2 echo "Reached the end of the plast slurm script"