Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fix number of download threads reportedly being 1 on some machines regardless of the configuration #253

Merged
merged 2 commits into from
Dec 14, 2023
Merged

fix: fix number of download threads reportedly being 1 on some machines regardless of the configuration #253

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
768913b
fix: now download commands use MAX_DOWNLOAD_THREADS cores
leoisl Dec 14, 2023
ee5f768
fix: removing priority queue from sbatch command
leoisl Dec 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ else
SMK_PARAMS=--cores all --rerun-incomplete --printshellcmds --keep-going --use-conda --resources max_download_threads=10000000 max_io_heavy_threads=10000000 max_ram_mb=1000000000 $(SMK_CLUSTER_ARGS)
endif

DOWNLOAD_PARAMS=--cores $(MAX_DOWNLOAD_THREADS) -j $(MAX_DOWNLOAD_THREADS) --restart-times $(DOWNLOAD_RETRIES)


######################
## General commands ##
Expand All @@ -33,7 +35,7 @@ all: ## Run everything (the default rule)
DIFF_CMD=diff -q <(gunzip --stdout output/reads_1___reads_2___reads_3___reads_4.sam_summary.gz | cut -f -3) <(xzcat data/reads_1___reads_2___reads_3___reads_4.sam_summary.xz | cut -f -3)

test: ## Quick test using 3 batches
snakemake download $(SMK_PARAMS) -j 99999 --config batches=data/batches_small.txt # download is not benchmarked
snakemake download $(SMK_PARAMS) $(DOWNLOAD_PARAMS) --config batches=data/batches_small.txt # download is not benchmarked
scripts/benchmark.py --log logs/benchmarks/test_match_$(DATETIME).txt "snakemake match $(SMK_PARAMS) --config batches=data/batches_small.txt nb_best_hits=1"
scripts/benchmark.py --log logs/benchmarks/test_map_$(DATETIME).txt "snakemake map $(SMK_PARAMS) --config batches=data/batches_small.txt nb_best_hits=1"
@if $(DIFF_CMD); then \
Expand Down Expand Up @@ -75,13 +77,13 @@ conda: ## Create the conda environments
snakemake $(SMK_PARAMS) --conda-create-envs-only

download: ## Download the assemblies and COBS indexes
snakemake download $(SMK_PARAMS) -j 99999 --restart-times $(DOWNLOAD_RETRIES)
snakemake download $(SMK_PARAMS) $(DOWNLOAD_PARAMS)

download_asms: ## Download only the assemblies
snakemake download_asms_batches $(SMK_PARAMS) -j 99999
snakemake download_asms_batches $(SMK_PARAMS) $(DOWNLOAD_PARAMS)

download_cobs: ## Download only the COBS indexes
snakemake download_cobs_batches $(SMK_PARAMS) -j 99999
snakemake download_cobs_batches $(SMK_PARAMS) $(DOWNLOAD_PARAMS)

match: ## Match queries using COBS (queries -> candidates)
scripts/benchmark.py --log logs/benchmarks/match_$(DATETIME).txt "snakemake match $(SMK_PARAMS)"
Expand Down Expand Up @@ -110,7 +112,6 @@ report: ## Generate Snakemake report
cluster_slurm: ## Submit to a SLURM cluster
sbatch \
-c 10 \
-p priority \
--mem=80GB \
-t 0-08:00:00 \
--wrap="make"
Expand Down
Loading