SGE qsub not requesting enough virtual memory to initialize Java virtual machine

[xiaofanjin]

Describe the bug
Hi I am running the pipeline in an SGE environment, but running into issues at the encode_task_filter step with virtual memory.

Currently it seems the default memory request for encode_task_filter step is 20GB, and so the SGE qsub requests 4 cores with 5GB each of virtual memory (-pe smp 4 -l h_vmem=5GB -l s_vmem=5GB). However, to run picard in this task, the java virtual machine upon initialization will request 20GB for its heap, plus usually 1GB default for metaspace, on top of other requirements. As a result, this step ends up requiring more than 20GB in virtual memory, resulting in the error:

Error occurred during initialization of VM
Could not reserve enough space for 20971520KB object heap

I have played around with this and increasing the qsub virtual memory request to 23GB seems likely to solve the problem. Less than that, but more than 20GB and the JVM has trouble allocating metaspace, resulting in something like:

Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes

For now, adding the following line to the default.conf file fixes the issue, but its a little hacky
sge-extra-param=-l h_vmem=23G -l s_vmem=23G

Would it be possible to modify the qsub -l h_vmem and -l s_vmem parameters in a future release so instead of requesting X/#threads of virtual memory, where X=memory-limit for a particular task, have qsub request something like (X+5GB)/#threads so there is a little extra space to initialize the JVM? Thanks

OS/Platform

• OS/Platform: CentOS Linux 7
• Conda version: conda 4.8.0
• Pipeline version: 1.3.6
• Caper version: 0.4.1

Caper configuration file
backend=sge
sge-pe=smp

tmp-dir=/wynton/scratch

Input JSON file
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v1/hg38_chr19_chrM_caper.tsv",
"chip.fastqs_rep1_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R1.subsampled.50.fastq.gz"
],
"chip.fastqs_rep1_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R2.subsampled.50.fastq.gz"
],
"chip.fastqs_rep2_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R1.subsampled.50.fastq.gz"
],
"chip.fastqs_rep2_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R2.subsampled.50.fastq.gz"
],
"chip.ctl_fastqs_rep1_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R1.subsampled.80.fastq.gz"
],
"chip.ctl_fastqs_rep1_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R2.subsampled.80.fastq.gz"
],
"chip.ctl_fastqs_rep2_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R1.subsampled.80.fastq.gz"
],
"chip.ctl_fastqs_rep2_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
"chip.always_use_pooled_ctl" : true,
"chip.title" : "ENCSR936XTK (subsampled 1/50, chr19 and chrM Only)",
"chip.description" : "ZNF143 ChIP-seq on human GM12878"
}

Error log
errorlog.txt

[ambeys]

Hey!! how did it go finally. were u able to get it resolved?? I am in exactly same error. so just wandering if you can giude me please. thanks

[xiaofanjin]

If this is still an issue, try adding the following line to the ~/.caper/default.conf file:
sge-extra-param=-l h_vmem=23G -l s_vmem=23G

[ambeys]

Right now its running with qsub lets see. if not will try. thanks a lot for responding.

[leepc12]

Sorry for late response, I think I missed this issue.

The latest pipeline automatically finds required resources for a task based on input file sizes (BAM for filter task, FASTQs for align task, ...). Also, JAVA heap size (java -Xmx) is taken as 90% of task's memory.

Please try with the latest pipeline and let me know if it doesn't work.