spp. chip.call_peak RetryableFailure

[yatianyang2019]

Describe the bug
Hi, i was running the pipeline for TF with error. And the error was pasted in the bottom.

....
[Caper] troubleshooting 49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74 ...
Found failures:
[
{
"causedBy": [
{
"message": "Job chip.call_peak_pooled:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job chip.call_peak:1:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job chip.call_peak_pr2:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job chip.call_peak:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job chip.call_peak_ppr2:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
}
],
"message": "Workflow failed"
}
]

chip.call_peak RetryableFailure. SHARD_IDX=0, RC=1, JOB_ID=20508, RUN_START=2020-02-28T19:20:16.079Z, RUN_END=2020-02-28T23:39:13.988Z, STDOUT=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/execution/stdout, STDERR=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in
main()
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 91, in main
args.fraglen, args.cap_num_peak, args.nth, args.out_dir)
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 66, in spp
run_shell_cmd(cmd0)
File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
raise Exception(err_str)
Exception: PID=16, PGID=16, RC=1
STDERR=Loading required package: caTools
There were 24 warnings (use warnings() to see them)
Error in data.frame(..., check.names = FALSE) :
arguments imply differing number of rows: 1, 0
Calls: find.binding.positions ... calculate.enrichment.estimates -> lapply -> FUN -> cbind -> cbind -> data.frame
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Execution halted
STDOUT=################
ChIP data: /cromwell-executions/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/inputs/-623737054/reverb_SR8278.merged.nodup.tagAlign.gz
Control data: /cromwell-executions/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/inputs/812910509/ctl_for_rep1.tagAlign.gz
:

---

• OS/Platform:Ubuntu 18.01
• Pipeline version: 1.3.3
• Caper version: 0.6.1

Caper configuration file
backend=local
tmp-dir=temp

Input JSON file
"chip.title" : "reverb",
"chip.description" : "reverb chip seq",

"chip.pipeline_type" : "tf",
"chip.aligner" : "bwa",
"chip.align_only" : false,
"chip.true_rep_only" : false,

"chip.genome_tsv" : "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/hg38_caper.tsv",

"chip.paired_end" : false,
"chip.ctl_paired_end" : false,

"chip.always_use_pooled_ctl" : false,

"chip.fastqs_rep1_R1" : [ "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/reverb_SR8278.fq.gz" ],
"chip.fastqs_rep2_R1" : [ "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/reverb_Vehicle.fq.gz" ],

"chip.ctl_fastqs_rep1_R1" : [ "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/Input_SR8278.fq.gz" ],
"chip.ctl_fastqs_rep2_R1" : [ "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/Input_Vehicle.fq.gz" ],


"chip.align_cpu": 16,
"chip.align_mem_mb": 32768,

"chip.filter_cpu": 4,
"chip.filter_mem_mb": 24576,

"chip.bam2ta_cpu": 2,
"chip.bam2ta_mem_mb": 24576,

"chip.spr_mem_mb": 24576,

"chip.jsd_cpu": 4,
"chip.jsd_mem_mb": 24576,

"chip.xcor_cpu": 4,
"chip.xcor_mem_mb": 24576,

"chip.call_peak_cpu": 4,
"chip.call_peak_mem_mb": 24576,

"chip.macs2_signal_track_mem_mb": 24576

}

Error log
[Caper] troubleshooting 49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74 ...
Found failures:
[
{
"causedBy": [
{
"message": "Job chip.call_peak_pooled:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job chip.call_peak:1:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job chip.call_peak_pr2:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job chip.call_peak:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job chip.call_peak_ppr2:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
}
],
"message": "Workflow failed"
}
]

chip.call_peak RetryableFailure. SHARD_IDX=0, RC=1, JOB_ID=20508, RUN_START=2020-02-28T19:20:16.079Z, RUN_END=2020-02-28T23:39:13.988Z, STDOUT=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_Eva
nsLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/execution/stdout, STDERR=/home/hwclab/NGS_Data/reverb-project/201904_DCai_R
EV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in
main()
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 91, in main
args.fraglen, args.cap_num_peak, args.nth, args.out_dir)
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 66, in spp
run_shell_cmd(cmd0)
File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
raise Exception(err_str)
Exception: PID=16, PGID=16, RC=1
STDERR=Loading required package: caTools
There were 24 warnings (use warnings() to see them)
Error in data.frame(..., check.names = FALSE) :
arguments imply differing number of rows: 1, 0
Calls: find.binding.positions ... calculate.enrichment.estimates -> lapply -> FUN -> cbind -> cbind -> data.frame
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Execution halted
STDOUT=################
ChIP data: /cromwell-executions/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/inputs/-623737054/reverb_SR8278.merged.nodup.tagAlign.gz
Control data: /cromwell-executions/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/inputs/812910509/ctl_for_rep1.tagAlign.gz
:

[leepc12]

What is your pipeline version?

[yatianyang2019]

1.3.3

[leepc12]

Did you use Conda? Then check spp version in pipeline's Conda env. I think this is already fixed in v1.3.6. What is your r-spp version?

$ conda activate encode-chip-seq-pipeline
$ conda list r-spp

[yatianyang2019]

I was using the docker image. I will give a try on the v1.3.6. Thanks!

[yatianyang2019]

Hi, I was using the docker on the V1.3.6. But it still have same error as below:
.......
chip.call_peak_pooled RetryableFailure. SHARD_IDX=-1, RC=1, JOB_ID=11888, RUN_START=2020-03-03T02:24:04.408Z, RUN_END=2020-03-03T07:07:19.256Z, STDOUT=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker_1.3.6/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/execution/stdout, STDERR=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker_1.3.6/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 109, in
main()
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 95, in main
args.nth, args.out_dir)
File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 69, in spp
run_shell_cmd(cmd0)
File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
raise Exception(err_str)
Exception: PID=16, PGID=16, RC=1
STDERR=Loading required package: caTools
There were 24 warnings (use warnings() to see them)
Error in data.frame(..., check.names = FALSE) :
arguments imply differing number of rows: 1, 0
Calls: find.binding.positions ... calculate.enrichment.estimates -> lapply -> FUN -> cbind -> cbind -> data.frame
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Execution halted
STDOUT=################
ChIP data: /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/inputs/1096230463/basename_prefix.pooled.tagAlign.gz
Control data: /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/inputs/-832788197/basename_prefix.pooled.tagAlign.gz
strandshift(min): -500
strandshift(step): 5
strandshift(max) 1500
user-defined peak shift 148
exclusion(min): 10
exclusion(max): NaN
num parallel nodes: NA
FDR threshold: 0.01
NumPeaks Threshold: 3e+05
Output Directory: /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/execution
narrowPeak output file name: NA
regionPeak output file name: basename_prefix.pooled_x_basename_prefix.pooled.300K.regionPeak.gz.tmp
Rdata filename: NA
plot pdf filename: NA
result filename: NA
Overwrite files?: TRUE

Decompressing ChIP file
Decompressing control file
Reading ChIP tagAlign/BAM file /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/inputs/1096230463/basename_prefix.pooled.tagAlign.gz
opened /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/tmp.7e7cdd0d/Rtmpo99IrN/basename_prefix.pooled.tagAlign1256eec687
done. read 40532677 fragments
ChIP data read length 50
[1] TRUE
Reading Control tagAlign/BAM file /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/inputs/-832788197/basename_prefix.pooled.tagAlign.gz
opened /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/tmp.7e7cdd0d/Rtmpo99IrN/basename_prefix.pooled.tagAlign1251b8815d
done. read 36009580 fragments
Control data read length 50
Calculating peak characteristics
Minimum cross-correlation value 0.365511
Minimum cross-correlation shift 1500
Top 3 cross-correlation values 0.431022635744594
Top 3 estimates for fragment length 148
Window half size 455
Phantom peak location 55
Phantom peak Correlation 0.4120462
Normalized Strand cross-correlation coefficient (NSC) 1.179233
Relative Strand cross-correlation Coefficient (RSC) 1.407788
Phantom Peak Quality Tag 1
Removing read stacks
Finding peaks
finding background exclusion regions ... done
determining peaks on provided 1 control datasets:
using reversed signal for FDR calculations
....

[leepc12]

I was able to reproduce the problem. I will look into it and let you know.

[yatianyang2019]

thanks, I tried to change to use reference genome hg19.tsv on the docker v 1.3.6. It run successfully. But there is still the same error with hg38. So I have no idea about this issue.

[snewberry]

I have run into this issue as well.

I removed everything except chr1-22, X, Y, and M from the rep bams and that seemed to fix the issue. This isn't ideal as I had to edit the bams and then re-run from the bams on; however, perhaps the underlying issue has something to do with calling on the contigs.