"CESAR wrapper command crashed"

[mprincipato]

Hello, dev team!

I'm very new to TOGA, and I'm trying it out on some public data for a class project.

When I run it on my personal Windows machine (using Windows Subsystem for Linux), it runs for about eight and a half hours before saying:

`### CESAR jobs done ###

Checking whether all CESAR results are complete
!!CRITICAL: Too many (4576) CESAR jobs died, please check your input data and re-run TOGA
!!CRITICAL: Too many (4576) CESAR jobs died, please check your input data and re-run TOGA
Program finished with exit code 1`

When I look in the CESAR job logs, I find several error messages like this:
(Slightly edited to remove personal information.)

!!CESAR wrapper command crashed: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del CESAR failed run for ['/home/m/code/TOGA/CESAR2.0/cesar', '/dev/stdin', '-x', '3']:  
!!FAILED COMMAND: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del
!!At attempt 0
!!CESAR wrapper command crashed: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del CESAR failed run for ['/home/m/code/TOGA/CESAR2.0/cesar', '/dev/stdin', '-x', '3']:  
!!FAILED COMMAND: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del
!!At attempt 1
!!FAILED TO EXECUTE COMMAND /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del in 2 attempts
cesar_runner::cesar_job_1_0: return code: 1
cesar_runner::cesar_job_1_0: CESAR JOB FAILED: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del
cesar_runner::cesar_job_1_0: ERROR MESSAGE: CESAR wrapper command crashed: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del CESAR failed run for ['/home/m/code/TOGA/CESAR2.0/cesar', '/dev/stdin', '-x', '3']:   

Here was the command I originally used to run TOGA:
./toga.py ../toga_input/hg38.HLpipKuh2.allfilled.chain supply/hg38.wgEncodeGencodeCompV34.bed ../toga_input/hg38.2bit ../toga_input/HLpipKuh2.2bit --kt -i supply/hg38.wgEncodeGencodeCompV34.isoforms.txt --cesar_mem_limit 5 --cesar_jobs_num 500

Do you have any guesses what might be going wrong to cause the CESAR wrapper command to crash?

As far as my inputs go, I got hg38.HLpipKuh2.allfilled.chain and HLpipKuh2.fa from this Bat1K page, and I converted HLpipKuh2.fa to 2bit using faToTwoBit.

[kirilenkobm]

Hi @mprincipato

CESAR distribution provides a few test samples
Could you pls try running CESAR on a couple of them?
Like https://github.com/hillerlab/CESAR2.0/blob/master/extra/example4.fa

Then pls try cat XXX.fa | ./cesar /dev/stdin

I suspect the current implementation is not quite compatible with WSL.
(but I need to check the docs, probably there is a simple way to adjust subprocess calls to make it work)

[mprincipato]

Thank you, @kirilenkobm !

I spotted one thing in the documentation (the fact that Python 3.11 was recommended, but my default version was Python 3.10.12) and went down an ill-advised rabbit hole trying to set my default Python3 to 3.11 and wound up breaking things further. I think I was able to undo what I did, though, and these results are from running CESAR after that!

example1.fa

m@MPC:~/code/TOGA/CESAR2.0$ ./cesar extra/example1.fa
>referenceExon
                                                                                                    gCCTGGGAACTTCACCTACCACATCCCTGTCAGTAGTGGCACCCCACTGCACCTCAGCCTGACTCTGCAGATGaa                                                                 
>mouse
ctttcctcatttcctcaggcttcagtatagcatgaggctgaggaggagagagggagaccggcaaagtggccttgcttaggtaccatctttgcccctttagGCTTGGCAACTTCACCTACCACATCCCTGTCAGCAGCAGCACACCACTGCACCTCAGCCTGACCCTGCAGATGAAgtgagtgctggtgtgtgggtatgtgtgggggaccatgtggaagccctcagaaaagtgaaagccaagtgcttactaaatttattacgtggagggtccaggc

example4.fa

m@MPC:~/code/TOGA/CESAR2.0$ ./cesar extra/example4.fa
>referenceExon
                                                                                                                                                                                                                                                                                                                                 ---CCCCTGTCCCGCTGGTTGAGATCTGTGGGGGTCTTCCTGCTGCCAGCCCCCTACTGGGCACCCCGGGAGAGGTGGCTGGGTTCCCTACGGCGGCCCTCCCTGGTGCACGGGTACCCAGTCCTG---GCCTGGCACAGTGCCCGCTGCTGGTGCCAAGCGTGGACAGAGGAACCTcg                                                                                  aGCCCTTTGCTCCTCCCTCAGAATGAACGGAGACCAGAATTCAGATGTTTATGCCCAAGAAAAGCAGGATTTCGTTCAGCACTTCTCCCAGATCGTTAGGGTGCTGACTGAGGATGAGATGGGGCACCCAGAGATAGGAGATGCTATTGCCCGGCTCAAGGAG                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      GTCCTGGAGTACAATGCCATTGGAGGCAAGTATAACCGGGGTTTGACGGTGGTAGTAGCATTCCGGGAGCTGGTGGAGCCAAGGAAACAGGATGCTGATAGTCTCCAGCGGGCCTGGACTGTGGGCTGGTGTGTGGAACTG                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    CTGCAAGCTTTCTTCCTGGTGGCAGATGACATCATGGATTCATCCCTTACCCGCCGGGGACAGATCTGCTGGTATCAGAAG>>>>>>>>>>>>>>>>>>>CCGGGCGTGGGTTTGGATGCCATCAATGATGCTAACCTCCTGGAAGCATGTATCTACCGCCTGCTGAAGCTCTATTGCCGGGAGCAGCCCTATTACCTGAACCTGATCGAGCTCTTCCTGCAG>>>>>>>>>>>>>>>>>>>AGTTCCTATCAGACTGAGATTGGGCAGACCCTGGACCTCCTCACAGCCCCCCAGGGCAATGTGGATCTTGTCAGATTCACTGAAAAGag                                                                                     gTACAAATCTATTGTCAAGTACAAGACAGCTTTCTACTCCTTCTACCTTCCTATAGCTGCAGCCATGTACATG                                                                                                                                                                                                                                                                                                                                                                                                                                                         GCAGGAATTGATGGCGAGAAGGAGCACGCCAATGCCAAGAAGATCCTGCTGGAGATGGGGGAGTTCTTTCAGATTCAG                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            GATGATTACCTTGACCTCTTTGGGGACCCCAGTGTGACCGGCAAAATTGGCACTGACATCCAGGACAACAAATGCAGCTGGCTGGTGGTTCAGTGTCTGCAACGGGCCACTCCAGAACAGTACCAGATCCTGAAG                                                                                                                                                                                                                                                                                                                                                                                      GAAAATTACGGGCAGAAGGAGGCTGAGAAAGTGGCCCGGGTGAAGGCGCTATATGAGGAGCTGGATCTGCCAGCAGTGTTCTTGCAATATGAGGAAGACAGTTACAGCCACATTATGGCTCTCATTGAACAGTACGCAGCACCCCTGCCCCCAGCCGTCTTTCTGGGGCTTGCGCGCAAAATCTACAAGCGGAGAAAG                                                                                                                                                                                                         
>rn6
aatagtgtcactttaggaccttccatcccataattctccgtaccgaattgtagtgaacgcagctttggaagtagaaagcgttcagttttagccatttgcggtttaagtgaagatctgttggctctcagtcactccgatatactttttatttcctgtgtgttatttatgtaattcttgaatttagaaacagggtggggcattctaatgcattctaaaagcttgggttgcggggtaccttaggtaacagccaccaagcatctgccttcggtgcttgctcctgcagggagtgcttggtgtcccccacccccatgcccacccaggATGCCCCTGTCCCGCTGGCTGAGATCTCTGGGGGTCTTCCTGCTGCCAGCCCCCTGCTGGGCACCCCGGGAGAGGTGGCTTGGTTACCTACAACGACCCTCCCTGGCATATGGGTGTCCAGTCCTTGGGGCTTGGCACAGTGCCCGCTGGTGGTGCCAAGTGTGGACAGAGGAGCCTCGgtgagtgtggttggggtgtggggcttcggggagggaagcaccggagccaccctctcactcatttgtgtgttttttccctcagAGCATTTAGCTCCTCTGTCAGAATGAATGGGGACCAGAAACTGGATGTTCATAACCAAGAAAAGCAGAATTTCATCCAGCACTTCTCCCAGATTGTCAAGGTGCTGACTGAGGATGAACTGGGACACCCAGAGAAGGGAGATGCTATTACCCGGATCAAAGAGgtgagggattcaggactgaagaagtcagtagaggtgtggattcgctgccagggttttgataagagcagaagtaacgtttttacctgtgggtcccgactgtacttcaacccacccagatatttggctccctgactttggaccatcccatcagctttgcttgagagcagagccctggacgtcatggtttgattttatgccctgagactataagataggatcttagcagataccaaaaggctgttggcactgaggcctgagggaagttactgtattgttaaagtctgctctaaaaagggagttgttcctcttctgcttgagttacctatttccctccagtctcttgagtttattttctttttttttggggggggggggtctttttttcggagctggggaccgaacccagggccttccgcttcctaggtaagcgctctaccactgagctaaatccccagccccttgagtttattttcttttcttttttttttttttttttttggttctctttttcggagctggggaccgaacccagggccttgtgcttcctaggtaagcgctctaccactgagctaaatccccagccccttgagtttattttcttaagacagagtttcaatttcttttttttttttttttttttttttttttggttcttttttcggagctggggaccgaacccagggcctttcgcttcctaggcaagcgctctaccactgaactaaatccccagccccgagtttcaatttcttttttttttttttttttttttttttggttcgttttttcggagctggggaccgaacccagggccttgcgcttcctaggtaagtgctctaccactgagctaaatccccagccccgagtttcaatttctaaacccacatgtcctcacacacacaaccccctgcctcagcctactgtgtgctgtatcaattacagtacatgtctagattgggaattctttttttttttttttttcaaaaaaagatttattatatataagaacactgtagctgtcttcagatacaccagaagaaggcatcggatcccattacagatggttgtgagccaccatgtggttgctgggaattgaactcagaacctctggaagagcagtcagtgctcttaaccactgaaccatctctccagcccaggaaacttttttttttttttttttttttttttttttttttttttttttttttttttgggttctttttttcggagctggggaccgaacccagggccttgcgcttcctaggtaagcgctctaccactgagctaaatccccagcccccaggaaacttttttttaaagacatacattttttttaaaaaattgtgcattgactttttgcctgcctctgtgtgtgtgtgtgtgtgtgtacacacttgtgagctgccatgtaggtgctgggaattgaacccaggtcctctggaggggcagtcagtgctcttcactgccaagccatctctagcctggggaattcttttttttataaatgtctcatctttgaggaaaacaccaaggagaattttctttgtctatactgtgcaggtattaaagagctatctctttagttagttagggctgctgtctggcagaaggcacagttgagaaggctccaggaggaggacattagaagggttagacgtagccctgactcttgctgtcccttcaatgccttcttttggctaactttggagaagccttgtcttttgttttgtttctttgttttctgtagccaaggctgtccagaactcattctgtatctgaggctggcctcatactcatcagtctgcctgcctccctgctgagtattgagcttataggagttagccatcactctacgctagggcccttgtcttcactacaagcttgtttgcaggtggtccaagttctcggcttgagggaagaactgggggtatgttatgtagcaggtacttgggttccatatgctataggactttggagaagaaatgaagaggggactgacccagggactccccaaggggagttggggttcaaagagggagaaggaaaggggaaacctcttcaggtcccttctgtgtggaccctgatgtcctctcaccctgcctcagGTCCTGGAGTACAACACTGTAGGAGGCAAGTACAATCGGGGTCTGACGGTGGTACAGACCTTCCAGGAACTGGTGGAACCAAGGAAACAGGATGCTGAGAGCCTACAGCGGGCCCTGACGGTGGGCTGGTGTGTAGAACTGgtaagagcggtcggagaagcagagttcccacagttggggcttccttggtgaggagcagaagcttgtttcttgcttgtgttgatgtgtgagtctcttgtgtgctacagtcattcgggcagtgtggatcccttcaggggctgacagatgggcacagagagttttcagaaatgctcaccttaaatatcaccatgtggtggtgtcccatgcctttgatcttatctcagggagcagaggcaagtggatctctatgagttctaggctagcctgatctatagagtaagttctaggatggccagggctactcaagaaaccctagtttcaaataacagaaaccttaaattcttaaacttaacaaagaagacatgccattaaaaaagaaagtagacttttaaagtctggttagatggcttggccggtgagagcatttgccatcagaagagccgaatgtgtgagccagggccgtagaaacccagttgtgtctctggacacagaatctgcatggtaggtagagccagcaggctggccaggctgcctggctcagtgagagagaccctgtttcaaaggaacattgatgtgatagacaggacacctggcatcttcctttggcttctccgcccctgttttcatgcccacactgaatgcacccaagtgtgtgcttggtggtacacatctgtaatacatacactcacctagtgctggggcatggagataggcgggtccccagagttcgctggcaagctaggttagccaatcagtcagttccgggctcagaaaggtttgtctcccaaaactgaggacacctgatactcacctcttgctgctacagatgcgcccagttcagatgcccatccatatatacaacttgtgacacagacacaagcacacaaggcctgctgggcaggcaggcaggcagacacacttgtgttcagtttctcactatctatatgacaactgtgtgcatctgttgcacccttcccggagttgggatagatgtgcagatcctggagcttgttggccagtctgaccatttcagtgaccaccctgtctcaaagaataaggtggagaggctggagagatgggccagtgtttaagagctctggctgcttttccagaggaccggggttcagtggttagggatctgacaccctcttctggccttcatgggcaccaggcatgcacatggttagcagacagacttgcagaccagcaccagtgcatttaaaataagtagtcctctgtttggcagcacatatactgaaattggaacaatacagagaagattagcttggcccctttgtaaggatgacacaaattcatgaggcagtcactctctttttttcctttctgtcttttcatttctatcaaatgtgtgaaaaaatacataaagattggttataaccaaaaatatgttatataaaattttaaaaatgaggaagacatctcacagaattcccccaaaatgaggacatccaaggtcgacctctgacctccacatgattgcatttgtaactcatgtacacagagcacatgcatacatgaaacatgtgtgcacaatataatatgtgcacacaattctacaaatctataataacatgtatgcatacactacacacatgcacacatgtacacacataaacagttcagatcactggcatgttcacttaatgaaatgattttccagagtgaattaaaagctgcttccctaatgtagggtacgtgctccacctatggctctgattttcagtcctggctcttgaatgtctgactttgttgtttcctgggtgggtgagaggtttccaatttggtattgaagtccagggccttgtctattgtaagcacgctcttatcccttaactatgtccctgccctttggtttcggtttttgatttttcactggctgtcctggaacttactctgtagatcaggctggcctcaaactcagctccacctgccctatctcccaagtgctgggattagaggtgagcgccaccagtgcctggctgtttttgtttatgagacaaggtctcactgtgtacagctcacacatactcaagatcatctcgcttcagccttccaagtgctgggatcatagatgtgtgccatcacgcctgactcatactgacttactaaattattggtctattaaatagaccagtttgcggtgtttggatgatgtaacgatagacttctgttgagaatcataggggtggaaggatgagtcaaagttctggagccctgagtttgatccttagaacccatgtaagaagtgaactgactcccagaagtcaccctctgacatcactgcatgctgtgacgtgtgagcatccacaccgagacacaccttcctcacccctgactagtgaggggctggcagtgtggttctagtgggtaaaggcactctgtcgaaagcccagcaaagtgagttctgtcccaggattcatgggaagatggaaagagaaaaacaactcctagaaattgtcctctgacctttatttgtgtgctgtggcatgtatgtgcccttgcacacaataataatagaagtttgttaaaacagaaaaaaaatagataattttgttgtgggggtcaaaagccttccatcccatatgtaaggccctgggttccatcctcagcactttaacacagtaaagccaaggcatgcagttaaagcaattgccacgtaacaaacgggtagccttgccggaaggtgcgtgggccagtgttcaggagatgggctgaggaccttgagatcgaagccagcctggccttcataccgaggttctgtctcaaggaatcaaaacccaaaagtgtcctggatgttgttgagtgttcagtcttccatctccctaatagagagcgcgtgggaggcaccactctgtggtcagggctaggctctgggaatacctgaggagccagcattgctgaactggaagagcatttaccagcaggcagagtggctcagtggtagcatgaggtcccgggttcagcactcgcagtgcacagaaccaaacatgacagtagacagctgtgacctcggcacctgggaaggagcggtaaaggcagaagtacagatccttggccacggagggactttggctcatctaagggaagacgccattgattgccaaggcggtgctcttggggtggtgctcctggaagtgtgactctcatagcccgtcactgagggaagtcagaatgggagttgatccagatgccgtgcagagtgatggtgactggtttgttcccacagctttctcagcctgttcccttacacaccccagaactgtctgcccagggatgcagcacccactgtgaaaatgcctcacagccttgctagaagccagtctggtgagggcatttttttcagttgaggctcccttttgccaaatgactatagcttgggtcaagttgacctcaaactagccaacacagtcacttactggaacaaaagttacggatgttttgggtaagagtgaggtacgtgaaggtgtttcatactcaacttactaccgagggagtcctagagccgattagaggcggcagagctgtgttttagtctgttagcctgaaaatgagggtgcagacatgggatcctgtaatctgtcagagttcccataaccaccccttccacccagtgcacggcagtggccagggcagcagccgtcactgagagagggcccctttcagagccctggggtcttactgtgttccttctccagCTCCAGGCTTTCTTCCTCGTGTTAGATGACATCATGGACTCTTCCCACACTCGCCGGGGGCAGATCTGCTGGTATCAGAAG-------------------CCGGGCATAGGCTTGGATGCCATCAACGATGCTCTGCTTCTGGAAGCCGCTATCTACCGCCTGCTTAAGTTCTACTGCAGGGAGCAGCCCTACTACCTCAACCTGCTGGAGCTCTTTCTACAG-------------------AGTTCCTATCAGACTGAGATCGGGCAGACTCTCGACCTCATCACAGCACCCCAGGGCCAAGTGGATCTTGGTAGATACACTGAAAAGAGgtgaggcccctggcaaccatgtgtagactttgaggcactcaacatgggcctagcccttaggagtgcatcttctcccctgactcagGTACAAATCTATCGTCAAGTACAAGACAGCTTTCTACTCTTTCTACCTGCCTATCGCGGCTGCCATGTACATGgtgagtcagtggcacctccactcttttcccttgggggcattgggatgggaggacatggagtagacattcagggtatgatggtataacttggagcaagccacccagtccctctgacactggaaagtgagcagagctgagtcctatctgagtgggaggggcagaggaaaatttaggctgagcaagagactaggtgtggcggtggctcttgggtccatcctggccagcctcttcaggttttctcttcattttgtaagaggttggagttggaagtggagctgtaatggctttccaggggaagatggcctgatgccttcagaatggattcattccttaggcctcccggggaggggctgaggttggactgcagttacgcagaacccatggaaagtgtcaagaagggcagggtgctgtggccatgctgccttacctgtctccctgtagGCTGGAATTGATGGGGAGAAGGAACACGCTAATGCCCTGAAGATCCTGCTGGAGATGGGCGAGTTCTTCCAGATCCAGgtaggagggtctgcagcgggaagtaataagaggattttccaccccacgctatgccgggcaccatggggtgttctcatctcggcagactgtgacccattgttcttaacactctccacccgactcgctaagttcccgtggtgggtcgctgccactccagccccatttaacaactgcccacctcacggttggctgtcacagatatctataccctggcaaaatcaaaactctagaggcttgtaatttattaatcagatttatatcagtaaattctcaccccacaaaacgcccacataataaactcagagccaattgatactgatataaagggcccacctagataagatgtcctgtagtcatctatcccttatataatactcatagttacccgtggctgtttaaagctacacggatccgggtggtcctcttcctccgtcttccttctccccttcctgtgctctttgtctctagaattctcggcccacttttctttttcactgtccaatcacaggctcttgccttgtcttctacctgcccttccctgcttacagacagcagtgtacacaggtcctccagccacactgcatggagcttctcatctctcctttctgcctgctccctagGACGACTACCTTGATCTCTTTGGAGACCCCAGTGTGACCGGAAAGGTCGGCACTGACATCCAGGACAACAAATGCAGCTGGCTGGTGGTTCAGTGTCTGCTACGAGCCACTCCTCAGCAGCGCCAGATCTTAGAGgtgcccaagtggggcttgggggtggcatgtccctttgctaagaggggatgggggaggagtggttagaacagtgattttgctgtccaggagacatttgggagctcagtgcaatttggtgtatgtgctgaaggtagcaggcaggaccaggaagattttgtacagtgccagacaacacatgcccagtaactgtcagtggtgtcgaggctgagaatccatagaggaagtagaatcctgagccaatgcctgaactttggggttggggaggcaatgcttgagattggattcccagagagagagctttgccagtgttctggaatgtaggagttgtgagggcctttgaatccagccaaaaaaagctgtttctctgcttccagGAGAATTATGGGCAGAAGGACCCAGAAAAAGTGGCGCGGGTGAAAGCACTGTACGAGGAGCTGGATCTGCGGAGTGTGTTCTTCAAGTACGAGGAAGACAGTTACAACCGCCTCAAGAGTCTCATAGAGCAGTGCTCCGCGCCCCTGCCCCCATCCATCTTCCTGGAACTAGCAAACAAGATCTACAAGCGGAGAAAGtaacctcgaattgtagaggctgcgagggaggggtctcaataaattattgttcaacatcctgtggttttatgcttgtgtcaggagttcaaaatggaggggtgggagagggcaacttggaatccaggttggcaaaagaattccttgagacccaaagctgctgcgttgttagatgggtgacataggcacagcgtcaagtgttgcgtggaggttgggaggcggtcctgagctatgccacgcccatgcgtgaaggggtgcggctgcgcagccactgggttttgctggtgcccctcccctccgctctg

Then I tried cat extra/example1.fa | ./cesar /dev/stdin, and here's what I got:

m@MPC:~/code/TOGA/CESAR2.0$ cat extra/example1.fa | ./cesar /dev/stdin
>referenceExon
                                                                                                    gCCTGGGAACTTCACCTACCACATCCCTGTCAGTAGTGGCACCCCACTGCACCTCAGCCTGACTCTGCAGATGaa                                                                 
>mouse
ctttcctcatttcctcaggcttcagtatagcatgaggctgaggaggagagagggagaccggcaaagtggccttgcttaggtaccatctttgcccctttagGCTTGGCAACTTCACCTACCACATCCCTGTCAGCAGCAGCACACCACTGCACCTCAGCCTGACCCTGCAGATGAAgtgagtgctggtgtgtgggtatgtgtgggggaccatgtggaagccctcagaaaagtgaaagccaagtgcttactaaatttattacgtggagggtccaggc

Does anything look unusual here? Any red flags for what might be going on?

[kirilenkobm]

@mprincipato

I apologise for the long delay...
And also, I would recommend using virtualenv to manage python versions between different projects =).

I should say, there are absolutely 0 red flags in the provided output, which makes everything more complicated.
The part that crashes is /home/m/code/TOGA/CESAR2.0/cesar /dev/stdin -x 3. It consumes the input directly from stdin, without creating any temporary input files. My guess was that it does not work on WSL, however, it seems like there is another reason for such a behaviour.
I need to add some additional logging and maybe provide an option to create these temporary input files.
Will keep you updated.

[MauriAndresMU1313]

Hi! Any update on this issue? I got the same issue in the STEP 7, using dog as reference and ferret as target (GCF_014441545.1
-GCF_011764305.1)
My output from toga_2025_01_15_at_16_41.log:

### STEP 7: Execute CESAR jobs: parallel step

Pushing 2 CESAR job lists
Pushing memory bucket 3Gb to the executor
Selected parallelization strategy: nextflow
Parallel manager: pushing job nextflow /scratch/mmora30/workspace/TOGA/execute_joblist.nf --joblist /users/mmora30/mmora30/workspace/TOGA/run_01_dog_ferret/temp/cesar_joblist_queue_3.txt -c /users/mmora30/mmora30/workspace/TOGA/run_01_dog_ferret/temp/cesar_config_3_queue.nf
Pushing memory bucket 5Gb to the executor
Selected parallelization strategy: nextflow
Parallel manager: pushing job nextflow /scratch/mmora30/workspace/TOGA/execute_joblist.nf --joblist /users/mmora30/mmora30/workspace/TOGA/run_01_dog_ferret/temp/cesar_joblist_queue_5.txt -c /users/mmora30/mmora30/workspace/TOGA/run_01_dog_ferret/temp/cesar_config_5_queue.nf
## Stated polling cluster jobs until they done
Polling iteration 0; already waiting 0 seconds.
Polling iteration 1; already waiting 60 seconds.
Polling iteration 2; already waiting 120 seconds.
Polling iteration 3; already waiting 180 seconds.
Polling iteration 4; already waiting 240 seconds.
Polling iteration 5; already waiting 300 seconds.
Polling iteration 6; already waiting 360 seconds.
Polling iteration 7; already waiting 420 seconds.
Polling iteration 8; already waiting 480 seconds.
### CESAR jobs done ###

Checking whether all CESAR results are complete
!!CRITICAL: Too many (6353) CESAR jobs died, please check your input data and re-run TOGA
!!CRITICAL: Too many (6353) CESAR jobs died, please check your input data and re-run TOGA
Program finished with exit code 1

However, if I go to cesar_jobs, I can not find details about !!CESAR wrapper command crashed:. When I check for error in the toga_2025_01_15_at_16_41.log, I do not have errors. However, I know that I need STEP 8 to finish the process.
Any recommendations?

[mprincipato]

I just noticed these responses; apologies for my late response! Back in December 2023, I was finally able to get it to work by running CESAR on a borrowed computer with more memory and (I think) a newer graphics card. (I still used WSL on the new computer.)

While I'm not positive the memory amount was the bottleneck, that's my current theory, since I haven't had similar issues with other ML tasks since I upgraded the memory in my computer.