Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added liftoff, integrated with pfr/nxf-modules, and some minor improvements #2

Merged
merged 59 commits into from
Jan 11, 2024
Merged
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
ee9d0b9
Added updated info about nf-core modules
GallVp Nov 6, 2023
7f03697
Turned off SortMeRNA by default
GallVp Nov 7, 2023
232493a
Decouple target assemblies and read qc/align
GallVp Nov 8, 2023
4f2ed8e
A bit of reformatiing
GallVp Nov 9, 2023
8b83c61
Started implementing liftoff
GallVp Nov 9, 2023
37ce74e
Checkpoint before major reshuffle
GallVp Nov 10, 2023
d148f18
Reformatted local modules
GallVp Nov 10, 2023
e63e22f
Now using galaxy containers
GallVp Nov 10, 2023
1978308
Extracted some subworkflows
GallVp Nov 10, 2023
10a0158
Extracted a few subworkflows
GallVp Nov 12, 2023
2031790
Extracted subworkflows uptill BRAKER3
GallVp Nov 13, 2023
f315467
Inc liftoff
GallVp Nov 13, 2023
f10ae94
Added polished out channel to liftoff
GallVp Nov 13, 2023
80db270
Added gffread before liftoff
GallVp Nov 13, 2023
bb9b8b0
Updated flowchart
GallVp Nov 13, 2023
c982946
Added liftoff options
GallVp Nov 15, 2023
c8ff8da
Started moving to nf-core/tools
GallVp Nov 21, 2023
f285461
Reimported modules using nf-core/tools
GallVp Nov 21, 2023
bea59ef
Reimported kherronism modules with nf-core/tools
GallVp Nov 22, 2023
2dda752
Updated braker3
GallVp Nov 22, 2023
8efa34d
Updated repeatmasker
GallVp Nov 22, 2023
8d31976
Updated modules
GallVp Nov 22, 2023
5eaa87b
Imported fastavalidate and liftoff from pfr/nxf-modules
GallVp Nov 22, 2023
0a96c7f
Updated modules and subworkflows
GallVp Dec 12, 2023
8a6c5fe
Added EDTA from pfr/nxf-modules
GallVp Dec 12, 2023
26c33fa
Updated config
GallVp Dec 18, 2023
9e58314
Integrated fastavalidator
GallVp Dec 18, 2023
4534684
Added patch for star/genomegenerate
GallVp Dec 18, 2023
ec7ffc1
Incorporated fasta_edta_lai
GallVp Dec 19, 2023
2de0d22
Trying to add FASTQ_FASTQC_UMITOOLS_FASTP
GallVp Dec 20, 2023
48e7271
Updated modules and applied prettier
GallVp Dec 20, 2023
c526933
FASTP now has stub
GallVp Dec 20, 2023
ac45bbb
SORTMERNA now has stub
GallVp Dec 20, 2023
ed6aa33
Cleaned up prepare_assembly
GallVp Dec 21, 2023
144edb2
Cleaned up preprocess_rnaseq
GallVp Dec 21, 2023
a79c187
Reformatted and inc ALIGN_RNASEQ
GallVp Dec 21, 2023
6f61f5d
Cleaned up and inc PREPARE_EXT_PROTS
GallVp Dec 21, 2023
1184795
Cleanedup BRAKER3
GallVp Dec 21, 2023
a9f1fc6
Updated fastp, sortmerna and liftoff
GallVp Dec 21, 2023
d0faf8d
Cleaned up fasta_liftoff
GallVp Dec 21, 2023
27a1293
Updated fasta_edta_lai
GallVp Jan 7, 2024
3867ed5
Added script for local stub run
GallVp Jan 7, 2024
4761491
Samplesheet now accepts relative paths
GallVp Jan 7, 2024
8ced027
Updated modules
GallVp Jan 7, 2024
fce65a0
Removed -exclude_partial and updated flowchart
GallVp Jan 7, 2024
582f3fa
Separated test config for local and pfr
GallVp Jan 8, 2024
e213bd3
Fixed local script typo
GallVp Jan 8, 2024
c961ab0
Fixed apptainer scope bug in base config
GallVp Jan 8, 2024
260c706
Updated README
GallVp Jan 8, 2024
565e7d7
Readded -exclude_partial and now using teambraker container
GallVp Jan 8, 2024
47b7c40
Added config for test data and quay.io container for braker3
GallVp Jan 9, 2024
664fba1
Now using repeatmodeler by default
GallVp Jan 9, 2024
dea4bb5
BRAKER3 now runnable with test data
GallVp Jan 9, 2024
0535b1a
Added editor config
GallVp Jan 10, 2024
457a643
Disabled sortmerna by default added option to save cat bam
GallVp Jan 10, 2024
784bb54
Added pre-commit
GallVp Jan 10, 2024
a4ada59
Updated manifest
GallVp Jan 10, 2024
4de72ab
Fixed linting errors
GallVp Jan 11, 2024
12968f4
Updated base config for docker
GallVp Jan 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated modules
GallVp committed Jan 11, 2024
commit 8ced02787818d8da7c3e6e7b158da58512137138
4 changes: 2 additions & 2 deletions modules.json
Original file line number Diff line number Diff line change
@@ -73,7 +73,7 @@
},
"custom/dumpsoftwareversions": {
"branch": "master",
"git_sha": "bba7e362e4afead70653f84d8700588ea28d0f9e",
"git_sha": "37dee863936732fe7e05dc598bf6e183a8e7ef73",
"installed_by": ["modules"]
},
"fastavalidator": {
@@ -88,7 +88,7 @@
},
"fastqc": {
"branch": "master",
"git_sha": "65ad3e0b9a4099592e1102e92e10455dc661cf53",
"git_sha": "617777a807a1770f73deb38c80004bac06807eef",
"installed_by": ["fastq_fastqc_umitools_fastp", "modules"]
},
"gffread": {
Original file line number Diff line number Diff line change
@@ -31,7 +31,12 @@ nextflow_process {
then {
assertAll(
{ assert process.success },
{ assert snapshot(process.out).match() }
{ assert snapshot(
process.out.versions,
file(process.out.mqc_yml[0]).readLines()[0..10],
file(process.out.yml[0]).readLines()[0..7]
).match()
}
)
}
}
50 changes: 28 additions & 22 deletions modules/nf-core/custom/dumpsoftwareversions/tests/main.nf.test.snap
Original file line number Diff line number Diff line change
@@ -1,27 +1,33 @@
{
"Should run without failures": {
"content": [
{
"0": [
"software_versions.yml:md5,1c851188476409cda5752ce971b20b58"
],
"1": [
"software_versions_mqc.yml:md5,2570f4ba271ad08357b0d3d32a9cf84d"
],
"2": [
"versions.yml:md5,3843ac526e762117eedf8825b40683df"
],
"mqc_yml": [
"software_versions_mqc.yml:md5,2570f4ba271ad08357b0d3d32a9cf84d"
],
"versions": [
"versions.yml:md5,3843ac526e762117eedf8825b40683df"
],
"yml": [
"software_versions.yml:md5,1c851188476409cda5752ce971b20b58"
]
}
[
"versions.yml:md5,3843ac526e762117eedf8825b40683df"
],
[
"data: \"<style>\\n#nf-core-versions tbody:nth-child(even) {\\n background-color: #f2f2f2;\\n\\",
" }\\n</style>\\n<table class=\\\"table\\\" style=\\\"width:100%\\\" id=\\\"nf-core-versions\\\"\\",
" >\\n <thead>\\n <tr>\\n <th> Process Name </th>\\n <th>\\",
" \\ Software </th>\\n <th> Version </th>\\n </tr>\\n </thead>\\n\\",
" \\n<tbody>\\n<tr>\\n <td><samp>CUSTOM_DUMPSOFTWAREVERSIONS</samp></td>\\n <td><samp>python</samp></td>\\n\\",
" \\ <td><samp>3.12.0</samp></td>\\n</tr>\\n\\n<tr>\\n <td><samp></samp></td>\\n \\",
" \\ <td><samp>yaml</samp></td>\\n <td><samp>6.0.1</samp></td>\\n</tr>\\n\\n</tbody>\\n\\",
" <tbody>\\n<tr>\\n <td><samp>TOOL1</samp></td>\\n <td><samp>tool1</samp></td>\\n\\",
" \\ <td><samp>0.11.9</samp></td>\\n</tr>\\n\\n</tbody>\\n<tbody>\\n<tr>\\n <td><samp>TOOL2</samp></td>\\n\\",
" \\ <td><samp>tool2</samp></td>\\n <td><samp>1.9</samp></td>\\n</tr>\\n\\n</tbody>\\n\\",
" <tbody>\\n<tr>\\n <td><samp>Workflow</samp></td>\\n <td><samp>Nextflow</samp></td>\\n\\"
],
[
"CUSTOM_DUMPSOFTWAREVERSIONS:",
" python: 3.12.0",
" yaml: 6.0.1",
"TOOL1:",
" tool1: 0.11.9",
"TOOL2:",
" tool2: '1.9'",
"Workflow:"
]
],
"timestamp": "2023-11-03T14:43:22.157011"
"timestamp": "2024-01-05T00:18:43.461970077"
}
}
}
271 changes: 191 additions & 80 deletions modules/nf-core/fastqc/tests/main.nf.test
Original file line number Diff line number Diff line change
@@ -3,23 +3,21 @@ nextflow_process {
name "Test Process FASTQC"
script "../main.nf"
process "FASTQC"

tag "modules"
tag "modules_nfcore"
tag "fastqc"

test("Single-Read") {
test("sarscov2 single-end [fastq]") {

when {
params {
outdir = "$outputDir"
}
process {
"""
input[0] = [
[ id: 'test', single_end:true ],
[
file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true)
]
[ id: 'test', single_end:true ],
[
file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true)
]
]
"""
}
@@ -28,82 +26,195 @@ nextflow_process {
then {
assertAll (
{ assert process.success },

// NOTE The report contains the date inside it, which means that the md5sum is stable per day, but not longer than that. So you can't md5sum it.
// looks like this: <div id="header_filename">Mon 2 Oct 2023<br/>test.gz</div>
// https://github.com/nf-core/modules/pull/3903#issuecomment-1743620039
{ assert process.out.html.get(0).get(1) ==~ ".*/test_fastqc.html" },
{ assert path(process.out.html.get(0).get(1)).getText().contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },
{ assert snapshot(process.out.versions).match("versions") },
{ assert process.out.zip.get(0).get(1) ==~ ".*/test_fastqc.zip" }

{ assert process.out.html[0][1] ==~ ".*/test_fastqc.html" },
{ assert process.out.zip[0][1] ==~ ".*/test_fastqc.zip" },
{ assert path(process.out.html[0][1]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },

{ assert snapshot(process.out.versions).match("versions") }
)
}
}

test("sarscov2 paired-end [fastq]") {

when {
process {
"""
input[0] = [
[id: 'test', single_end: false], // meta map
[
file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true),
file(params.test_data['sarscov2']['illumina']['test_2_fastq_gz'], checkIfExists: true)
]
]
"""
}
}

then {
assertAll (
{ assert process.success },

{ assert process.out.html[0][1][0] ==~ ".*/test_1_fastqc.html" },
{ assert process.out.html[0][1][1] ==~ ".*/test_2_fastqc.html" },
{ assert process.out.zip[0][1][0] ==~ ".*/test_1_fastqc.zip" },
{ assert process.out.zip[0][1][1] ==~ ".*/test_2_fastqc.zip" },
{ assert path(process.out.html[0][1][0]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },
{ assert path(process.out.html[0][1][1]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },

{ assert snapshot(process.out.versions).match("versions") }
)
}
}

test("sarscov2 interleaved [fastq]") {

when {
process {
"""
input[0] = [
[id: 'test', single_end: false], // meta map
file(params.test_data['sarscov2']['illumina']['test_interleaved_fastq_gz'], checkIfExists: true)
]
"""
}
}

then {
assertAll (
{ assert process.success },

{ assert process.out.html[0][1] ==~ ".*/test_fastqc.html" },
{ assert process.out.zip[0][1] ==~ ".*/test_fastqc.zip" },
{ assert path(process.out.html[0][1]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },

{ assert snapshot(process.out.versions).match("versions") }
)
}
}

test("sarscov2 paired-end [bam]") {

when {
process {
"""
input[0] = [
[id: 'test', single_end: false], // meta map
file(params.test_data['sarscov2']['illumina']['test_paired_end_sorted_bam'], checkIfExists: true)
]
"""
}
}

then {
assertAll (
{ assert process.success },

{ assert process.out.html[0][1] ==~ ".*/test_fastqc.html" },
{ assert process.out.zip[0][1] ==~ ".*/test_fastqc.zip" },
{ assert path(process.out.html[0][1]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },

{ assert snapshot(process.out.versions).match("versions") }
)
}
}
// TODO
// //
// // Test with paired-end data
// //
// workflow test_fastqc_paired_end {
// input = [
// [id: 'test', single_end: false], // meta map
// [
// file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true),
// file(params.test_data['sarscov2']['illumina']['test_2_fastq_gz'], checkIfExists: true)
// ]
// ]

// FASTQC ( input )
// }

// //
// // Test with interleaved data
// //
// workflow test_fastqc_interleaved {
// input = [
// [id: 'test', single_end: false], // meta map
// file(params.test_data['sarscov2']['illumina']['test_interleaved_fastq_gz'], checkIfExists: true)
// ]

// FASTQC ( input )
// }

// //
// // Test with bam data
// //
// workflow test_fastqc_bam {
// input = [
// [id: 'test', single_end: false], // meta map
// file(params.test_data['sarscov2']['illumina']['test_paired_end_sorted_bam'], checkIfExists: true)
// ]

// FASTQC ( input )
// }

// //
// // Test with multiple samples
// //
// workflow test_fastqc_multiple {
// input = [
// [id: 'test', single_end: false], // meta map
// [
// file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true),
// file(params.test_data['sarscov2']['illumina']['test_2_fastq_gz'], checkIfExists: true),
// file(params.test_data['sarscov2']['illumina']['test2_1_fastq_gz'], checkIfExists: true),
// file(params.test_data['sarscov2']['illumina']['test2_2_fastq_gz'], checkIfExists: true)
// ]
// ]

// FASTQC ( input )
// }

// //
// // Test with custom prefix
// //
// workflow test_fastqc_custom_prefix {
// input = [
// [ id:'mysample', single_end:true ], // meta map
// file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true)
// ]

// FASTQC ( input )
// }

test("sarscov2 multiple [fastq]") {

when {
process {
"""
input[0] = [
[id: 'test', single_end: false], // meta map
[
file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true),
file(params.test_data['sarscov2']['illumina']['test_2_fastq_gz'], checkIfExists: true),
file(params.test_data['sarscov2']['illumina']['test2_1_fastq_gz'], checkIfExists: true),
file(params.test_data['sarscov2']['illumina']['test2_2_fastq_gz'], checkIfExists: true)
]
]
"""
}
}

then {
assertAll (
{ assert process.success },

{ assert process.out.html[0][1][0] ==~ ".*/test_1_fastqc.html" },
{ assert process.out.html[0][1][1] ==~ ".*/test_2_fastqc.html" },
{ assert process.out.html[0][1][2] ==~ ".*/test_3_fastqc.html" },
{ assert process.out.html[0][1][3] ==~ ".*/test_4_fastqc.html" },
{ assert process.out.zip[0][1][0] ==~ ".*/test_1_fastqc.zip" },
{ assert process.out.zip[0][1][1] ==~ ".*/test_2_fastqc.zip" },
{ assert process.out.zip[0][1][2] ==~ ".*/test_3_fastqc.zip" },
{ assert process.out.zip[0][1][3] ==~ ".*/test_4_fastqc.zip" },
{ assert path(process.out.html[0][1][0]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },
{ assert path(process.out.html[0][1][1]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },
{ assert path(process.out.html[0][1][2]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },
{ assert path(process.out.html[0][1][3]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },

{ assert snapshot(process.out.versions).match("versions") }
)
}
}

test("sarscov2 custom_prefix") {

when {
process {
"""
input[0] = [
[ id:'mysample', single_end:true ], // meta map
file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true)
]
"""
}
}

then {
assertAll (
{ assert process.success },

{ assert process.out.html[0][1] ==~ ".*/mysample_fastqc.html" },
{ assert process.out.zip[0][1] ==~ ".*/mysample_fastqc.zip" },
{ assert path(process.out.html[0][1]).text.contains("<tr><td>File type</td><td>Conventional base calls</td></tr>") },

{ assert snapshot(process.out.versions).match("versions") }
)
}
}

test("sarscov2 single-end [fastq] - stub") {

options "-stub"

when {
process {
"""
input[0] = [
[ id: 'test', single_end:true ],
[
file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true)
]
]
"""
}
}

then {
assertAll (
{ assert process.success },
{ assert snapshot(process.out.html.collect { file(it[1]).getName() } +
process.out.zip.collect { file(it[1]).getName() } +
process.out.versions ).match() }
)
}
}

}
Loading