Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation of csvkit for stream merging csv #136

Open
tillenglert opened this issue Sep 30, 2024 · 0 comments
Open

Implementation of csvkit for stream merging csv #136

tillenglert opened this issue Sep 30, 2024 · 0 comments
Labels
enhancement Improvement for existing functionality

Comments

@tillenglert
Copy link
Collaborator

Description of feature

Currently some processes, like MERGE_PREDICTIONS use a custom concat_tsv.py script enabling streamed merging of many cvs, as prediction tables tend to get very large and multiple 100ed files are concatenated.

To reduce maintenance load we could implement an nf-core module for:
https://csvkit.readthedocs.io/en/latest/contributing.html#streaming-versus-buffering
@tillenglert tillenglert added the enhancement Improvement for existing functionality label Sep 30, 2024
@tillenglert tillenglert mentioned this issue Sep 30, 2024
Merged
@tillenglert tillenglert changed the title Implementation of csvkit for stream merging css Implementation of csvkit for stream merging csv Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Improvement for existing functionality
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tillenglert