Add leader fallback for worker file imports #5189
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamnovak
requested changes
Dec 18, 2024
src/toil/cwl/cwltoil.py
Outdated
| ) | ||
|
|
||
| # files with a associated filesize that are valid to be imported on workers | ||
| valid_files_to_data = dict() |
There was a problem hiding this comment.
This isn't really mapping to file data (bits in the file), right? This is metadata consisting of file size and... something.
Maybe this should be renamed and have a comment explaining what's actually in the values? Or an inline type hint and not just auto-typing?
Co-authored-by: Adam Novak <[email protected]>
adamnovak
requested changes
Jan 8, 2025
src/toil/cwl/cwltoil.py
Outdated
Comment on lines
2968
to
2977
| files_to_data = get_file_sizes( | ||
| filenames, toil._jobStore, include_remote_files=options.reference_inputs | ||
| ) | ||
|
|
||
| # Mapping of files to metadata for files that will be imported on the worker | ||
| # This will consist of files that we were able to get a file size for | ||
| worker_files_to_data: dict[str, FileMetadata] = dict() | ||
| # Mapping of files to metadata for files that will be imported on the leader | ||
| # This will consist of files that we were not able to get a file size for | ||
| leader_files_to_data = dict() |
There was a problem hiding this comment.
I still don't think we want to name these variables _to_data. They don't contain file data.
We could maybe change files_to_data, leader_files_to_data, worker_files_to_data to instead be metadata, leader_metadata, worker_metadata?
There was a problem hiding this comment.
I think metadata makes sense
adamnovak
approved these changes
Jan 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #5135
Changelog Entry
To be copied to the draft changelog by merger:
Reviewer Checklist
issues/XXXX-fix-the-thingin the Toil repo, or from an external repo.camelCasethat want to be insnake_case.docs/running/{cliOptions,cwl,wdl}.rstMerger Checklist