Add filesize sniffing and parallelize importing on jobs (#5133)

* Basic WDL implementation

* Fix bug with importworker options

* Asbtract WDL and CWL imports and enforce a better method

* Remove unused import

* satisfy mypy

* Fix WDL import in CWL

* Fix WDL comments

* move url functions to job, remove dead imports, format with black

* Address comments

* mypy

* Detect running out of disk space with OSError instead

* Fix logic in worker import loop

* Fix bad caching

* Add wdl-conformance-tests to gitignore and sphinx config + remove unsued imports in job + get rid of wdl dependency in job.py

* Update src/toil/cwl/cwltoil.py

Co-authored-by: Michael R. Crusoe <[email protected]>

* Move URLNotImplemented exception out to exceptions.py to prevent circular import and add new argument to make user deal with import worker disk size when streaming is not available. Fix a bug with file mutation as well.

* Update src/toil/job.py

Co-authored-by: Adam Novak <[email protected]>

* Some type hinting improvements

* Use base disk instead of disk_size

Without the stream switch there's no reason to use disk_size instead of disk.

* Fix mypy types

* Fix bug with importing tool associated files

* Mypy issues

* Visit all steps

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Michael R. Crusoe <[email protected]>
Co-authored-by: Adam Novak <[email protected]>

.gitignore

@@ -14,6 +14,7 @@ venv*
 v3nv/
 tmp/
 /src/toil/test/cwl/spec*
+/src/toil/test/wdl/wdl-conformance-tests
 /cwltool_deps/
 /docs/generated_rst/
 /docker/Dockerfile

docs/conf.py

@@ -160,7 +160,8 @@ def setup(app):
 autoapi_dirs = ["../src/toil"]
 autodoc_typehints = "description"
 autoapi_keep_files = True
-autoapi_ignore = ["*.pyi", "*/test/cwl/spec*/*.py", "*/fake_mpi_run.py", "*/tutorial_*.py", "*/example_*.py", "*/mkFile.py", "*/debugWorkflow.py"]
+autoapi_ignore = ["*.pyi", "*/test/cwl/spec*/*.py", "*/fake_mpi_run.py", "*/tutorial_*.py", "*/example_*.py", "*/mkFile.py", "*/debugWorkflow.py",
+                  "*/test/wdl/wdl-conformance-tests/*"]
 autoapi_options = [
     "members",
     "undoc-members",

src/toil/common.py

@@ -750,7 +750,7 @@ def addOptions(
     add_cwl_options(parser, suppress=not cwl)
     add_wdl_options(parser, suppress=not wdl)
     # Add shared runner options
-    add_runner_options(parser)
+    add_runner_options(parser, cwl=cwl, wdl=wdl)
 
     def check_arguments(typ: str) -> None:
         """
@@ -764,7 +764,7 @@ def check_arguments(typ: str) -> None:
             add_cwl_options(check_parser)
         if typ == "cwl":
             add_wdl_options(check_parser)
-        add_runner_options(check_parser)
+
         for action in check_parser._actions:
             action.default = SUPPRESS
         other_options, _ = check_parser.parse_known_args(