-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
APPS-1271 Update improve manifest docs #354
Conversation
(fix) ExpertOptions.md - clarification of manifest inputs. Fixed example
(fix) ExpertOptions.md - review changes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some changes, a request for examples and some questions to improve my understanding of manifest mode.
by @emiloslavsky Co-authored-by: emiloslavsky <[email protected]>
(fix) ExpertOptions.md - review changes
(fix) ExpertOptions.md - review changes
(fix) ExpertOptions.md - review changes
in GH web Co-authored-by: emiloslavsky <[email protected]>
in GH web Co-authored-by: emiloslavsky <[email protected]>
from GH web Co-authored-by: emiloslavsky <[email protected]>
from GH web Co-authored-by: emiloslavsky <[email protected]>
from GH web Co-authored-by: emiloslavsky <[email protected]>
GH web Co-authored-by: emiloslavsky <[email protected]>
GH web Co-authored-by: emiloslavsky <[email protected]>
GH web Co-authored-by: emiloslavsky <[email protected]>
GH web Co-authored-by: emiloslavsky <[email protected]>
Thanks for the reviews/fixes @emiloslavsky |
Co-authored-by: emiloslavsky <[email protected]>
Co-authored-by: emiloslavsky <[email protected]>
(fix) ExpertOptions.md - review changes
@emiloslavsky latest changes incorporated. thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missing ` otherwise LGTM
Co-authored-by: emiloslavsky <[email protected]>
(fix) ExpertOptions.md - review changes
@commandlinegirl - could you approve if my comments satisfy your request for changes? I tried to outdate them but it didn't work |
In extreme cases, running compiled workflows can fail due to DNAnexus platform limits on the total size of the input and | ||
output JSON documents of a job. An example is a task with many inputs/outputs that is called in scatter over a large collection. | ||
In such a case, you can enable manifest support at compile time with the `-useManifests` option. | ||
This option causes each generated applet or workflow to accept inputs as an array of manifests, and to produce outputs as a single manifest. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In extreme cases, running compiled workflows can fail due to DNAnexus platform limits on the total size of the input and | |
output JSON documents of a job. An example is a task with many inputs/outputs that is called in scatter over a large collection. | |
In such a case, you can enable manifest support at compile time with the `-useManifests` option. | |
This option causes each generated applet or workflow to accept inputs as an array of manifests, and to produce outputs as a single manifest. | |
In extreme cases, running compiled workflows can fail due to DNAnexus platform limits on the total size of the input and output JSON documents of a job. An example is a task with many inputs/outputs that is called in scatter over a large collection. In such a case, you can enable manifest support at compile time with the `-useManifests` option. This option causes each generated applet or workflow to accept inputs as an array of manifests, and to produce outputs as a single manifest. |
It's a minor thing but it's easier to read (now and in the future) if there are no newlines introduced.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here and below wherever the breakline is added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, this new line here looks ugly to show the difference in versions. In the raw source it looks fine and there's no change in the rendered variant. But without this break line - the source looks like one stretched line and you have to scroll sidewise to read it. So I would really insist on having those break lines in the markdown docs from now on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is how it looks now:
dxCompiler/doc/ExpertOptions.md
Line 1444 in 455de1d
In extreme cases, running compiled workflows can fail due to DNAnexus platform limits on the total size of the input and |
This is how the rest of the doc looks without new lines:
dxCompiler/doc/ExpertOptions.md
Line 1296 in 455de1d
When calling a workflow with `dx run`, jobs and analyses launched by this workflow will have their temporary workspaces to store resources and intermediate outputs. By default, when a job or an analysis has transitioned to a terminal state (done, failed, or terminated), its temporary workspace will be destroyed by the system. |
To me second example is harder to read compared to my version
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please click the links to actually see it in the raw .md
(fix) ExpertOptions.md - review changes
(fix) ExpertOptions.md - review changes
(fix) ExpertOptions.md - review changes
(fix) ExpertOptions.md - review changes
(fix) ExpertOptions.md - review changes
Co-authored-by: Aleksandra Zalcman <[email protected]>
section can be used to pass manifest output from a stage of one workflow (including the `output` stage) as input to another workflow. A | ||
typical use case for this scenario is when a user wants to pass manifest output file from a stage (including `output` stage) | ||
directly to a new workflow. Also, this scenario might be useful when debugging individual stages of a failing workflow. | ||
|
||
## Manifest JSON | ||
|
||
When manifest support is enabled, each applet has an `input_manifest___` input field of type `hash`, which means that it | ||
When manifest support is enabled, applet/workflow outputs which are passed from one stage to another (or to the final output |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't want to inform the user of what the input_manifest___
is?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not as simple. It does have this field, as well as a bunch of others, which I could not find the documentation for. APPS-1309 ticket should address that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gotcha. Thanks!
(fix) ExpertOptions.md - clarification of manifest inputs. Fixed example