Adding WDL and Docker Contribution Guides

_quarto.yml

@@ -19,6 +19,8 @@ book:
     - codereview.qmd
     - packagedocs.qmd
     - maintenance.qmd
+    - wdlconfig.qmd
+    - docker.qmd
     - security.qmd
     - conventions.qmd
   site-url: https://getwilds.org/guide/

docker.qmd

@@ -0,0 +1,49 @@
+
+# Docker Configuration Guide {{< iconify mdi docker >}} {#sec-docker}
+
+The mindset with regard to Docker images is different for WILDS WDLs compared to other projects. Normally, repositories are relatively self-contained and only need one image that can just be directly linked to that repository. However, WDL pipelines often require a different image for each step, creating the need for a laundry list of Docker images for each repository. In addition, our bioinformatics workflows will have a large amount of image overlap in that the same tools get used, just in a different fashion depending on the workflow. To avoid unnecessary image duplication, this repository will contain all Dockerfiles and images relevant to WILDS and all future workflows refer back to these images.
+
+## Docker Image Guidelines
+
+- Because these Docker images will be used for individual steps within WDL workflows, they should be as minimal as possible in terms of the number of tools installed in each image (1 or 2 max).
+- As a general (but flexible) rule, try to start from as basic of a parent image as possible, e.g. `scratch`, `ubuntu`, `python`, `r-base`, etc.
+    - Outside parent images are fine, as long as they are from a VERY trusted source, e.g. Ubuntu, Python, Conda, Rocker, etc.
+- To speed up build and deployment of containers, try to keep image sizes relatively small (a few hundred MB on average, 2GB max).
+    - For this reason, reference data should not be stored in an image unless absolutely necessary.
+    - Unnecessary tools should also be avoided, even if they serve a "just-in-case" functionality.
+
+## Dockerfile Guidelines
+- Every Dockerfile must contain the labels below at a minimum. This provides users with increased visibility in terms of where the image came from and open access to the necessary resources in case they have any questions or concerns.
+```
+LABEL org.opencontainers.image.title="awesomeimage" # Name of the image in question
+LABEL org.opencontainers.image.description="Short description of awesomeimage and its purpose"
+LABEL org.opencontainers.image.version="1.0" # Version tag of the image
+LABEL org.opencontainers.image.authors="[email protected]" # Author email address
+LABEL org.opencontainers.image.url=https://hutchdatascience.org/ # Home page
+LABEL org.opencontainers.image.documentation=https://getwilds.org/ # Documentation page
+LABEL org.opencontainers.image.source=https://github.com/getwilds/wilds-docker-library # GitHub repo to link with
+LABEL org.opencontainers.image.licenses=MIT # License type for the image in question
+```
+- When creating a different version of an existing image, use one of the other Dockerfiles as a starting template and modify it as needed.
+    - This will help to ensure that the only thing that has changed between image versions is the version of tool in question, not any strange formatting/configuration issues.
+- Try to be as specific as possible in terms of tool versions within the Dockerfile, especially the parent image.
+    - If you just specify "latest", a tag that get updated frequently over time, your image could be completely different the next time you build it, even though it uses the exact same Dockerfile.
+    - On the other hand, specifying "v1.2.3" will always pull the same instance of the tool every time, providing greater reproducibility over time.
+
+## Repository Guidelines
+
+- In terms of the repo organization, each image should have its own directory named after the tool being used in the image. Each version of the image should have its own Dockerfile in that directory following the naming convention of `[IMAGENAME]/Dockerfile_[VERSIONTAG]`.
+    - If formatted correctly, a GitHub Action will automatically build and upload the image to the [WILDS GitHub container registry](https://github.com/orgs/getwilds/packages) upon merging into the `main` branch.
+
+- Before merging your changes to `main` (and therefore uploading a new image to the WILDS package registry), try uploading it to your user-specific package registry using the command below and make sure it works for the WDL task in question.
+```
+docker build --platform linux/amd64 -t ghcr.io/GITHUBUSERNAME/IMAGENAME:VERSIONTAG -f IMAGENAME/Dockerfile_VERSIONTAG --push .
+```
+
+- Upon creation or modification of a pull request in this repo, a GitHub Action will run a check using linting tool specific to Dockerfiles called [Hadolint](https://github.com/hadolint/hadolint).
+    - If any major warnings pop up, the check will fail and the user will be unable to merge the branch into `main` until the warning is resolved.
+    - Smaller stylistic issues will still be reported, but they will not restrict you from merging your branch into `main`.
+    - Details about the location and root cause of each warning can be found in the details of the check.
+
+
+

index.qmd

@@ -13,6 +13,8 @@ The book covers important aspects of software development, including how to get
 - Code review: @sec-review
 - Package documentation: @sec-docs
 - Package maintenance: @sec-maintenance
+- WDL Configuration: @sec-wdlconfig
+- Docker Configuration: @sec-docker
 - Security: @sec-security
 - Conventions: @sec-conventions
 

wdlconfig.qmd

@@ -0,0 +1,58 @@
+
+# WDL Configuration Guide {{< iconify file-icons wdl >}} {#sec-wdlconfig}
+
+So as not to reinvent the wheel, WILDS WDL's should follow guidelines similar to the ones provided by [BioWDL](https://biowdl.github.io/styleGuidelines.html) and [WARP](https://broadinstitute.github.io/warp/docs/Best_practices/suggested_formats). However, because of the pedagogical "proof-of-concept" nature of the WILDS, they will not be identical and even differ significantly in a few places.
+
+## WILDS Philosophy
+
+- The mindset behind WILDS is for each repository to be a self-contained demonstration of a particular bioinformatic functionality:
+    1. Researcher scans the workflow to deem whether it is relevant to their needs.
+    2. Researcher clones the repository as is, makes minimal updates to the inputs, and easily executes the code locally or otherwise.
+    3. Researcher forks the repository and customizes it as necessary to fit their exact research needs.
+- To that end, WILDS WDL repositories will usually consist of a single WDL script containing the workflow as well as the tasks that make up the workflow.
+    - This contradicts the recommendations from BioWDL, i.e. tasks should be written in a separate script and imported into the workflow script as a module.
+    - We believe the "one-stop-shop" nature of this setup will aid from a readability/learning standpoint.
+
+## Structural Guidelines
+
+- Structs should be at the top of the WDL script, followed by the workflow itself, followed by all of its corresponding tasks.
+- Tasks should be broken down into as small of operations as possible.
+    - If a task uses more than one or two command line tools, it should probably be broken up into individual tasks.
+- Docker containers should be assigned to every task to ensure uniform execution, regardless of local context.
+    - Outside of very basic images from very trusted sources, Docker images should be pulled directly from [WILDS' Docker Library](https://github.com/getwilds/wilds-docker-library) whenever possible.
+    - If you think a particular tool should be added to that library, [submit an issue](https://github.com/getwilds/wilds-docker-library/issues) or email us at [email protected].
+- In general, runtime attributes should be defined whenever possible in order to enable execution on as many backends as possible.
+
+## Stylistic Guidelines
+
+- **Indentation**: braces contents, inputs, and line continuations should all be indented by two spaces (not four).
+- **White Space**: different input groups and code blocks should be separated by a single blank line.
+- **Line Breaks**: line breaks should only occur in the following places:
+    - After a comma
+    - Following an opening parenthesis/bracket
+    - Before the `else` of an `if` statement
+    - Between inputs
+    - Opening and closing braces
+- **Line Character Limit**: lines should be a maximum of 100 characters.
+- **Expression Spacing**: spaces should surround operators to increase clarity and readability.
+- **Naming Conventions**:
+    - Tasks, workflows, and structs should follow upper camel case (`SuperAwesomeTask`)
+    - Call aliases should follow lower camel case (`superAwesomeCall`)
+    - Variables should follow lowercase underscore (`super_awesome_variable`)
+- **Descriptive Commenting**:
+    - Comments should be placed above each task in the workflow describing its function.
+    - Input descriptors should be provided in the `parameter_meta` component.
+
+## Repository Guidelines
+
+- As with all repositories, each workflow should include a detailed README containing:
+    - Purpose and functionality of the workflow
+    - Basic diagram illustration the flow of data
+    - Contact information in case issues pop up 
+    - [WILDS Badge](https://github.com/getwilds/badges) at the top describing the development status of the workflow
+- Make sure to include an example input json in the repository for users to modify and easily execute the workflow.
+    - For a skeleton template, try the `inputs` action of [WOMtool](https://cromwell.readthedocs.io/en/stable/WOMtool/#inputs).
+- A GitHub Action executing [WOMtool](https://cromwell.readthedocs.io/en/stable/WOMtool/#validate) `validate` is highly recommended as a check before merging new features into main.
+    - If you're feeling adventurous, try automating an entire test run using a very small validation dataset.
+
+