Formatting actions as runnable scripts

[zsusswein]

Hey folks,

I want to start a discussion on standardizing how we write actions. We on NNH have been seeing ~weekly outages on the self-hosted runners (I'm curious if you've seen similar @dylanhmorris @damonbayer). My understanding is that the Tools team is looking into strategies to increase reliability, but we've also received feedback that we should update our systems so they don't rely on the runners to work.

My interpretation of that feedback is that we should format workflows as a wrapper around a script, with environment variables passed as arguments. I'm proposing that a workflow would do some basic authentication, maybe start a Docker container as a running environment, and run the script. This script should be easily locally runnable and not require any extra GitHub environment magic to function or manually pulling commands out of YAML. I tried to implement a version of this idea here which runs this script.

I'm going to make sure our EpiNow2 pipeline updates in response to this feedback and I was hopeful there would be interest in matching the format here too. I think the standardization and shared tooling this repo implements is a valuable thing and I'd like to be able to continue relying on the work here while still being able to decouple from GHA as necessary (e.g., runners are down so we clone this repo and run some of the scripts manually to run a pipeline).

I don't think this proposal would be a heavy lift to implement right now -- maybe some small tweaks here or there.

Thoughts? @gvegayon @CDCgov/cfa-nnh @dylanhmorris @damonbayer

[natemcintosh]

I think the idea of having a script that can be run in the VAP or in GHA is a good idea.

As well as bash scripts, we could also use python scripts with inline metadata to specify the desired python version, dependencies, and dependency versions, and run them with uv's support for inline script metadata. This would allow us to use any python packages we like, while still getting pinned versions of everything.

[dylanhmorris]

I prefer limiting scripting within yaml files, even leaving aside the specific question of testing and/or running code outside of GHA. Code-within-yaml is harder to read and debug.

Proposed rule of thumb: if scripting code (whether shell, Python, R, etc) is more than a one-liner, it probably belongs in a .sh/.py/.R file rather than in an action.yaml or workflow.yaml file.

[natemcintosh]

Oh yes, we should keep such scripts in their own files, not in the yaml; that would be painful.

[kgostic]

Let's raise this at the vap working group meeting later today. I think that's our new BAH Tools point of contact.

[zsusswein]

Dropping a few notes from the meeting:

• @jkislin and @giomrella had a generally positive reaction to moving to formatting this way
• @giomrella mentioned there was some exploration underway for lighter weight non-Batch options
• My takeaway is that it would then also be good to be careful to keep Batch-specific workflows segmented from other logic
  • I think all of @gvegayon's work in here already does a good job of that -- mostly noting for me so I don't forget
• Also some discussion of Actions debugging being painful and lots of complexity in Actions being a sign that we should move to a more purpose-built tool if possible

[gvegayon]

I like what you are proposing, @zsusswein. This is something I have mentioned to a few people around: I am not comfortable with 100% depending on actions for things (and you stated the main reason clearly). A hybrid approach could be having the actions rely on scripts and have a way of sharing that script with the actions' users. One example of this is in the post artifact action: While an important part of the workflow direct scripting in the file, we do have three Python scripts included in the action under the scripts folder which could be shared with the user.

I propose the following:

• Move most (if not all) of the steps of the actions into bash/Python scripts.
• Add a step that posts an annotation after each run, like what docker does with the build and push action, pointing to the script used to execute the steps.
• Repositories can copy these scripts, and keep them under scripts or .github/scripts, as @zsusswein did, with a README.md stating the origin of these.
• For more complex actions (@natemcintosh), we can go further and use the container actions approach instead of the composit action.

Happy to take this discussion to Teams :).

[zsusswein]

I think this all sounds good and we're all in agreement! From my perspective, I think the most urgent is the container build. Everything else is a nice-to-have, but we need to be able to reproducibly build and tag the images if the runners are down.