Add new template cylc-dev (and test building in GitHub actions)

[climbfuji]

Summary

This PR adds a new template cylc-dev to build a special environment for running cylc. Because of the way cylc works, this is best accomplished by creating an environment view and then a cylc wrapper that contains the minimum settings necessary to use cylc from the view's bin directory. See cylc/cylc-flow#6532 for more information. Note that the creation of the cylc wrapper is not part of this PR.

Associated changes:

• To compile cylc-dev on the Ubuntu self-hosted runner, exclude meson from the spack external find call. Spack found a deprecated version [email protected] and despite it being deprecated it tried to use it - with the result that the harfbuzz build failed (needed for cylc-dev). Letting spack build meson (it picks [email protected]) solved the problem.
• Remove the +excel variant from py-pandas, which was something the jedi-tools-env used, but that is no longer needed and that caused build errors (and added several more dependencies).
• Fix a bug in the GitHub actions that would allow the tests to fail silently (replace set +e with set -e).
• To avoid duplicate packages, require py-cython@3 and [email protected]
• With Intel Classic and LLVM, build qt with gcc (has no effect on pre-configured sites, where qt is configured as an external package)

Testing

• CI (also tests cylc-dev template with gcc on Ubuntu)
• Atlantis (built cylc-dev with gcc, created cylc wrapper, tested flow, gui and NEPTUNE workflow)
• Narwhal (built cylc-dev with gcc, created cylc wrapper, tested flow, gui)
• Nautilus (built cylc-dev with gcc, created cylc wrapper, tested flow, gui)

Applications affected

None

Systems affected

None

Dependencies

• waiting on Add recent versions of py-cylc-flow, py-cylc-rose, py-cylc-uiserver, and their dependency py-protobuf (spack PR #48052); pull spack PRs #48251 ([email protected] - intel classic conflict) and #48478 (comment out [email protected]) spack#491
• Follow-up bug fix in freetype when using external libz; add conflict for node-js@21: with [email protected] spack#499

Issue(s) addressed

Resolves #1413
Resolves #1444

Checklist

• This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.

[climbfuji]

Note. For ede071b, GitHub actions reports that the tests succeeded, but in fact the build of cylc-dev failed: https://github.com/JCSDA/spack-stack/actions/runs/12662753274/job/35288186039?pr=1410

[eap]

Thanks for this, based on description I'm still a little puzzled, I thought we were also intending to have a cylc meta module that could be loaded along side of other modules as in our unified dev environment.

I'm approving because this code looks fine and I assume you'll have a good response to my question above.

[eap]

Good catch here and elsewhere.

[eap]

Wouldn't the concretizer unify: true prevent duplication?

[climbfuji]

It will lead to concretization errors if unify: true is set. Unfortunately, the concretizer isn't perfect, it still has its bugs. But I am happy to move this (numpy and cython) into the cylc-dev template instead of applying it to all environments for now (but new packages that are coming down the road will want the same settings for unified-env/neptune-env, too).

[climbfuji]

Thanks for this, based on description I'm still a little puzzled, I thought we were also intending to have a cylc meta module that could be loaded along side of other modules as in our unified dev environment.

I'm approving because this code looks fine and I assume you'll have a good response to my question above.

Initially, the idea was to add cylc to the unified-env and other larger environments. But, as we are expanding the use of Python in our environments (more and more packages with complicated dependencies), this is getting harder and harder.

Especially cylc has so many tight version requirements that it's impossible to build it in unified-env without duplicate packages ( (and is very hard to build with anything else but gcc). Then you also need the view+wrapper (wrapper is currently a manual process) steps to make it work with CYLC_PYTHONPATH and keep the environment to run cylc in and the one that cylc tasks use completely separate. This is a non-trivial setup to get right, and the link to the cylc issue/discussion is only the entry point. I've come to learn that cylc is designed to operate in its own, separate environment (which does make a lot of sense), and this can be done best with two separate spack-stack environments.