Merge spack/develop as of 2024/01/26 into jcsda_emc_spack_stack #401

climbfuji · 2024-01-27T00:02:40Z

Description

Pull in spack develop as of 2024/01/26 - needed for JEDI+GEOS development, and generally good to stay up to date. Every single package has changed because of the copyright date change (2013-2023 --> 2013-2024).

I would like the reviewers to specifically look at the following packages for which I had to resolve merge conflicts that weren't entirely trivial, or for which I needed to add more code (nco and openmpi).

var/spack/repos/builtin/packages/cmake/package.py
var/spack/repos/builtin/packages/ecmwf-atlas/package.py
var/spack/repos/builtin/packages/libjpeg-turbo/package.py
var/spack/repos/builtin/packages/mapl/package.py
var/spack/repos/builtin/packages/met/package.py
var/spack/repos/builtin/packages/mysql/package.py
var/spack/repos/builtin/packages/nco/package.py
var/spack/repos/builtin/packages/openmpi/package.py

Issues

This is all part of compiling JEDI+GEOS with spack-stack (i.e. not using Baselibs) on more than just Discover: https://github.com/JCSDA-internal/AOP23/issues/33

Testing

Usual CI tests, see below
See Update spack submodule from spack/develop as of 2024/01/26 spack-stack#977 for spack-stack testing

Part 3 of reworking all package metadata to key by `when` conditions. Changes conflict dictionary structure from this: { (requirement_spec, ...): [(when_spec, policy, msg)] } to this: { when_spec: [((requirement_spec, ...), policy, msg), ...] }

`make_when_spec()` was being used in the solver, but it has semantics that are specific to parsing when specs from `package.py`. In particular, it returns `None` when the `when` spec is `False`, and directives are responsible for ignoring that case and not adding requirements, deps, etc. when there's an actual `False` passed in from `package.py`. In `asp.py`, we know that there won't ever be a raw boolean when spec or constraint, so we know we can parse them without any of the special boolean handling. However, we should report where in the file the error happened on error, so this adds some parsing logic to extract the `mark` from YAML and alert the user where the bad parse is. - [x] refactor `config.py` so that basic `spack_yaml` mark info is in its own method - [x] refactor `asp.py` so that it uses the smarter YAML parsing routine - [x] refactor `asp.py` so that YAML input validation for requirements is done up front

Part 4 of reworking all package metadata to key by `when` conditions. Changes conflict dictionary structure from this: { provided_spec: {when_spec, ...} } to this: { when_spec: {provided_spec, ...} }

Shows how to modify environment variables using "setup_build_environment" instead of overriding "edit"

…spack#41986) This fixes an issue where pkg.stage throws because a patch cannot be found, but the patch is redundant because the spec is reused from a build cache and will be installed from existing binaries.

…k#41727) * Move in vs. satisfies to a note and mention special cases of in * Address feedback: oveoverlap -> intersect * Re-word the satisfies versus in note. --------- Co-authored-by: Massimiliano Culpo <[email protected]>

Adds DateTime::Format::MySQL

Adds DateTime::Format::Oracle

Adds MetaCPAN::Client

Adds Email::MIME::ContentType

* perl-memoize: add new package with version 1.16= * fix styling

* glow: add latest version v1.5.1 * update: glow build from source

…le load time. Add 1.0.8.1 update branch. (spack#42010)

* py-pyvista: fix import tests * Skip additional modules

…tel (not patched)

…penmpi@5:, autoreconf is enforced and we need to apply the two-level namespace patch again after the autoreconf step

…into feature/merge_spack_dev_20240126

AlexanderRichert-NOAA · 2024-02-06T19:10:52Z

The only thing that stands out to me is MySQL-- Let's restore the +download_boost functionality since we're still using it in common/packages.yaml, unless we have a different solution there.

RatkoVasic-NOAA

I checked mentioned package.py's and differences with old ones. Looks good to me.

climbfuji · 2024-02-06T20:43:00Z

The only thing that stands out to me is MySQL-- Let's restore the +download_boost functionality since we're still using it in common/packages.yaml, unless we have a different solution there.

Do we need this functionality? On all the HPCs as well as on all the user platforms we rely on mysql being provided as an external package? I'd rather remove that option unless you remind me of why we need it ;-)

AlexanderRichert-NOAA · 2024-02-06T20:45:09Z

JCSDA/spack-stack#705

climbfuji · 2024-02-06T21:19:15Z

JCSDA/spack-stack#705

That would have been my guess but I don't recall what the download_boost option gives us? We need to build boost in spack anyway for other reasons (not just ecflow or mysql, but JEDI itself requires it).

AlexanderRichert-NOAA · 2024-02-06T21:24:47Z

It downloads and uses its own internal boost libraries, so it doesn't impose version constraints on the Spack-installed boost.

climbfuji · 2024-02-06T21:26:34Z

It downloads and uses its own internal boost libraries, so it doesn't impose version constraints on the Spack-installed boost.

Ok, so because of the version constraints for mysql. Let me try to put it back in. CI won't test it though.

climbfuji · 2024-02-06T22:29:11Z

@AlexanderRichert-NOAA I have another suggestion. What about if we take mysql out of the default dependencies and only enable it (as external package) for macOS users who wish to run JEDI-Skylab experiments on their laptops, and on 2 particular systems managed by JCSDA (on AWS)? There is only one particular use case for a local mysql server, and that is running JEDI-Skylab in "localhost" mode. This only happens on macOS laptops at JCSDA, on the JEDI Singlenode AMI for AWS, and on AWS ParallelCluster - all HPCs, all non-JCSDA user installations, all other cloud systems can be installed w/o mysql.

AlexanderRichert-NOAA · 2024-02-06T22:38:13Z

Take it off the HPC systems? Yes that sounds lovely :)

srherbener

The changes in the listed package.py scripts seem okay. I noticed that the openmpi patch for apple-clang two level namespace has this addition:

+    @when("@5.0.0:5.0.1")
+    def autoreconf(self, spec, prefix):
+        perl = which("perl")
+        perl("autogen.pl", "--force")
+        if "+two_level_namespace" in spec and spec.satisfies("platform=darwin"):
+            print("Re-applying configure patch for two_level_namespace on MacOS after autoreconf")
+            filter_file(r"-flat_namespace", "-commons,use_dylibs", "configure")
+

which looks like re-applying the patch for two level namespace when autoreconf is run. This seem like a good thing for us to pick up. Just wanted to point that out.

climbfuji · 2024-02-06T22:55:15Z

Take it off the HPC systems? Yes that sounds lovely :)

Ok, then I'll do that. I leave the PR as is, and file a bug to redo the mysql dependency in the way I described above. Is that ok with you?

AlexanderRichert-NOAA · 2024-02-06T22:56:01Z

Yep works for me! Thanks

climbfuji · 2024-02-06T22:56:39Z

The changes in the listed package.py scripts seem okay. I noticed that the openmpi patch for apple-clang two level namespace has this addition:
+    @when("@5.0.0:5.0.1")
+    def autoreconf(self, spec, prefix):
+        perl = which("perl")
+        perl("autogen.pl", "--force")
+        if "+two_level_namespace" in spec and spec.satisfies("platform=darwin"):
+            print("Re-applying configure patch for two_level_namespace on MacOS after autoreconf")
+            filter_file(r"-flat_namespace", "-commons,use_dylibs", "configure")
+
which looks like re-applying the patch for two level namespace when autoreconf is run. This seem like a good thing for us to pick up. Just wanted to point that out.

Yes, that is needed. As you can see from the code snippet, for openmpi@5: it currently enforces an autoreconf, which overwrites the configure file you patched earlier. I kept the original logic above in place (always patch the default configure) in case someone removes these forced autoreconf in the future.

climbfuji · 2024-02-06T23:03:54Z

@AlexanderRichert-NOAA See JCSDA/spack-stack#990 - if satisfied, please approve. Thanks!

AlexanderRichert-NOAA

Built unified-dev and ufs-weather-model-static on Gaea, everything checks out. Checked recipe diffs and everything looks reasonable.

climbfuji · 2024-02-07T14:36:12Z

Thanks @AlexanderRichert-NOAA and all other reviewers!

tgamblin and others added 30 commits January 9, 2024 00:26

refactor: Index requirements by when spec

d2a9e3f

Part 3 of reworking all package metadata to key by `when` conditions. Changes conflict dictionary structure from this: { (requirement_spec, ...): [(when_spec, policy, msg)] } to this: { when_spec: [((requirement_spec, ...), policy, msg), ...] }

refactor: Index provided virtuals by when spec

6753cc0

Part 4 of reworking all package metadata to key by `when` conditions. Changes conflict dictionary structure from this: { provided_spec: {when_spec, ...} } to this: { when_spec: {provided_spec, ...} }

ParaView: add 5.12.0-RC2 (spack#41992)

651b63d

lfortran: add v0.30.0 (spack#42000)

13e8029

Improve "makefile" build system documentation (spack#41781)

5919bd0

Shows how to modify environment variables using "setup_build_environment" instead of overriding "edit"

py-numpy: add v1.26.3 (spack#41977)

62c7dfc

installer.py: don't dereference stage before installing from binaries (…

d3fb298

…spack#41986) This fixes an issue where pkg.stage throws because a patch cannot be found, but the patch is redundant because the spec is reused from a build cache and will be installed from existing binaries.

paml: adding new version 4.10.7 (spack#41968)

f5d6727

imagemagick: add SVG support (spack#41956)

40bda00

git commit shasums bump (spack#41935)

87f03dd

fpm: add v0.10.0 (spack#42002)

705e0c6

hip-tensor: add new package (spack#40991)

097c041

py-kornia: add v0.7.1 (spack#41888)

362b54c

neko: add v0.7.0, v0.7.1, v0.7.2 and fix package (spack#42012)

7583cf2

fix gpg version parsing for macos (spack#41995)

ba547a7

perl-datetime-format-mysql: New package (spack#42004)

74de7c8

Adds DateTime::Format::MySQL

perl-datetime-format-oracle: New package (spack#42003)

9ffbf5d

Adds DateTime::Format::Oracle

perl-log-any-adapter-callback: New package (spack#41965)

346e04d

perl-metacpan-client: New package (spack#41963)

7f7fcde

Adds MetaCPAN::Client

perl-email-mime-contenttype: New package (spack#41951)

f63f7f4

Adds Email::MIME::ContentType

perl-memoize: add new package with version 1.16 (spack#41938)

b66f2ee

* perl-memoize: add new package with version 1.16= * fix styling

perl-compress-raw-zlib: add new version (spack#41909)

969c643

py-fsspec: fix import tests (spack#41860)

86730c7

glow: add new package (spack#41967)

9ed3f37

* glow: add latest version v1.5.1 * update: glow build from source

dorado: add new package (spack#41988)

c5309b0

Update mpi dependency so we do not load the build mpi version at modu…

5048cd3

…le load time. Add 1.0.8.1 update branch. (spack#42010)

libpspio: move to correct location (spack#42014)

ed4cdb9

py-pyvista: fix import tests (spack#41868)

f330586

* py-pyvista: fix import tests * Skip additional modules

Add patch for [email protected]%intel, add conflicts for [email protected]:5.1.5 % in…

967a929

…tel (not patched)

climbfuji marked this pull request as ready for review January 31, 2024 18:47

climbfuji requested review from RatkoVasic-NOAA, ulmononian and AlexanderRichert-NOAA January 31, 2024 18:47

climbfuji added 2 commits February 2, 2024 11:13

Bug fix in var/spack/repos/builtin/packages/openmpi/package.py: for o…

dda37c3

…penmpi@5:, autoreconf is enforced and we need to apply the two-level namespace patch again after the autoreconf step

Merge branch 'jcsda_emc_spack_stack' of https://github.com/jcsda/spack …

980218a

…into feature/merge_spack_dev_20240126

srherbener mentioned this pull request Feb 5, 2024

Implement workaround for [email protected] linker/loader issue JCSDA/spack-stack#971

Closed

climbfuji requested a review from srherbener February 5, 2024 21:44

climbfuji mentioned this pull request Feb 6, 2024

Upgrade the GSI to Spack-Stack version 1.6.0 NOAA-EMC/GSI#684

Merged

6 tasks

RatkoVasic-NOAA approved these changes Feb 6, 2024

View reviewed changes

srherbener approved these changes Feb 6, 2024

View reviewed changes

climbfuji mentioned this pull request Feb 6, 2024

By default, don't depend on mysql in ewok-env JCSDA/spack-stack#990

Closed

ulmononian approved these changes Feb 7, 2024

View reviewed changes

AlexanderRichert-NOAA approved these changes Feb 7, 2024

View reviewed changes

climbfuji merged commit 03ecae6 into JCSDA:jcsda_emc_spack_stack Feb 7, 2024
15 checks passed

climbfuji deleted the feature/merge_spack_dev_20240126 branch February 7, 2024 14:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge spack/develop as of 2024/01/26 into jcsda_emc_spack_stack #401

Merge spack/develop as of 2024/01/26 into jcsda_emc_spack_stack #401

climbfuji commented Jan 27, 2024 •

edited

Loading

AlexanderRichert-NOAA commented Feb 6, 2024

RatkoVasic-NOAA left a comment

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA commented Feb 6, 2024

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA commented Feb 6, 2024

climbfuji commented Feb 6, 2024

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA commented Feb 6, 2024

srherbener left a comment •

edited

Loading

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA commented Feb 6, 2024

climbfuji commented Feb 6, 2024

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA left a comment

climbfuji commented Feb 7, 2024

Merge spack/develop as of 2024/01/26 into jcsda_emc_spack_stack #401

Merge spack/develop as of 2024/01/26 into jcsda_emc_spack_stack #401

Conversation

climbfuji commented Jan 27, 2024 • edited Loading

Description

Issues

Testing

AlexanderRichert-NOAA commented Feb 6, 2024

RatkoVasic-NOAA left a comment

Choose a reason for hiding this comment

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA commented Feb 6, 2024

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA commented Feb 6, 2024

climbfuji commented Feb 6, 2024

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA commented Feb 6, 2024

srherbener left a comment • edited Loading

Choose a reason for hiding this comment

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA commented Feb 6, 2024

climbfuji commented Feb 6, 2024

climbfuji commented Feb 6, 2024

AlexanderRichert-NOAA left a comment

Choose a reason for hiding this comment

climbfuji commented Feb 7, 2024

climbfuji commented Jan 27, 2024 •

edited

Loading

srherbener left a comment •

edited

Loading