Spack CI + MAPL 3 Issues

[mathomp4]

I am making this issue mainly as a pointer as to why I'm blocking #3298 .

Build Issues with ESMF 8.8.0

In testing Spack with MAPL 2, I found I needed ESMF 8.8.0 (see #3298). But when I tried MAPL3 with that same environment I got:

[ 20%] Building Fortran object field/CMakeFiles/MAPL.field.dir/FieldDelta.F90.o
/home/ubuntu/Models/MAPL3/field/FieldDelta.F90:14:7:

   14 |    use esmf
      |       1
......
  429 |    impure elemental logical function ESMF_GeomEqual(geom1, geom2)
      |                                                   2
Error: Ambiguous interfaces in intrinsic '==' operator for ‘esmf_geomeq’ at (1) and ‘esmf_geomequal’ at (2)
gmake[2]: *** [field/CMakeFiles/MAPL.field.dir/build.make:130: field/CMakeFiles/MAPL.field.dir/FieldDelta.F90.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:6670: field/CMakeFiles/MAPL.field.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2

Now, in those files I see:

MAPL/field/FieldDelta.F90

Lines 51 to 54 in 80afc89

	! Will be in next release of ESMF (8.8?)
	interface operator(==)
	procedure :: ESMF_GeomEqual
	end interface operator(==)

and:

MAPL/field/FieldDelta.F90

Lines 427 to 430 in 80afc89

	! TODO - delete when next ESMF release provides support.
	impure elemental logical function ESMF_GeomEqual(geom1, geom2)
	type(ESMF_Geom), intent(in) :: geom1, geom2

So we'll need to remove those as stated.

Weird zstandard compression issue

I'm also seeing an error with spack + MAPL3 (on bare metal, haven't tried in cloud).

To wit, in my Spack env, netcdf-c was built with zstd support:

[email protected]%[email protected]+blosc~byterange+dap~fsync~hdf4~jna~logging+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared+szip+zstd build_system=autotools patches=0161eb8 arch=linux-ubuntu24.04-x86_64_v3

and during CMake MAPL3 seems to see it:

-- Performing Test NETCDF_HAS_ZSTD
-- Performing Test NETCDF_HAS_ZSTD - Success
-- netCDF has zstandard capability

but the ExtData Zstd tests fail:

...
 Writing:      4 Slices to File:  case1.2004.nc4
pe=00000 FAIL at line=00789    pfio/NetCDF4_FileFormatter.F90           <status=-136>
pe=00000 FAIL at line=00327    pfio/NetCDF4_FileFormatter.F90           <status=-136>
pe=00000 FAIL at line=00064    pfio/HistoryCollection.F90               <status=-136>
pe=00000 FAIL at line=00806    pfio/ServerThread.F90                    <status=-136>
...
The following tests FAILED:
	 65 - ExtData2G_case35 (Failed)                         ESSENTIAL EXTDATA2G_SMALL_TESTS
	 66 - ExtData2G_case36 (Failed)                         ESSENTIAL EXTDATA2G_SMALL_TESTS

The odd thing is, these tests PASS with MAPL2 + Spack:

      Start 51: ExtData2G_case35
35/36 Test #51: ExtData2G_case35 .................   Passed    0.33 sec
      Start 52: ExtData2G_case36
36/36 Test #52: ExtData2G_case36 .................   Passed    0.33 sec

I'll try and figure this out, but I might need to invoke @edwardhartnett as maybe he has knowledge/insight/memories of Spack, netcdf, zstd issues?

[edwardhartnett]

Do you have the failure message from zstd? Is this a LD_LIBRARY_PATH issue, where the zstd library is not being found at runtime? But other times, when LD_LIBRARY_PATH is set correctly, unit tests pass?

[mathomp4]

@edwardhartnett I mean, the error seems to be:

#define NC_ENOFILTER     (-136)    // Filter not defined on variable.

which is...weird. I mean, this code around is essentially identical between MAPL2 and MAPL3 (the only differences are gFTL v1 to v2 interfaces, so nothing to do with netCDF). And I'm using the exact same Spack environment on the same machine!

I'll try and debug with @bena-nasa next week. Just...baffling.

[edwardhartnett]

OK make sure environment var HDF5_PLUGIN_PATH is set to the place where the zstd plugin code is installed. (Check the netcdf-c build output to find where that is.)

IIRC the error you're getting is filter not found, which means that HDF5 is trying to read/write some data, and needs a filter for it, and cannot find that filter in it's filter path. The filter path can be set by setting HDF5_PLUGIN_PATH.

If this is the answer, you have my apology, because this awkwardness is my fault. However, in the next release of netCDF, there is a way to set the plugin path programatically, so that an environment var does not have to be used.

[mathomp4]

@edwardhartnett Ahh. Interesting. I see this:

    def setup_run_environment(self, env):
        if self.spec.satisfies("@4.9.0:+shared"):
            # Both HDF5 and NCZarr backends honor the same environment variable:
            env.append_path("HDF5_PLUGIN_PATH", self.prefix.plugins)

in the spack netcdf-c package, but I wonder if what we need is an "if +zstd, then add a zstd path to that".

I see things like that in, say, sz3:

    def setup_run_environment(self, env):
        if "+hdf5" in self.spec:
            env.prepend_path("HDF5_PLUGIN_PATH", self.prefix.lib64)

Something to test on Monday! (And, I guess, I should test SZ3... 😄 )