Installation issue: py-scipy / spack-stack @ develop fails to configure with oneAPI @ 2024.2.1 on hercules and orion

[rickgrubin-noaa]

Describe the issue

py-scipy fails to configure / build / install when spec'd for oneAPI @ 2024.2.1 for spack-stack @ develop (updated with two PRs)

py-scipy @ 1.14.1 spec: py-scipy.spec.txt

This failure prevents a unified-dev env from building / installing.

Pertinent error output:

• log.install.txt
• build-stage.txt

The failure appears identical to that which is reported in Installation issue: [email protected] with Intel Classic compilers #48243 and fixed in Add conflict for [email protected]: with Intel Classic #48251, despite that PR being applied.

To Reproduce

spack-stack @ develop with changes incorporated from:

Avoid linking to libirc.so in spack (parallel-netcdf), turn off crypt variant for Python, and update Orion site config to fix tar issue #1435

and

Bug fix in parallel-netcdf to avoid linking to libirc.so AND cherry-pick spack develop PR 48251 (conflict Intel Classic with [email protected]) #495.

The fix from Add conflict for [email protected]: with Intel Classic #48251 is already present in the JCSDA spack specified in spack-stack @ develop / .gitmodules -- the build-stage output clearly shows that a valid oneAPI compiler is found.

Configuration files:

• site-compilers.yaml site-compilers.yaml.txt

• site-packages_oneapi.yaml site-packages_oneapi.yaml.txt

• site-packages.yaml site-packages.yaml.txt

• common-packages_oneapi.yaml common-packages_oneapi.yaml.txt

• common-packages.yaml common-packages.yaml.txt

Expected behavior

py-scipy configures, builds, and installs with oneAPI @ 2024.2.1 compilers (as a preface to spack-stack @ release/1.9.0)

System:

tier1 EPIC hosts hercules (linux-rocky9-icelake) and orion (linux-rocky9-skylake_avx512)

Additional context

A test env to install only py-scipy % gcc (and dependencies) successfully configures / builds / installs.

py-scipy @ 1.14.1 % gcc spec: py-scipy-spec.gcc.txt

Given this success, I'm unclear if something is misconfigured or if this is a bug somewhere.

[climbfuji]

I see the meson error in the install log. Can you also upload the meson log file, please?

/work/noaa/epic/role-epic/spack-stack-testing/spack-stack-1.9.0/cache/build_stage/spack-stage-py-scipy-1.14.1-hsd4od3zcazqqupk5lhpinnr3y4lig2g/spack-src/build/meson-logs/meson-log.txt

[rickgrubin-noaa]

I see the meson error in the install log. Can you also upload the meson log file, please?

That's referenced as build-stage.txt in the attached files.

[climbfuji]

I see it now, thanks. I am wondering why this is a problem with icx. What version is the backend GNU compiler?

[rickgrubin-noaa]

I see it now, thanks. I am wondering why this is a problem with icx. What version is the backend GNU compiler?

System-provided: 11.3.1 which ought to be adequate. The meson-build output didn't care for GNU ld version 2.35.2-24 as a linker; the cfg for icx says, if it's helpful:

-Wl,-rpath,/apps/spack-managed-x86_64_v3-v1.0/gcc-11.3.1/intel-oneapi-compilers-2024.2.1-podbez65l57ms4uba527kg7pomxk5y3m/compiler/2024.2/lib --gcc-toolchain=/usr

---

Module available to use, if desired: 14.2.0

The successful py-scipy only env build with %gcc was with 14.2.0, so it makes sense to reconfigure a unified-dev build that sets the prepend_path vars for PATH / LD_LIBRARY_PATH / CPATH to use those available with gcc/14.2.0. It's time-consuming because py-scipy is one of the last packages built for a unified-dev build. Then again, the icx cfg settings may render this moot.

Separate but potentially related question: does it make sense to require %gcc for building glibc ? Doesn't seem to have been at issue before, as far as I can tell, but am curious.

[climbfuji]

gcc@11 is more than enough for C++-17 - I never tried using gcc@14 as backend for Intel oneAPI.

Not sure about glibc either. We don't need to build this with gcc on any of the other platforms.

I am pretty sure I can build py-scipy on the NRL machines with oneAPI 2024 (icx, icpx, ifort) using gcc@11 as backend.

[rickgrubin-noaa]

gcc@11 is more than enough for C++-17 - I never tried using gcc@14 as backend for Intel oneAPI.

That's why I chose to use the system-supplied gcc@11 -- it met the requirements.

Not sure about glibc either. We don't need to build this with gcc on any of the other platforms.

Understood, was just curious.

I am pretty sure I can build py-scipy on the NRL machines with oneAPI 2024 (icx, icpx, ifort) using gcc@11 as backend.

Hence raising this issue -- I suspect it's probably not showing up elsewhere, and am concerned that it's a config issue / error on my part. What holds me back a bit there is the meson-build error about GNU ld.

[rickgrubin-noaa]

py-scipy installs with oneAPI @ 2024.2.1 on other hosts (ursa, for example). That particular installation was built with intel-oneapi-mkl.

On hercules and orion, I was building with openblas. This works with GNU compilers, but not with oneAPI compilers.

Building a test env for just py-scipy, and using --keep-stage to retain the build dirs, portions of the meson build dirs for py-scipy are incomplete, when compared to a build_cache dir on ursa. In particular, the directory that contains test C / C++ / Fortran files to verify the compilers, as well as ancillary config files, is empty, as is a dir (meson-info) that is clearly written when configuring, as it contains host-specific pathing to compilers and such.

The error that shows up in the meson-log.txt file happens because there's no test file to compile. This is quite odd.

Trying again (standalone py-scipy env) with intel-oneapi-mkl instead of openblas yielded the same negative result.

For whatever reason(s), py-scipy is not configuring correctly / configuring incompletely to build / install. There are no ninja-generated make / etc. files -- files that exist on other hosts with successful builds.

[rickgrubin-noaa]

Still see this behavior only on hercules and orion, and nowhere else -- be it EPIC hosts, or elsewhere.

Closing as it can be solved with:

  py-scipy:
    require:
      '%gcc'

in configs/sites/tier1/hercules/packages_oneapi.yaml

[rickgrubin-noaa]

Observations when building py-scipy in release/1.9.0

Building py-scipy @ 1.14.1 has been something of an odyssey for release/1.9.0 and oneAPI @ v2024.2.1 compilers on various EPIC tier1 hosts. A brief summary:

• hera
  • successfully builds with oneAPI (see notes below)
• hercules
  • fails to build with oneAPI (must specify '%gcc' to build)
• orion
  • fails to build with oneAPI (must specify '%gcc' to build)
• ursa
  • successfully builds with oneAPI (see notes below)

A note (from an above comment) with respect to hercules and orion and why GNU compilers are used to build on those hosts:

[for oneAPI compilers] Building a test env for just py-scipy, and using --keep-stage to retain the build dirs, portions of the meson build dirs for py-scipy are incomplete, when compared to a build_cache dir on ursa. In particular, the directory that contains test C / C++ / Fortran files to verify the compilers, as well as ancillary config files, is empty, as is a dir (meson-info) that is clearly written when configuring, as it contains host-specific pathing to compilers and such.

The error that shows up in the meson-log.txt file happens because there's no test file to compile <-- the empty dir noted above.

Given that, there are some differences between these hosts with respect to system-provided compiler modules:

• hera
  • module intel/2024.2.1
    • does not set env vars for CC / CXX / FC / F77
• hercules
  • module intel-oneapi-compilers/2024.2.1 (requires loading spack-managed-x86-64_v3 first)
    • sets env vars for CC / CXX / FC / F77
    • FC and F77 are set to ifx
• orion
  • module intel-oneapi-compilers/2024.2.1 (requires loading spack-managed-x86-64_v3 first)
    • sets env vars for CC / CXX / FC / F77
    • FC and F77 are set to ifx
• ursa
  • module intel-oneapi-compilers/2024.2.1
    • sets env vars for CC / CXX / FC / F77
    • FC and F77 are set to ifx

For each of these hosts, site-specific compilers.yaml files set FC and F77 to /path/to/ifort (as specified in the system module).

---

Reviewing install log files for each of the hosts that successfully build with oneAPI (hera, ursa):

• py-scipy configures to use FC == ifort (as expected)

Fortran compiler for the host machine: /contrib/spack-stack/spack-stack-1.9.0-release/spack/lib/spack/env/oneapi/ifx (intel 2021.13.1 "ifort (IFORT) 2021.13.1 20240703")

• on hera and ursa, some Fortran objects within the package generate a warning message when building, e.g.:

[66/1383] Compiling Fortran object scipy/sparse/linalg/_propack/liblib__spropack.a.p/PROPACK_single_smgs.risc.F.o
 ../scipy/sparse/linalg/_propack/PROPACK/single/smgs.risc.F(59): warning #6477: Fortran 90 does not allow this statement or directive.
 CDIR$ LOOP COUNT(10000)
 ------^

[102/1383] Compiling Fortran object scipy/sparse/linalg/_propack/liblib__zpropack.a.p/PROPACK_complex16_zblasext.F.o
../scipy/sparse/linalg/_propack/PROPACK/complex16/zblasext.F(236): warning #6916: Fortran 90 does not allow this length specification.   [16]
        complex*16 x(*)
  --------------^

../scipy/sparse/linalg/_propack/PROPACK/complex16/zblasext.F(506): warning #7416: Fortran 90 does not allow this intrinsic procedure.   [DCMPLX]
        if (alpha.eq.dcmplx(zero,zero) .and.
  -------------------^

[202/1383] Compiling Fortran object scipy/integrate/liblsoda_lib.a.p/odepack_bnorm.f.o
../scipy/integrate/odepack/bnorm.f(24): warning #6028: DO termination on a statement other than an END DO or CONTINUE is an obsolescent feature in Fortran 90.
 10       sum = sum + dabs(a(i1-j,j))/w(j)
-^

[289/1383] Compiling Fortran object scipy/interpolate/libfitpack_lib.a.p/fitpack_fptrnp.f.o
../scipy/interpolate/fitpack/fptrnp.f(42): warning #6033: Sharing of a DO termination statement by more than one DO statement is an obsolescent feature in Fortran 90.  Use an END DO or CONTINUE statement for each DO statement.   [100]
 100  continue
-^

There exist a lot of these warnings. None of these warnings occur when building with GNU compilers (v11.x and 12.x)

---

py-scipy's configure steps verifies (what it considers to be) valid C / CXX/ FC compilers, as well as cmake / cython / pkg-config / python (all built within the stack) and a few other tools (e.g. f2py).

In the case of each host, stack versions for cython / pkg-config / python are found and used; this is not the case for cmake. A review of py-scipy/package.py shows that the first three components are dependencies (depends_on()), and cmake is not.

Reviewing install log files for all hosts, regardless of compiler (all hosts in question here):

• the stack's cmake package is not used
  • if cmake exists on the host, it is found / used (e.g. /usr/bin/cmake)
  • if cmake does not exist (e.g. not installed) on the host (e.g. ursa), cmake is not found / not configured
  • this situation doesn't seem to impact build / install
• should cmake be a dependency for py-scipy (this is a somewhat rhetorical question) ?