flux-core: 0.67.0 builds failing due to Python environment

[vsoch]

@trws does this look salient to you? Why would we need to define these python envars in spack now?

[trws]

That's a new one on me, maybe it's having trouble getting them from the python it found, but I don't know what it tries that fails to get it to that point. Maybe this is another bit of fallout from setup tools not being there by default without request anymore?

[vsoch]

I've been burned by that for two other projects this week! I don't have any ideas either, so I'll give that a shot. The ubuntu images are definitely latest so we are getting a newer Python. I can't reproduce the error locally so having trouble debugging. I'll try installing setuptools and report back.

[vsoch]

Same bug, so wasn't that!

[trws]

What config or base produces this? I could try and poke at it but just realized I'm not sure where it came from.

[vsoch]

This is the sciworks action that checks for new updates, and then updates the spack package.py with the new version, and then basically issues a spack install. So you can see the entire environment here: https://github.com/sciworks/spack-updater/blob/fafb5f17a4a82cfd3d5de358c7ea6803fdcdd727/release-check/action.yaml#L115-L119. So from tonight's failed run:

https://github.com/flux-framework/spack/actions/runs/11357977767/job/31591736851

If you take as much of that environment as possible and then add:

version("0.67.0", sha256="9406e776cbeff971881143fd1b94c42ec912e5b226401d2d3d91d766dd81de8c")

To the package.py and install it explicitly, it might reproduce. Versions of stuff installed on the runner:

==> The following specs have been detected on this system and added to /home/runner/.spack/packages.yaml
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Let me know if you learn anything interesting!

[vsoch]

@trws I'm disabling all automation for flux-core - it's a huge number of notifications I get daily and I don't know how to fix. Flux core won't be automatically updated in spack, but the others will continue to be.

[trws]

I've been completely unable to repro this issue unfortunately, and the builds don't seem to store off the config.log or anything else I can use to debug in the action. The error happens when the autoconf macro can't determine the version (it thinks the python version is less than 2.1.0). The only way this makes any sense is if the test for version is busted. This is the python one-liner it uses to output either True or False:

$PYTHON -c "import sys; \
		ver = sys.version.split ()[0]; \
		print (ver >= '2.1.0')"

This is not a good test, but I'm really not sure how it could fail without more information.

[vsoch]

Could it be an issue somewhere of needing to specify python3 instead of just python?

[trws]

It's possible, autoconf would look for both but if something set PYTHON=python or something like that it could do it.

[vsoch]

@trws I was able to get passed that error by telling spack exactly where the PYTHON is (I don't know how to do this from the self.spec["python"] - it has to be for the environment view - in there somewhere?

def setup_build_environment(self, env):
        #  Ensure ./fluxometer.lua can be found during flux's make check
        env.append_path("LUA_PATH", "./?.lua", separator=";")
       # This is what I added
        env.set('PYTHON', '/opt/views/._view/jxzbhbafj75ohx5f5kzx7pqdzkmjivvx/bin/python')

But that hits another error that is far uglier during build:

[+] /opt/software/linux-rocky9-x86_64/gcc-11.4.1/openssh-9.7p1-ykqdpe3xtfho3vvdrulxr3fwm5xgcgxp
==> Installing flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3 [76/87]
==> No binary for flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3 found: installing from source
==> No patches needed for flux-core
==> flux-core: Executing phase: 'autoreconf'
==> flux-core: Executing phase: 'configure'
==> flux-core: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16' 'V=1'

14 errors found in build log:
     96      autoreconf: configure.ac: not using Intltool
     97      autoreconf: configure.ac: not using Gtkdoc
     98      autoreconf: running: aclocal --force -I config
     99      autoreconf: running: /opt/software/linux-rocky9-x86_64/gcc-11.4.1/autoconf-2.72-d2ogfh2eqhv7jxfhsve4sjlbvldsknla/bin/autoconf --force
     100     autoreconf: running: /opt/software/linux-rocky9-x86_64/gcc-11.4.1/autoconf-2.72-d2ogfh2eqhv7jxfhsve4sjlbvldsknla/bin/autoheader --force
     101     autoreconf: running: automake --add-missing --copy --force-missing
  >> 102     configure.ac:125: installing 'config/compile'
  >> 103     configure.ac:9: installing 'config/config.guess'
  >> 104     configure.ac:9: installing 'config/config.sub'
  >> 105     configure.ac:22: installing 'config/install-sh'
  >> 106     configure.ac:22: installing 'config/missing'
     107     doc/Makefile.am: installing 'config/depcomp'
     108     parallel-tests: installing 'config/test-driver'
  >> 109     src/bindings/python/_flux/Makefile.am:35: installing 'config/py-compile'
     110     autoreconf: 'config/config.sub' is updated
     111     autoreconf: 'config/config.guess' is updated
     112     autoreconf: 'config/install-sh' is updated
     113     autoreconf: Leaving directory '.'
     114     Now run ./configure.
     115     ==> flux-core: Executing phase: 'configure'

     ...

     2695    gcc: fatal error: cannot specify '-o' with '-c', '-S' or '-E' with multiple files
     2696    compilation terminated.
     2697    gcc: fatal error: cannot specify '-o' with '-c', '-S' or '-E' with multiple files
     2698    compilation terminated.
     2699    gcc: fatal error: cannot specify '-o' with '-c', '-S' or '-E' with multiple files
     2700    compilation terminated.
  >> 2701    make[3]: *** [Makefile:1053: flux-lua.lo] Error 1
     2702    make[3]: *** Waiting for unfinished jobs....
  >> 2703    make[3]: *** [Makefile:1053: lutil.lo] Error 1
     2704    gcc: fatal error: cannot specify '-o' with '-c', '-S' or '-E' with multiple files
     2705    compilation terminated.
     2706    gcc: fatal error: cannot specify '-o' with '-c', '-S' or '-E' with multiple files
     2707    compilation terminated.
  >> 2708    make[3]: *** [Makefile:1053: zmsg-lua.lo] Error 1
  >> 2709    make[3]: *** [Makefile:1053: lalarm.lo] Error 1
  >> 2710    make[3]: *** [Makefile:1053: jansson-lua.lo] Error 1
     2711    make[3]: Leaving directory '/tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-src/spack-build/src/bindings/lua'
  >> 2712    make[2]: *** [Makefile:500: all-recursive] Error 1
     2713    make[2]: Leaving directory '/tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-src/spack-build/src/bindings'
  >> 2714    make[1]: *** [Makefile:516: all-recursive] Error 1
     2715    make[1]: Leaving directory '/tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-src/spack-build/src'
  >> 2716    make: *** [Makefile:597: all-recursive] Error 1

See build log for details:
  /tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-build-out.txt

==> Warning: Skipping build of flux-pmix-0.5.0-7chcreeshvg2psipdgogg6w6u27uzijj since flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3 failed
==> Warning: Skipping build of flux-sched-0.33.1-pagscakavv4yanucmxpudihejibqeoug since flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3 failed
==> Error: Terminating after first install failure: ProcessError: Command exited with status 2:
    'make' '-j16' 'V=1'

[vsoch]

This is the last line attempted from the log:

libtool: compile:  /opt/spack/lib/spack/env/gcc/gcc -DHAVE_CONFIG_H -I. -I/tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-src/src/bindings/lua -I../../../config -I/tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-src -I/tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-src/src/include -I/tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-src/src/common/libccan -I../../../src/common/libflux /opt/views/._view/jxzbhbafj75ohx5f5kzx7pqdzkmjivvx/include/lualib.h -Wall -Werror -Wno-strict-aliasing -Wno-error=deprecated-declarations -Werror=missing-field-initializers -Wno-parentheses -Wno-error=parentheses -g -O2 -MT lalarm.lo -MD -MP -MF .deps/lalarm.Tpo -c /tmp/root/spack-stage/spack-stage-flux-core-master-ymkorfx7viwtmie5z5qfedcjwwiewjo3/spack-src/src/bindings/lua/lalarm.c  -fPIC -DPIC -o .libs/lalarm.o
gcc: fatal error: cannot specify '-o' with '-c', '-S' or '-E' with multiple files

[trws]

@vsoch, I really would like to get this fixed, but I keep having no luck actually reproing it locally. Have you been able to do that, and if so could you send me a script/command? Debugging it in GHA is proving deeply challenging.

[vsoch]

Let me re-enable with a PR and reproduce with a fresh environment (in GitHub actions) and go from there! With spack, sometimes things just magically fix. 🪄

[vsoch]

Failed again - I am re-running with an upload of the spack stage (as an artifact) so we can see the whole log.

[vsoch]

Ah there we go - it wants setuptools?! 🤔

==> flux-core: Executing phase: 'autoreconf'
==> flux-core: Executing phase: 'configure'
==> [2024-12-31-02:39:53.651421] Find (max depth = None): ['spack-build'] ['configure']
==> [2024-12-31-02:39:53.651884] find must skip /tmp/runner/spack-stage/spack-stage-flux-core-0.67.0-wmjofn4zaroi2encgt44k5rq4s5ttqym/spack-src/spack-build: ENOENT [Errno 2] No such file or directory: '/tmp/runner/spack-stage/spack-stage-flux-core-0.67.0-wmjofn4zaroi2encgt44k5rq4s5ttqym/spack-src/spack-build'
==> [2024-12-31-02:39:53.651932] Find complete: ['spack-build'] ['configure']
==> [2024-12-31-02:39:53.653448] '/tmp/runner/spack-stage/spack-stage-flux-core-0.67.0-wmjofn4zaroi2encgt44k5rq4s5ttqym/spack-src/configure' '--prefix=/opt/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-14.2.0/flux-core-0.67.0-wmjofn4zaroi2encgt44k5rq4s5ttqym' '--enable-pylint=no' '--disable-docs'
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking installation directory variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking how to create a pax tar archive... gnutar
checking whether make supports nested variables... (cached) yes
checking whether to enable maintainer-specific portions of Makefiles... yes
checking for a sed that does not truncate output... /usr/bin/sed
checking Major version... 0
checking Minor version... 67
checking Point version... 0
checking whether version number is sane... yes
checking for pkg-config... /opt/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-14.2.0/pkgconf-2.2.0-3jpgeezss7c7t646o6jrruy2yjfoul7g/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for gcc... /opt/spack/lib/spack/env/gcc/gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether /opt/spack/lib/spack/env/gcc/gcc accepts -g... yes
checking for /opt/spack/lib/spack/env/gcc/gcc option to enable C11 features... none needed
checking whether /opt/spack/lib/spack/env/gcc/gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of /opt/spack/lib/spack/env/gcc/gcc... gcc3
checking for C compiler vendor... gnu
checking for C compiler version... 14.2.0
checking If -Werror=missing-field-initializers can be used... yes
checking whether to enable a sanitizer tool... no
checking for rsh... no
checking for ssh... /usr/bin/ssh
checking how to print strings... printf
checking for a sed that does not truncate output... (cached) /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by /opt/spack/lib/spack/env/gcc/gcc... /opt/spack/lib/spack/env/ld
checking if the linker (/opt/spack/lib/spack/env/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 3145728
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /opt/spack/lib/spack/env/ld option to reload object files... -r
checking for file... file
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from /opt/spack/lib/spack/env/gcc/gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for mt... mt
checking if mt is a manifest tool... no
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if /opt/spack/lib/spack/env/gcc/gcc supports -fno-rtti -fno-exceptions... no
checking for /opt/spack/lib/spack/env/gcc/gcc option to produce PIC... -fPIC -DPIC
checking if /opt/spack/lib/spack/env/gcc/gcc PIC flag -fPIC -DPIC works... yes
checking if /opt/spack/lib/spack/env/gcc/gcc static flag -static works... yes
checking if /opt/spack/lib/spack/env/gcc/gcc supports -c -o file.o... yes
checking if /opt/spack/lib/spack/env/gcc/gcc supports -c -o file.o... (cached) yes
checking whether the /opt/spack/lib/spack/env/gcc/gcc linker (/opt/spack/lib/spack/env/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for gawk... (cached) gawk
checking for pthread.h... yes
checking for getopt.h... yes
checking for fcntl.h... yes
checking for limits.h... yes
checking for strings.h... (cached) yes
checking for syslog.h... yes
checking for unistd.h... (cached) yes
checking for sys/cdefs.h... yes
checking for sys/param.h... yes
checking for stdarg.h... yes
checking for locale.h... yes
checking for xlocale.h... no
checking for endian.h... yes
checking for inttypes.h... (cached) yes
checking whether byte ordering is bigendian... no
checking for an ANSI C-conforming const... yes
checking for size_t... yes
checking size of int... 4
checking size of long... 8
checking size of long long... 8
checking size of uintptr_t... 8
checking size of ptrdiff_t... 8
checking size of size_t... 8
checking for getopt_long... yes
checking for vsnprintf... yes
checking for vsscanf... yes
checking for realloc... yes
checking for strcasecmp... yes
checking for strdup... yes
checking for strerror... yes
checking for snprintf... yes
checking for vsnprintf... (cached) yes
checking for vasprintf... yes
checking for open... yes
checking for vsyslog... yes
checking for strncasecmp... yes
checking for setlocale... yes
checking for uselocale... yes
checking for strlcpy... yes
checking for strlcat... yes
checking for argz_add... yes
checking for envz_add... yes
checking how to link against pthreads... -lpthread
checking for clock_gettime in default libs... yes
checking for dlerror in default libs... yes
checking for floor in -lm... yes
checking for python3... /opt/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-14.2.0/python-venv-1.0-xud43f27cqu3enutqy6xv3krycsyamec/bin/python3
checking for a version of Python >= '2.1.0'... Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
no
configure: WARNING:
This version of the AC_PYTHON_DEVEL macro
doesn't work properly with versions of Python before
2.1.0. You may need to re-run configure, setting the
variables PYTHON_CPPFLAGS, PYTHON_LIBS, PYTHON_SITE_PKG,
PYTHON_EXTRA_LIBS and PYTHON_EXTRA_LDFLAGS by hand.
Moreover, to disable this check, set PYTHON_NOVERSIONCHECK
to something else than an empty string.

configure: error: in `/tmp/runner/spack-stage/spack-stage-flux-core-0.67.0-wmjofn4zaroi2encgt44k5rq4s5ttqym/spack-src/spack-build':
configure: error: Giving up
See `config.log' for more details

[vsoch]

I'll try adding it. I'm not sure why it's not triggering for other flux-core builds that are using 24.04 too.

[vsoch]

I think I fixed it, see #274