Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Building shared ESMF on macOS fails in linker stage #956

Closed
climbfuji opened this issue Jan 17, 2024 · 6 comments
Closed

Building shared ESMF on macOS fails in linker stage #956

climbfuji opened this issue Jan 17, 2024 · 6 comments
Assignees
@climbfuji
Labels
bug Something is not working NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center

Comments

@climbfuji
Copy link
Collaborator

Describe the bug

See PR #953 for the code changes to trigger this error (in CI and on my macOS):

     9748    /Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/openmpi-4.1.6-atso6dm/bin/mpic++ -c  -std=c++11 -O -DNDEBUG -fPIC -x c++ -mmacosx-version-min=10.7 -stdlib=libc++ -pthread -I/Users/heinzell/work/sp
             ack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure/Trace/preload -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w
             7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure/Trace/preload/../include -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/build_config/Darwin.
             gfortranclang.default -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/bui
             ld_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Superstructure -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastr
             ucture/stubs/pthread  -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/include  -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-en
             v/install/apple-clang/13.1.6/netcdf-c-4.9.2-vufz7vj/include -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/parallelio-2.5.10-nkc6ohk/include -I/Users/heinzell/work/spack-stack/spack-stack-geos-
             gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/prologue/yaml-cpp/include -DESMF_NO_INTEGER_1_BYTE -DESMF_NO_INTEGER_2_BYTE -DESMF_LOWERCASE_SINGLEUNDERSCORE -DESMF_MOAB=1 -DESMF_LAPACK=1 -DESMF_LAPACK_
             INTERNAL=1 -DESMF_NO_ACC_SOFTWARE_STACK=1 -DESMF_NETCDF=1 -DESMF_YAMLCPP=1 -DESMF_YAML=1 -DESMF_PIO=1 -DESMF_NO_OPENMP -DESMF_NO_OPENACC -DESMF_BOPT_O -DESMF_TESTCOMPTUNNEL -DS64=1 -DESMF_OS_Darwin=1 -DESMF_COMM=openmpi -DESMF_DIR=/Users/heinzell
             /work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src -D__SDIR__='"src/Infrastructure/Trace/preload"' -DESMF_CXXSTD=11 -DESMF_NO_SIGUSR2 preload.C -o preload.o
     9749    /Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/openmpi-4.1.6-atso6dm/bin/mpic++ -c  -std=c++11 -O -DNDEBUG -fPIC -x c++ -mmacosx-version-min=10.7 -stdlib=libc++ -pthread -I/Users/heinzell/work/sp
             ack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure/Trace/preload -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w
             7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure/Trace/preload/../include -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/build_config/Darwin.
             gfortranclang.default -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/bui
             ld_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Superstructure -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastr
             ucture/stubs/pthread  -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/include  -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-en
             v/install/apple-clang/13.1.6/netcdf-c-4.9.2-vufz7vj/include -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/parallelio-2.5.10-nkc6ohk/include -I/Users/heinzell/work/spack-stack/spack-stack-geos-
             gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/prologue/yaml-cpp/include -DESMF_NO_INTEGER_1_BYTE -DESMF_NO_INTEGER_2_BYTE -DESMF_LOWERCASE_SINGLEUNDERSCORE -DESMF_MOAB=1 -DESMF_LAPACK=1 -DESMF_LAPACK_
             INTERNAL=1 -DESMF_NO_ACC_SOFTWARE_STACK=1 -DESMF_NETCDF=1 -DESMF_YAMLCPP=1 -DESMF_YAML=1 -DESMF_PIO=1 -DESMF_NO_OPENMP -DESMF_NO_OPENACC -DESMF_BOPT_O -DESMF_TESTCOMPTUNNEL -DS64=1 -DESMF_OS_Darwin=1 -DESMF_COMM=openmpi -DESMF_DIR=/Users/heinzell
             /work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src -D__SDIR__='"src/Infrastructure/Trace/preload"' -DESMF_CXXSTD=11 -DESMF_NO_SIGUSR2 preload_mpi.C -o preload_mpi.o
     9750    /Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/openmpi-4.1.6-atso6dm/bin/mpif90  -dynamiclib -o /Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q6
             5v7wblrc7xwb52al7l5smpg2r2yx/spack-src/lib/libO/Darwin.gfortranclang.64.openmpi.default/libesmftrace_preload.dylib preload.o preload_mpi.o wrappers.o wrappers_mpi.o   -pthread -L/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_st
             age/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/lib/libO/Darwin.gfortranclang.64.openmpi.default -L/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/netcdf-c-4.9.2-vufz7vj/lib
             -L/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/netcdf-fortran-4.6.1-zmvqkmu/lib -L/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/parallelio-2.
             5.10-nkc6ohk/lib -L/opt/homebrew/Cellar/gcc@12/12.3.0/bin/../lib/gcc/12/gcc/aarch64-apple-darwin21/12/../../../ -Wl,-rpath,/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2
             r2yx/spack-src/lib/libO/Darwin.gfortranclang.64.openmpi.default -Wl,-rpath,/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/netcdf-c-4.9.2-vufz7vj/lib -Wl,-rpath,/Users/heinzell/work/spack-stack/sp
             ack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/netcdf-fortran-4.6.1-zmvqkmu/lib  -Wl,-rpath,/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/parallelio-2.5.10-nkc6ohk/lib -Wl,-rpath
             ,/opt/homebrew/Cellar/gcc@12/12.3.0/bin/../lib/gcc/12/gcc/aarch64-apple-darwin21/12/../../../  -lgfortran -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -lesmf
  >> 9751    Undefined symbols for architecture arm64:
     9752      "__ZNKSt3__121__basic_string_commonILb1EE20__throw_length_errorEv", referenced from:
     9753          __ZNKSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE20__throw_length_errorEv in wrappers_mpi.o
     9754      "__ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcm", referenced from:
     9755          __ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC2IDnEEPKc in wrappers_mpi.o
     9756      "__ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEED1Ev", referenced from:
     9757          ___wrap_MPI_Allgather in wrappers_mpi.o

     ...

     9773          ___wrap_MPI_Alltoall in wrappers_mpi.o
     9774          ___wrap_MPI_Alltoallv in wrappers_mpi.o
     9775          ___wrap_MPI_Alltoallw in wrappers_mpi.o
     9776          ___wrap_MPI_Barrier in wrappers_mpi.o
     9777          ...
     9778    ld: symbol(s) not found for architecture arm64
  >> 9779    collect2: error: ld returned 1 exit status
  >> 9780    make[2]: *** [tracelib_preload] Error 1
  >> 9781    make[1]: *** [build_tracelibs] Error 2
  >> 9782    make: *** [lib] Error 2

To Reproduce
See #953

Expected behavior
It builds!

System:
macOS (CI, @climbfuji's macOS)

Additional context
openmpi is built by/in spack, not as an external package
@climbfuji climbfuji added the bug Something is not working label Jan 17, 2024
@climbfuji climbfuji added NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center labels Jan 17, 2024
@climbfuji
Copy link
Collaborator Author

@AlexanderRichert-NOAA @ulmononian @mathomp4 FYI

@climbfuji climbfuji mentioned this issue Jan 17, 2024
Open
@mathomp4
Copy link
Collaborator

Interesting. As @oehmke said in esmf-org/esmf#209, try setting ESMF_TRACE_LIB_BUILD=OFF in the build. We've only had to that in Baselibs when using NAG + ESMF 8.6.0 + macOS, but maybe it is more general?

@climbfuji
Copy link
Collaborator Author

That worked! I'll submit a PR to spack with the change for macOS when building shared ESMF. Sorry for the late reply, all-day meeting today ...

This was referenced Jan 18, 2024
Closed
Merged
@climbfuji
Copy link
Collaborator Author

See:

spack/spack#42133 # issue in spack develop
spack/spack#42134 # pull request in spack develop (cherry-picked from PR to our spack fork)
JCSDA/spack#393 # pull request to our spack fork

@climbfuji climbfuji mentioned this issue Jan 18, 2024
Merged
8 tasks
@oehmke
Copy link

oehmke commented Jan 18, 2024 via email

@climbfuji
Copy link
Collaborator Author

Closed via JCSDA/spack#393

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@climbfuji climbfuji

Labels
bug Something is not working NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mathomp4 @climbfuji @oehmke @ulmononian @AlexanderRichert-NOAA