Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update GNU container recipe, add Clang container #320

Merged
merged 24 commits into from
Sep 12, 2022

Conversation

climbfuji
Copy link
Collaborator

@climbfuji climbfuji commented Aug 26, 2022

Description

  1. Update ubuntu-gnu-openmpi docker recipe:
    • use the default [email protected] compiler instead of [email protected] - there were problems with both of them in the container build
    • use a few more external packages such as qt@5 and wget
    • set environment variable TZ to avoid the container build hanging indefinitely in the final phase when installing tzdata
    • do not strip binaries after build to avoid warnings at end of build process
    • Use new functionality of extra instructions in pre-build and pre-final phases
  2. Add ubuntu-clang-mpich docker recipe, almost identical to ubuntu-gnu-openmpi with minimal necessary changes
  3. Add README.md in configs/containers to list the specs to install for each spack-stack release and application

Notes

@climbfuji climbfuji marked this pull request as ready for review August 30, 2022 18:54
@climbfuji climbfuji changed the title Feature/update gnu container Update GNU container recipe, add Clang container Sep 2, 2022
@climbfuji
Copy link
Collaborator Author

Excellent: on the new clang+mpich container, only the folllowing three tests fail in jedi-bundle:

Total Test time (real) = 9047.02 sec

The following tests FAILED:
	1165 - ufo_test_tier1_test_ufo_conventionalprofileprocessing_UInterp_OPScomparison (Failed)
	1317 - test_soca_convertstate (Failed)
	1375 - test_soca_3dvarlowres_soca (Failed)
Errors while running CTest
Output from these tests are in: /root/jedi/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

@climbfuji
Copy link
Collaborator Author

For jedi-bundle ctest failures with gcc+openmpi, see https://github.com/JCSDA-internal/jedi-bundle/issues/12

@climbfuji
Copy link
Collaborator Author

@mer-a-o This is definitely ready for testing from your side. See comments in this PR about my testing etc.

pre_final: |
#Set environment variables for installing tzdata
ENV DEBIAN_FRONTEND=noninteractive && \
TZ=Etc/UTC && \
Copy link
Collaborator

@mer-a-o mer-a-o Sep 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line doesn't need && \. However, after removing && \ I still get and error with docker build command complaining about extra && in the Dockerfile. I had remove the space between && and \ in for it to work, which doesn't make much sense. So with this I didn't get the error about extra &&

ENV DEBIAN_FRONTEND=noninteractive &&\
TZ=Etc/UTC

The error I get:

Error response from daemon: failed to parse Dockerfile: Syntax error - can't find = in "&&". Must be of the form: name=value

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I figured it out. You don't need && in ENV. So it should be

        ENV DEBIAN_FRONTEND=noninteractive 
        TZ=Etc/UTC


# Extra instructions
extra_instructions:
pre_final: |
#Set environment variables for installing tzdata
ENV DEBIAN_FRONTEND=noninteractive &&\
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove &&\ from here too. Thanks.

@mer-a-o
Copy link
Collaborator

mer-a-o commented Sep 7, 2022

Building clang container failed with this error:

==> Error: ProcessError: Command exited with status 2:
    'make' '-j2'

2 errors found in build log:
     1     ==> pigz: Executing phase: 'edit'
     2     ==> [2022-09-07-19:16:46.811453] Using default implementation: skipp
           ing edit phase.
     3     ==> pigz: Executing phase: 'build'
     4     ==> [2022-09-07-19:16:46.833180] 'make' '-j2'
     5     gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -Wcast-qual   -c -o pigz.
           o pigz.c
     6     gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -Wcast-qual   -c -o yarn.
           o yarn.c
  >> 7     pigz.c:415:10: fatal error: zlib.h: No such file or directory
     8       415 | #include "zlib.h"       // deflateInit2(), deflateReset(), d
           eflate(),
     9           |          ^~~~~~~~
     10    compilation terminated.
  >> 11    make: *** [<builtin>: pigz.o] Error 1
     12    make: *** Waiting for unfinished jobs....

See build log for details:
  /tmp/root/spack-stage/spack-stage-pigz-2.7-fhhp3sk4w6ipgtmjsc7cdinvbpuaqe6x/spack-build-out.txt

@climbfuji
Copy link
Collaborator Author

Building clang container failed with this error:

==> Error: ProcessError: Command exited with status 2:
    'make' '-j2'

2 errors found in build log:
     1     ==> pigz: Executing phase: 'edit'
     2     ==> [2022-09-07-19:16:46.811453] Using default implementation: skipp
           ing edit phase.
     3     ==> pigz: Executing phase: 'build'
     4     ==> [2022-09-07-19:16:46.833180] 'make' '-j2'
     5     gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -Wcast-qual   -c -o pigz.
           o pigz.c
     6     gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -Wcast-qual   -c -o yarn.
           o yarn.c
  >> 7     pigz.c:415:10: fatal error: zlib.h: No such file or directory
     8       415 | #include "zlib.h"       // deflateInit2(), deflateReset(), d
           eflate(),
     9           |          ^~~~~~~~
     10    compilation terminated.
  >> 11    make: *** [<builtin>: pigz.o] Error 1
     12    make: *** Waiting for unfinished jobs....

See build log for details:
  /tmp/root/spack-stage/spack-stage-pigz-2.7-fhhp3sk4w6ipgtmjsc7cdinvbpuaqe6x/spack-build-out.txt

I just fixed a minor bug in the pre_final/final instructions for the clang container. With those, unrelated to your error, I could build the container just fine. I am using Docker version 20.10.14, build a224086 on my macOS.

@climbfuji climbfuji requested a review from mer-a-o September 12, 2022 19:34
@climbfuji
Copy link
Collaborator Author

@mer-a-o This is ready for review.

@climbfuji climbfuji enabled auto-merge (squash) September 12, 2022 19:35
@climbfuji
Copy link
Collaborator Author

I am going to merge this now. Any updates needed will be made in a follow-up PR. Thanks for the review, @mer-a-o!

@climbfuji climbfuji disabled auto-merge September 12, 2022 22:24
@climbfuji climbfuji merged commit d6a3e8d into JCSDA:develop Sep 12, 2022
@climbfuji climbfuji deleted the feature/update_gnu_container branch September 12, 2022 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Need container extra instructions prior to build phase
2 participants