Skip to content

IAL data location

Jump to bottom
Oldřich Španiel edited this page May 23, 2024 · 53 revisions

Files on ATOS

  • CY46T1 installation:

    /home/snh02/pack/46t1_main.01.OMPIIFC2104.x (root pack)

    /home/snh02/sources/cy46t1/cy46t1_bf.07.src.tgz – source code

    /home/snh02/sources/cy46t1/ignore4compil.46T1_bf.07 – the list of an "obsolete" files

    .genesis – saved command line for creating the root pack

    /home/rme/public/bin/gmkpack.6.9.4/util/gmkpack -f /home/snh02/pack -r 46t1 -b main -K -a -n 01 -l OMPIIFC2104 -o x -p masterodb

    .gmkfile/OMPIIFC2104.AA – configuration file for compilation/linking

    ics_masterodb – script for compilation/linking of MASTERODB executable

    OUT_1 – output from ics_masterodb

    /home/snh02/pack/46t1_fix.01.OMPIIFC2104.x – home pack under root pack 46t1_main.01.OMPIIFC2104.x

    src/local/ifsaux/fa/facgrm.F90 – fix for subhourly inline FP

  • CY46T1 testbed ALARO:

    /home/snh02/work/cy46t1/alaro

    data/lbc - LBC/INIT for different domains (4.5km, 2.3 km hor. res.)

    namelist

    script - scripts and output from slurm job

    /ec/res4/scratch/snh02 ($SCRATCH) - some outputs files from the jobs

    cy46t1_alaro_grb2 cy46t1_alaro_Praha cy46t1_alaro_shmu

Remark 1:

ECCODES_DEFINITION_PATH=/home/snh02/opt/share/eccodes/definitions:/perm/rme/packages/eccodes-2.21.0/INTEL/2021.4/share/eccodes/definitions ECCODES_SAMPLES_PATH=/home/snh02/opt/share/eccodes/ifs_samples/grib1_mlgrib2

[snh02@ad6-100 cy46t1_grb2]$ grib_ls /ec/res4/scratch/snh02/cy46t1_grb2/GRIBPFSHMUGRB2+0002:30 | head

/ec/res4/scratch/snh02/cy46t1_grb2/GRIBPFSHMUGRB2+0002:30

edition centre date dataType gridType stepRange typeOfLevel level shortName packingType_

2 lfpw 20220816 fc lambert_lam 150 surface 0 z grid_ieee

2 lfpw 20220816 fc lambert_lam 150 surface 0 t grid_second_order

2 lfpw 20220816 fc lambert_lam 150 surface 0 sd grid_second_order

Remark 2:

Some rules for slurm have been changed:

#SBATCH --time=

#SBATCH --mem=

  • CY46T1 testbed AROME:

    /home/snh02/work/cy46t1/arome

    data - LBC/INIT + SFX input files

    namelist - A-to-tos-cy46t1 AROME forecast,

    script - scripts and output from slurm job

    /ec/res4/scratch/snh02 ($SCRATCH) - some outputs files from the jobs

    cy46t1_arome cy46t1_arome_grb2

  • CY48T3 installation:

    /home/snh02/pack/48t3_main.01.OMPIIFC2104.x

    /home/snh02/sources/cy48t3/IAL-CY48T3.tar.gz - sources

    /home/snh02/sources/cy48t3/hub.tar - sources ecSDK (ecbuild, eckit, fckit)

    /home/rme/public/bin/gmkpack.6.9.4/util/gmkpack -f /home/snh02/pack -r 48t3 -b main -K -a -n 01 -l OMPIIFC2104 -o x -p masterodb

    /home/snh02/pack/48t3_fix.01.OMPIIFC2104.x – home pack under root pack 48t3_main.01.OMPIIFC2104.x

    src/local/ifsaux/fa/facgrm.F90 – fix for subhourly inline FP

    /home/snh02/pack/48t3_main_SP.02.OMPIIFC2104.x - single precision - root pack

    For compilation SP code are recommended following fixes:

    • /home/snh02/sources/cy48t3/48t3_48t3_sp.tgz - Ryad's fix for SP

    arpifs/module/field_module.fypp arpifs/module/field_registry_mod.fypp arpifs/module/field_variables_mod.fypp arpifs/module/variable_module.fypp

    • /home/snh02/sources/cy48t3/mods_fix_simul.tgz - Ryad's fix for GNU compiler gcc-11.2.0

    algor/internal/minim/mlis0.F90 algor/internal/minim/m1qn3a.F90 algor/interface/simul_interface.h

    • /home/snh02/sources/cy48t3/facgrm_SP_subhrl.tgz - Florian's fix for subhourly output + Ryad's fix for SP

    ifsaux/fa/facgrm.F90

  • CY48T3 testbed:

    /home/snh02/work/cy48t3/alaro

    namelist - fort.4_cy48t3_128, fort.4_cy48t3_128_IO, fort.4_cy48t3_128_grb2, fort.4_cy48t3_grb2_288

    script - scripts and output from slurm job

    /ec/res4/scratch/snh02 ($SCRATCH) - some outputs files from the jobs

    cy48t3_alaro cy48t3_alaro_grb2 cy48t3_alaro_grb2_288

    /home/snh02/work/cy48t3/arome

    namelist - fort.4_arome - forecast, fort.4_arome_grb2 - hourly FP to grib2, sort_fort.4 - subhourly FP to grib2

    script - scripts and output from slurm job

    /ec/res4/scratch/snh02 ($SCRATCH) - some outputs files from the jobs

    cy48t3_arome cy48t3_arome_grb2 cy48t3_arome_grb2_subh

  • gl: /home/snh02/bin/gl

Files on LUMI

  • sources

/users/ospaniel/sources/

  • CY46T1 installation:

    /users/ospaniel/pack/46t1_bf07.01.FTNGNU1120.x (root pack - without FFTW)

    /users/ospaniel/pack/46t1_bf07.02.FTNGNU1120.x (final root root pack - with -DWITH_FFTW)

    /users/ospaniel/pack/46t1_bf07.03.FTNGNU1120.x (development root pack - with -DWITH_FFTW and additional gnu options)

    /users/ospaniel/sources/cy46t1/cy46t1_bf.07.src.tgz – source code

    /users/ospaniel/sources/cy46t1/ignore4compil.46T1_bf.07 – the list of an "obsolete" files

    .genesis – saved command line for creating the root pack

    /users/ospaniel/opt/gmkpack.6.9.4/util/gmkpack -f /users/ospaniel/pack -r 46t1 -b bf07 -a -n 02 -l FTNGNU1120 -o x -p masterodb

    .gmkfile/GFORTRAN.LUMI – configuration file for compilation/linking

    ics_masterodb – script for compilation/linking of MASTERODB executable

    OUT_? – output from ics_masterodb

    /home/ospaniel/pack/46t1_bf07_grb2_fix.02.FTNGNU1120.x – home pack under root pack 46t1_bf07.02.FTNGNU1120.x

    ifsaux/fa/facgrm.F90 - fix for subhourly inline FP

  • CY46T1 testbed ALARO:

    /users/ospaniel/work/cy46t1/alaro

    A-to-tos-cy46t1_OMPIIFC_128-alaro - ALARO 1b -> /users/ospaniel/work/cy46t1/tmp-alaro

    A-to-tos-cy46t1_OMPIIFC_128-alaro_shmu - ALARO 1b, different init fields -> /users/ospaniel/work/cy46t1/tmp-alaro_shmu

    A-to-tos-cy46t1_OMPIIFC_128-alaro_shmu_grb2 - ALARO 1b, inline FP and subhourly output to grib2 -> /users/ospaniel/work/cy46t1/tmp-alaro_shmu_grb2

    A-to-tos-cy46t1_OMPIIFC_128-alaro_shmu_Yogesh - ALARO 1b with different MASTERODB (created by cmake) -> /users/ospaniel/work/cy46t1/tmp-alaro_shmu_Yogesh

  • CY46T1 testbed AROME:

    /users/ospaniel/work/cy46t1/alaro

    A-to-tos-cy46t1_OMPIIFC_128-arome -> /users/ospaniel/work/cy46t1/tmp-arome

    A-to-tos-cy46t1_OMPIIFC_128-arome_grb2 -> /users/ospaniel/work/cy46t1/tmp-arome_grb2

  • CY48T3 installation:

    The Cray compiler has been tested and some issues are listed below.

  • Compile:

    ftn -c -hbyteswapio -hflex_mp=conservative -hadd_paren -emf -fopenmp -f free -DLINUX -DLITTLE_ENDIAN -DLITTLE -DGRIB_API_1 -r am -DADDRESS64 local/arpifs/obs_preproc/decis.F90

    ftn-2116 ftn: INTERNAL

    "/opt/cray/pe/cce/14.0.2/cce/x86_64/bin/optcg" was terminated due to receipt of signal 013: Segmentation fault (core dumped).

  • During compilation of ecfftw/module/tpm_fftw.F90 source occurred error message:

    ftn-130 ftn: ERROR FFTWL_PLAN_DFT, File = fftw3.f03.h, Line = 1240,

    Column = 15 The kind type parameter value -1 is not valid for type COMPLEX.

    complex(C_LONG_DOUBLE_COMPLEX), dimension(*), intent(out) :: out

    Looks like Cray Fortran ISO_C_BINDING module does not support C_LONG_DOUBLE_COMPLEX. There are some options from LUMI support:

    Is the build system trying to build it's own FFTW3? In that case check if you can disable that and use the readily provided cray-fftw instead.

    I'd say that the build system should probably use C_DOUBLE_COMPLEX instead of C_LONG_DOUBLE_COMPLEX if C_LONG_DOUBLE_COMPLEX is not defined. This should be discussed with the developers, if you pursue that approach.

    Change from PrgEnv-cray to PrgEnv-gnu. GNU compiler's ISO_C_BINDING supports C_LONG_DOUBLE_COMPLEX.

    The GNU compiler

    Some compiler options have to be added and the code has been fixed here:

    • Error: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) -fallow-argument-mismatch

    • Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to - -fno-allow-invalid-boz

    • Error: Symbol 'norgwd_sat' in namelist 'namnorgwd' at (1) must be declared before the namelist is declared - change ordering of includes in the code

    • Error: Explicit interface required for polymorphic argument at (1)

      /users/ospaniel/pack/48t3_main.04.FTNGNU1120.x/src/local/algor/internal/minim/m1qn3a.F90

      /users/ospaniel/pack/48t3_main.04.FTNGNU1120.x/src/local/algor/internal/minim/mlis0.F90

      The fix from Ryad has been used here: belenos:/home/gmap/mrpm/khatib/public/modset/mods_fix_simul.tgz It is a backphasing from pre-cycle 49.

    Currently Loaded Modules:

    1) gcc/11.2.0 6) cray-fftw/3.3.8.12 11) craype-x86-rome 16) partition/L (S)

    2) craype/2.7.13 7) perftools-base/21.12.0 12) craype-accel-host 17) LUMI/21.12 (S)

    3) cray-dsmml/0.2.2 8) cray-mpich/8.1.12 13) libfabric/1.15.0.0 18) ModuleLabel/label (S)

    4) cray-libsci/21.08.1.2 9) cray-hdf5-parallel/1.12.0.7 14) craype-network-ofi 19) init-lumi/0.1 (S)

    5) PrgEnv-gnu/8.2.0 10) cray-netcdf-hdf5parallel/4.7.4.7 15) xpmem/2.4.4-2.3_9.1__gff0e1d9.shasta

    /users/ospaniel/pack/48t3_main.05.FTNGNU1120.x - root pack - with FFTW

    /users/ospaniel/pack/48t3_test.05.FTNGNU1120.x - home pack under root pack 48t3_main.05.FTNGNU1120.x (ALARO)

    src/local/ifsaux/fa/facgrm.F90 – fix for subhourly inline FP

    /users/ospaniel/pack/48t3_debug.05.FTNGNU1120.x - home pack under root pack 48t3_main.05.FTNGNU1120.x (AROME)

    surfex/SURFEX/read_pgd_tebn.F90 - temporary hardcoded fix for ABOR1 CALLED SURFEX FIELD CACHE:THER_PDAY WAS NOT FOUND (solution ongoing)

    src/local/ifsaux/fa/facgrm.F90 – fix for subhourly inline FP

    /users/ospaniel/pack/48t3_main_SP.06.FTNGNU1120.x - root pack Single Precision

    /users/ospaniel/pack/48t3_main_SP_fix.06.FTNGNU1120.x - home pack Single Precision

    algor/internal/minim/m1qn3a.F90 algor/internal/minim/mlis0.F90 algor/interface/simul_interface.h arpifs/module/field_variables_mod.fypp arpifs/module/field_module.fypp arpifs/module/variable_module.fypp arpifs/module/field_registry_mod.fypp ifsaux/fa/facgrm.F90

    • CY48T3 testbed ALARO:

    /users/ospaniel/work/cy48t3/alaro/A-to-tos-cy48t2_OMPIIFC_128-alaro -> /users/ospaniel/work/cy48t3/tmp-alaro

    /users/ospaniel/work/cy48t3/alaro/A-to-tos-cy48t2_OMPIIFC_128-alaro_grb2 -> /users/ospaniel/work/cy48t3/tmp-alaro_grb2

    • CY48T3 testbed AROME:

    /users/ospaniel/work/cy48t3/arome/A-to-tos-cy48t3_OMPIIFC_128-arome

    /users/ospaniel/work/cy48t3/arome/A-to-tos-cy48t3_OMPIIFC_128-arome_grb2

    /users/ospaniel/work/cy48t3/arome/A-to-tos-cy48t3_OMPIIFC_128-arome_grb2_SP - 40% save elepased time compare with A-to-tos-cy48t3_OMPIIFC_128-arome_grb2

Remark:

I received very occasionally message from runtime:

libfabric:228787:1674574110:cxi:core:cxip_ux_onload_cb():2259 nid001031:

RXC (0x3bc3:31:0): PtlTE 111LE resources not recovered during flow control.

FI_CXI_RX_MATCH_MODE=[hybrid|software] is required.

Program received signal SIGABRT: Process abort signal.

export FI_CXI_RX_MATCH_MODE=hybrid this setting in script is recommended

LUMI Cray Fortran : Version 16.0.1

Sources

/scratch/project_465000527/ospaniel/work/sources/IAL-CY48T3_deode.zip github repo 2.5.2024

/scratch/project_465000527/ospaniel/work/sources/eccodes-2.34.1-Source.tgz

Compilation eccodes-2.34.1

  • module --force purge module use /scratch/project_465000527/jasinskas/scl/modules module load scl-ftn16_23

  • ftn -V export CC=gcc export CXX=g++ export FC=ftn cmake /scratch/project_465000527/ospaniel/tmp/eccodes-2.34.1-Source -DCMAKE_INSTALL_PREFIX=/scratch/project_465000527/ospaniel/opt/eccodes-2.34.1_threads -DENABLE_ECCODES_THREADS=ON -DENABLE_NETCDF=ON -DENABLE_EXTRA_TESTS=1 -DENABLE_JPG=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_FORTRAN=ON -DENABLE_AEC=ON make ctest make install

Compilation cy48t3 by gmkpack

/users/ospaniel/pack/48t3_main.68.CRAYFTN1601.x/bin/MASTERODB

  • ecfftw/module/fftw3.f03.h substitution of C_LONG_DOUBLE_COMPLEX -> DOUBLE_COMPLEX .... is it still in repo?

  • cat > $GMKWRKDIR/.ignored_files <<end_of_ignored_files ifsaux/py_interface/init_gfortran.c end_of_ignored_files

  • /arpifs/control/cprep4.F90 ftn-2116 ftn: INTERNAL "/opt/cray/pe/cce/16.0.1/cce/x86_64/bin/optcg" was terminated due to receipt of signal 013: Segmentation fault (core dumped). srun -p standard -A project_465000527 --time=00:45:00 --nodes=1 --pty bash -i ics_masterodb > log.00 2>&1 &

  • touch src/unsxref/verbose/MTOCOMP_static_init

  • module --force purge module load LUMI/23.09 partition/C module load libaec/1.0.6-cpeCray-23.09 module load cpe/23.09 module load PrgEnv-cray module load cray-mpich cray-libsci cray-fftw cray-python module load cray-hdf5-parallel cray-netcdf-hdf5parallel module load ncurses/6.4-cpeCray-23.09 module load cce/16.0.1 ftn -V

  • -Wl,--copy-dt-needed-entries -Wl,-Bstatic -Wl,--start-group -L/project/project_465000527/dadegrau/accord/aux/lib -lgribex -lbufr -lnetcdff -lnetcdf -L/scratch/project_465000527/ospaniel/opt/eccodes-2.34.1/lib64 -leccodes_f90 -L/scratch/project_465000527/ospaniel/opt/eccodes-2.34.1/lib64 -leccodes -L/appl/lumi/SW/LUMI-23.09/C/EB/libaec/1.0.6-cpeCray-23.09/lib64 -laec -lhdf5hl_fortran -lhdf5_fortran -lhdf5 -lfdbdummy -lwamdummy -lnaglitedummy -loasisdummy -libmdummy -Wl,--end-group -Wl,-Bdynamic

  • export LIBSCI_ARCH_OVERRIDE=broadwell echo $LIBSCI_ARCH_OVERRIDE

  • /users/ospaniel/pack/48t3_main.68.CRAYFTN1601.x/.gmkfile/CRAYFTN.LUMI - compilation setting

  • /users/ospaniel/pack/48t3_main.68.CRAYFTN1601.x/ics_masterodb - compiation and linking script

Forecast ALARO

  • script : /users/ospaniel/work/DEODE_testbed/script/LUMI-cy48t3_CRAYFTN1601_128-alaro1_2km_grb2_67_34.1

  • output files: /users/ospaniel/work/DEODE_testbed/wrk/alaro1_2km_grb2_CRAYFTN1601_67_34.1

Forecast AROME

  • script : /users/ospaniel/work/DEODE_testbed/script/LUMI-cy48t3_CRAYFTN1601_640-arome_1500_grb2_67_34.1_proto_IO_server

  • output files: /users/ospaniel/work/DEODE_testbed/wrk/arome_1.5km_grb2_CRAYFTN1601_67_34.1_proto_IO_server

Remarks:

  • #SBATCH --mem=0

  • export LIBSCI_ARCH_OVERRIDE=broadwell echo $LIBSCI_ARCH_OVERRIDE

  • export OMP_STACKSIZE=8G export KMP_STACKSIZE=8G

  • ECCODES_SAMPLES_PATH=/users/ospaniel/opt/share/eccodes/ifs_samples/grib1_mlgrib2 ECCODES_DEFINITION_PATH=/users/ospaniel/opt/share/eccodes/definitions/grib2/localConcepts/lfpw:/scratch/project_465000527/ospaniel/opt/eccodes-2.34.1/share/eccodes

  • module --force purge module load LUMI/23.09 partition/C module load libaec/1.0.6-cpeCray-23.09 module load cpe/23.09 module load PrgEnv-cray module load cray-mpich cray-libsci cray-fftw cray-python module load cray-hdf5-parallel cray-netcdf-hdf5parallel module load ncurses/6.4-cpeCray-23.09 module load cce/16.0.1

  • ALARO

NPROMA=-32, NFPROMA=-32, NFPROMA_DEP=-32,

#SBATCH --partition=debug # Partition (queue) name

#SBATCH --nodes=4 # Total number of nodes

#SBATCH --ntasks-per-node=32 # Number of mpi tasks per node

#SBATCH --cpus-per-task=4 # Number of cores (threads) per task

  • AROME

NPROMA=-16, NFPROMA=-16, NFPROMA_DEP=-16, CMICRO='OLD3',

#SBATCH --partition=standard # Partition (queue) name

#SBATCH --nodes=20 # Total number of nodes

#SBATCH --ntasks-per-node=32 # Number of mpi tasks per node

#SBATCH --cpus-per-task=4 # Number of cores (threads) per task

Clone this wiki locally