forked from NOAA-EMC/global-workflow
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Splitting up the archive task (NOAA-EMC#3242)
In this PR the archive task will be split into two parts. The first will always run and will just be for the copying of verification data to the `VRFY_ARC` and `ARCDIR` directories. The second will only run when `HPSSARCH` or `LOCALARCH` is set to YES and will generate and store tarballs in the `ATARDIR`, either on `HPSS` or locally. Resolves NOAA-EMC#3152
- Loading branch information
1 parent
1939dac
commit c9011ab
Showing
36 changed files
with
655 additions
and
123 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| #! /usr/bin/env bash | ||
|
|
||
| source "${HOMEgfs}/ush/preamble.sh" | ||
| source "${HOMEgfs}/ush/jjob_header.sh" -e "earc_vrfy" -c "base earc_vrfy" | ||
|
|
||
|
|
||
| ############################################## | ||
| # Set variables used in the script | ||
| ############################################## | ||
| YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_TOP | ||
| MEMDIR="ensstat" YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \ | ||
| COMIN_ATMOS_ANALYSIS_ENSSTAT:COM_ATMOS_ANALYSIS_TMPL \ | ||
| COMIN_ATMOS_HISTORY_ENSSTAT:COM_ATMOS_HISTORY_TMPL \ | ||
| COMIN_SNOW_ANALYSIS_ENSSTAT:COM_SNOW_ANALYSIS_TMPL | ||
|
|
||
| ############################################################### | ||
| # Run archive script | ||
| ############################################################### | ||
|
|
||
| "${SCRgfs}/exgdas_enkf_earc_vrfy.py" | ||
| status=$? | ||
| [[ ${status} -ne 0 ]] && exit "${status}" | ||
|
|
||
| ############################################################### | ||
|
|
||
| ############################################## | ||
| # End JOB SPECIFIC work | ||
| ############################################## | ||
|
|
||
| ############################################## | ||
| # Final processing | ||
| ############################################## | ||
| if [[ -e "${pgmout}" ]] ; then | ||
| cat "${pgmout}" | ||
| fi | ||
|
|
||
| ########################################## | ||
| # Remove the Temporary working directory | ||
| ########################################## | ||
| cd "${DATAROOT}" || (echo "${DATAROOT} does not exist. ABORT!"; exit 1) | ||
| [[ ${KEEPDATA} = "NO" ]] && rm -rf "${DATA}" | ||
|
|
||
| exit 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| #! /usr/bin/env bash | ||
|
|
||
| source "${HOMEgfs}/ush/preamble.sh" | ||
| source "${HOMEgfs}/ush/jjob_header.sh" -e "arch_vrfy" -c "base arch_vrfy wave" | ||
|
|
||
|
|
||
| ############################################## | ||
| # Set variables used in the script | ||
| ############################################## | ||
| YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \ | ||
| COMIN_ATMOS_ANALYSIS:COM_ATMOS_ANALYSIS_TMPL \ | ||
| COMIN_ATMOS_GENESIS:COM_ATMOS_GENESIS_TMPL \ | ||
| COMIN_ATMOS_HISTORY:COM_ATMOS_HISTORY_TMPL \ | ||
| COMIN_ATMOS_TRACK:COM_ATMOS_TRACK_TMPL \ | ||
| COMIN_CHEM_ANALYSIS:COM_CHEM_ANALYSIS_TMPL \ | ||
| COMIN_SNOW_ANALYSIS:COM_SNOW_ANALYSIS_TMPL \ | ||
| COMIN_OBS:COM_OBS_TMPL \ | ||
| COMOUT_ATMOS_TRACK:COM_ATMOS_TRACK_TMPL | ||
|
|
||
| for grid in "0p25" "0p50" "1p00"; do | ||
| YMD=${PDY} HH=${cyc} GRID=${grid} declare_from_tmpl -rx \ | ||
| "COMIN_ATMOS_GRIB_${grid}:COM_ATMOS_GRIB_GRID_TMPL" | ||
| done | ||
|
|
||
| ############################################################### | ||
| # Run archive script | ||
| ############################################################### | ||
|
|
||
| ${GLOBALARCHIVESH:-${SCRgfs}/exglobal_archive_vrfy.py} | ||
| status=$? | ||
| [[ ${status} -ne 0 ]] && exit "${status}" | ||
|
|
||
| ############################################## | ||
| # End JOB SPECIFIC work | ||
| ############################################## | ||
|
|
||
| ############################################## | ||
| # Final processing | ||
| ############################################## | ||
| if [[ -e "${pgmout}" ]] ; then | ||
| cat "${pgmout}" | ||
| fi | ||
|
|
||
| ########################################## | ||
| # Remove the Temporary working directory | ||
| ########################################## | ||
| cd "${DATAROOT}" || (echo "${DATAROOT} does not exist. ABORT!"; exit 1) | ||
| [[ ${KEEPDATA} = "NO" ]] && rm -rf "${DATA}" | ||
|
|
||
| exit 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| #! /usr/bin/env bash | ||
|
|
||
| source "${HOMEgfs}/ush/preamble.sh" | ||
|
|
||
| ############################################################### | ||
| # Source FV3GFS workflow modules | ||
| . "${HOMEgfs}"/ush/load_fv3gfs_modules.sh | ||
| status=$? | ||
| [[ ${status} -ne 0 ]] && exit "${status}" | ||
|
|
||
| ############################################################### | ||
| # setup python path for workflow utilities and tasks | ||
| PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${HOMEgfs}/ush/python" | ||
| export PYTHONPATH | ||
|
|
||
| export job="arch_vrfy" | ||
| export jobid="${job}.$$" | ||
|
|
||
| ############################################################### | ||
| # Execute the JJOB | ||
| "${HOMEgfs}"/jobs/JGLOBAL_ARCHIVE_VRFY | ||
| status=$? | ||
|
|
||
| exit "${status}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| #! /usr/bin/env bash | ||
|
|
||
| source "${HOMEgfs}/ush/preamble.sh" | ||
|
|
||
| ############################################################### | ||
| # Source FV3GFS workflow modules | ||
| . "${HOMEgfs}/ush/load_fv3gfs_modules.sh" | ||
| status=$? | ||
| [[ ${status} -ne 0 ]] && exit "${status}" | ||
|
|
||
| ############################################################### | ||
| # setup python path for workflow utilities and tasks | ||
| PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${HOMEgfs}/ush/python" | ||
| export PYTHONPATH | ||
|
|
||
| export job="earc_vrfy" | ||
| export jobid="${job}.$$" | ||
|
|
||
| ############################################################### | ||
| # Execute the JJOB | ||
| "${HOMEgfs}/jobs/JGDAS_ENKF_ARCHIVE_VRFY" | ||
| status=$? | ||
|
|
||
| exit "${status}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8 changes: 4 additions & 4 deletions
8
parm/config/gefs/config.arch → parm/config/gefs/config.arch_tars
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,15 @@ | ||
| #! /usr/bin/env bash | ||
|
|
||
| ########## config.arch ########## | ||
| ########## config.arch_tars ########## | ||
| # Archive specific | ||
|
|
||
| echo "BEGIN: config.arch" | ||
| echo "BEGIN: config.arch_tars" | ||
|
|
||
| # Get task specific resources | ||
| . "${EXPDIR}/config.resources" arch | ||
| . "${EXPDIR}/config.resources" arch_tars | ||
|
|
||
| export ARCH_GAUSSIAN="YES" | ||
| export ARCH_GAUSSIAN_FHMAX=${FHMAX_GFS} | ||
| export ARCH_GAUSSIAN_FHINC=${FHOUT_GFS} | ||
|
|
||
| echo "END: config.arch" | ||
| echo "END: config.arch_tars" |
8 changes: 4 additions & 4 deletions
8
parm/config/gfs/config.arch → parm/config/gefs/config.arch_vrfy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,15 @@ | ||
| #! /usr/bin/env bash | ||
|
|
||
| ########## config.arch ########## | ||
| ########## config.arch_vrfy ########## | ||
| # Archive specific | ||
|
|
||
| echo "BEGIN: config.arch" | ||
| echo "BEGIN: config.arch_vrfy" | ||
|
|
||
| # Get task specific resources | ||
| . "${EXPDIR}/config.resources" arch | ||
| . "${EXPDIR}/config.resources" arch_vrfy | ||
|
|
||
| export ARCH_GAUSSIAN="YES" | ||
| export ARCH_GAUSSIAN_FHMAX=${FHMAX_GFS} | ||
| export ARCH_GAUSSIAN_FHINC=${FHOUT_GFS} | ||
|
|
||
| echo "END: config.arch" | ||
| echo "END: config.arch_vrfy" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.