Splitting up the archive task #3242

AntonMFernando-NOAA · 2025-01-21T17:02:13Z

Description

In this PR the archive task will be split into two parts. The first will always run and will just be for the copying of verification data to the VRFY_ARC and ARCDIR directories. The second will only run when HPSSARCH or LOCALARCH is set to YES and will generate and store tarballs in the ATARDIR, either on HPSS or locally.
Resolves Split up the archive task #3152

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

CI tests in Hera

Example:

Cycled test on Hera
Forecast-only on Hera

Checklist

…o-NOAA/global-workflow into feature/arch-split

emcbot · 2025-01-28T20:34:14Z

Experiment C96_atm3DVar FAILED on Hera in Build# 3 in
/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/EXPDIR/C96_atm3DVar_b2bf61c8

emcbot · 2025-01-28T21:07:06Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 3 with error logs:

/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_b2bf61c8/logs/2024022400/gfs_arch_tars.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-28T21:07:14Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 3 in
/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_b2bf61c8

emcbot · 2025-01-28T21:16:34Z

Experiment C48_S2SW FAILED on Hera in Build# 3 with error logs:

/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/COMROOT/C48_S2SW_b2bf61c8/logs/2021032312/gfs_arch_tars.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-28T21:16:43Z

Experiment C48_S2SW FAILED on Hera in Build# 3 in
/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/EXPDIR/C48_S2SW_b2bf61c8

emcbot · 2025-01-28T21:16:51Z

CI Failed on Hera in Build# 3
Built and ran in directory /scratch1/NCEPDEV/global/CI/3242


Experiment C48_ATM_b2bf61c8 Completed 1 Cycles: *SUCCESS* at Tue Jan 28 19:26:49 UTC 2025
Experiment C48mx500_3DVarAOWCDA_b2bf61c8 Completed 2 Cycles: *SUCCESS* at Tue Jan 28 19:38:58 UTC 2025
Experiment C48mx500_hybAOWCDA_b2bf61c8 Completed 2 Cycles: *SUCCESS* at Tue Jan 28 19:45:03 UTC 2025
Experiment C96_atm3DVar_b2bf61c8 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Jan 28 20:33:47 UTC 2025
Experiment C96_atm3DVar_b2bf61c8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/COMROOT/C96_atm3DVar_b2bf61c8/logs/2021122100/gfs_arch_tars.log
Experiment C96C48_hybatmDA_b2bf61c8 Completed 3 Cycles: *SUCCESS* at Tue Jan 28 20:33:54 UTC 2025
Experiment C96C48_hybatmaerosnowDA_b2bf61c8 Completed 3 Cycles: *SUCCESS* at Tue Jan 28 20:58:14 UTC 2025
Experiment C96C48_ufs_hybatmDA_b2bf61c8 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Jan 28 21:06:50 UTC 2025
Experiment C96C48_ufs_hybatmDA_b2bf61c8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_b2bf61c8/logs/2024022400/gfs_arch_tars.log
Experiment C48_S2SW_b2bf61c8 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Jan 28 21:16:24 UTC 2025
Experiment C48_S2SW_b2bf61c8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/COMROOT/C48_S2SW_b2bf61c8/logs/2021032312/gfs_arch_tars.log

emcbot · 2025-01-29T10:42:47Z

Experiment C96_atm3DVar FAILED on Hera in Build# 4 with error logs:

/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/COMROOT/C96_atm3DVar_b4eee81f/logs/2021122100/gfs_arch_tars.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-29T10:42:57Z

Experiment C96_atm3DVar FAILED on Hera in Build# 4 in
/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/EXPDIR/C96_atm3DVar_b4eee81f

emcbot · 2025-01-29T12:15:07Z

CI Failed on Hera in Build# 4
Built and ran in directory /scratch1/NCEPDEV/global/CI/3242


Experiment C48_ATM_b4eee81f Completed 1 Cycles: *SUCCESS* at Wed Jan 29 09:29:16 UTC 2025
Experiment C96_S2SWA_gefs_replay_ics_b4eee81f Completed 1 Cycles: *SUCCESS* at Wed Jan 29 09:35:25 UTC 2025
Experiment C48mx500_hybAOWCDA_b4eee81f Completed 2 Cycles: *SUCCESS* at Wed Jan 29 09:47:31 UTC 2025
Experiment C96_atm3DVar_b4eee81f Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Jan 29 10:42:30 UTC 2025
Experiment C96_atm3DVar_b4eee81f Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3242/RUNTESTS/COMROOT/C96_atm3DVar_b4eee81f/logs/2021122100/gfs_arch_tars.log
Experiment C96C48_hybatmDA_b4eee81f Completed 3 Cycles: *SUCCESS* at Wed Jan 29 10:42:38 UTC 2025
Experiment C48_S2SW_b4eee81f Completed 1 Cycles: *SUCCESS* at Wed Jan 29 11:12:57 UTC 2025
Experiment C96C48_hybatmaerosnowDA_b4eee81f Completed 3 Cycles: *SUCCESS* at Wed Jan 29 11:13:14 UTC 2025
Experiment C48_S2SWA_gefs_b4eee81f Completed 1 Cycles: *SUCCESS* at Wed Jan 29 11:32:13 UTC 2025
Experiment C96C48_ufs_hybatmDA_b4eee81f Completed 3 Cycles: *SUCCESS* at Wed Jan 29 11:50:07 UTC 2025
Experiment C48mx500_3DVarAOWCDA_b4eee81f Completed 2 Cycles: *SUCCESS* at Wed Jan 29 12:14:48 UTC 2025

DavidHuber-NOAA · 2025-01-29T13:25:40Z

The gfs_arch_tars job failed for the C96_atm3DVar test on Hera twice due to log file sizes changing during execution of htar. The first failure was for the log file gfs_arch_vrfy.log and the second for gfs_metpg2g1.log.

First, the parm/archive/gfsa.yaml.j2 file needs to be amended here

global-workflow/parm/archive/gfsa.yaml.j2

Line 9 in a783f9b

{% if not "gfs_arch.log" in log %}

to screen out gfs_arch_tars.log instead of gfs_arch.log.

Next, I think the dependencies need to be reworked for the gfs_arch_tars job so that it runs after gfs_arch_vrfy and the gfs_metp metatask. I could be convinced that the gfs_metp* tasks do not need to run before gfs_arch_tars, but then the screening in gfsa.yaml.j2 will need to also exclude gfs_metp*.log files.

workflow/rocoto/gfs_tasks.py

AntonMFernando-NOAA · 2025-01-29T17:49:13Z

@WalterKolczynski-NOAA @DavidHuber-NOAA C96_atm3DVar test passed with new changes.
test: /scratch1/NCEPDEV/global/Anton.Fernando/RUNTESTS/EXPDIR/gefs_C96_atm3DVar

DavidHuber-NOAA · 2025-01-29T18:20:56Z

workflow/rocoto/gfs_tasks.py

+        if self.options['do_archtar']:
+            dep_dict = {'type': 'task', 'name': f'{self.run}_arch_tars'}
+            deps.append(rocoto.add_dependency(dep_dict))


The arch_tars job shouldn't be a dependency for the metp jobs. I don't think you have to rerun your local test after you fix this.

AntonMFernando-NOAA added 30 commits January 17, 2025 19:24

initial modification to split arch job to two parts

12696d7

modified workflow\applications\gefs.py

183a303

modified arch_vrfy var

c585fe1

modified arch_vrfy vars in gefs config files

8ac2162

modified config files

b903470

modified exglobal_archive_vrfy.py

c74910d

modified defaults and JJOB

c9d414d

modified jobs\JGLOBAL_ARCHIVE_TARS

7d42e36

modified jobs\JGLOBAL_ARCHIVE_TARS

fc69dcd

modified config files

8e523a1

modified parm\config\gefs\config.base

5cafe8a

modify jjobs

3b2074f

correct a typo

3008b33

modified gefs.py

4c145ca

modified workflow\applications\gefs.py

cdd5acf

modified gefs.py

5c9b3f5

modified workflow\applications\gefs.py

260354e

modifed gefs.py

e880082

modified gefs.py

4fab72f

modified workflow\applications\gefs.py

9e54cc0

modifed gefs.py

0f0c7c2

modifed workflow\applications\gefs.py

1b73697

modified workflow\rocoto\gefs_tasks.py

14fc710

modifed workflow\rocoto\gefs_tasks.py

4493271

change permission

4d6ad0c

modified archive.py functions

8d97a78

Merge branch 'feature/arch-split' of https://github.com/AntonMFernand…

913188f

…o-NOAA/global-workflow into feature/arch-split

modified archive.py

a07e405

modified configure function

95c0eb2

modified ush\python\pygfs\task\archive.py

c0cdcb6

AntonMFernando-NOAA added 2 commits January 28, 2025 15:45

modified ARCH jscripts

3db17b8

modified scripts\exglobal_archive_vrfy.py

5c50e6a

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jan 28, 2025

Merge branch 'develop' into feature/arch-split

b4eee81

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jan 29, 2025

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jan 29, 2025

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jan 29, 2025

added updated that failed 3DVar job

62892b5

DavidHuber-NOAA reviewed Jan 29, 2025

View reviewed changes

workflow/rocoto/gfs_tasks.py Outdated Show resolved Hide resolved

modified workflow\rocoto\gfs_tasks.py

5da1889

DavidHuber-NOAA reviewed Jan 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Splitting up the archive task #3242

Splitting up the archive task #3242

AntonMFernando-NOAA commented Jan 21, 2025 •

edited

Loading

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 29, 2025

emcbot commented Jan 29, 2025

emcbot commented Jan 29, 2025

DavidHuber-NOAA commented Jan 29, 2025

AntonMFernando-NOAA commented Jan 29, 2025

DavidHuber-NOAA Jan 29, 2025

Splitting up the archive task #3242

Are you sure you want to change the base?

Splitting up the archive task #3242

Conversation

AntonMFernando-NOAA commented Jan 21, 2025 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 28, 2025

emcbot commented Jan 29, 2025

emcbot commented Jan 29, 2025

emcbot commented Jan 29, 2025

DavidHuber-NOAA commented Jan 29, 2025

AntonMFernando-NOAA commented Jan 29, 2025

DavidHuber-NOAA Jan 29, 2025

Choose a reason for hiding this comment

AntonMFernando-NOAA commented Jan 21, 2025 •

edited

Loading