feature/rm_cpreq #643
feature/rm_cpreq #643
Conversation
malloryprow
requested review from
PerryShafran-NOAA,
AliciaBentley-NOAA and
AndrewBenjamin-NOAA
|
Starting with the vhr=00 run of the cam_href_severe_prep. |
|
@malloryprow The vhr=00 href cam prep is done. .o file is here: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr643test/EVS/dev/drivers/scripts/prep/cam/jevs_cam_href_severe_prep.o176418876 There is nothing in COMOUT. There are many not found messages in here:
I set COMIN to point to your directory as instructed, but it appears that I need to run something else before I run the href severe prep job. I'm moving on to global_ens prep and we'll circle back to here once we figure out what to do here. |
|
Oh, did you mean emc.vpppg and not mallory.row? Maybe that's the issue? |
|
Hahaha, I definitely meant for that to be emc.vpppg and not mallory.row. 🙃 I fixed it. |
|
Ok cool...take two. |
|
The job is complete: .o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr643test/EVS/dev/drivers/scripts/prep/cam/jevs_cam_href_severe_prep.o176423216 There is no working directory as that is actually removed once the job is completed. I'll submit the vhr=12 job. |
|
The vhr=12 job is complete: .o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr643test/EVS/dev/drivers/scripts/prep/cam/jevs_cam_href_severe_prep.o176423906 There is no output, because t12z data is missing from emc.vpppg.
|
|
The global_ens prep jobs are all underway. |
|
cam prep is good. I think it might be missing due to the production switch since the job is suppose to run at 12Z and that is when the switch started. I don't see any cpreq in the log and the file in COMOUT matches the parallel. |
|
Let me know if you wish to run cam prep again, just to get in a run where there actually is data. |
|
I think we are good! |
|
Also note as I said above, the cam prep job removes the working directory (which means KEEPDATA must be set to NO). Might we want to change that? |
|
For global_ens, which ran yesterday: global_ens atmos prep: global_ens headline prep: global_ens wave prep: |
|
Note on global_ens wave prep:
|
|
Just pushed a commit to set KEEPDATA to YES. It was set to NO for multiple cam prep jobs. |
|
Just pushed a commit for PB2NC_SKIP_VALID_TIMES. |
|
As you look at the global_ens prep, I'll start on the aqm stats. |
|
global_ens prep for atmos and headline is good. Counts match the parallel and no cpreq in the logs. Do you want to remove /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/prep/global_ens/wave.20250106 and re-run the job after pulling in the new changes? |
|
No need to remove since 20250106 was run yesterday; today's run would be 20250107. Unless you want me to run the same date as previously. |
|
aqm plots is good! Some warnings about missing fils and thresholds not being met, but the final tar files match the parallel and no cpreq usage in the logs file. |
|
I also submitted the mesoscale plots jobs. For the precip and snowfall, I am submitting with a 10 hr walltime to see how long these jobs take. However, when I tried to submit the mesoscale headline job, I got this weird error: No clue why these errors are there. I'm quite befuddled. What do you think? |
|
The global_ens wave plots job is finished. .o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr643test/EVS/dev/drivers/scripts/plots/global_ens/jevs_global_ens_wave_grid2obs_plots.o176516061 |
|
I added exclhost to the PBS options for jevs_mesoscale_headline_plots.sh, and the job was submitted normally. |
Wow okay you got this too! So I had been seeing this in the parallel I thought maybe it was something with the cron and I have emailed NCO about it. So it looks like it is a not problem with the submission coming from the cron. Wild thing is NCO said this: "This is the first time I've ever seen an error from that hook over millions of executions." |
|
global_ens wave plots is all good! |
|
Huh, you've been seeing this same error? With this specific job or with other jobs as well? |
|
Just this job! |
|
Interesting! I wonder why. Well, adding exclhost seems to correct things... |
|
2 of the 4 plot jobs are complete: mesoscale grid2obs: mesoscale headline: |
|
Everything looks in order for these two jobs! |
|
Good news! I think the other two jobs are going to finish after your workday is over, so I guess we'll pick this up on Friday. |
|
Did the other two jobs complete? |
|
Whoops! My apologies, I was going to post the info first thing in the morning yesterday, but I got caught up in other stuff. I'll get you that info in a minute or so. |
|
meso precip plot: .o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr643test/EVS/dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_precip_plots.o176516471 meso snowfall plot: .o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr643test/EVS/dev/drivers/scripts/plots/mesoscale/jevs_mesoscale_snowfall_plots.o176516475 |
|
Wow! Those are really long run times. Were there new plots added? There haven't been problems in production so I'm not sure what is causing these jobs to run so long. |
|
I don't recall adding any new plots to these jobs, but they do seem quite long. I'm curious if the processes that added in the mpmd processing added time to the jobs. I think we talked about this previously when doing the restart but moving on because it was so long, and I think all plot processes will be split into 31day and 90day jobs soon. |
|
Oh yes, now I am remembering that now. |
|
I can't compare the final tar files to the parallel but the logs look good! |
|
Oh, I see, you can't compare the tar files because they weren't created due to the walltime exceedance - I see that they are not there on emc.vpppg. Ah well. Looks like you gave it the check mark, so time to check the code and then I'll merge. |
|
Before I merge, is this code up to date or do we need to bring in develop? |
|
It's good! |
|
Great! Here we go! |
…A/EVS into feature/href_MPMD * 'feature/href_MPMD' of https://github.com/BinbinZhou-NOAA/EVS: feature/rm_cpreq (NOAA-EMC#643)
Note to developers: You must use this PR template!
Description of Changes
This removes the usage of
cpreqin EVS (an item under coder manger fixes and additions). It also sets KEEPDATA to YES for cam prep jobs, uses PB2NC_SKIP_VALID_TIMES for global_ens wave prep, and adds exclhost for jevs_mesoscale_headline_plots.It also fixes typos in the PR template.
Developer Questions and Checklist
${USER}where necessary throughout the code.HOMEevsare removed from the code.dev/drivers/scriptsordev/modulefileshave been made in the correspondingecf/scriptsandecf/defs/evs-nco.def?Testing Instructions
Set-up
ln -sf /lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix fixcd sorc;./buildFor everything below, be sure to set
HOMEevsto the location of the clone andCOMINto /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2.✔️ cam prep
cd dev/drivers/scripts/prep/camjevs_cam_href_severe_prep.sh✔️ global_ens prep
cd dev/drivers/scripts/prep/global_ensjevs_global_ens_wave_grid2obs_prep.sh,jevs_global_ens_atmos_prep.sh,jevs_global_ens_headline_prep.sh✔️ aqm stats
cd dev/drivers/scripts/stats/aqmjevs_aqm_grid2obs_stats.sh✔️ global_ens stats
cd dev/drivers/scripts/stats/global_ensjevs_global_ens_wave_grid2obs_stats.shjevs_global_ens_gefs_chem_grid2obs_aeronet_stats.sh✔️ aqm plots
cd dev/drivers/scripts/plots/aqmjevs_aqm_grid2obs_plots.sh✔️ global_ens plots
cd dev/drivers/scripts/plots/global_ensjevs_global_ens_wave_grid2obs_plots.sh✔️ mesoscale plots
cd dev/drivers/scripts/plots/mesoscalejevs_mesoscale_grid2obs_plots.sh,jevs_mesoscale_headline_plots.sh,jevs_mesoscale_precip_plots.sh,jevs_mesoscale_snowfall_plots.sh