Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add the Multigrid Beta Filter (MGBF) for ensemble localization (#699) #700

Merged
merged 1 commit into from
Mar 26, 2024
Merged

Add the Multigrid Beta Filter (MGBF) for ensemble localization (#699) #700

merged 1 commit into from
Mar 26, 2024

Conversation

shoyokota
Copy link
Collaborator

@shoyokota shoyokota commented Feb 16, 2024 •
edited
Loading

DUE DATE for merger of this PR into develop is 3/29/2024 (six weeks after PR creation).

Description

Resolves #699

This PR is to add the option to apply Multigrid Beta Filter (MGBF; Purser et al. 2022) for ensemble localization instead of Recursive Filter (RF). This work includes to add an initial version of the MGBF as a subdirectory in GSI.

To apply the MGBF, set "l_mgbf_loc=true" in the namelist and additionally input "mgbf_loc01.nml". (In Scale/Variable-Dependent Localization, input also "mgbf_locXX.nml" (XX=02,03,...) with the same number of grid points.)

How to set MGBF parameters in mgbf_locXX.nml

An example of mgbf_locXX.nml is as follows:

&PARAMETERS_MGBETA
  mg_ampl01=1.125,            ! length of vertical beta filter (standard deviation; filter grid unit)
  mg_ampl02=2.4,              ! length of horizontal beta filter (standard deviation; filter grid unit)
  mg_ampl03=0.85,             ! length of 3D beta filter (standard deviation; filter grid unit)
  mg_weig1=0.,                ! weight of generation 1
  mg_weig2=0.,                ! weight of generation 2
  mg_weig3=0.,                ! weight of generation 3
  mg_weig4=1.,                ! weight of generation 4
  hx=5,                       ! number of halo grid points in x-direction
  hy=5,                       ! number of halo grid points in y-direction
  hz=3,                       ! number of halo grid points in z-direction
  p=2,                        ! beta filter exponent
  mgbf_line=.false.,          ! set false except for mgbf_proc=2,4,7
  mgbf_proc=8,                ! 1-2: 3D filter; 3-5: 2D filter for static B; 6-8: 2D filter for localization (1,3,6: radial filter; 2,4,7: line filter; 5,8: isotropic line filter)
  lm_a=65,                    ! number of vertical layers in analysis grid
  lm=33,                      ! number of vertical layers in filter grid
  km2=0,                      ! number of 2D variables (set 0 for localization)
  km3=1,                      ! number of 3D variables (set 1 for localization)
  n_ens=30,                   ! ensemble size
  l_loc=.true.,               ! set true in localization
  l_filt_g1=.false.,          ! set false in skipping generation 1
  l_lin_vertical=.true.,      ! set true in applying linear vertical interpolation for analysis-filter mapping
  l_lin_horizontal=.true.,    ! set true in applying linear horizontal interpolation for analysis-filter mapping
  l_quad_horizontal=.false.,  ! set true in applying quadratic horizontal interpolation for analysis-filter mapping
  l_new_map=.true.,           ! set true in applying efficient vertical interpolation for analysis-filter mapping
  l_vertical_filter=.true.,   ! set true in applying vertical beta filter outside 2D filter
  ldelta=.false.,             ! (not used)
  lquart=.false.,             ! set true in applying quadratic horizontal interpolation for up/down-sending
  lhelm=.false.,              ! set true in applying Helmholtz differential operator for weighting
  nm0=1975,                   ! number of analysis grid points in x-direction
  mm0=1350,                   ! number of analysis grid points in y-direction
  gm_max=4,                   ! highest generation (max: 4)
  nxPE=79,                    ! number of MPI processors in x-direction
  nyPE=54,                    ! number of MPI processors in y-direction
  im_filt=8,                  ! number of filter grid points in each MPI processor in x-direction
  jm_filt=8,                  ! number of filter grid points in each MPI processor in y-direction
 /

Here, to make the result of MGBF-based localization similar to RF-based one, we can set the beta filter length ( mg_ampl0[12] ) from the recursive filter length in the GSI namelist ( s_ens_[vh] ) as:

  • $\text{mg\_ampl01} = \left[\text{s\_ens\_v (grid unit)} * \frac{1}{\sqrt{2}} * \frac{\text{lm}-1}{\text{lm\_a}-1} \right]^2$
  • $\text{mg\_ampl02} = \left[\frac{\text{s\_ens\_h (km)}}{\text{analysis grid interval (km)}} * \frac{1}{\sqrt{2}} * \frac{\text{im\_filt} * \text{nxPE}}{\text{nm0}} * \frac{1}{2} * \frac{1}{2} * \frac{1}{2} \right]^2$ (in case mg_weig[1-4]=[0,0,0,1])

Please note there are some limitations for the other MGBF parameters such as:

  • The number of MPI processors input in GSI should be nxPE x nyPE
  • (nm0, mm0, lm_a) should be the same as the GSI analysis grid
  • nm0 should be divisible by nxPE
  • mm0 should be divisible by nyPE
  • nm0 / nxPE = mm0 / nyPE

How to run RRFS regression tests with MGBF-based localization

Change settings in regression/ as follows, and run Test#3 (rrfs_3denvar_glbens)

diff --git a/regression/regression_namelists.sh b/regression/regression_namelists.sh
index 7ca183ef3..671d028ff 100755
--- a/regression/regression_namelists.sh
+++ b/regression/regression_namelists.sh
@@ -457,7 +457,7 @@ OBS_INPUT::
    beta_s0=0.15,s_ens_h=110,s_ens_v=3,
    regional_ensemble_option=1,
    pseudo_hybens = .false.,
-   grid_ratio_ens = 3,
+   grid_ratio_ens = 5.1,
    l_ens_in_diff_time=.true.,
    ensemble_path='',
    i_en_perts_io=1,
@@ -465,6 +465,7 @@ OBS_INPUT::
    fv3sar_bg_opt=0,
    readin_localization=.true.,
    ens_fast_read=.false.,
+   l_mgbf_loc=.true.,
  /
  &RAPIDREFRESH_CLDSURF
    dfi_radar_latent_heat_time_period=20.0,
diff --git a/regression/regression_namelists_db.sh b/regression/regression_namelists_db.sh
index e03917e88..36b8b6a22 100755
--- a/regression/regression_namelists_db.sh
+++ b/regression/regression_namelists_db.sh
@@ -438,7 +438,7 @@ OBS_INPUT::
    beta_s0=0.15,s_ens_h=110,s_ens_v=3,
    regional_ensemble_option=1,
    pseudo_hybens = .false.,
-   grid_ratio_ens = 3,
+   grid_ratio_ens = 5.1,
    l_ens_in_diff_time=.true.,
    ensemble_path='',
    i_en_perts_io=1,
@@ -446,6 +446,7 @@ OBS_INPUT::
    fv3sar_bg_opt=0,
    readin_localization=.true.,
    ens_fast_read=.false.,
+   l_mgbf_loc=.true.,
  /
  &RAPIDREFRESH_CLDSURF
    dfi_radar_latent_heat_time_period=20.0,
diff --git a/regression/regression_param.sh b/regression/regression_param.sh
index 2ac615fc4..6186acdbb 100755
--- a/regression/regression_param.sh
+++ b/regression/regression_param.sh
@@ -87,23 +87,23 @@ case $regtest in
     rrfs_3denvar_glbens)

         if [[ "$machine" = "Hera" ]]; then
-           topts[1]="0:15:00" ; popts[1]="20/1/"  ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="20/2/"  ; ropts[2]="/1"
+           topts[1]="0:15:00" ; popts[1]="30/2/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="15/4/"  ; ropts[2]="/1"
         elif [[ "$machine" = "Orion" ]]; then
-           topts[1]="0:15:00" ; popts[1]="20/1/" ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="20/2/" ; ropts[2]="/2"
+           topts[1]="0:15:00" ; popts[1]="30/2/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="15/4/"  ; ropts[2]="/2"
         elif [[ "$machine" = "Hercules" ]]; then
-           topts[1]="0:15:00" ; popts[1]="20/1/" ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="20/2/" ; ropts[2]="/2"
+           topts[1]="0:15:00" ; popts[1]="30/2/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="15/4/"  ; ropts[2]="/2"
         elif [[ "$machine" = "Jet" ]]; then
-           topts[1]="0:15:00" ; popts[1]="20/1/"  ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="20/2/"  ; ropts[2]="/1"
+           topts[1]="0:15:00" ; popts[1]="15/4/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="10/6/"  ; ropts[2]="/1"
         elif [[ "$machine" = "Gaea" ]]; then
-           topts[1]="0:15:00" ; popts[1]="18/1/"  ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="18/2/"  ; ropts[2]="/1"
+           topts[1]="0:15:00" ; popts[1]="15/4/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="10/6/"  ; ropts[2]="/1"
         elif [[ "$machine" = "wcoss2" ]]; then
-           topts[1]="0:15:00" ; popts[1]="64/1/" ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="128/2/" ; ropts[2]="/1"
+           topts[1]="0:15:00" ; popts[1]="60/1/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="30/2/"  ; ropts[2]="/1"
         fi

         if [ "$debug" = ".true." ] ; then
diff --git a/regression/rrfs_3denvar_glbens.sh b/regression/rrfs_3denvar_glbens.sh
index af5da5117..04fd73d57 100755
--- a/regression/rrfs_3denvar_glbens.sh
+++ b/regression/rrfs_3denvar_glbens.sh
@@ -272,6 +272,46 @@ $gsi_namelist

 EOF

+cat << EOF > mgbf_loc01.nml
+&PARAMETERS_MGBETA
+  mg_ampl01=1.125,            ! length of vertical beta filter (standard deviation; filter grid unit)
+  mg_ampl02=1.615,            ! length of horizontal beta filter (standard deviation; filter grid unit)
+  mg_ampl03=0.85,             ! length of 3D beta filter (standard deviation; filter grid unit)
+  mg_weig1=0.,                ! weight of generation 1
+  mg_weig2=1.,                ! weight of generation 2
+  mg_weig3=0.,                ! weight of generation 3
+  mg_weig4=0.,                ! weight of generation 4
+  hx=4,                       ! number of halo grid points in x-direction
+  hy=4,                       ! number of halo grid points in y-direction
+  hz=3,                       ! number of halo grid points in z-direction
+  p=2,                        ! beta filter exponent
+  mgbf_line=.false.,          ! set false except for mgbf_proc=2,4,7
+  mgbf_proc=8,                ! 1-2: 3D filter; 3-5: 2D filter for static B; 6-8: 2D filter for localization (1,3,6: radial filter; 2,4,7: line filter; 5,8: isotropic line filter)
+  lm_a=65,                    ! number of vertical layers in analysis grid
+  lm=33,                      ! number of vertical layers in filter grid
+  km2=0,                      ! number of 2D variables (set 0 for localization)
+  km3=1,                      ! number of 3D variables (set 1 for localization)
+  n_ens=10,                   ! ensemble size
+  l_loc=.true.,               ! set true in localization
+  l_filt_g1=.false.,          ! set false in skipping generation 1
+  l_lin_vertical=.true.,      ! set true in applying linear vertical interpolation for analysis-filter mapping
+  l_lin_horizontal=.true.,    ! set true in applying linear horizontal interpolation for analysis-filter mapping
+  l_quad_horizontal=.false.,  ! set true in applying quadratic horizontal interpolation for analysis-filter mapping
+  l_new_map=.true.,           ! set true in applying efficient vertical interpolation for analysis-filter mapping
+  l_vertical_filter=.true.,   ! set true in applying vertical beta filter outside 2D filter
+  ldelta=.false.,             ! (not used)
+  lquart=.false.,             ! set true in applying quadratic horizontal interpolation for up/down-sending
+  lhelm=.false.,              ! set true in applying Helmholtz differential operator for weighting
+  nm0=40,                     ! number of analysis grid points in x-direction
+  mm0=24,                     ! number of analysis grid points in y-direction
+  gm_max=2,                   ! highest generation (max: 4)
+  nxPE=10,                    ! number of MPI processors in x-direction
+  nyPE=6,                     ! number of MPI processors in y-direction
+  im_filt=4,                  ! number of filter grid points in each MPI processor in x-direction
+  jm_filt=4,                  ! number of filter grid points in each MPI processor in y-direction
+ /
+EOF
+
 # Copy executable and fixed files to $tmpdir
 if [[ $exp == *"updat"* ]]; then
    $ncp $gsiexec_updat  ./gsi.x

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

EnVar for NA-domain RRFS was tested with "mgbf_locXX.nml" (XX=01) shown above on Orion. The resulting analysis increment was similar to the original and the computation time for localization became short.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

@shoyokota
Copy link
Collaborator Author

@ShunLiu-NOAA Thank you for managing this PR. Could you additionally request review from @MiodragRancic-NOAA? This PR is mainly based on his code.

@shoyokota shoyokota removed their assignment Feb 16, 2024
@shoyokota
Copy link
Collaborator Author

@ShunLiu-NOAA Sorry, this unassignment was my mistake... Could you assign me again?

@TingLei-NOAA
Copy link
Contributor

@shoyokota As we discussed off-line, now I have two suggestions: 1) re-organize to let the mgbf lib as a separate lib. The availability of such a form is one of the benefit for such a oops-style standing alone mgbf lib. 2), I suggest to add log information about authors/major coding histories in the codes especially for the mgbf lib part. Such a mgbf lib with enough information about its' authors and major developing progresses could also be used for JEDI's mgbf component. I hope the major mgbf developers at EMC like @MiodragRancic-NOAA @JimPurser-NOAA and @ManuelPondeca-NOAA could help with this.

shoyokota pushed a commit to shoyokota/GSI that referenced this pull request Feb 20, 2024
Add MGBF-based localization (NOAA-EMC#700)
1c2b7e6
@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from eeadfd6 to 1c2b7e6 Compare February 20, 2024 19:05
shoyokota pushed a commit to shoyokota/GSI that referenced this pull request Feb 20, 2024
Add MGBF-based localization (NOAA-EMC#700)
997a53e
@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from 1c2b7e6 to 997a53e Compare February 20, 2024 21:57
@shoyokota
Copy link
Collaborator Author

@TingLei-NOAA Thank you for the suggestions.
(1) I moved src/gsi/mgbf to src/mgbf and re-organized it as a separate lib. This mgbf lib is linked in compiling gsi as like the other libraries.
(2) I added GSI-style headers in src/mgbf/*f90. @JimPurser-NOAA and @MiodragRancic-NOAA, these headers are based on the comments in your MGBF code. If you find wrong information in the headers or want to revise them, please let me know.

@ShunLiu-NOAA
Copy link
Contributor

@ManuelPondeca-NOAA, @guoqing-noaa and @TingLei-NOAA Could you please review this PR? MGBF might be implemented in RTMA in operation. Given Sho's return to JMA at the end of the next month, it is better that we can address the isusses before Sho departs EMC.

@MiodragRancic-NOAA
Copy link
Contributor

Parameters for localization and background error will be need to be held separately. Application to background error will use six grid generations (probably) because of very large correlation lengths of stream function. In addition, aspect tensors and scale weights will all be depended of model level and variable. Also, the filter will be applied to 2D variable, not only to 3D variables as in the case of filtering.

MiodragRancic-NOAA
MiodragRancic-NOAA reviewed Feb 26, 2024
View reviewed changes
Copy link
Contributor

@MiodragRancic-NOAA MiodragRancic-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to order subroutines as they are ordered in the case select statement.

shoyokota pushed a commit to shoyokota/GSI that referenced this pull request Feb 26, 2024
Add MGBF-based localization (NOAA-EMC#700)
e296508
@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from 997a53e to e296508 Compare February 26, 2024 20:53
@shoyokota
Copy link
Collaborator Author

@MiodragRancic-NOAA Thank you for reviewing the code.

Parameters for localization and background error will be need to be held separately. Application to background error will use six grid generations (probably) because of very large correlation lengths of stream function. In addition, aspect tensors and scale weights will all be depended of model level and variable. Also, the filter will be applied to 2D variable, not only to 3D variables as in the case of filtering.

Thank you for clarifying what is additionally required for MGBF-based static background error. Yes, what you are suggesting should be implemented in another PR to apply MGBF-based static background error.

With the MGBF library added in this PR, we can input two independent parameter files for localization and static background error by separately calling mg_initialize even without adding another MGBF library. So, I expect that MGBF-based static background error also will be implemented based on this MGBF library with some modifications.

It would be nice to order subroutines as they are ordered in the case select statement.

I modified the order in "select case" in mg_filtering.f90.

MiodragRancic-NOAA
MiodragRancic-NOAA reviewed Feb 26, 2024
View reviewed changes
Copy link
Contributor

@MiodragRancic-NOAA MiodragRancic-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be also good to add comments in mg_paramater and the namelist from which flag mg_filt gets values that clarify its meaning

shoyokota pushed a commit to shoyokota/GSI that referenced this pull request Feb 29, 2024
Add MGBF-based localization (NOAA-EMC#700)
6aea0f0
@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from e296508 to 6aea0f0 Compare February 29, 2024 16:48
@MiodragRancic-NOAA
Copy link
Contributor

MiodragRancic-NOAA commented Mar 1, 2024 via email

@TingLei-NOAA
Copy link
Contributor

TingLei-NOAA commented Mar 1, 2024 via email

@MiodragRancic-NOAA
Copy link
Contributor

MiodragRancic-NOAA commented Mar 1, 2024 via email

@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from c7d5585 to ce2f96b Compare March 21, 2024 15:34
@TingLei-NOAA
Copy link
Contributor

TingLei-NOAA commented Mar 21, 2024 •
edited
Loading

@shoyokota and @GangZhao-NOAA On wcoss2, when gsi was built with debug mode, for rrfs regression test , the first rrfs_3denvar_glbens_loproc_updat failed for

 nlat,nlon=nya,nxa=          117         199                   
forrtl: severe (71): integer divide by zero
Image              PC                Routine            Line        Source     
                   
gsi.x              0000000007DBB2BB  Unknown               Unknown  Unknown
libpthread-2.31.s  00001486B586D8C0  Unknown               Unknown  Unknown    gsi.x              000000000091627F  general_sub2grid_         447  general_sub
2grid_mod.f90      
gsi.x              0000000000C6E593  gridmod_mp_init_g         680  gridmod.F90

I am trying to see what the problem is.
Could you also try this regression tests using debug mode GSI ?
Thanks
An update: the cultprit is an empty anavinfo file. I will see how it will be fixed.

@GangZhao-NOAA
Copy link
Contributor

@shoyokota and @GangZhao-NOAA On wcoss2, when gsi was built with debug mode, for rrfs regression test , the first rrfs_3denvar_glbens_loproc_updat failed for

 nlat,nlon=nya,nxa=          117         199                   
forrtl: severe (71): integer divide by zero
Image              PC                Routine            Line        Source     
                   
gsi.x              0000000007DBB2BB  Unknown               Unknown  Unknown
libpthread-2.31.s  00001486B586D8C0  Unknown               Unknown  Unknown    gsi.x              000000000091627F  general_sub2grid_         447  general_sub
2grid_mod.f90      
gsi.x              0000000000C6E593  gridmod_mp_init_g         680  gridmod.F90

I am trying to see what the problem is. Could you also try this regression tests using debug mode GSI ? Thanks An update: the cultprit is an empty anavinfo file. I will see how it will be fixed.

@TingLei-NOAA @shoyokota Hi Ting and Sho,
I haven't run the ctest in debug mode before. As you mentioned, to run the regression test in debug mode, I need to re-compile the GSI code , I mean both control code and updated code, right?

Then after the compilations, to run the regression test, is there any specific option for running? For example, in the command line, "ctest -R rrfs_3denvar_glbens", do I need to add any option to it, like "-debug"?

Sho mentioned the configuration file "regression_namelist_db.txt", is this the file used for debug mode. Then how should I let the ctest to use it, instead of regular "regression_namelist.txt"?

Thank you!

-Gang

@shoyokota
Copy link
Collaborator Author

@GangZhao-NOAA In my understanding, all we have to do to run ctest in the debug-mode is to re-compile GSI with BUILD_TYPE="Debug" both for control and updated codes. In running ctests, we don't need to add any options, and regression_namelist_db.txt is automatically applied.

@TingLei-NOAA Is my understanding correct? I also will run the debug-mode ctests on Orion.

@TingLei-NOAA
Copy link
Contributor

@GangZhao-NOAA Yes. As @shoyokota , just re-compile gsi using the debug mode and run the ctests. Regression tests with debug mode GSI are not required. But I feel that will be very helpful to verify the codes.

@TingLei-NOAA
Copy link
Contributor

@GangZhao-NOAA I had finished debug mode GSI 's regression tests on wcoss2. I will give an update below. I think you don't have to do them again on other machines. But if you could run a rtma operational case with this PR, that will be great while I would run a rrfs operational case with this PR.
Debug mode regression tests results: rtma, rrfs_3denvar_glbens, netcdf_fv3_regional passed. global_4denvar and hafs regression tests passed for the known issues to be fixed by PR #722 and #679. Hence, it is concluded debug mode GSI regression tests on wcoss2 haven't revealed any issues with this PR.

@GangZhao-NOAA
Copy link
Contributor

@GangZhao-NOAA I had finished debug mode GSI 's regression tests on wcoss2. I will give an update below. I think you don't have to do them again on other machines. But if you could run a rtma operational case with this PR, that will be great while I would run a rrfs operational case with this PR. Debug mode regression tests results: rtma, rrfs_3denvar_glbens, netcdf_fv3_regional passed. global_4denvar and hafs regression tests passed for the known issues to be fixed by PR #722 and #679. Hence, it is concluded debug mode GSI regression tests on wcoss2 haven't revealed any issues with this PR.

@TingLei-NOAA @shoyokota Hi, Ting and Sho,
Thank you for the instructions and suggestions on the debug mode run.
I ran the regression tests in debug mode on Hera. Here are the brief reports:

(base) [Gang.Zhao@hfe08:build] (feature/PR_NOAA-EMC_EnVar-MGBF)$ ctest -j 7
Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_mgbf_syokota/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #6: hafs_3denvar_hybens ..............***Failed   61.55 sec
2/7 Test #5: hafs_4denvar_glbens ..............***Failed  182.56 sec
3/7 Test #4: netcdf_fv3_regional ..............   Passed  856.88 sec
4/7 Test #3: rrfs_3denvar_glbens ..............   Passed  980.15 sec
5/7 Test #2: rtma .............................***Failed  6153.79 sec

netcdf_fv3_regional and rrfs_3denvar_glbens passed, rtma almost passed (just one test took too much time which is beyond the limit and count towards failure); two hafs tests crashed very quickly; two global tests failed due to running time beyond the wall clock limits.

As you mentioned to run a rtma operational case with this PR, the GSI code of the current (2D)RTMA is somehow fairly different to the current EMC/GSI code. We tested it one month ago, it could run but with an issue related to analysis of the obs of total cloud amount.

Thank you!

TingLei-NOAA
TingLei-NOAA reviewed Mar 22, 2024
View reviewed changes
Copy link
Contributor

@TingLei-NOAA TingLei-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR passed a test using a modified rrfs operational case (while miter=2) in which it produced identical results compared with the head of EMC GSI.
Approval is pending upon the suggested changes

@TingLei-NOAA
Copy link
Contributor

TingLei-NOAA commented Mar 22, 2024 •
edited
Loading

@GangZhao-NOAA I had finished debug mode GSI 's regression tests on wcoss2. I will give an update below. I think you don't have to do them again on other machines. But if you could run a rtma operational case with this PR, that will be great while I would run a rrfs operational case with this PR. Debug mode regression tests results: rtma, rrfs_3denvar_glbens, netcdf_fv3_regional passed. global_4denvar and hafs regression tests passed for the known issues to be fixed by PR #722 and #679. Hence, it is concluded debug mode GSI regression tests on wcoss2 haven't revealed any issues with this PR.

@TingLei-NOAA @shoyokota Hi, Ting and Sho, Thank you for the instructions and suggestions on the debug mode run. I ran the regression tests in debug mode on Hera. Here are the brief reports:

(base) [Gang.Zhao@hfe08:build] (feature/PR_NOAA-EMC_EnVar-MGBF)$ ctest -j 7
Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_mgbf_syokota/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #6: hafs_3denvar_hybens ..............***Failed   61.55 sec
2/7 Test #5: hafs_4denvar_glbens ..............***Failed  182.56 sec
3/7 Test #4: netcdf_fv3_regional ..............   Passed  856.88 sec
4/7 Test #3: rrfs_3denvar_glbens ..............   Passed  980.15 sec
5/7 Test #2: rtma .............................***Failed  6153.79 sec

netcdf_fv3_regional and rrfs_3denvar_glbens passed, rtma almost passed (just one test took too much time which is beyond the limit and count towards failure); two hafs tests crashed very quickly; two global tests failed due to running time beyond the wall clock limits.

As you mentioned to run a rtma operational case with this PR, the GSI code of the current (2D)RTMA is somehow fairly different to the current EMC/GSI code. We tested it one month ago, it could run but with an issue related to analysis of the obs of total cloud amount.

Thank you!

@GangZhao-NOAA Thanks. I think your tests a alongside with the test using rrfs-like real case areported re enough, at least for this reviewing , to verify this PR!

shoyokota pushed a commit to shoyokota/GSI that referenced this pull request Mar 22, 2024
Add MGBF-based localization (NOAA-EMC#700)
a1e875d
@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from ce2f96b to a1e875d Compare March 22, 2024 15:50
shoyokota pushed a commit to shoyokota/GSI that referenced this pull request Mar 22, 2024
Add MGBF-based localization (NOAA-EMC#700)
61652b6
@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from a1e875d to 61652b6 Compare March 22, 2024 15:52
shoyokota pushed a commit to shoyokota/GSI that referenced this pull request Mar 22, 2024
Add MGBF-based localization (NOAA-EMC#700)
e333a07
@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from 61652b6 to e333a07 Compare March 22, 2024 16:29
@shoyokota shoyokota force-pushed the feature/PR_NOAA-EMC_EnVar-MGBF branch from e333a07 to b1b5a96 Compare March 25, 2024 13:48
@shoyokota
Copy link
Collaborator Author

I just merged the latest GSI develop branch in the last push.

@TingLei-NOAA
Copy link
Contributor

@shoyokota Thanks for clarification!

@shoyokota
Copy link
Collaborator Author

I confirmed all regression tests with the latest code were passed on Orion. (For global_4denvar and global_enkf, the 2022110900 case was tested.)

Test project /home/syokota/da/gsi/ctest/PR_NOAA-EMC_EnVar-MGBF_09/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  1923.88 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  2167.45 sec
3/7 Test #7: global_enkf ......................   Passed  2351.31 sec
4/7 Test #2: rtma .............................   Passed  2829.07 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  2899.09 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  2962.67 sec
7/7 Test #1: global_4denvar ...................   Passed  3405.76 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 3405.77 sec

For rrfs_3denvar_glbens, I confirmed that the debug-mode run was also passed. (The debug-mode failures except for rrfs_3denvar_glbens should not be related to this PR.)

@GangZhao-NOAA
Copy link
Contributor

GangZhao-NOAA commented Mar 25, 2024 •
edited
Loading

@shoyokota @ShunLiu-NOAA @TingLei-NOAA
With Sho's latest updated code (commit #b1b5a962) vs the control code (EMC-GSI develop branch #f7e93abf), I ran the regression tests on WCOSS2 (Cactus) and Hera (Rocky-8), all the 7 tasks passed successfully. Here is the brief reports:
on WCOSS2/Cactus:
regression results are under directory "/lfs/h2/emc/da/noscrub/gang.zhao/noscrub/regression_b1b5a962_mgbf_vs_f7e93abf_wcoss2_cactus"
regression tests were running under directory "/lfs/h2/emc/ptmp/gang.zhao/gsi_mgbf_syokota"

[gang.zhao@clogin07:build] (feature/PR_NOAA-EMC_EnVar-MGBF)$ ctest -j 7
Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/ProdGSI_Dev/gsi_mgbf_syokota/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  482.90 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  545.67 sec
3/7 Test #7: global_enkf ......................   Passed  856.54 sec
4/7 Test #2: rtma .............................   Passed  975.33 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  1154.16 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  1214.35 sec
7/7 Test #1: global_4denvar ...................   Passed  1682.69 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 1682.70 sec

On Hera (Rocky-8):
regression results are under directory "/scratch1/NCEPDEV/da/Gang.Zhao/noscrub/regression_b1b5a962_mgbf_vs_f7e93abf_hera"
regression tests were running under directory "/scratch1/NCEPDEV/stmp2/Gang.Zhao/gsi_mgbf_syokota"

(base) [Gang.Zhao@hfe08:build] (feature/PR_NOAA-EMC_EnVar-MGBF)$ ctest -j 7
Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_mgbf_syokota/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  493.18 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  561.20 sec
3/7 Test #2: rtma .............................   Passed  975.66 sec
4/7 Test #7: global_enkf ......................   Passed  1252.31 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  1290.15 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  1408.73 sec
7/7 Test #1: global_4denvar ...................   Passed  1979.08 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 1979.27 sec

@ShunLiu-NOAA
Copy link
Contributor

@GangZhao-NOAA thank you for regression test!

@ShunLiu-NOAA ShunLiu-NOAA merged commit 1cc934e into NOAA-EMC:develop Mar 26, 2024
4 checks passed
@RussTreadon-NOAA RussTreadon-NOAA mentioned this pull request Jul 3, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ShunLiu-NOAA ShunLiu-NOAA ShunLiu-NOAA approved these changes

@MiodragRancic-NOAA MiodragRancic-NOAA MiodragRancic-NOAA approved these changes

@TingLei-NOAA TingLei-NOAA TingLei-NOAA approved these changes

@JimPurser-NOAA JimPurser-NOAA Awaiting requested review from JimPurser-NOAA

@ManuelPondeca-NOAA ManuelPondeca-NOAA Awaiting requested review from ManuelPondeca-NOAA

@guoqing-noaa guoqing-noaa Awaiting requested review from guoqing-noaa

Assignees

@ShunLiu-NOAA ShunLiu-NOAA

@shoyokota shoyokota

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add the Multigrid Beta Filter (MGBF) for ensemble localization in preparation for the 3DRTMAv1 implementation
6 participants
@shoyokota @TingLei-NOAA @ShunLiu-NOAA @MiodragRancic-NOAA @RussTreadon-NOAA @GangZhao-NOAA