Input convective precipitation flux is zero in GEOS-FP (6/2020+) and GEOS-IT

[lizziel]

Your name

Lizzie Lundgren

Your affiliation

Harvard University

What happened? What did you expect to happen?

Viral Shah reports:

There was an error in archiving the convective precipitation fields (PFLCU and PFICU) in our GEOS-IT and GEOS-FP data after the switch to Grell-Freitas (the fields are just set to zero). These fields are used in GEOS-Chem to calculate convective washout and re-evaporation

This bug impacts all GEOS-FP fields starting in Jun 2020 and the entire GEOS-IT inventory.

[yuanjianz]

Glad to find this after I have just reported this to Randall (What a coincidence! 😧 ). So it is confirmed to be a bug instead of a feature. It seems GEOS-IT has a much large convective rainfall re-evaporation than MERRA-2 here. Does it mean that we might be missing a lot of downward transport through convective precipitation and re-evaporation?

plot above is from 1 year MERRA-2 v.s. GEOS-IT transport tracer simulation for 2019

I am also curious if a complete re-run of GEOS is needed to regenerate these fields. If so, how long would it be estimated to take? @lizziel @viral211

[viral211]

Hi @yuanjianz. Thanks for the plots. We cannot redo the entire GEOS-IT run, but we might be able to implement a fix moving forward, at least for GEOS-FP. We could also try to calculate the missing fields from DQRCU and REEVAPCN. I am trying to find out if that would work.

[yuanjianz]

According to #1454, MERRA-2 precipitation flux reconstructed from DQR* - REEVAP* is lower than PFI* + PFL* by about 30-40%. In GEOS-IT, my tests shows that at least LSAN is lower when reconstructing from DQRLSAN and REEVAPLS. Maybe fixed GEOS-FP runs can give us more detailed hints, @viral211.

* from 1 year GEOS-IT transport tracer simulation for 2019
** reconstruction is by DRQLSAN - REEVAPLS and cumulative summation from lev=0 to lev=72 (caveat: a center diagnostic while PF* is level diganostic)

I also notice that PRECCON is preserved in GEOS-IT archive. I wonder if it is possible to utlize this information. For example, scale grid-wise reconstructed precipitation flux with PRECCON(surface flux)/reconstructed surface flux.

[viral211]

In GF, DQRCU is the net precipitation source, and we can calculate the precipitation fluxes and the re-evaporated fraction from it alone - with just unit conversion. DQR is in units of kg(precip) / kg (air) / s, and precip fluxes in kg(precip) / m2 / s. REEVAPCN is the cumulative (over layers) re-evaporated flux, which is different from the previous definition, but we don’t need to use that field now anyway.

Below are some plots comparing the different fields and showing minor differences (probably because I am using dry air density instead of total for the unit conversion).

Surface convective precip flux from a 1-hour GEOS (1 degree) run:

Convective precip flux profile from the GEOS run (GEOS levels are flipped compared to GEOS-Chem):

Monthly convective precip flux from a GEOS-Chem run with GEOS-IT fields (from Lizzie’s transport tracer run at 2x2.5)

[viral211]

@yuanjianz

According to #1454, MERRA-2 precipitation flux reconstructed from DQR* - REEVAP* is lower than PFI* + PFL* by about 30-40%. In GEOS-IT, my tests shows that at least LSAN is lower when reconstructing from DQRLSAN and REEVAPLS. Maybe fixed GEOS-FP runs can give us more detailed hints,

This is expected because DQRLSAN does not include all sources of precipitation (eg. accretion of cloud drops by rain). We don't have a separate 3D field with these precip sources, but they can be calculated from PFL+PFILSAN, and REEVAP if needed.

[lizziel]

@lizziel:

Thank you Viral. Would this approach work with RAS-generated fields as well? If yes, we could retire PFLC* met arrays and avoid awkward switches between using them or not based on meteorology and simulation date.

@viral211:

Unfortunately, this doesn’t work for RAS. DQRCU is defined differently there. Yes, it would be awkward to have a switch based on simulation time or met product. I suggest a switch based on a comparison of PFLCN+PFICN at level 1 with PRECCON. For RAS, they should be equal. If PRECCON>0, but PFLCN+PFICN==0 (or say < 0.1 * PRECCON), then calculate PFLCN+PFICN from DQR.

This is a great suggestion. Let's follow up after the GCSC meeting on next steps. I can put together a PR and test, or you can if you prefer. This fix will go into 14.5.1.

[lizziel]

The plan moving forwards is I will implement a fix in the code. Since we will use existing fields in the logic paths the data processing will not need to change. Tagging @yidant.

[yuanjianz]

@lizziel:

Thank you Viral. Would this approach work with RAS-generated fields as well? If yes, we could retire PFLC* met arrays and avoid awkward switches between using them or not based on meteorology and simulation date.

@viral211:

Unfortunately, this doesn’t work for RAS. DQRCU is defined differently there. Yes, it would be awkward to have a switch based on simulation time or met product. I suggest a switch based on a comparison of PFLCN+PFICN at level 1 with PRECCON. For RAS, they should be equal. If PRECCON>0, but PFLCN+PFICN==0 (or say < 0.1 * PRECCON), then calculate PFLCN+PFICN from DQR.

This is a great suggestion. Let's follow up after the GCSC meeting on next steps. I can put together a PR and test, or you can if you prefer. This fix will go into 14.5.1.

Sorry for the late catch up!

Adding some details about using DQRCU + REEVAPCN to determine the cloud base:

@yuanjianz:

GEOS-IT DQRCU level1 is all zero. But in MERRA-2, (PFICU + PFLCU) level1 - level2 are all negatives (re-evaporation). This might be one of the reasons of the inconsistency of your cumulative DQRCU and PF* at the surface.

@viral211:

Good catch about the issues with DQRCU and REEVAP at level 1. You are right that it doesn’t make sense and could be the reason for some of the differences between the 3D PF* and 2D PREC* field. We’ll just have to live with this. I don’t expect the diff to be a large. But to get around this issue at level 1, I suggest we set REEVAP also to 0 at that level. That way the cloud base, calculated by adding DQRCU and REEVAP, will not be at level 1.

I implemented the fix itself, but I am also a bit worried about the switch. PRECCON is around 80% zero under C180 (0.5x0.625) resolution. If we use the on-the-fly switch based on PRECCON and PFLCU+PFICU, 80% of grids would be treated as non-GF grids and only 20% are GF grids. (I assume no precipitation at the surface does not mean no convective activities.)

I would prefer to treat all grids with the same convection procedure (PDOWN, DQRCU and REEVAPCN). In this sense, we might inevitably need the switch based on date and meteorlogy (in GCHP and all other MPI implementations, we cannot loop over all grids to determine a global setting).

@lizziel, for evaluation, may I open a draft pull request to share the codes? Besides, I can run two GCHP Transport-Tracer simulations with GEOS-FP and GEOS-IT and compare with your results mentioned in #1409. Are they based on v14.4.1, using c24 20190101 restart to simulate 2021 whole year?

[viral211]

Thanks, @yuanjianz. I agree that it should be a global setting and I also can't think of a better way to set it than based on date and meteorology given the limitation in MPI implementations you pointed out.

[yuanjianz]

@lizziel, I can implement the switch based on date and meteorlogy but it seems GCHP does not have built-in input%opt of meteorology:

geos-chem/GeosCore/input_mod.F90

Line 615 in bef56c6

Input_Opt%MetField = 'See ExtData.rc'

I wonder might it complicate things too much if we add this entry into geoschem_config.yml for GCHP?

[lizziel]

I think it would be fine to add met source to geoschem_config.yml for GCHP. This makes GEOS-IT easy.

FP is still very complicated. If this is an issue only for GCHP then I think date-checking should be within a MODEL_GCHP block to keep GC-Classic easy. Ideally it also would only happen if simulation start and end actually crossed the time boundary at which the convection switch happened. If the simulation is entirely before or after the switch then a logical could be set once during initialization. Is there a resolution dependency for the need to do this? If yes, perhaps the date checking could also be skipped if sufficiently low resolution.

This would leave us with only having to check the date if using FP, running at higher resolution than a threshold, and simulating across the convection change date. If the simulation crosses the time boundary then there is a further issue of what is the date of the switch. This is complicated because our processed files and the raw GMAO files switch on different dates.

I recall @sdeastham long ago was very against checking actual dates in the model. Seb, do you still feel this way?

[yuanjianz]

For GC-Classics, we can loop through all grids and determine the convection scheme. Then no need for switch based on meteorlogy or date.

For GCHP and other MPI jobs,
@lizziel:

If the simulation is entirely before or after the switch then a logical could be set once during initialization.

I think in principle, we do not recommend run the simulation with GEOS-FP across the major model update (GF convection). (And we need to warn users of this issue during run directory set up). So it would be much more easier if we can just set the logical during initialization based on meteorology and the first day of the simulation.

[lizziel]

For GC-Classics, we can loop through all grids and determine the convection scheme.

Do you mean PRECCON zero test? It occurred to me that some GC-Classic simulations actually do run at very high resolution, specifically nested grid runs and global carbon. This means we need to handle those cases as well.

It is true we do not recommend running FP across the convection switch, but we do not prohibit it. Differences due to the convection change are expected, but the convective precipitation flux bug is not. I agree about keeping things simple, and having a warning at run directory creation. I also realized last night that our FP raw fields only start in 2021 so would always be GF.

In summary, I think this is what we are converging on as a way forwards.

1. Add meteorology source name to GCHP run directory geoschem_config.yml
2. Create new Input_Opt logical for whether source met convective scheme is Grell-Friedas, e.g. Input_Opt%Met_Conv_Is_Grell_Friedas
3. Set logical during initialization as true if met is GEOS-IT or if met is FP and start date is 6/1/2020 or later
4. Implement Viral's recommended criteria for bypassing zero convective precipitation flux
5. Execute the Viral's fix only if the Input_Opt logical is true.
6. Add warning during run directory creation if user chooses FP meteorology. We could actually pause the run directory creation and force the user to acknowledge they understand.

Am I missing anything?

[yuanjianz]

Hi @lizziel, I basically agree with the solutions you posted here. The only concern is (4), do we still need the criteria(PFICU+PFLCU < 0.1 PRECCON) if we already have the logical set up during initialization?

[lizziel]

By implementation of Viral's recommended criteria I actually meant implement Viral's fix, namely computation involving DQRCU. The criteria for applying the alternate method would indeed use the new Input_Opt logical.

[yuanjianz]

I see. Another question is,

It is true we do not recommend running FP across the convection switch, but we do not prohibit it. Differences due to the convection change are expected, but the convective precipitation flux bug is not.

3. Set logical during initialization as true if met is GEOS-IT or if met is FP and start date is 6/1/2020 or later

If we set up the logical at the intialization, those who wish to run FP across the convection switch may suffer from the zero PF* and cloud base bug. In this sense, we might need to set this logical based on the current date time.

I am sharing my codes through a draft pull request #2523

[lizziel]

If we set up the logical at the intialization, those who wish to run FP across the convection switch may suffer from the zero PF* and cloud base bug.

Correct. There would be a warning about this during run directory creation. The fix would be for users to split up their run such that one run ends on 6/1/2020 and the next one begins where it left off. This can be clearly explained as part of the warning.

[sdeastham]

I recall @sdeastham long ago was very against checking actual dates in the model. Seb, do you still feel this way?

Unfortunately I do sill feel that this would be a mistake. GCHP is meant to be both time and grid agnostic, and it is difficult to actually know what date some data are coming from. For example, it is easy (and sometimes desirable) to loop a year of met data, with the rest of the simulation evolving. If we have hard coded date checks then that would result in unexpected behaviour, with the same met data being treated differently in different years. The simulation would not know that the data is from a different year unless we propagated information through the ExtData layer, which would mean more exceptions and brittle code.

It sounds like we are all on the same page broadly - a run really shouldn't cross a major model change, so in general we don't want/need a truly seamless solution. But felt I should answer the question!

[yuanjianz]

Considering @sdeastham 's suggestion, might it be more straightforward to just add an entry for whether to enable GF convection in yml configuration? (We can explain this entry in documentation and state our recommendation.)

[lizziel]

might it be more straightforward to just add an entry for whether to enable GF convection in yml configuration?

This would be more straightforward for implementation but less so for the user. It adds risk to all simulations rather than just the case of FP crossing the boundary. If both options were scientifically valid in different ways then having it as an option would be useful. However, putting the wrong setting introduces a bug that could be avoided by looking at met and configured start (except for the FP at the boundary which users would be warned about). Users may or may not remember to do this.

That being said, I'm open to other arguments for this setting. For example, it could be auto-updated from setCommonRunSettings.sh. I generally err on the side of not adding more to that file, however. @sdeastham, do you have any strong opinions about this?

[lizziel]

Also note the start time is already saved in Input_Opt, so no new code is needed to extract the start time of the simulation. Stop time is also available if we also want to print a warning during init to log should the run use FP and cross the boundary.

[sdeastham]

@lizziel My point though is that we don't know the time of the met file - I could specify 2024 as the time of the simulation, but if I only make met data for 2010 available it will get looped (a situation which might be desirable if evaluating a steady-state response to changing emissions data, or if a user only has limited data). The problem to my mind is that the simulation date and time are not reliably connected to the date and time in the file (by design). In fact, if we hardcode a switch under the hood which uses the simulation date to decide on whether to use GF convection, users would need to edit the code to use looping meteorology.

Personally I think that it's better to be explicit than implicit. I remember Andrea Molod arguing years ago that we should strongly discourage users from running simulations which cross model update boundaries, and I'm inclined to agree. I would actually argue that it's better to have the model fail because a field is missing (i.e. a clear indicator that something has changed) than to hardcode a switchover.

[lizziel]

Ugh, I see what you are saying now. That makes sense.

[lizziel]

So to summarize:

• There will be a switch in geoschem_config.yml for specifying whether to assume input met uses GF in convection.
• The switch will automatically be set during run directory creation to true for GEOS-IT, and false for all other cases including FP. This is because the default restart file and start date for FP run directories is 2019.
• If using FP then there will be a warning during run directory creation with the following information:
  • The convection scheme used for met generation changed from RAS to Grell-Freidas in archived GEOS-FP meteorology files starting June 1, 2020 with impact on vertical transport. Discussion and analysis of the impact is available at github.com/geoschem/geos-chem/issues/1409.
  • There is a bug in convective precipitation flux fields following the switch where all values are zero. To avoid this bug users should manually update geoschem_config.yml entry (to be filled in once we have a final name/location) between runs before and after the time boundary based on dates of meteorology used.
  • Due to these issues it is not recommended to run across the June 1, 2020 in a single simulation using GEOS-FP meteorology. It is better to either break up the run in time before and after the switch or use a different meteorology inventory.
• This warning will stop run directory creation and require acknowledgement from the user to continue.

Did I miss anything?

[yuanjianz]

Thanks @lizziel, I think it is good to go.

[viral211]

I agree with what @lizziel had said earlier: we should not have this switch as an option in geoschem_config.yml because it is not optional. It should be under the hood, but obviously not tied to the simulation date, given what @sdeastham pointed out.

For GEOS-IT this is straightforward. For FP, couldn't we test the PRECCON, PF condition for each column at, say, the beginning of each day, and then flip the (global) GF switch to true if the condition is not met for any column? We need to do this only until the GF switch is false, and once set to true, we don't have to test again. Just something to consider, if it is doable.

It is still a good idea to warn users about the discontinuities in FP during run directory creation.

[yuanjianz]

For FP, couldn't we test the PRECCON, PF condition for each column at, say, the beginning of each day, and then flip the (global) GF switch to true if the condition is not met for any column?

I think it is doable in GC-Classic. But in my understanding, GEOS-Chem GridComp is designed to work as individual columns and would not allow inter-column communication in MPI implementation like GCHP and GEOS (Or we can only do this during the initialization of the whole GEOS-Chem GridComp?) @lizziel since this is a structure question.

[lizziel]

GEOS-Chem GridComp is designed to work as individual columns and would not allow inter-column communication in MPI implementation like GCHP and GEOS

If one instance of the PRECCON, PF condition indicates GF then we could broadcast the logical across all nodes.

@sdeastham, what are your thoughts on this new suggestion?

[sdeastham]

In this case I'd argue that it's not an option, but rather a model configuration switch. People like Lee Murray and myself are looking to translate met data from other sources into input for GEOS-Chem, which means we do need to be able to specify how the met data is handled - including which convection scheme to use. It seems risky to rely on detection of an invalid configuration of inputs in order to decide which convection scheme to use.

That having been said, I'm probably a bit too much of a purist in this regard. Using an MPI broadcast with a logical is doable for sure, as long as we can absolutely reliably detect (based only on mangled met data) that we should be using GF convection. I'd still like to have some way - even if it's a hardcoded boolean - of forcing the met data to perform convective washout using a specific approach though.

One last thought. Going back through the thread, is there a specific reason we can't use the new reconstruction scheme for all met data (including pre-2020 GFP), rather than switching? Is the new approach actually worse or just slightly different for unclear reasons? It's not like we are actually changing how we perform convection/convective washout calculations in GC as far as I can tell, just how we reconstruct a quantity.

[viral211]

I see @sdeastham's point. The config could be named something like "reconstruct_convective_precip" because the bug is not an inherent issue of the GF scheme, it was just introduced during the transition. I would still advocate for a runtime check to make sure that the switch is set correctly and stop the run if not. That way a user doesn't mistakenly use precip fluxes from the met product if they are zero throughout.
The reconstruction scheme doesn't work for the RAS output because its precipitation source term doesn't include all sources (e.g. collection of cloud droplets by rain) but they are included in the precipitation fluxes.

[lizziel]

If there is a run-time check that stops the model regardless of what the user sets then I am not seeing the advantage of having the manual setting. I am going to check with some of our FP users on their thoughts for more user perspectives. We have a lot of FP users for GC-Classic carbon.

[lizziel]

Actually, never mind about carbon since it does not undergo wet deposition.

[viral211]

I am probably being overcautious and maybe just a warning would suffice. Something that alerts the user if they are not using the recommended configuration.

[lizziel]

We discussed this at the GCST meeting with Randall and Daniel. The consensus is to exclude the switch from configuration and go with my original proposal:

1. Add meteorology source name to GCHP run directory geoschem_config.yml
2. Create new Input_Opt logical for whether to reconstruct convective precipitation flux
3. Set the switch automatically during run-time to true for GEOS-IT, true for GEOS-FP if start date is June 1 2020 or later, and false for all other cases.
4. Implement Viral's recommended criteria for bypassing zero convective precipitation flux and use it if the Input_Opt logical is true.
5. If using FP then there will be a warning during run directory creation with the following information:
   • The convection scheme used for met generation changed from RAS to Grell-Freidas in archived GEOS-FP meteorology files starting June 1, 2020 with impact on vertical transport. Discussion and analysis of the impact is available at github.com/geoschem/geos-chem/issues/1409.
   • There is a bug in convective precipitation flux input fields following the switch where all values are zero. This is automatically fixed by computing fluxes online for runs starting on or after June 1 2020. This fix assumes meteorology year corresponds to simulation year. Please create a GEOS-Chem GitHub issue if you wish to use a GEOS-FP meteorology year different from your simulation year.
   • Due to these issues we recommend splitting up GEOS-FP runs in time such that a single simulation does not run across June 1, 2020. Instead set one run to stop on June 1 2020 and then restart a new run from there.
6. This warning will stop run directory creation and require acknowledgement from the user to continue.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the issue from closing this issue.

[github-actions]

Closing due to inactivity