-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MAPL 2.34.0 failed in ufs weather model #207
Comments
@junwang-noaa I think your libraries are fine, the line it is dying on is: call ESMF_AttributeGet(grid, name='GridType', value=grid_type, _RC)
if(trim(grid_type) == "Cubed-Sphere") then This came in between 2.23.1 and 2.34.0 in changes from @aoloso. Before it was: if (im_world*6==jm_world) then So it's like it doesn't know your grid is a cubed-sphere grid. We might need to ping @weiyuan-jiang and @bena-nasa to see if maybe you need to set something in a file somewhere? |
@junwang-noaa The grid you are using in the UFS application, however it is created, that is passed to GOCART clearly does not have this attribute set. The solution is to add the appropriate attribute to the grid ('Gridtype' as the key, and value is 'Cubed-Sphere'), that is used in GOCART so that this routine can know what the grid type is. @weiyuan-jiang since you know how to build UFS can you take it from here? I've no idea where the grid comes from within UFS so can't really be of further advice. |
I think that we need to come up with a more robust way to do this. For instance, what if we run this on a regional grid (a single tile). We should not expect that the only solution is the global cube sphere. |
Thank you all for looking into this issue. @weiyuan-jiang I transferred the run directory to Orion at:
It has all the configure .rc files. Thanks |
@junwang-noaa I don't have the permission to that folder |
@weiyuan-jiang Please try it again. Thanks |
@junwang-noaa I still cannot access gocart directory. To build mapl_v2.34.0, where can I load the esmf8.4.0? I got error |
@weiyuan-jiang Sorry, please try again. The library team installed the library on acorn. UFS failed with ESMF 840 due to a bug in ESMF 840. The ESMF 841b07 works in UFS, GOCART runs with MAPL v2.23.1. Just when we try MAPL v2.34.0, we got error message. We don't have the library installed on Orion yet. Please let me know if you need to run tests on Orion, and I can see if EPIC team can install them. |
The whole reason we changed the logic to what is there now is for MORE robubstness. We were doing some open work where the component had a "grid" was still a cube-sphere in the sense that it had a copy of the local cubed-sphere domain. In which case we still want to go through this code path since the cells are still great circles on all 4 sides and same search algorithm can should be used. A fully robust, efficient grid agnostic implementation of this (additive point binning to an arbitrary grid) where the cell boundaries may not be great circles (like a tripolar grid for example), is beyond the scope of what MAPL can do but I fully agree that is what we need. ESMF does have an action item I believe to someday implement this, a generic point binning that is additive to a grid given a set of points as an extension of existing regridding methods. |
@junwang-noaa I think the problem is this line GOCART/ESMF/UFS/Aerosol_Cap.F90 Line 342 in 984fc49
The temporary fix should be call cap % cap_gc % set_grid(grid, lm=nlev, grid_type ="Cubed-Sphere", _RC) I am wondering how to pass the type info into the cap options. |
Oh, Sorry, I spoke to soon. Let me verify again . Yes, the change should work. Does UFS always work with cubed-sphere grid ? @junwang-noaa |
@weiyuan-jiang Currently UFS is using cubed sphere grid atmosphere model fv3atm, it has 6 tiles for global domain and 1 or multiple domains for regional domains. I am not sure if UFS will always use cubed sphere grid though as people might use other grids if they want to integrate other dycores. |
We also have the regional application. We have to think more than just
operational when designing these.
On Sat, Feb 25, 2023 at 5:59 PM Jun Wang ***@***.***> wrote:
@weiyuan-jiang <https://github.com/weiyuan-jiang> Currently UFS is using
cubed sphere grid atmosphere model fv3atm, it has 6 tiles for global domain
and 1 or multiple domains for regional domains. I am not sure if UFS will
always use cubed sphere grid though as people might use other grids if they
want to integrate other dycores.
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFIUVN2MX5GSFPCJUBFPVN3WZKFE3ANCNFSM6AAAAAAVHAFRV4>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
Barry Baker
National Oceanic and Atmospheric Administration
Air Resources Laboratory
Physical Research Scientist
Chemical Modeling and Emissions Group Leader
NCWCP, R/ARL, Rm. 4204
5830 University Research Court
College Park, Maryland 20740
Phone: (301) 683-1395
|
bbakernoaa I want to confirm that the regional application in UFS also runs on native cubed sphere grid as I explained in the message, and currently UFS does not have a regional fv3atm coupled with GOCART yet, please clarify if your group is working on a different grid or you know what other grid it will be. Also I believe coupling GOCART in regional requires additional work on boundaries, which is beyond the issue here.. Thanks |
@weiyuan-jiang There is an issue on our wcoss2 test platform, it may take some time to verify your fix in UFS. Thanks |
You are correct in that there currently is not any application in regional
for GOCART but this doesn’t mean that we shouldn’t make it a possibility.
Right now (I’ve tested before and brought it up in our joint meetings with
nasa before) gocart cannot except the regional grid because of the way the
grid is defined within the cap. We should find a way to more generally
define the grid.
On Sat, Feb 25, 2023 at 10:01 PM Jun Wang ***@***.***> wrote:
bbakernoaa <https://github.com/bbakernoaa> I want to confirm that the
regional application in UFS also runs on native cubed sphere grid grid as I
explained in the message, and currently UFS does not have a regional fv3atm
coupled with GOCART yet, please clarify if your group is working on a
different grid or on GOCART coupling in regional domain. Thanks
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFIUVN2W3WJNQROCFSOJ7QLWZLBQNANCNFSM6AAAAAAVHAFRV4>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
Barry Baker
National Oceanic and Atmospheric Administration
Air Resources Laboratory
Physical Research Scientist
Chemical Modeling and Emissions Group Leader
NCWCP, R/ARL, Rm. 4204
5830 University Research Court
College Park, Maryland 20740
Phone: (301) 683-1395
|
@weiyuan-jiang The fix you provided resolved the issue. Now the test ran successfully in UFS WM. Please let us know if you'd like to have a new MAPL library or you will update GOCART with the fix. If you are going to update MAPL, we will stop installing MAPL 2.34.0. Thanks |
@junwang-noaa If I can get my tests done, I hope to put out a MAPL 2.35 today that should have this fix. |
@junwang-noaa MAPL 2.35.0 has been released: https://github.com/GEOS-ESM/MAPL/releases/tag/v2.35.0 I think @weiyuan-jiang can tell you how to use this release for this issue. |
@junwang-noaa , Without changing ufs but adding this line to AERO.rc should work GridType: Cubed-Sphere |
@mathomp4 @weiyuan-jiang Thank you very much for fixing the issue! I will ask our library team to install MAPL 2.35.0 and test in UFS with the change Weiyuan suggested. Will let you know how it goes. |
@junwang-noaa You might actually want to wait for 2.35.1 which I'll issue soon. 2.35.0 had a bug in handling monthly history output. My guess is you don't use thatm but you might as well not have a buggy version! |
Sure, we will wait for 2.35.1. Thanks |
Whoops. Forgot to update this. 2.35.1 is out now! My guess is 2.35.0 is fine for your runs, but now the bug isn't there. 😄 |
@mathomp4 Thanks a lot!
Now I got error message:
I see the line 342 in Aerosol_Cap.F90 is:
Do I need to make any change in this line? Thanks |
@weiyuan-jiang I tried the following in the AERO.rc, I got the same error. GridType: "Cubed-Sphere" |
@junwang-noaa I know what happens here. The cf_root is not created when we set_grid. It seems that we need to exchange the two lines. GOCART/ESMF/UFS/Aerosol_Cap.F90 Lines 345 to 348 in df0e5a8
However, the grid should be set at this point. That creates a circular dependence: Maybe the best solution is to just change this line 342 of Aerosol_Cap.F90 and get back to the old set_grid of MAPL? call cap % cap_gc % set_grid(grid, lm=nlev, grid_type ="Cubed-Sphere", _RC) |
@weiyuan-jiang Thanks for looking into this. I confirm that with the change above in Aerosol_cap.F90 and MAPL 2.34.0, the UFS WM tests ran successfully. Thanks |
@junwang-noaa Could you please try this branch? https://github.com/GEOS-ESM/MAPL/tree/fix/wjiang/set_grid_fix ? The cf_root is not created when setting the grid. So you would need to move this line to CAP.rc |
@weiyuan-jiang I tried your branch and I got this error:
|
@junwang-noaa I have no problem running the new MAPL branch with unchanged ufs. There may be two reasons for the crash:
|
@weiyuan-jiang Thanks for looking into the issue. I added the GridType to CAP.rc, instead of AERO.rc, the UFS WM test finished successfully. Please let us know if you have a MAPL release version available. |
@junwang-noaa MAPL 2.35.2 is now out: https://github.com/GEOS-ESM/MAPL/releases/tag/v2.35.2 @weiyuan-jiang Can you inform us how to use your new fixes? |
@mathomp4 I want to confirm that the MAPL 2.35.2 has the fixes in Weiyuan's MAPL branch: https://github.com/GEOS-ESM/MAPL/tree/fix/wjiang/set_grid_fix, right? Thanks. |
Yes. It has GEOS-ESM/MAPL#2003 inside ETA: For Git Flow reasons, @weiyuan-jiang made a new branch against |
Yes. That is right |
@weiyuan-jiang I want to confirm with you, when using the new MAPL library, our gocart history files (gocart.inst_aod.20130401_0600z.nc4) now have one additional dimension: lev with value 1 to 4 as shown below:
Is this what we expect, may I ask what the "lev" means? Thanks |
It should be the number of wavelength. But I am not sure why it uses this confusing name "lev" which usually represents the levels. @bena-nasa, I think we should change this dimension name. |
@weiyuan-jiang I'll let @bena-nasa chime in, but I think one reason we did that (at least in the past) was for plotting packages like GrADS that could only handle 3rd dimensions that were layer or level. Without some post processing, the variables were unviewable. I suppose we should do whatever CF says is "right" for these sorts of things, but I want to say these might be part of the "discrete axis" part of the Conventions which is confusing to read. |
@mathomp4 @weiyuan-jiang Thanks for the information. I think we are OK to use the "lev" for 3rd dimension. Is it possible that we have attributes to specify what wavelengths the lev values represent? |
@junwang-noaa I believe that is specified in the GOCART2G_GridComp.rc file In the global-workflow it is here: https://github.com/NOAA-EMC/global-workflow/blob/develop/parm/chem/GOCART2G_GridComp.rc#L41 |
@bbakernoaa Thanks for the info. So do we want to have the information in the gocart files as attributes or it is OK to leave with lev value 1-4? I mean for downstream jobs (post or verification jobs) |
@junwang-noaa I think it is ok. In production we may send it through the UPP anyway. We currently do not pass the AOD back to the physics radiation. |
@bbakernoaa Thanks for confirming. I will ask the library team to install the MAPL 2.35.2 for UFS. |
I have run some tests with threads using the MAPL 2.35.2 and GOCART develop branch e2245c2. It looks to me that the threading does not work in UFS. We may still need some work to enable threads in UFS WM: single thread:
4 threads:
|
@junwang-noaa I think @weiyuan-jiang and @aoloso are looking at this... |
@weiyuan-jiang @aoloso I have two test cases ready on Orion for you to run some tests. Case 1: single thread (total PETs 192 for atm forecast and chem, layout 4x8, 1 thread)
Timing profile is in ESMF_Profile.summary
Case 2: 4 threads (total PETs 192 for atm forecast and chem, layout 1x8, 4 threads)
Timing profile is in ESMF_Profile.summary
My branch is: https://github.com/junwang-noaa/ufs-weather-model/blob/newmapl
You can compile the code by: cd ufs-weather-model/tests Please let me know if you have any questions, Thanks. |
We are trying to update MAPL library from 2.23.1 to MAPL 2.34.0 in ufs-weather-model. However we got following error message in GOCART:
pe=00098 FAIL at line=03053 Base_Base_implementation.F90 <status=57>
pe=00098 FAIL at line=00685 SU2G_GridCompMod.F90 <status=57>
pe=00098 FAIL at line=01817 MAPL_Generic.F90 <status=57>
pe=00098 FAIL at line=00193 BaseProfiler.F90 <Timer does not match start timer >
pe=00098 FAIL at line=01838 MAPL_Generic.F90 <status=1>
pe=00098 FAIL at line=00161 Aerosol_GridComp.F90
pe=00098 FAIL at line=01817 MAPL_Generic.F90 <status=1>
I tried the latest GOCART version acc574f in the develop branch (https://github.com/GEOS-ESM/GOCART/tree/develop), I got the same error. The code ran fine when switching back to MAPL 2.23.1. The ESMF library is: 8.4.1b07.
May I ask if anything needs to be updated to use MAPL 2.34.0? Thanks
The text was updated successfully, but these errors were encountered: