Monthly ESMValTool meeting August

[bouweandela]

The next monthly ESMValTool meeting will be in August. The result of the poll is that it will be on

Thursday, August 5 at 11:00 CEST (10:00 BST).

The duration will be one hour. Join the zoom meeting.

Calendar invitations to the meeting are sent to the mailing list, join it here.

@ESMValGroup/esmvaltool-developmentteam

Format and agenda

• Anyone with an interest in ESMValTool is welcome to join and ask questions
• We start the meeting by doing a round where everyone gets one minute to introduce themselves and tell what they will be working on in the next month or what they would like help with.
• Using the input from the introduction round, we can decide to talk about specific topics with the entire group or make smaller breakout groups to continue the discussion there.
• If you already know that you would like to bring up a particular topic in the meeting, you can mention the GitHub issue on the topic in a comment in this issue. That will make it easy to find for other people in the meeting and we can already start discussing there before the meeting. Just create a new GitHub issue about your topic if there is no open issue yet.
• GitHub issues are our preferred way to communicate outside the meeting and keep track of work, so any feature requests/action items/etc discussed in the meeting should ideally result in a GitHub issue.

Notes from the July meeting can be found here: #2189 (comment)

[bouweandela]

Thanks for attending everyone!

Here are some updates about what everyone is working on:

• @hb326 Paper for ozone analysis with esmvaltool submitted
• @schlunma Successfully defended his PhD 🥳
• @jvegasbsc Working with @remi-kazeroni on improving the ease with which observational datasets can be downloaded and CMORized in Streamline observations download #1657
• @katjaweigel Is still looking for reviewers for Splitting of flato13ipcc.yml into separate recipes and adding recipes for regional Figures #2156, please give her a hand @ESMValGroup/tech-reviewers and @ESMValGroup/science-reviewers! She is also working on improving support for ERA5 evapotranspiration in the ESMValCore, it looks like there is a problem with the sign.
• @bettina-gier Just joining the meeting to see our lovely faces 🥳
• @axel-lauer Working on proposals and reports
• @annefou Working on training material and developing new tutorials based on https://esmvalgroup.github.io/ESMValTool_Tutorial/. Interested in setting up a local data pool for Norway, see the discussion topic below.
• @remi-kazeroni Helped @LisaBock with IPCC, next steps will be to merge recipes and diagnostics. How to distribute Tier3 data? This data has licenses attached that we would violate if we share this. @bouweandela recommended talking to the Jasmin and Mistral infrastructure maintainers to make sure we can use their shared data collection for ERA5, because it is just too large to duplicate. @jvegasbsc Voiced concerns that accessing many files (Jasmin) or files in GRIB format (Mistral) would be too slow. @bouweandela recommended first implementing something that works, even if it's slow, with the existing data pools and then talking with their administrators and/or the iris developers if this turns out to be too slow.
• @egalytska Introduced herself, she is a colleague of @katjaweigel and @bettina-gier and will be writing diagnostics scripts related to her research
• @sloosvel Worked on time ranges related pull request Generalise clip_timerange to custom time periods. ESMValCore#1214 and now fixing extract_levels and statistics preprocessor to preserve fx variables fx_data not preserved as 'cell_measures' after iris aggregated_by and 'extract_levels' processor ESMValCore#1189
• @schlunma Bug fixes and testing for release, working on new diagnostics together with @hb326
• @thomascrocker Working on EUCP hourly convection permitting runs, will work in the LAUNCHPAD project with Oxford University and African climate scientists to implement diagnostics for African regions in the ESMValTool over the next few months
• @ledm Working on IPCC analyses with ESMValTool
• @bouweandela is working on improved automatic data downloads from ESGF for CMIP3, CMIP5, CMIP6, CORDEX, and obs4MIPs data in Add an option to download missing data from ESGF ESMValCore#1217

Discussion topics

2-factor authentication

In the interest of security, we would like to ask all members of the development team to enable 2-factor authentication for their GitHub accounts. Most people in the meeting are familiar with 2-factor authentication and would not consider it a problem to enable this for their GitHub accounts. That means we can ask our contributors to enable this.

@ESMValGroup/esmvaltool-developmentteam Please enable 2-factor authentication for your GitHub account. Instructions are available here. If you have any trouble, please comment on the issue and we'll try to help.

Next workshop

Workshop later this year, maybe October or November, hopefully in-person but probably online

Testing recipes

• @bouweandela For a release we need to test all recipes as they are, because there is so much variety in the input data that testing with a subset does not guarantee that the complete recipe works. A cylc suite for running all recipes is available: https://docs.esmvaltool.org/en/latest/utils.html#run-all-recipes. This can be used to test before a release and to generate a set of reference outputs after a release.
• @bouweandela We would like to have a subset of CMIP6 models that we want most diagnostics to work with, for quick testing with the ESMValBot
• @hb326 Will organize a half-day session to dive into testing recipes in more detail. In particular, to talk about finding meaningful subsets of all the available data that are representative of the variety of calendars, grids, etc found in CMIP6.
• @thomascrocker Mentioned that the --skip-nonexistent command line flag could be used, but this is only possible for some recipes because many diagnostics do not support this, see also Add warning about usability of run command line flags ESMValCore#936 (comment)
• See issues Synthetic / Sample data needed for diagnostic tests #80, How to test diagnostic scripts #316, Automated testing of diagnostic options #1115, Populate repository with data ESMValTool_sample_data#1, for some previous discussion on this topic (and maybe more issues that I couldn't find just now)

Setting up a local data pool

@annefou asked about the options for setting up a local data pool. We have some information in our documentation about this. The common setup is to locate compute next to an ESGF node, examples of this are accessing the CEDA archive from Jasmin and accessing the DKRZ CMIP datapool from Mistral. There were a few recommendations:

• Look at the IS-ENES3 transnational access call to get access to Jasmin and Mistral to see how things work there and maybe even run your analyses there instead of setting up your own infrastructure (depending on your needs).
• Use synda or sproket to set up a local data pool
• Use the features currently under implementation in Add an option to download missing data from ESGF ESMValCore#1217 (automatically download data available on ESGF while running a recipe) and Streamline observations download #1657 (download and CMORize observational and reanalysis datasets).

If you've read all the way to the end and are missing something, please comment below!

[Peter9192]

* @schlunma Successfully defended his PhD 🥳

Congrats!! 🎉 🍻

[schlunma]

Thanks @Peter9192 😄