Out-of-schedule ESMValTool date for developer discussion about testing strategy - DATE FIXED (Thursday Oct 14, 2pm-5pm CEST)

[hb326]

Hey all,
after we had some problems with the recipe testing lately, and after we had a brief discussion about a possible ESMValTool testing strategy at the last monthly meeting, we decided that it would be a good thing to set aside some time to be able to discuss this in detail.
So there will be a half-day out-of-schedule ESMValTool developers meeting to talk about a testing strategy. Please indicate your availability below. Thanks!

https://terminplaner4.dfn.de/KzhsW6Ruac8OtByP

@ESMValGroup/esmvaltool-developmentteam
@ESMValGroup/esmvaltool-coreteam

The doodle will be open until the first week of September.

[hb326]

Hey all @ESMValGroup/esmvaltool-developmentteam @ESMValGroup/esmvaltool-coreteam,
please don't forget to fill out the poll, so that we can set a date and time for the disucssion about the recipe testing.

Thanks!!

[hb326]

Hey all @ESMValGroup/esmvaltool-developmentteam @ESMValGroup/esmvaltool-coreteam,
we have a date and time for the out-of-schedule ESMValTool discussion about testing strategy:
Thursday Oct 14, 2pm-5pm CEST

Please mark this date in your calendars. We will send around information about the meeting details closer to the meeting date.

[hb326]

Hey all @ESMValGroup/esmvaltool-developmentteam @ESMValGroup/esmvaltool-coreteam ,
just a brief reminder that we have the ESMValTool half day meeting about the testing strategy next Thursday afternoon!
We will post the link to access the meeting here next week closer to the meeting day.

For now, you can familiarize yourself with the recipe testing problem by looking at the following issues:

• Test esmvalcore=2.3.0 with current batch of ESMValTool recipes #2198
• How to test diagnostic scripts #316
• Allow to load all files, first X years or last X years in an experiment ESMValCore#1133

If you have any ideas about the testing strategy or know about an issue that could be relevant, please post them in this issue here.
Thanks!

[bouweandela]

Some more relevant issues:

• Synthetic / Sample data needed for diagnostic tests #80
• Automated testing of diagnostic options #1115
• Populate repository with data ESMValTool_sample_data#1

@hb326 How is ESMValGroup/ESMValCore#1133 relevant for this discussion?

[remi-kazeroni]

Here was the link for the meeting taking place today, 14:00-17:00 CEST.

[remi-kazeroni]

The summary of the meeting can be found in the three issues listed below. During the meeting, we have decided to considerer three different categories of automatized recipes testing performed at different frequencies (from most to least frequent ones):

1. (#1361) run special recipes on a nightly basis to test the latest changes in the ESMValCore and check if and how these would break recipes. The development of such test recipes for the core will be discussed at the November workshop (ESMValTool workshop Nov 2021 #2311)
2. ([recipe testing strategy] Testing recipes on a weekly basis #2346) run batch of recipes in an automatized way on a weekly basis to check if these are affected by the latest changes in the Core. This would help to identify possible problems earlier than close to a release.
3. ([recipe testing strategy] Testing all recipes before a release #2345) run all recipes prior to a release in an automatized way to check whether recipes work fine with the latest versions of the Core and the Tool.

[valeriupredoi]

many thanks for the notes taken during the meeting and summary @remi-kazeroni 🍺