Update develop-ref after #2872 #2879
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…ed for tests can be easily overridden to test changes in a MET PR before merging
* per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface
… and conda env image
…ge is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none>
… docker image pruning step
* new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings
…if that resolves the disk space issues
* changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3
… updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update
This reverts commit e7dffb6.
This reverts commit c1cb3c4.
…anch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d.
…nder a category
…the RTD web interface
…TC web server to be done for the rc1 release instead of after the official release
* resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates
* Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file.
…' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting
…nd/or output formats.
…w reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca.
…he develop branch content.
* update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#2872 adds PairStat basic use case output and changes the name of the file_list files for a few use cases (see PR description for complete list)
Created by @georgemccabe