Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Wrapper: PairStat #2781

Closed
9 of 23 tasks
JohnHalleyGotway opened this issue Nov 12, 2024 · 3 comments · Fixed by #2872
Closed
9 of 23 tasks

New Wrapper: PairStat #2781

JohnHalleyGotway opened this issue Nov 12, 2024 · 3 comments · Fixed by #2872
Assignees
Labels
alert: NEED MORE DEFINITION Not yet actionable, additional definition required component: python wrapper priority: high High Priority requestor: Navy/NRL Naval Research Laboratory type: new feature Make it do something new
Milestone

Comments

@JohnHalleyGotway
Copy link
Collaborator

JohnHalleyGotway commented Nov 12, 2024

Describe the New Feature

Issue dtcenter/MET#3006 adds a new tool named Pair-Stat to read already paired forecast/observation data from input files, filter as requested, and compute/write ASCII output statistics. This logic is pretty similar to the existing Point-Stat tool, however we'll only have a single input data source rather than separate ones for forecasts and observations. The initial implementation will support two data source types:

  • ASCII MPR data generated by Point-Stat or provided through Python-embedding
  • IODA pairs

The specifics of the usage statement and configuration file contents are still under development, but will be noted here once they are finalized.

Pair-Stat Usage:
TDB

Pair-Stat Configuration File:
TBD

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

Approx 1 week?

Sub-Issues

Consider breaking the new feature down into sub-issues.
None needed.

Relevant Deadlines

Do not charge 7730022 past 12/30/2024.

Funding Source

7730022 NRL METplus

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required: @georgemccabe will implement
  • Select scientist(s) or no scientist required: @DanielAdriaansen will advise.

Labels

  • Review default alert labels
  • Select component(s)
  • Select priority
  • Select requestor(s)

Milestone and Projects

  • Select Milestone as a METplus-Wrappers-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
  • For a METplus-Wrappers-X.Y.Z version, select the METplus-Wrappers-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

New Feature Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • For any new datasets, an entry to the METplus Verification Datasets Guide.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issue
    Select: Milestone as the next official version
    Select: METplus-Wrappers-X.Y.Z Development project for development toward the next official release
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added component: python wrapper priority: high High Priority type: new feature Make it do something new requestor: Navy/NRL Naval Research Laboratory alert: NEED MORE DEFINITION Not yet actionable, additional definition required labels Nov 12, 2024
@JohnHalleyGotway JohnHalleyGotway added this to the METplus-6.1.0 milestone Nov 12, 2024
@github-project-automation github-project-automation bot moved this to 🩺 Needs Triage in METplus-6.1.0 Development Nov 12, 2024
@DanielAdriaansen DanielAdriaansen moved this from 🩺 Needs Triage to 🛑 Not Ready in METplus-6.1.0 Development Nov 15, 2024
@DanielAdriaansen
Copy link
Contributor

At the same time this work is done for the new wrapper, we should develop a corresponding met_tool_wrapper use case to demonstrate it's functionality. Maybe that's a separate issue but @JohnHalleyGotway mentioned that @georgemccabe may have off-the-shelf scripts or code to set up a basic use case to demonstrate the tool, so it may be contained within this issue. @georgemccabe please advise if you feel strongly about a separate issue. Thanks!

@georgemccabe georgemccabe changed the title Create new PairStat wrapper to configure and run the MET Pair-Stat tool New Wrapper: PairStat Nov 15, 2024
@georgemccabe
Copy link
Collaborator

Part of the process of adding a new wrapper is to include a met_tool_wrapper basic use case to demonstrate its use.

@JohnHalleyGotway
Copy link
Collaborator Author

Current usage statement as of 12/16/24:

*** Model Evaluation Tools (METV12.1.0) ***

Usage: pair_stat
	-pairs file
	-format type
	-config config_file
	[-outdir path]
	[-log file]
	[-v level]

	where	"-pairs file" is one or more files containing forecast/observation pairs. May be used multiple times (required).
		"-format type" defines the input pairs file format and may be set to "mpr" or "ioda" (required).
		"-config config_file" is a PairStatConfig file containing the desired configuration settings (required).
		"-outdir path" overrides the default output directory (.) (optional).
		"-log file" outputs log messages to the specified file (optional).
		"-v level" overrides the default level of logging (2) (optional).

Recommend that we change to -pairs file_list to support file list input to handle a very long list of inputs.

@georgemccabe georgemccabe moved this from 🛑 Not Ready to 🏗 In progress in METplus-6.1.0 Development Dec 16, 2024
georgemccabe added a commit that referenced this issue Dec 27, 2024
@georgemccabe georgemccabe linked a pull request Jan 23, 2025 that will close this issue
16 tasks
@georgemccabe georgemccabe moved this from 🏗 In progress to 🔎 In review in METplus-6.1.0 Development Jan 28, 2025
georgemccabe added a commit that referenced this issue Jan 28, 2025
* update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release

* cleanup

* per #2781, start first implementation of PairStat wrapper

* appease SonarQube by increasing code coverage for component versions script

* add new wrapper to lookup dictionary

* per #2781 add wrapped MET config file

* use runtime freq logic to find input files

* start unit tests (broken)

* reorder assert to align expected and actual values with PyCharm notation

* update unit tests

* rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command

* reorder assert to align actual and expected in PyCharm

* refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code

* change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this.

* remove line that is not needed

* fix bug introduced with recent changes and run use cases that failed to test that the fix is correct

* fix multi-variate MODE run to pass all fields to the call instead of just the first one

* update SeriesAnalysis test to use logic that is actually used when running

* turn off use cases that now succeed and turn on use case to test fix to multi-variate mode

* turn off use case after confirming that it now runs successfully

* refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports

* remove functions that are not used

* handle -pairs flag like other command line args for input files. Add documentation to new functions

* updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers

* remove unused import

* remove config variable that is no longer used in MET

* remove match_month support from climo_mean/stdev

* remove obs_valid_beg/end from command line args because it is not supported by pair_stat

* remove test config files that are not being used

* configure basic use case to run command that is being used to test/develop the app

* error if invalid tool name is provided

* add missing tests

* remove support for config variables that did not make it into the final implementation of the MET pair_stat tool

* fix formatting on a few config values

* correct input file info

* turn on use case to test

* fix command line argument for output directory that changed since the last test

* refactor and clean up to reduce SonarQube issues

* update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis

* Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function

* per feedback in PR #2872, remove support for setting output_prefix

* add PairStat to list of wrappers that use the MODEL config variable

* per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat

* remove output_prefix from tables
@github-project-automation github-project-automation bot moved this from 🔎 In review to 🏁 Done in METplus-6.1.0 Development Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alert: NEED MORE DEFINITION Not yet actionable, additional definition required component: python wrapper priority: high High Priority requestor: Navy/NRL Naval Research Laboratory type: new feature Make it do something new
Projects
Status: 🏁 Done
Development

Successfully merging a pull request may close this issue.

3 participants