Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate configs based on most primitive JSON types #683

Merged
merged 16 commits into from
Jan 13, 2025
5 changes: 4 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,10 @@
html_static_path = ["static"]
html_theme = "sphinx_rtd_theme"
intersphinx_mapping = {"python": ("https://docs.python.org/3", None)}
nitpick_ignore_regex = [("py:class", r"^uwtools\..*"), ("py:class", "f90nml.Namelist")]
nitpick_ignore = [
("py:class", "Path"),
("py:class", "f90nml.Namelist"),
]
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved
numfig = True
numfig_format = {"figure": "Figure %s"}
project = "Unified Workflow Tools"
Expand Down
2 changes: 1 addition & 1 deletion docs/sections/user_guide/api/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@
:target: https://mybinder.org/v2/gh/ufs-community/uwtools/main?labpath=notebooks%2Fconfig.ipynb

.. automodule:: uwtools.api.config
:inherited-members: UserDict
:members:
:show-inheritance:
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved
26 changes: 14 additions & 12 deletions notebooks/config.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -1228,15 +1228,17 @@
"text": [
"Help on function validate in module uwtools.api.config:\n",
"\n",
"validate(schema_file: Union[pathlib.Path, str], config: Union[dict, str, uwtools.config.formats.yaml.YAMLConfig, pathlib.Path, NoneType] = None, stdin_ok: bool = False) -> bool\n",
"validate(schema_file: Union[pathlib.Path, str], config_data: Union[bool, dict, float, int, list, str, uwtools.config.formats.yaml.YAMLConfig, NoneType] = None, config_path: Union[str, pathlib.Path, NoneType] = None, stdin_ok: bool = False) -> bool\n",
" Check whether the specified config conforms to the specified JSON Schema spec.\n",
"\n",
" If no config is specified, ``stdin`` is read and will be parsed as YAML and then validated. A\n",
" ``dict`` or a YAMLConfig instance may also be provided for validation.\n",
" Specify at most one of config_data or config_path. If no config is specified, ``stdin`` is read\n",
" and will be parsed as YAML and then validated.\n",
"\n",
" :param schema_file: The JSON Schema file to use for validation.\n",
" :param config: The config to validate.\n",
" :param config_data: A config to validate.\n",
" :param config_path: A path to a file containing a config to validate.\n",
" :param stdin_ok: OK to read from ``stdin``?\n",
" :raises: TypeError if both config_* arguments specified.\n",
" :return: ``True`` if the YAML file conforms to the schema, ``False`` otherwise.\n",
"\n"
]
Expand Down Expand Up @@ -1322,7 +1324,7 @@
"id": "8c61a2d2-473c-45c6-9c6c-6c07fc5bf940",
"metadata": {},
"source": [
"The schema file and config from above are passed to the respective `schema_file` and `config` parameters. Config file paths should be passed as a string or <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> object. Files should be of YAML format, or parseable as YAML. Alternatively, a `YAMLConfig` object or a Python `dict` can be provided. `validate()` returns `True` if the config conforms to the JSON schema, and `False` otherwise. With a logger initialized, details about any validation errors are reported.\n",
"The schema file and config from above are passed to the respective `schema_file` and `config_path` parameters. Config file paths should be passed as a string or <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> object. Files should be of YAML format, or parseable as YAML. Alternatively, a `YAMLConfig` object or a Python `dict` can be provided. `validate()` returns `True` if the config conforms to the JSON schema, and `False` otherwise. With a logger initialized, details about any validation errors are reported.\n",
"<!--cell 72-->"
]
},
Expand Down Expand Up @@ -1353,7 +1355,7 @@
"source": [
"config.validate(\n",
" schema_file='fixtures/config/validate.jsonschema',\n",
" config='fixtures/config/get-config.yaml'\n",
" config_path='fixtures/config/get-config.yaml'\n",
")"
]
},
Expand All @@ -1362,7 +1364,7 @@
"id": "8d151205-4a95-4b46-aa1d-30ec30d96e88",
"metadata": {},
"source": [
"The `config` argument also accepts a dictionary. In the next example, validation errors exist, and the logger reports the number of errors found along with their locations and details.\n",
"A mutually-exclusive alternative to the `config_path` argument, the `config_data` argument commonly accepts a `dict` object, but can also validate configs based on `bool`, `float`, `int`, `list`, or `str` values. In the next example, validation errors exist, and the logger reports the number of errors found along with their locations and details.\n",
"<!--cell 74-->"
]
},
Expand Down Expand Up @@ -1395,7 +1397,7 @@
"source": [
"config.validate(\n",
" schema_file='fixtures/config/validate.jsonschema',\n",
" config={'greeting':'Hello', 'recipient':47}\n",
" config_data={'greeting':'Hello', 'recipient':47}\n",
")"
]
},
Expand Down Expand Up @@ -1470,10 +1472,10 @@
" | ----------------------------------------------------------------------\n",
" | Methods inherited from uwtools.config.formats.base.Config:\n",
" |\n",
" | __repr__(self) -> str\n",
" | __repr__(self) -> 'str'\n",
" | Return the string representation of a Config object.\n",
" |\n",
" | compare_config(self, dict1: dict, dict2: Optional[dict] = None, header: Optional[bool] = True) -> bool\n",
" | compare_config(self, dict1: 'dict', dict2: 'Optional[dict]' = None, header: 'Optional[bool]' = True) -> 'bool'\n",
" | Compare two config dictionaries.\n",
" |\n",
" | Assumes a section/key/value structure.\n",
Expand All @@ -1482,10 +1484,10 @@
" | :param dict2: The second dictionary (default: this config).\n",
" | :return: True if the configs are identical, False otherwise.\n",
" |\n",
" | dereference(self, context: Optional[dict] = None) -> None\n",
" | dereference(self, context: 'Optional[dict]' = None) -> 'Config'\n",
" | Render as much Jinja2 syntax as possible.\n",
" |\n",
" | update_from(self, src: Union[dict, collections.UserDict]) -> None\n",
" | update_from(self, src: 'Union[dict, UserDict]') -> 'None'\n",
" | Update a config.\n",
" |\n",
" | :param src: The dictionary with new data to use.\n",
Expand Down
4 changes: 2 additions & 2 deletions notebooks/template.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
"text": [
"Help on function render in module uwtools.api.template:\n",
"\n",
"render(values_src: Union[dict, str, pathlib.Path, NoneType] = None, values_format: Optional[str] = None, input_file: Union[str, pathlib.Path, NoneType] = None, output_file: Union[str, pathlib.Path, NoneType] = None, overrides: Optional[dict[str, str]] = None, env: bool = False, searchpath: Optional[list[str]] = None, values_needed: bool = False, dry_run: bool = False, stdin_ok: bool = False) -> str\n",
"render(values_src: Union[dict, pathlib.Path, str, NoneType] = None, values_format: Optional[str] = None, input_file: Union[str, pathlib.Path, NoneType] = None, output_file: Union[str, pathlib.Path, NoneType] = None, overrides: Optional[dict[str, str]] = None, env: bool = False, searchpath: Optional[list[str]] = None, values_needed: bool = False, dry_run: bool = False, stdin_ok: bool = False) -> str\n",
" Render a Jinja2 template to a file, based on specified values.\n",
"\n",
" Primary values used to render the template are taken from the specified file. The format of the\n",
Expand Down Expand Up @@ -314,7 +314,7 @@
"text": [
"Help on function render_to_str in module uwtools.api.template:\n",
"\n",
"render_to_str(values_src: Union[dict, str, pathlib.Path, NoneType] = None, values_format: Optional[str] = None, input_file: Union[str, pathlib.Path, NoneType] = None, overrides: Optional[dict[str, str]] = None, env: bool = False, searchpath: Optional[list[str]] = None, values_needed: bool = False, dry_run: bool = False) -> str\n",
"render_to_str(values_src: Union[dict, pathlib.Path, str, NoneType] = None, values_format: Optional[str] = None, input_file: Union[str, pathlib.Path, NoneType] = None, overrides: Optional[dict[str, str]] = None, env: bool = False, searchpath: Optional[list[str]] = None, values_needed: bool = False, dry_run: bool = False) -> str\n",
" Render a Jinja2 template to a string, based on specified values.\n",
"\n",
" See ``render()`` for details on arguments, etc.\n",
Expand Down
2 changes: 1 addition & 1 deletion recipe/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,5 @@
"requests =2.32.*"
]
},
"version": "2.5.0"
"version": "2.6.0"
}
30 changes: 20 additions & 10 deletions src/uwtools/api/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from pathlib import Path
from typing import Optional, Union

from uwtools.config.formats.base import Config as _Config
from uwtools.config.formats.base import Config
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved
from uwtools.config.formats.fieldtable import FieldTableConfig
from uwtools.config.formats.ini import INIConfig
from uwtools.config.formats.nml import NMLConfig
Expand All @@ -15,6 +15,8 @@
from uwtools.config.support import YAMLKey
from uwtools.config.tools import compare_configs as _compare
from uwtools.config.tools import realize_config as _realize
from uwtools.config.validator import ConfigDataT, ConfigPathT
from uwtools.config.validator import validate_check_config as _validate_check_config
from uwtools.config.validator import validate_external as _validate_external
from uwtools.exceptions import UWConfigError
from uwtools.utils.api import ensure_data_source as _ensure_data_source
Expand Down Expand Up @@ -111,9 +113,9 @@ def get_yaml_config(


def realize(
input_config: Optional[Union[_Config, Path, dict, str]] = None,
input_config: Optional[Union[Config, Path, dict, str]] = None,
input_format: Optional[str] = None,
update_config: Optional[Union[_Config, Path, dict, str]] = None,
update_config: Optional[Union[Config, Path, dict, str]] = None,
update_format: Optional[str] = None,
output_file: Optional[Union[Path, str]] = None,
output_format: Optional[str] = None,
Expand Down Expand Up @@ -143,9 +145,9 @@ def realize(


def realize_to_dict( # pylint: disable=unused-argument
input_config: Optional[Union[dict, _Config, Path, str]] = None,
input_config: Optional[Union[dict, Config, Path, str]] = None,
input_format: Optional[str] = None,
update_config: Optional[Union[dict, _Config, Path, str]] = None,
update_config: Optional[Union[dict, Config, Path, str]] = None,
update_format: Optional[str] = None,
key_path: Optional[list[YAMLKey]] = None,
values_needed: bool = False,
Expand All @@ -163,25 +165,32 @@ def realize_to_dict( # pylint: disable=unused-argument

def validate(
schema_file: Union[Path, str],
config: Optional[Union[dict, YAMLConfig, Path, str]] = None,
config_data: Optional[ConfigDataT] = None,
config_path: Optional[ConfigPathT] = None,
stdin_ok: bool = False,
) -> bool:
"""
Check whether the specified config conforms to the specified JSON Schema spec.

If no config is specified, ``stdin`` is read and will be parsed as YAML and then validated. A
``dict`` or a YAMLConfig instance may also be provided for validation.
Specify at most one of config_data or config_path. If no config is specified, ``stdin`` is read
and will be parsed as YAML and then validated.

:param schema_file: The JSON Schema file to use for validation.
:param config: The config to validate.
:param config_data: A config to validate.
:param config_path: A path to a file containing a config to validate.
:param stdin_ok: OK to read from ``stdin``?
:raises: TypeError if both config_* arguments specified.
:return: ``True`` if the YAML file conforms to the schema, ``False`` otherwise.
"""
_validate_check_config(config_data, config_path)
if config_data is None:
config_path = _ensure_data_source(_str2path(config_path), stdin_ok)
try:
_validate_external(
schema_file=_str2path(schema_file),
desc="config",
config=_ensure_data_source(_str2path(config), stdin_ok),
config_data=config_data,
config_path=config_path,
)
except UWConfigError:
return False
Expand Down Expand Up @@ -252,6 +261,7 @@ def validate(
).strip()

__all__ = [
"Config",
"FieldTableConfig",
"INIConfig",
"NMLConfig",
Expand Down
2 changes: 1 addition & 1 deletion src/uwtools/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ def _dispatch_config_validate(args: Args) -> bool:
"""
return uwtools.api.config.validate(
schema_file=args[STR.schemafile],
config=args[STR.infile],
config_path=args[STR.infile],
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved
stdin_ok=True,
)

Expand Down
10 changes: 8 additions & 2 deletions src/uwtools/config/formats/base.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
from __future__ import annotations

import difflib
import os
import re
Expand All @@ -23,14 +25,17 @@ class Config(ABC, UserDict):
several configuration-file formats.
"""

def __init__(self, config: Optional[Union[dict, str, Path]] = None) -> None:
def __init__(self, config: Optional[Union[dict, str, Config, Path]] = None) -> None:
"""
:param config: Config file to load (None => read from stdin), or initial dict.
"""
super().__init__()
if isinstance(config, dict):
self._config_file = None
self.update(config)
elif isinstance(config, Config):
self._config_file = config._config_file
self.update(config.data)
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved
else:
self._config_file = str2path(config) if config else None
self.data = self._load(self._config_file)
Expand Down Expand Up @@ -216,7 +221,7 @@ def config_file(self) -> Optional[Path]:
"""
return self._config_file

def dereference(self, context: Optional[dict] = None) -> None:
def dereference(self, context: Optional[dict] = None) -> Config:
"""
Render as much Jinja2 syntax as possible.
"""
Expand All @@ -234,6 +239,7 @@ def logstate(state: str) -> None:
break
self.data = new
logstate("final")
return self
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved

@abstractmethod
def dump(self, path: Optional[Path]) -> None:
Expand Down
Loading
Loading