CLI - Kedro Catalog Create fails (so does Catalog list) if a dataset is defined with a `${runtime_params:<xxx>}` resolver exists, and passing --params is not permitted #4173

gitgud5000 · 2024-09-17T22:07:37Z

Description

When creating or listing a catalog in Kedro that includes a dataset with a ${runtime_params:<xxx>} resolver, it fails when --params is not provided. This results in an InterpolationResolutionError indicating that the runtime parameter <xxx> is not found and no default value is provided.

Context

This bug prevents the successful creation or listing of the catalog when the entry in the existing catalog depends on a runtime parameter,.

Steps to Reproduce

1 - Define a dataset in catalog.yml using ${runtime_params:} resolver.
2 - Try to use kedro catalog create/list.

Expected Result

Allow users to pass --params during the catalog creation or listing process to provide runtime parameters, resolving the issue and avoiding the InterpolationResolutionError.

Actual Result

Traceback (most recent call last):
  File "/anaconda/envs/reprecios/bin/kedro", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 269, in main
    cli_collection()
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 162, in main
    super().main(
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/kedro/framework/cli/catalog.py", line 181, in create_catalog
    for ds_name in context.catalog._datasets.keys()
                   ^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/kedro/framework/context/context.py", line 190, in catalog
    return self._get_catalog()
           ^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/kedro/framework/context/context.py", line 226, in _get_catalog
    conf_catalog = self.config_loader["catalog"]
                   ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/kedro/config/omegaconf_config.py", line 208, in __getitem__
    base_config = self.load_and_merge_dir_config(  # type: ignore[no-untyped-call]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/kedro/config/omegaconf_config.py", line 355, in load_and_merge_dir_config
    for k, v in OmegaConf.to_container(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 573, in to_container
    return BaseContainer._to_content(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 292, in _to_content
    value = get_node_value(key)
            ^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 247, in get_node_value
    value = BaseContainer._to_content(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 292, in _to_content
    value = get_node_value(key)
            ^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 244, in get_node_value
    conf._format_and_raise(key=key, value=None, cause=e)
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 242, in get_node_value
    node = node._dereference_node()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/base.py", line 246, in _dereference_node
    node = self._dereference_node_impl(throw_on_resolution_failure=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/base.py", line 277, in _dereference_node_impl
    return parent._resolve_interpolation_from_parse_tree(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/base.py", line 584, in _resolve_interpolation_from_parse_tree
    resolved = self.resolve_parse_tree(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/base.py", line 764, in resolve_parse_tree
    return visitor.visit(parse_tree)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
    return visitor.visitConfigValue(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
    return self.visit(ctx.getChild(0))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
    return visitor.visitText(self)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 301, in visitText
    return self._unescape(list(ctx.getChildren()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 389, in _unescape
    text = str(self.visitInterpolation(node))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
    return self.visit(ctx.getChild(0))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1041, in accept
    return visitor.visitInterpolationResolver(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 179, in visitInterpolationResolver
    return self.resolver_interpolation_callback(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/base.py", line 750, in resolver_interpolation_callback
    return self._evaluate_custom_resolver(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/base.py", line 694, in _evaluate_custom_resolver
    return resolver(
           ^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 445, in resolver_wrapper
    ret = resolver(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/reprecios/lib/python3.11/site-packages/kedro/config/omegaconf_config.py", line 411, in _get_runtime_value
    raise InterpolationResolutionError(
omegaconf.errors.InterpolationResolutionError: Runtime parameter 'run_date' not found and no default value provided.
    full_key: {namespace}.model.filepath
    object_type=dict

Your Environment

Kedro: 0.19.8
Python: 3.11.9
OS: Ubuntu 20.04.6 LTS

The text was updated successfully, but these errors were encountered:

merelcht · 2024-09-18T14:30:43Z

Hi @gitgud5000 , thanks for flagging this issue. Can I ask why you wouldn't just provide a default value in that case? I would assume you don't know yet what runtime parameters to use when creating the catalog.

gitgud5000 · 2024-09-18T15:21:40Z

@merelcht I ended up doing exactly that, setting a default for all the runtime_params—it just required a bit of catalog surgery.

It would be nice, though, if we could pass these parameters as args, so I suppose convenience is the ultimate ergonomic feature 😅.

merelcht · 2024-09-18T16:00:24Z

Okay that makes sense! I'll leave the issue open to get a gauge of how many users would want this feature.

astrojuanlu · 2024-11-08T18:32:45Z

Opened an enhancement proposal at #4310, I'm not 100 % I captured it correctly so please feel free to chime in. In the meantime, I'm closing this as resolved!

merelcht added the Community label Sep 18, 2024

merelcht added this to Kedro Wizard 🪄 Oct 31, 2024

astrojuanlu closed this as completed Nov 8, 2024

github-project-automation bot moved this to Done in Kedro Wizard 🪄 Nov 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLI - Kedro Catalog Create fails (so does Catalog list) if a dataset is defined with a `${runtime_params:<xxx>}` resolver exists, and passing --params is not permitted #4173

CLI - Kedro Catalog Create fails (so does Catalog list) if a dataset is defined with a `${runtime_params:<xxx>}` resolver exists, and passing --params is not permitted #4173

gitgud5000 commented Sep 17, 2024

merelcht commented Sep 18, 2024

gitgud5000 commented Sep 18, 2024

merelcht commented Sep 18, 2024

astrojuanlu commented Nov 8, 2024

CLI - Kedro Catalog Create fails (so does Catalog list) if a dataset is defined with a ${runtime_params:<xxx>} resolver exists, and passing --params is not permitted #4173

CLI - Kedro Catalog Create fails (so does Catalog list) if a dataset is defined with a ${runtime_params:<xxx>} resolver exists, and passing --params is not permitted #4173

Comments

gitgud5000 commented Sep 17, 2024

Description

Context

Steps to Reproduce

Expected Result

Actual Result

Your Environment

merelcht commented Sep 18, 2024

gitgud5000 commented Sep 18, 2024

merelcht commented Sep 18, 2024

astrojuanlu commented Nov 8, 2024

CLI - Kedro Catalog Create fails (so does Catalog list) if a dataset is defined with a `${runtime_params:<xxx>}` resolver exists, and passing --params is not permitted #4173

CLI - Kedro Catalog Create fails (so does Catalog list) if a dataset is defined with a `${runtime_params:<xxx>}` resolver exists, and passing --params is not permitted #4173