From 5410d8436eeaac116c5bd28153accfcdf6fc9428 Mon Sep 17 00:00:00 2001 From: Sander Van Balen Date: Thu, 7 Dec 2023 15:58:33 +0100 Subject: [PATCH 1/6] added unknown docs --- changelogs/unreleased/6056-docs-unknowns.yml | 8 ++ docs/glossary.rst | 4 + docs/language.rst | 73 +++++++++++++++++++ .../platform_developers/modelexportformat.rst | 21 +++--- 4 files changed, 95 insertions(+), 11 deletions(-) create mode 100644 changelogs/unreleased/6056-docs-unknowns.yml diff --git a/changelogs/unreleased/6056-docs-unknowns.yml b/changelogs/unreleased/6056-docs-unknowns.yml new file mode 100644 index 0000000000..a1c6ecaa90 --- /dev/null +++ b/changelogs/unreleased/6056-docs-unknowns.yml @@ -0,0 +1,8 @@ +description: Added documentation about unknowns +change-type: patch +sections: + minor-improvement: "{{description}}" +issue-nr: 6056 +destination-branches: + - master + - iso6 diff --git a/docs/glossary.rst b/docs/glossary.rst index a39b84e018..dbd5d5abcd 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -102,6 +102,10 @@ Glossary a cloud provider will not be known upfront. Inmanta marks this parameters as **unknown**. The state of any resource that uses such an unknown parameter becomes undefined. + For more context, see + :ref:`how unknowns propagate through the configuration model ` and + :ref:`how the exporter deals with them `. + entity Concepts in the infrastructure are modelled in the configuration with entities. An entity defines a new type in the configuration model. See :ref:`lang-entity`. diff --git a/docs/language.rst b/docs/language.rst index fc027cf3f9..77d66a2a77 100644 --- a/docs/language.rst +++ b/docs/language.rst @@ -864,3 +864,76 @@ Plug-ins For more complex operations, python plugins can be used. Plugins are exposed in the Inmanta language as function calls, such as the template function call. A template accepts parameters and returns a value that it computed out of the variables. Each module that is included can also provide plug-ins. These plug-ins are accessible within the namespace of the module. The :ref:`module-plugins` section of the module guide provides more details about how to write a plugin. + + +.. _language_unknowns: +Unknowns +======== + +Wherever the configuration model interacts with the outside world (e.g. to fetch external values) :term:`unknown` values +may be present. These unknowns are propagated through the model to finally end up in the resources that require these unknown +values. This section describes how unknown values flow through the model, and perhaps equally importantly, where they do not +flow at all. + +.. note:: + Unknowns are a subtle concept. Luckily, for the majority of model development you don't really need to take them into + account. However, for some advanced scenarios it may be important to know how and where they may occur. + +For the most part, unknowns are simply propagated along the data flow: statements like assignment statements, constructors +and lists simply include the unknown in their result like they would any other value. Any expression that can not produce +a definite result without knowing the value, will return another unknown. And finally, statements that expand the model +with new blocks based on some value, like the if statement and the for loop, simply do not expand the model with their +respective blocks for unknowns. + +The model below presents some examples of how an unknown propagates. + +.. code-block:: inmanta + + # std::env returns an unknown if the environment variable is not (yet) set + my_unknown = std::get_env("THIS_ENV_VAR_DOES_NOT_EXIST") + + a = my_unknown # a is unknown + b = [1, 2, my_unknown, 3] # b is a list with 1 unknown element + c = my_unknown is defined # we can not know if c is null, so c is also unknown + d = true or my_unknown # trivial, value of my_unknown is irrelevant -> d is true + e = my_unknown or true # lazy boolean operator can not compute result without knowing the value -> e is unknown + f = (e == my_unknown) # both e and my_unknown are unknown but they aren't necessarily the same value -> f is unknown + + if my_unknown: + # this block is never executed + else: + # neither is this one + end + + for x in my_unknown: + # neither is this one + end + + for x in [1, 2, my_unknown]: + # this block is executed twice: x=1 and x=2 + end + + g = my_unknown ? true : false # condition is unknown -> neither branch is executed, result is unknown + h = [1 for x in [1, 2, my_unknown]] # the expression `1` is executed once with x=1 and once with x=2. Unknown is propagated as is -> h = [1, 1, unknown] + i = [1 for x in [1, 2, my_unknown] if not std::is_unknown(x)] # the unknown is filtered out -> i = [1, 1] + +Now that we've covered how unknowns flow through the model, we can discuss what an unknown value actually means. In most cases +it simply represents an unknown value. But because of the propagation semantics outlined above, if it happens to occur in a +list, it may in fact represent any number of values: not only the value is unknown, also its size. + +For example, consider a list comprehension that filters a list on some condition. If the list contains an unknown, the compiler +can not know if the filter applies so it will propagate the unknown to the result. When the unknown eventually becomes known, +it might remain in the result, or it might be filtered out, depending on whether it matches the condition. + +.. code-block:: inmanta + + my_unknown = std::get_env("THIS_ENV_VAR_DOES_NOT_EXIST") + my_unknown2 = std::get_env("THIS_ENV_VAR_DOES_NOT_EXIST2") + + l = [1, my_unknown, 3, my_unknown2, 5] + a = [x for x in l if x > 2] # l = [unknown, 3, unknown, 5] + + # an unknown can even represent more than one unknown value + b = my_unknown == 0 ? [1, 2] : [3, 4] # b = unknown -> when it becomes known it will be either [1, 2] or [3, 4] + + c = std::len(l) # c = unknown (l contains unknowns, so its length is also unknown) diff --git a/docs/platform_developers/modelexportformat.rst b/docs/platform_developers/modelexportformat.rst index 0e4925f200..492ef16105 100644 --- a/docs/platform_developers/modelexportformat.rst +++ b/docs/platform_developers/modelexportformat.rst @@ -1,20 +1,19 @@ +.. _model_export_format: Model Export Format ======================== - - #. top level is a dict with one entry for each instance in the model #. the key in this dict is the object reference handle #. the value is the serialized instance #. the serialized instance is a dict with three fields: type, attributes and relation. #. type is the fully qualified name of the type #. attributes is a dict, with as keys the names of the attributes and as values a dict with one entry. -#. An attribute can have one or more of tree keys: unknows, nones and values. The "values" entry has as value a list with the attribute values. - If any of the values is Unknown or None, it is removed from the values array and the index at which it was removed is recorded in respective the unknowns or nones value -#. relations is like attributes, but the list of values contains the reference handles to which this relations points +#. An attribute can have one or more of tree keys: unknows, nones and values. The "values" entry has as value a list with the attribute values. + If any of the values is :term:`unknown` or None, it is removed from the values array and the index at which it was removed is recorded in respective the unknowns or nones value +#. relations is like attributes, but the list of values contains the reference handles to which this relations points -Basic structure as pseudo jinja template +Basic structure as pseudo jinja template .. code-block:: js+jinja @@ -22,23 +21,23 @@ Basic structure as pseudo jinja template {% for instance in instances %} '{{instance.handle}}':{ "type":"{{instance.type.fqn}}", - "attributes":[ + "attributes":[ {% for attribute in instance.attributes %} "{{attribute.name}}": [ {{ attribute.values | join(",") }} ] {% endfor %} ] "relations" : [ {% for relation in instance.relations %} - "{{relation.name}}": [ + "{{relation.name}}": [ {% for value in relation.values %} {{value.handle}} {% endfor %} ] {% endfor %} ] - + {% endif %} - } + } Type Export Format ======================== @@ -46,4 +45,4 @@ Type Export Format .. automodule:: inmanta.model :members: :private-members: - + From 6f5dcd86be5106597632d969f0f6df2c9e2a41ec Mon Sep 17 00:00:00 2001 From: Sander Van Balen Date: Thu, 16 Jan 2025 14:21:34 +0100 Subject: [PATCH 2/6] Apply suggestions from code review Co-authored-by: Wouter De Borger --- docs/language.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/language.rst b/docs/language.rst index 312aa145ea..b381d66400 100644 --- a/docs/language.rst +++ b/docs/language.rst @@ -930,7 +930,7 @@ Unknowns ======== Wherever the configuration model interacts with the outside world (e.g. to fetch external values) :term:`unknown` values -may be present. These unknowns are propagated through the model to finally end up in the resources that require these unknown +may be present. These unknowns represent values we don't know yet, such as the IP address of a machine that hasn't been created yet. Because the compiler can handle unknowns, developers can write models as if all information is present up front, even when this is not the case. These unknowns are propagated through the model to finally end up in the resources that require these unknowns. values. This section describes how unknown values flow through the model, and perhaps equally importantly, where they do not flow at all. @@ -954,7 +954,7 @@ The model below presents some examples of how an unknown propagates. a = my_unknown # a is unknown b = [1, 2, my_unknown, 3] # b is a list with 1 unknown element c = my_unknown is defined # we can not know if c is null, so c is also unknown - d = true or my_unknown # trivial, value of my_unknown is irrelevant -> d is true + d = true or my_unknown # value of my_unknown is irrelevant -> d is true e = my_unknown or true # lazy boolean operator can not compute result without knowing the value -> e is unknown f = (e == my_unknown) # both e and my_unknown are unknown but they aren't necessarily the same value -> f is unknown From 35a58d31cbdbd4dcf1ef8b32ceacc6e3de455314 Mon Sep 17 00:00:00 2001 From: Sander Van Balen Date: Thu, 16 Jan 2025 14:21:49 +0100 Subject: [PATCH 3/6] updated destination branches --- changelogs/unreleased/6056-docs-unknowns.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/changelogs/unreleased/6056-docs-unknowns.yml b/changelogs/unreleased/6056-docs-unknowns.yml index a1c6ecaa90..06ed9adcd7 100644 --- a/changelogs/unreleased/6056-docs-unknowns.yml +++ b/changelogs/unreleased/6056-docs-unknowns.yml @@ -5,4 +5,5 @@ sections: issue-nr: 6056 destination-branches: - master - - iso6 + - iso8 + - iso7 From cb612fac51df628e6de0f3eec3d46a436da18842 Mon Sep 17 00:00:00 2001 From: Sander Van Balen Date: Thu, 16 Jan 2025 14:35:36 +0100 Subject: [PATCH 4/6] review comment --- docs/language.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/language.rst b/docs/language.rst index b381d66400..7dba629ad3 100644 --- a/docs/language.rst +++ b/docs/language.rst @@ -938,7 +938,9 @@ flow at all. Unknowns are a subtle concept. Luckily, for the majority of model development you don't really need to take them into account. However, for some advanced scenarios it may be important to know how and where they may occur. -For the most part, unknowns are simply propagated along the data flow: statements like assignment statements, constructors +For the most part, unknowns are simply propagated along the data flow: they're treated like any other value, except +when anything needs to be derived from them, the result simply becomes an unknown as well. More specifically: statements like +assignment statements, constructors and lists simply include the unknown in their result like they would any other value. Any expression that can not produce a definite result without knowing the value, will return another unknown. And finally, statements that expand the model with new blocks based on some value, like the if statement and the for loop, simply do not expand the model with their From 7cd7196dc55878be83fff918d04c5690e6db88d2 Mon Sep 17 00:00:00 2001 From: Sander Van Balen Date: Thu, 16 Jan 2025 14:51:16 +0100 Subject: [PATCH 5/6] updated snippet --- docs/language.rst | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/language.rst b/docs/language.rst index 7dba629ad3..c22bf5d3fa 100644 --- a/docs/language.rst +++ b/docs/language.rst @@ -962,21 +962,31 @@ The model below presents some examples of how an unknown propagates. if my_unknown: # this block is never executed + std::print("This message is never printed!") else: # neither is this one + std::print("This message is never printed!") end for x in my_unknown: # neither is this one + std::print("This message is never printed!") end for x in [1, 2, my_unknown]: # this block is executed twice: x=1 and x=2 + std::print(f"This message is printed twice! x={x}") end g = my_unknown ? true : false # condition is unknown -> neither branch is executed, result is unknown - h = [1 for x in [1, 2, my_unknown]] # the expression `1` is executed once with x=1 and once with x=2. Unknown is propagated as is -> h = [1, 1, unknown] - i = [1 for x in [1, 2, my_unknown] if not std::is_unknown(x)] # the unknown is filtered out -> i = [1, 1] + + entity E: + int n + end + implement E using std::none + + h = [E(n=x) for x in [1, 2, my_unknown]] # the constructor is executed once with n=1 and once with n=2. Unknown is propagated as is -> h = [E(n=1), E(n=2), unknown] + i = [E(n=x) for x in [1, 2, my_unknown] if not std::is_unknown(x)] # the unknown is filtered out -> i = [E(n=1), E(n=2)] Now that we've covered how unknowns flow through the model, we can discuss what an unknown value actually means. In most cases it simply represents an unknown value. But because of the propagation semantics outlined above, if it happens to occur in a From 73639d1e186b889a577c03d8b8fbbe8908fc88c0 Mon Sep 17 00:00:00 2001 From: Sander Van Balen Date: Thu, 16 Jan 2025 15:18:36 +0100 Subject: [PATCH 6/6] review comment --- docs/language.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/language.rst b/docs/language.rst index c22bf5d3fa..875d002fbc 100644 --- a/docs/language.rst +++ b/docs/language.rst @@ -1006,5 +1006,7 @@ it might remain in the result, or it might be filtered out, depending on whether # an unknown can even represent more than one unknown value b = my_unknown == 0 ? [1, 2] : [3, 4] # b = unknown -> when it becomes known it will be either [1, 2] or [3, 4] + # or none at all + c = [x for x in l if x > 1000] # c = [unknown, unknown] -> would become [] if the env var values are <= 1000 - c = std::len(l) # c = unknown (l contains unknowns, so its length is also unknown) + d = std::len(l) # d = unknown (l contains unknowns, so its length is also unknown)