Merge branch 'main' into fix_ioplacer

docs/source/reference/configuration.md

@@ -1,20 +1,22 @@
 # Design Configuration Files
+
 Unless the design uses the API directly, each OpenLane-compatible design must
 come with a configuration file. These configuration files can be written in one
 of two grammars: JSON or Tcl.
 
 Tcl offers more flexibility at the detriment of security, while JSON is more
-straightforward at the cost of flexbility. While Tcl allows you to do all
-manner of computation on your variables, JSON has a limited expression engine
-that will be detailed later in this document. Nevertheless, for security (and
+straightforward at the cost of flexbility. While Tcl allows you to do all manner
+of computation on your variables, JSON has a limited expression engine that will
+be detailed later in this document. Nevertheless, for security (and
 future-proofing), we recommend you use either the JSON format or write Python
 scripts using the API.
 
-The folder containing your `config.tcl`/`config.json` is known as the
-**Design Directory**. The design directory is special in that paths in the JSON
-configuration files can be resolved relative to this directory and that in TCL
-configuration files, it can be referenced via the environment variable
-`DESIGN_DIR`. This will be explained in detail in later sections.
+The folder containing your `config.tcl`/`config.json` is known as the **Design
+Directory** -- though the design directory can also be set explicitly over the
+command-line using `--design-dir`. The design directory is special in that paths
+in the JSON configuration files can be resolved relative to this directory and
+that in TCL configuration files, it can be referenced via the environment
+variable `DESIGN_DIR`. This will be explained in detail in later sections.
 
 ```{note}
 When using the API, you can provide the inputs directly as a Python dictionary,
@@ -23,12 +25,13 @@ JSON or Tcl files. You can still use `ref::` and such like JSON files though.
 ```
 
 ## JSON
+
 The JSON files are simple key-value pairs.
 
 <a name="scalars"></a>
 
 The values can be scalars (strings, numbers, booleans, and `null`s), lists or
-dictionaries, subject to validation.  
+dictionaries, subject to validation.
 
 All files must be ECMA404-compliant, i.e., pure JSON with no extensions such as
 comments or the new elements introduced in [JSON5](https://json5.org/).
@@ -55,8 +58,8 @@ An minimal demonstrative configuration file would look as follows:
 ### Pre-processing
 
 The JSON files are pre-processed at runtime. Features include conditional
-execution, a way to reference the design directory, other variables,
-and a basic numeric expression engine.
+execution, a way to reference the design directory, other variables, and a basic
+numeric expression engine.
 
 #### Conditional Execution
 
@@ -69,11 +72,10 @@ SCL matches those in the key, i.e., for `pdk::sky130A` as shown above, this
 particular `dict` will be evaluated and its values used if and only if the PDK
 is set to `sky130A`, meanwhile with say, `asap7`, it will not be evaluated.
 
-
-
-The match is evaluated using [`fnmatch`](https://docs.python.org/3.6/library/fnmatch.html),
-giving it limited wildcard support: meaning that `pdk::sky130*` would match both
-`sky130A` and `sky130B`.
+The match is evaluated using
+[`fnmatch`](https://docs.python.org/3.6/library/fnmatch.html), giving it limited
+wildcard support: meaning that `pdk::sky130*` would match both `sky130A` and
+`sky130B`.
 
 Note that ***the order of declarations matter here***: as seen in the following
 example, despite a more specific value for a PDK existing, the unconditionally
@@ -94,14 +96,14 @@ declared value later in the code would end up overwriting it:
     }
 }
 ```
+
 > In the first example, the final value for A would always be 4 given the order
 > of declarations. In the second example, it would be 40 is the PDK is sky130A
 > and 4 otherwise.
 
-It is worth nothing that the final resolved configuration would have the
-symbol in the parent object with no trace left of the conditionally-executed,
-dict i.e., the second example with the
-sky130A PDK simply becomes:
+It is worth nothing that the final resolved configuration would have the symbol
+in the parent object with no trace left of the conditionally-executed, dict
+i.e., the second example with the sky130A PDK simply becomes:
 
 ```json
 {
@@ -128,16 +130,18 @@ reference a variable that is declared after the current expression.
     "A": "ref::$B"
 }
 ```
+
 > In this example, the first configuration is invalid, as B is referenced before
-> it is declared, but the latter is OK, where the value will be "vdd gnd" as well.
+> it is declared, but the latter is OK, where the value will be "vdd gnd" as
+> well.
 
 Do note that unlike Tcl config files, environment variables (other than
 `DESIGN_DIR`, `PDK`, `PDKPATH`, `STD_CELL_LIBRARY`) are not exposed to
 `config.json` by default.
 
 If the files you choose lie **inside** the design directory, a different prefix,
-`refg::`, supports non-recursive globs, i.e., you can use an
-asterisk as a wildcard to pick multiple files in a specific folder.
+`refg::`, supports non-recursive globs, i.e., you can use an asterisk as a
+wildcard to pick multiple files in a specific folder.
 
 * Outside the design directory, this is disabled for security reasons and the
   final path will continue to include the asterisk.
@@ -156,25 +160,25 @@ in the `src` folder inside the design directory.
 ```
 
 There are some shorthands for the exposed default variables:
+
 * `dir::` is equivalent to `refg::$DESIGN_DIR/`
 * `pdk_dir::` is equivalent to `refg::$PDK_ROOT/$PDK`
 
-
 #### Expression Engine
 
 By adding `expr::` to the beginning of a string, you can write basic infix
 mathematical expressions. Binary operators supported are `**`, `*`, `/`, `+`,
-and `-`, while operands can be any floating-point value, and previously evaluated
-numeric variables prefixed with a dollar sign. Unary operators are not supported,
-though negative numbers with the - sign stuck to them are. Parentheses (`()`)
-are also supported to prioritize certain operations.
+and `-`, while operands can be any floating-point value, and previously
+evaluated numeric variables prefixed with a dollar sign. Unary operators are not
+supported, though negative numbers with the - sign stuck to them are.
+Parentheses (`()`) are also supported to prioritize certain operations.
 
-Your expressions must return exactly one value: multiple expressions in the
-same `expr::`-prefixed value are considered invalid and so are empty expressions.
+Your expressions must return exactly one value: multiple expressions in the same
+`expr::`-prefixed value are considered invalid and so are empty expressions.
 
-It is important to note that, like variable referencing and conditional execution,
-the order of declarations matter: i.e., you cannot reference a variable that is
-declared after the current expression.
+It is important to note that, like variable referencing and conditional
+execution, the order of declarations matter: i.e., you cannot reference a
+variable that is declared after the current expression.
 
 ```json
 {
@@ -187,8 +191,10 @@ declared after the current expression.
     "A": "expr::$B * 2"
 }
 ```
+
 > In this example, the first configuration is invalid, as B is used in a
-> mathematical expression before declaration, but the latter is OK, evaluating to 8.
+> mathematical expression before declaration, but the latter is OK, evaluating
+> to 8.
 
 You can also simply reference another number using this prefix:
 
@@ -198,9 +204,11 @@ You can also simply reference another number using this prefix:
     "B": "expr::$A"
 }
 ```
+
 > In this example, B will simply hold the value of A.
 
 ## Tcl
+
 These configuration files are simple Tcl scripts with environment variables that
 are sourced by the OpenLane flow. Again, Tcl config files are not recommended
 for newer designs, but is still maintained and supported at the moment.

openlane/__main__.py

@@ -45,7 +45,7 @@
 from . import common
 from .container import run_in_container
 from .plugins import discovered_plugins
-from .config import Config, InvalidConfig
+from .config import Config, InvalidConfig, PassedDirectoryError
 from .common.cli import formatter_settings
 from .flows import Flow, SequentialFlow, FlowException, FlowError, cloup_flow_opts
 
@@ -66,45 +66,54 @@ def run(
     with_initial_state: Optional[State],
     config_override_strings: List[str],
     _force_run_dir: Optional[str],
-    _force_design_dir: Optional[str],
-) -> int:
-    if len(config_files) == 0:
-        err("No config file has been provided.")
-        ctx.exit(1)
-    elif len(config_files) > 1:
-        err("OpenLane does not currently support multiple configuration files.")
-        ctx.exit(1)
-    config_file = config_files[0]
-    # Enforce Mutual Exclusion
+    design_dir: Optional[str],
+):
+    try:
+        if len(config_files) == 0:
+            err("No config file(s) have been provided.")
+            ctx.exit(1)
 
-    flow_description: Union[str, List[str]] = flow_name or "Classic"
+        flow_description: Optional[Union[str, List[str]]] = None
 
-    if meta := Config.get_meta(config_file, flow_override=flow_name):
-        if flow_ids := meta.flow:
-            flow_description = flow_ids
+        for config_file in config_files:
+            if meta := Config.get_meta(config_file, flow_override=flow_name):
+                if flow_ids := meta.flow:
+                    if flow_description is None:
+                        flow_description = flow_ids
 
-    TargetFlow: Type[Flow]
+        if flow_name is not None:
+            flow_description = flow_name
 
-    if not isinstance(flow_description, str):
-        TargetFlow = SequentialFlow.make(flow_description)
-    else:
-        if FlowClass := Flow.factory.get(flow_description):
-            TargetFlow = FlowClass
+        if flow_description is None:
+            flow_description = "Classic"
+
+        TargetFlow: Type[Flow]
+
+        if not isinstance(flow_description, str):
+            TargetFlow = SequentialFlow.make(flow_description)
         else:
-            err(
-                f"Unknown flow '{flow_description}' specified in configuration file's 'meta' object."
-            )
-            return -1
+            if FlowClass := Flow.factory.get(flow_description):
+                TargetFlow = FlowClass
+            else:
+                err(
+                    f"Unknown flow '{flow_description}' specified in configuration file's 'meta' object."
+                )
+                ctx.exit(1)
 
-    try:
         flow = TargetFlow(
-            config_file,
+            config_files,
             pdk_root=pdk_root,
             pdk=pdk,
             scl=scl,
             config_override_strings=config_override_strings,
-            _force_design_dir=_force_design_dir,
+            design_dir=design_dir,
         )
+    except PassedDirectoryError as e:
+        err(e)
+        info(
+            f"If you meant to pass this as a design directory alongside valid configuration files, pass it as '--design-dir {e.config}'."
+        )
+        ctx.exit(1)
     except InvalidConfig as e:
         info(f"[green]Errors have occurred while loading the {e.config}.")
         for error in e.errors:
@@ -115,12 +124,12 @@ def run(
             for warning in e.warnings:
                 warn(warning)
         info("OpenLane will now quit. Please check your configuration.")
-        return 1
+        ctx.exit(1)
     except ValueError as e:
         err(e)
         debug(traceback.format_exc())
         info("OpenLane will now quit.")
-        return 1
+        ctx.exit(1)
 
     try:
         flow.start(
@@ -137,14 +146,12 @@ def run(
         err(f"The flow has encountered an unexpected error: {e}")
         traceback.print_exc()
         err("OpenLane will now quit.")
-        return 1
+        ctx.exit(1)
     except FlowError as e:
         if "deferred" not in str(e):
             err(f"The following error was encountered while running the flow: {e}")
         err("OpenLane will now quit.")
-        return 2
-
-    return 0
+        ctx.exit(2)
 
 
 def print_version(ctx: Context, param: Parameter, value: bool):
@@ -250,7 +257,7 @@ def run_example(
             with_initial_state=None,
             config_override_strings=[],
             _force_run_dir=None,
-            _force_design_dir=None,
+            design_dir=None,
         )
         if status == 0:
             info("Smoke test passed.")
@@ -403,8 +410,8 @@ def cli(ctx, /, **kwargs):
     ]:
         if subcommand_flag in run_kwargs:
             del run_kwargs[subcommand_flag]
-
-    ctx.exit(run(ctx, **run_kwargs))
+    run(ctx, **run_kwargs)
+    ctx.exit(0)
 
 
 if __name__ == "__main__":

openlane/common/__init__.py

@@ -50,6 +50,7 @@
     is_string,
     Number,
     Path,
+    AnyPath,
     ScopedFile,
 )
 from .toolbox import Toolbox

openlane/common/generic_dict.py

@@ -211,11 +211,24 @@ def update(self, incoming: "Mapping[KT, VT]"):
         """
         A convenience function to update multiple values in the GenericDict object
         at the same time.
-        :param
+        :param incoming: The values to update
         """
         for key, value in incoming.items():
             self[key] = value
 
+    def update_reorder(self, incoming: "Mapping[KT, VT]"):
+        """
+        A convenience function to update multiple values in the GenericDict object
+        at the same time. Pre-existing keys are deleted first so the values in
+        incoming are emplaced at the end of the dictionary.
+
+        :param incoming: The values to update
+        """
+        for key, value in incoming.items():
+            if key in self:
+                del self[key]
+            self[key] = value
+
 
 class GenericImmutableDict(GenericDict[KT, VT]):
     __lock: bool

openlane/common/types.py

@@ -90,6 +90,9 @@ def rel_if_child(
             return Path(my_abspath)
 
 
+AnyPath = Union[str, os.PathLike]
+
+
 class ScopedFile(Path):
     """
     Creates a temporary file that remains valid while this variable is in scope,

openlane/config/__init__.py

@@ -20,5 +20,13 @@
 """
 from .preprocessor import Keys
 from .variable import Instance, Macro, Variable
-from .config import Meta, Config, InvalidConfig
+from .config import (
+    Meta,
+    Config,
+    InvalidConfig,
+    AnyConfig,
+    AnyConfigs,
+    PassedDirectoryError,
+    UnknownExtensionError,
+)
 from .flow import flow_common_variables as universal_flow_config_variables