From 3635df711d2e7414fbd64640dccd57e8de438341 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 29 Jan 2020 16:59:43 -0800 Subject: [PATCH 01/11] [RFC] Imports --- _docs/rfc/imports.md | 753 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 753 insertions(+) create mode 100644 _docs/rfc/imports.md diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md new file mode 100644 index 0000000000..bc74979e6a --- /dev/null +++ b/_docs/rfc/imports.md @@ -0,0 +1,753 @@ +# Imports + +**STATUS**: In proposal + + +## Background + +### Problem + +Currently, Terragrunt does not provide a lot of flexibility when it comes to reusing values from other configs. +The only feature available to users right now for config reuse is the `include` block, which allows you to include and +merge all the values from another `terragrunt.hcl` config. The `include` block has a few limitations: + +- You do not have fine grained control over what values get merged. The entire config has to be inherited. +- You can only include and inherit one level. +- You have limited ability to affect the parent config from the child config. + +The canonical use case where these limitations get in the way is if you want to construct a hierarchy of inputs that get +merged. For example, consider the following canonical terragrunt folder structure: + +``` +prod +└── us-east-1 + └── app + └── vpc + └── terragrunt.hcl +``` + +As you progress down the directory, there is a desire to automatically include variables that specify the encapsulated +environment such that you can predict the target environment based on where the config lies in the folder structure. For +example, when you are in the `us-east-1` folder, you will want to pass in the variable `aws_region = "us-east-1"` to +terraform. Similarly, you will want to set `vpc_name = "app"` when you are in the `app` folder of a particular region, +so that you deploy to that particular VPC. + +In Terragrunt 0.18 and Terraform 0.11, this was accomplished by specifying `tfvars` files in the folder hierarchy: + +``` +prod +├── us-east-1 +│ ├── app +│ │ ├── env.tfvars +│ │ └── vpc +│ │ └── terraform.tfvars +│ └── region.tfvars +└── terraform.tfvars +``` + +You would then include each `tfvars` in the hierarchy in the root `terraform.tfvars` file using the `extra_arguments` +setting with `optional_var_files`. + +While you can still implement the same mechanism with Terragrunt 0.19 and above, a change in behavior in Terraform 0.12 +made it so that you must have all the variables that are being included specified in the child modules. This means that +even if you had a module that did not depend on the AWS region (e.g a Kubernetes Service), you have to specify the +`aws_region` variable in order for this to work. + +Another limitation of this approach is if the region variable is under a different name in the module being deployed. +This assumes that all the variable names for the region must be the same, which is oftentimes not the case, especially +when you want to deploy third party modules as well. In this case, you want to specify the various permutations in the +`region.tfvars` file but you won't be able to do that because you will run into the limitation where no module will have +all the permutations defined as variables. + +The current workaround for this use case is to specify the variables in `json` or `yaml` and merge them into the +`inputs` attribute of the root `terragrunt.hcl` file using the `jsondecode` / `yamldecode` function with `merge`. While +this works, the configuration becomes fairly verbose as you try to workaround the fact that not all directories will +have all the yaml files in the hierarchy. See [this example config](https://github.com/gruntwork-io/terragrunt-infrastructure-live-example/blob/b796c371f631b9ba42189fef744601cdd16d48f5/non-prod/terragrunt.hcl#L30). + +Another limitation of `yaml` and `json` is that they are static configurations. This means that you can't share +complex variables in the middle of the hierarchy that might require more computation than hard coded values. For +example, suppose that you wanted to always include the `vpc_id` when you are at the `app` level of the folder hierarchy. +Ideally, you would be able to specify: + +```hcl +dependency "vpc" { + config_path = "/path/to/app/vpc/module" +} + +inputs = { + vpc_id = dependency.vpc.outputs.vpc_id +} +``` + +and then auto merge this configuration with the children. However, since we can't rely on terragrunt parsing for the +middle layers, and since we can only include one level of terragrunt configuration, there is no way to do the above +without introducing the dependency to all configurations, which is not what you want. + + +### Goals + +Given the problem statement, this RFC aims to propose a solution that addresses the following: + +- Provide a way to reuse partial terragrunt configurations. +- Provide a way to reuse from multiple terragrunt configuration files. +- Maintain design principles of explicit vs. implicit. Avoid confusing behavior that adds cognitive load, such as monkey + patching logic. +- Avoid verbosity where possible. +- Avoid expanding the number of constructs in terragrunt (e.g a helper function or block). If a new construct has to be + introduced, it should deprecate and replace an existing construct. + + +## Proposed solution + +The proposed solution is to introduce a new block `import` which replaces the functionality of `include`. We use a new +block instead of reusing `include` for backwards compatibility. As part of the implementation, `include` and the +relevant functions (`get_parent_terragrunt_dir`, `path_relative_to_include`, and `path_relative_from_include`) will be +deprecated and will throw a warning whenever someone uses it. + +The `import` block works as follows: + +- When the `import` block appears in a terragrunt config, the target config is parsed in full before the current config. +- Subsequent blocks that are parsed will be given the context of the `import` and any block and attribute of the + imported config can be referenced. +- Complex structures like `remote_state` will have their properties nested. E.g you can reference the backend of the + imported config under the name `remote_state.backend`. +- Blocks that support multiple declarations (e.g `dependency`) will be referenced by name. E.g if you had in the parent: + + dependency "vpc" {} + dependency "db" {} + + You can reference each dependency block in the child as `dependency.vpc` and `dependency.db` respectively. + +- Imports are one way: you can not have circular imports, and you can not influence the parent config to affect other + children (commonly known as monkey patching). +- You can chain imports. As in, imported configs can themselves import other config, as long as the chain of imports + does not form a cycle. +- Imports support auto merging via the `merge` setting. When `merge = true`, all the blocks and attributes of the + imported config will be merged with the child config. Note that since HCL blocks are sequential, imports will be + merged top to bottom. See below for more details. +- Imports will also export the absolute path to the directory of the terragrunt config file as the attribute + `terragrunt_dir`. +- Imports are parsed first in the [configuration parsing + order](https://terragrunt.gruntwork.io/docs/getting-started/configuration/#configuration-parsing-order). This is to + allow references to the imports in `locals`. This is because it is more likely that one would want to break down + imports attributes into local references than want to use locals in the `import` block given that it only has a single + attribute pointing to the target config. +- Imports are not compatible with `include`. Having both blocks will cause a terragrunt syntax error. + +Let's take a look at a few common use cases and how we might use `import` to address them: + +- [Hierarchical variables included across multiple terragrunt.hcl + files](#hierarchical-variables-included-across-multiple-terragrunt-hcl-files) +- [Reusing common variables](#reusing-common-variables) +- [Reusing dependencies](#reusing-dependencies) +- [Keeping remote state configuration DRY](#keeping-remote-state-configuration-dry) + +### Hierarchical variables included across multiple terragrunt.hcl files + +Consider the following folder structure from the canonical example: + +``` +prod +├── us-east-1 +│ ├── app +│ │ ├── env.hcl +│ │ └── vpc +│ │ └── terragrunt.hcl +│ └── region.hcl +└── account.hcl +``` + +With `import` blocks, you can implement the automatic variable merging in the following manner: + +prod/account.hcl + +```hcl +inputs = { + account_id = 0000000 +} +``` + +prod/us-east-1/region.hcl + +```hcl +import "account" { + config_path = "../account.hcl" +} + +inputs = merge( + import.account.inputs, + { + region = "us-east-1" + }, +) +``` + +prod/us-east-1/app/env.hcl + +```hcl +import "region" { + config_path = "../region.hcl" +} + +inputs = merge( + import.region.inputs, + { + env = "prod" + }, +) +``` + +prod/us-east-1/app/vpc/terragrunt.hcl + +```hcl +import "env" { + config_path = "../env.hcl" +} + +inputs = merge( + import.env.inputs, + { + # args to module + }, +) +``` + +Note how there is a chain of imports that add additional inputs as we progress down the hierarchy. Each level appends +additional inputs that are made available for use to deeper levels of the hierarchy. This means that by +the time we get to the leaf (`prod/us-east-1/app/vpc`), all the required inputs will have been merged in. The nice thing +about this behavior is that everything you need to know about the configuration is explicitly mentioned. There is no +implicit values that change based on who is importing the configuration. + +Here is another alternative that avoids deep imports: + +prod/account.hcl + +```hcl +inputs = { + account_id = 0000000 +} +``` + +prod/us-east-1/region.hcl + +```hcl +inputs = { + region = "us-east-1" +} +``` + +prod/us-east-1/app/env.hcl + +```hcl +inputs = { + env = "prod" +} +``` + +prod/us-east-1/app/vpc/terragrunt.hcl + +```hcl +import "root" { + config_path = "../../../root.hcl" + merge = true +} + +import "region" { + config_path = "../../region.hcl" + merge = true +} + +import "env" { + config_path = "../env.hcl" + merge = true +} + +inputs = { + # args to module +} +``` + +This trades off verbosity in the child config with a relatively simpler import path, where there is only one level of +imports. Note how all three imports have `merge = true`. This is equivalent to the following: + +``` +import "root" { + config_path = "../../../root.hcl" +} + +import "region" { + config_path = "../../region.hcl" +} + +import "env" { + config_path = "../env.hcl" +} + +inputs = merge( + import.root.inputs, + import.region.inputs, + import.env.inputs, + { + # args to module + }, +) +``` + +The `merge = true` option is a convenience feature to make the child config less verbose in case you have multiple +overridable configuration in your hierarchy. + + +### Reusing common variables + +Many resources are named with the region or environment that they belong to. A canonical terragrunt live configuration +uses hierarchical variables to pass in the region and environment settings to terraform. However, there is no way in the +current implementation to compose those variables to construct different inputs (e.g a `name` variable that includes the +`region` variable). You had to rely on doing the composition in terraform. This works if you have control over the +module, but sometimes you want to directly deploy a third party module (e.g from the registry) and it seems heavy to +have to fork or wrap that module just for the name. + +With `import` blocks, you can implement this by referencing the `region` input from the `import`: + +region config +``` +import "region" { + config_path = "../../region.hcl" +} + +inputs = merge( + import.region.inputs, + { + name = "${import.region.inputs["region"]}-unique-name" + }, +) +``` + + +### Reusing dependencies + +In the problem statement we discussed a use case where we want to pass in and merge the `vpc_id`. This can be +implemented with `import` blocks in two ways. + +- [Auto merge](#auto-merge) +- [Explicit reference](#explicit-reference) + +#### Auto merge + +With auto merge, you can `import` the configuration that declares the dependency and the input directly and have nothing +in the child config. For example: + +vpc_dependency_config.hcl + +``` +dependency "vpc" { + config_path = "/path/to/app/vpc/module" +} + +inputs = { + vpc_id = dependency.vpc.outputs.vpc_id +} +``` + +``` +import "vpc_dependency_config" { + config_path = "../vpc_dependency_config.hcl" + merge = true +} + +inputs = { + name = "unique-name" +} +``` + +This will pass in the input variables `vpc_id` and `name` to the terraform configuration, where the `vpc_id` input comes +from the `vpc_dependency_config.hcl` configuration import that is merged in. Note that we can add in additional +configurations by adding another `import` block. For example, if you had a root config that specifies remote state +configurations, you can add another `import` block for it to pull it in: + +``` +import "root" { + config_path = "../../root.hcl" + merge = true +} + +import "vpc_dependency_config" { + config_path = "../vpc_dependency_config.hcl" + merge = true +} + +inputs = { + name = "unique-name" +} +``` + +#### Explicit reference + +As an alternative to auto merge, you can also be explicit in referencing the inputs block in the import: + +``` +import "vpc_dependency_config" { + config_path = "../vpc_dependency_config.hcl" +} + +inputs = merge( + import.vpc_dependency_config.inputs, + { + name = "unique-name" + }, +) +``` + +Or, if you needed to pass in the under a different name (e.g `id_of_vpc`), you can access the vpc dependency across the +import: + +``` +import "vpc_dependency_config" { + config_path = "../vpc_dependency_config.hcl" +} + +inputs = { + name = "unique-name" + id_of_vpc = import.vpc_dependency_config.dependency.vpc.outputs.vpc_id +} +``` + + + +### Keeping remote state configuration DRY + +A canonical use case for terragrunt is to share the remote state configuration. The classic example is having a root +`terragrunt.hcl` configuration that specifies the `remote_state`, which is then imported using `include` to all the +other configurations: + +parent + +```hcl +remote_state { + backend = "s3" + config = { + bucket = "my-terraform-state" + key = "${path_relative_to_include()}/terraform.tfstate" + region = "us-east-1" + encrypt = true + dynamodb_table = "my-lock-table" + } +} +``` + +child + +```hcl +include { + path = find_in_parent_folders() +} +``` + +In this world, the parent configuration (in this case the `remote_state` block) is automatically "merged" into the child +configuration (NOTE: this is not an actual merge as it does not do a deep merge. If the child had a `remote_state` +block, the entire block is replaced.). + +A key feature here is the use of the `path_relative_to_include` to monkey patch the S3 key of the parent config based on +who is importing. For example: + +``` +├── terragrunt.hcl +├── backend-app +│ ├── main.tf +│ └── terragrunt.hcl +├── frontend-app +│ ├── main.tf +│ └── terragrunt.hcl +├── mysql +│ ├── main.tf +│ └── terragrunt.hcl +└── vpc + ├── main.tf + └── terragrunt.hcl +``` + +Here, the S3 key of the statefile for each of the child configurations will be as follows: + +``` +backend-app => backend-app/terraform.tfstate +frontend-app => frontend-app/terraform.tfstate +mysql => mysql/terraform.tfstate +vpc => vpc/terraform.tfstate +``` + +This is because the relative path from the parent config to the child config is the single level of folders in the +hierarchy. + +However, this can be confusing as now there is complex cognitive load on the reader to know the relative path between +the two configs to fully know what the key should be. This can be difficult to do in your head since you do not have the +paths in view in the config! + +With `import` blocks, we can implement this in the following way: + +parent + +```hcl +remote_state { + backend = "s3" + config = { + bucket = "my-terraform-state" + region = "us-east-1" + encrypt = true + dynamodb_table = "my-lock-table" + # `key` must be set by the child configs. + } +} +``` + +child + +```hcl +import "root" { + config_path = find_in_parent_folders("root.hcl") + merge = true +} + +remote_state { + config = { + # The relative path between the root terragrunt directory and the current terragrunt file directory. + key = relpath(import.root.terragrunt_dir, get_terragrunt_dir()) + } +} +``` + +Note how we explicitly set the relative path using only the context of the current file and the parent. There is no +circular reference here where you need the context of the child to know the exact values of the attributes being set in +the parent (although you won't know the full setting of `remote_state` without seeing the child). + +Additionally, here we take advantage of the `merge` feature to deep merge the two configurations for `remote_state`. +This is equivalent to having the following `remote_state` block in the child config: + +```hcl +remote_state { + backend = "s3" + config = { + bucket = "my-terraform-state" + region = "us-east-1" + # The relative path between the root terragrunt directory and the current terragrunt file directory. + key = relpath("/abspath/to/dir/containing/root.hcl", get_terragrunt_dir()) + encrypt = true + dynamodb_table = "my-lock-table" + } +} +``` + +If you wanted to be explicit and avoid the implicit `merge` of the blocks, you can also explicitly set the attributes of +the block in the child config: + +```hcl +import "root" { + config_path = find_in_parent_folders("root.hcl") +} + +remote_state { + backend = import.root.remote_state.backend + config = merge( + import.root.remote_state.config, + { + # The relative path between the root terragrunt directory and the current terragrunt file directory. + key = relpath(import.root.terragrunt_dir, get_terragrunt_dir()) + }, + ) +} +``` + +NOTE: `relpath` does not currently exist, and must be implemented as a terragrunt helper function. + +The advantage of this configuration is that we're only pulling the `remote_state` configuration to the child. The other +blocks and attributes (e.g `inputs`) is not inherited to the child config. This was something that was not possible to +do with `include`. + +#### Why can't we use the merge function? + +In the above example, we have to explicitly declare the block. A natural extension of this pattern is to use the `merge` +function directly on the `remote_state`: + +```hcl +remote_state = merge( + import.root.remote_state, + { + config = { + key = relpath(import.root.terragrunt_dir, get_terragrunt_dir()) + } + } +) +``` + +Unfortunately, this is a syntax error because `remote_state` is a block, and not an attribute. You can not directly set +blocks in this way. + + + + +## Alternatives + +### Enhancing include with import semantics + +Instead of having a dedicated block with the new functionality, we could enhance `include` to have all the semantics of +`import`. This reduces complexity of the configuration by being able to recycle a very similar construct that already +exists. + +However, this means that we are locked into supporting all the functions that allow manipulation of the parent +configuration based on who has included it (e.g `path_relative_to_include`). Users will be used to and expect continued +support for such semantics, and possibly request feature enhancements that further encourage monkey patching. + +Additionally, the backwards compatibility story of reusing the include block can lead to confusion. Reusing `include` +implies that we should default `merge` to `true` to maintain backwards compatibility. Defaulting to `false` not only +breaks existing configurations, but it breaks them in a very subtle way where the parent configuration is not merged +into the child. This can be especially problematic to a user who is only relying on `include` to keep their remote state +configuration DRY, as all their remote state configuration will be missing and a naive `apply-all` for a new environment +may not surface this fact because it will "just work" on the local state. Note that this is a realistic scenario: +imagine a new person on your team who installs the latest version of terragrunt with this change, but the team has been +using the older version without this update. They then start to add a new component, and when they run plan, everything +will look correct because they are adding new resources, but they may not realize that it is not storing to remote +state! + +To summarize: + +Pros: + +- Reduce configuration complexity by reusing an existing block. +- Avoid confusion from having two ways to do similar things (`import` with `merge = true` vs. `include`). + +Cons: + +- Potentially complex implementation due to having to touch and modify lots of existing code. +- Locks us into supporting existing semantics around "monkey patching" +- Backwards incompatibility has gotchas that can cause frustrations to existing teams. + +Ultimately, the potential for badly shooting yourself in the foot was enough to warrant a new block to start clean. + + +### globals + +Globals was a potential solution to the problem proposed by the community. `globals` work the same way as `locals`, +except they support merging across `include`. For example, to address the use case of [reusing common +variables](#reusing-common-variables), you could have the following: + +parent config + +``` +globals { + aws_region = "us-east-1" +} +``` + +child config + +``` +include { + path = find_in_parent_folders() +} + +inputs = { + aws_region = global.aws_region + name = "${global.aws_region}-unique-name" +} +``` + +Note how the child config accesses the `global` variable that is defined in the parent config, without having defined +the `globals` block. + +`globals` also had the ability to "monkey patch" the parent config. For example: + +parent config + +``` +globals { + aws_region = "us-east-1" +} + +inputs = { + aws_region = global.aws_region + name = "${global.aws_region}-${global.name_suffix}" +} +``` + +child config + +``` +globals { + name_suffix = "unique-name" +} + +include { + path = find_in_parent_folders() +} +``` + +Note how the child config updated the `global` variable in the parent config by specifying a `globals` block. + +You can read more about the proposal in [the issue](https://github.com/gruntwork-io/terragrunt/issues/814) and +[corresponding PR](https://github.com/gruntwork-io/terragrunt/pull/858). + +Pros: + +- Reuses existing import mechanism. +- Powerful construct that can be used to simulate custom functions in terragrunt configs by monkey patching. + +Cons: + +- Encourages monkey patching, which generally increases complexity and deteriorates readability/maintainability. +- Implementation can be complex due to graph logic between globals and locals to allow reuse across the two. +- Does not solve all the problems in this RFC. E.g `globals` does not address the need for multiple includes and + fine grained control over merging. + +Ultimately, the complexity of the implementation and the monkey patching behavior suggested that it may be better to +look for an alternative implementation. + + +### hcldecode or read_terragrunt_config helper function + +Instead of defining a dedicated block for `import`, we could define a helper function that parses the relevant config. +For example, the explicit triple import example in the [hierarchical variables use +case](#hierarchical-variables-included-across-multiple-terragrunt-hcl-files) can be implemented as: + +``` +locals { + root_config = read_terragrunt_config("../../../root.hcl") + region_config = read_terragrunt_config("../../region.hcl") + env_config = read_terragrunt_config("../env.hcl") +} + +inputs = merge( + local.root_config.inputs, + local.region_config.inputs, + local.env_config.inputs, + { + # args to module + }, +) +``` + +Pros: + +- Relatively simple implementation. It is significantly easier to add helper functions than it is to introduce new + blocks in terragrunt. +- Can address all the same use cases as `import` blocks. + +Cons: + +- Everything has to be explicit in the config. We can not support automatic merging capabilities when using helper + functions, as you can't manipulate the config in a helper. This can lead to very verbose configurations if one wants + to use this in place of `include`. + +Ultimately, the verbosity that this implementation entails would encourage users to continue to use `include`. While +this is not a bad thing, it was preferred to reduce constructs instead of introducing new ones. Note that while `import` +introduces a new construct, the proposal includes deprecating the `include` block in favor of `import` such that +eventually `import` replaces `include`. + + +## References + +This challenge has come up numerous times in the lifetime of Terragrunt. The following are relevant issues that raise +similar concerns: + +- [Shared and overridable variabls](https://github.com/gruntwork-io/terragrunt/issues/814) +- [Being able to merge maps from inputs](https://github.com/gruntwork-io/terragrunt/issues/744) +- [Request to allow more than one level of include](https://github.com/gruntwork-io/terragrunt/issues/303) +- [Request to reference inputs from another config](https://github.com/gruntwork-io/terragrunt/issues/967) +- [Partially override components of an input](https://github.com/gruntwork-io/terragrunt/issues/1011) From af65e515041b6973a7dd33c2b8922134f3fed102 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 29 Jan 2020 22:11:58 -0800 Subject: [PATCH 02/11] [skip ci] Include capability to deep merge config --- _docs/rfc/imports.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index bc74979e6a..6f78e270b5 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -125,6 +125,7 @@ The `import` block works as follows: - Imports support auto merging via the `merge` setting. When `merge = true`, all the blocks and attributes of the imported config will be merged with the child config. Note that since HCL blocks are sequential, imports will be merged top to bottom. See below for more details. +- Imports can also be deep merged using the `deep_merge` setting. This will work similar to `merge = true`. - Imports will also export the absolute path to the directory of the terragrunt config file as the attribute `terragrunt_dir`. - Imports are parsed first in the [configuration parsing @@ -503,7 +504,7 @@ child ```hcl import "root" { config_path = find_in_parent_folders("root.hcl") - merge = true + deep_merge = true } remote_state { @@ -518,8 +519,8 @@ Note how we explicitly set the relative path using only the context of the curre circular reference here where you need the context of the child to know the exact values of the attributes being set in the parent (although you won't know the full setting of `remote_state` without seeing the child). -Additionally, here we take advantage of the `merge` feature to deep merge the two configurations for `remote_state`. -This is equivalent to having the following `remote_state` block in the child config: +Additionally, here we take advantage of the `deep_merge` feature to deep merge the two configurations for +`remote_state`. This is equivalent to having the following `remote_state` block in the child config: ```hcl remote_state { From 37d253672baaac4d8491e0bb5d1c210cdc81bb30 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 29 Jan 2020 22:43:42 -0800 Subject: [PATCH 03/11] [skip ci] Open questions section with the first question: should we implement a few easy to understand functions that are in the context of the child? --- _docs/rfc/imports.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index 6f78e270b5..afb31c3d8d 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -742,6 +742,18 @@ introduces a new construct, the proposal includes deprecating the `include` bloc eventually `import` replaces `include`. +## Open Question + +- While a major design goal in this RFC is to keep the principles of explicit over implicit, practically speaking it may + not be feasible to achieve that design goal to support all the use cases of the community. In that regard, it may be + useful to implement a minimal set of features that don't take a whole lot of cognitive load to understand. Should the + following functions be implemented for use with `import`? + - `path_relative_to_import` and `path_relative_from_import`: These are the equivalent functions of the ones named + for `include`. + - `find_in_parent_folders_from_importing_config`: This function is the version of `find_in_parent_folders` that + works in the context of the config that is importing the current config. + + ## References This challenge has come up numerous times in the lifetime of Terragrunt. The following are relevant issues that raise From 542509f2e0dade1a0dd4ef67214cbc5b60020749 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Tue, 11 Feb 2020 17:46:38 -0800 Subject: [PATCH 04/11] [skip ci] Update read_terragrunt_config to walk through a few more examples --- _docs/rfc/imports.md | 164 ++++++++++++++++++++++++++++++++----------- 1 file changed, 122 insertions(+), 42 deletions(-) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index afb31c3d8d..a817646feb 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -586,7 +586,127 @@ blocks in this way. ## Alternatives -### Enhancing include with import semantics +### In consideration + +#### hcldecode or read_terragrunt_config helper function + +Instead of defining a dedicated block for `import`, we could define a helper function that parses the relevant config. +For example, the explicit triple import example in the [hierarchical variables use +case](#hierarchical-variables-included-across-multiple-terragrunt-hcl-files) can be implemented as: + +``` +locals { + root_config = read_terragrunt_config("../../../root.hcl") + region_config = read_terragrunt_config("../../region.hcl") + env_config = read_terragrunt_config("../env.hcl") +} + +inputs = merge( + local.root_config.inputs, + local.region_config.inputs, + local.env_config.inputs, + { + # args to module + }, +) +``` + +Pros: + +- Relatively simple implementation. It is significantly easier to add helper functions than it is to introduce new + blocks in terragrunt. +- Can address all the same use cases as `import` blocks. + +Cons: + +- Everything has to be explicit in the config. We can not support automatic merging capabilities when using helper + functions, as you can't manipulate the config in a helper. This can lead to very verbose configurations if one wants + to use this in place of `include`. + +Note that to take full advantage of this approach, all the blocks in `terragrunt.hcl` need to be redefined as attributes +so that we can use assignment to override them. + +Let's walk through a few more of the use cases: + +**Keeping remote state configuration DRY** + +parent + +```hcl +remote_state { + backend = "s3" + config = { + bucket = "my-terraform-state" + key = "${path_relative_to_include()}/terraform.tfstate" + region = "us-east-1" + encrypt = true + dynamodb_table = "my-lock-table" + } +} +``` + +child + +```hcl +locals { + root_config = read_terragrunt_config("../../../root.hcl") +} + +remote_state { + backend = local.root_config.remote_state.backend + config = merge( + local.root_config.remote_state.config, + { + # The relative path between the root terragrunt directory and the current terragrunt file directory. + key = relpath(import.root.terragrunt_dir, get_terragrunt_dir()) + }, + ) +} +``` + +Note that we can't do: + +```hcl +remote_state = deep_merge(local.root_config.remote_state, { config = { key = relpath } }) +``` + +due to the fact that `remote_state` is a block and not an attribute. + + +**Reusing dependencies** + +We can't reuse `dependency` blocks in this implementation because there is no way to auto merge the blocks. + +If `dependency` was instead an attribute, we could use the following alternative syntax: + +vpc_dependency_config.hcl + +``` +dependency = { + vpc = { + config_path = "/path/to/app/vpc/module" + } +} +``` + +``` +locals { + common_deps = read_terragrunt_config("../vpc_dependency_config.hcl") +} + +dependency = local.common_deps.dependency + +inputs = { + name = "unique-name" + vpc_id = dependency.vpc.outputs.vpc_id +} +``` + + + +### Rejected + +#### Enhancing include with import semantics Instead of having a dedicated block with the new functionality, we could enhance `include` to have all the semantics of `import`. This reduces complexity of the configuration by being able to recycle a very similar construct that already @@ -623,7 +743,7 @@ Cons: Ultimately, the potential for badly shooting yourself in the foot was enough to warrant a new block to start clean. -### globals +#### globals Globals was a potential solution to the problem proposed by the community. `globals` work the same way as `locals`, except they support merging across `include`. For example, to address the use case of [reusing common @@ -701,46 +821,6 @@ Ultimately, the complexity of the implementation and the monkey patching behavio look for an alternative implementation. -### hcldecode or read_terragrunt_config helper function - -Instead of defining a dedicated block for `import`, we could define a helper function that parses the relevant config. -For example, the explicit triple import example in the [hierarchical variables use -case](#hierarchical-variables-included-across-multiple-terragrunt-hcl-files) can be implemented as: - -``` -locals { - root_config = read_terragrunt_config("../../../root.hcl") - region_config = read_terragrunt_config("../../region.hcl") - env_config = read_terragrunt_config("../env.hcl") -} - -inputs = merge( - local.root_config.inputs, - local.region_config.inputs, - local.env_config.inputs, - { - # args to module - }, -) -``` - -Pros: - -- Relatively simple implementation. It is significantly easier to add helper functions than it is to introduce new - blocks in terragrunt. -- Can address all the same use cases as `import` blocks. - -Cons: - -- Everything has to be explicit in the config. We can not support automatic merging capabilities when using helper - functions, as you can't manipulate the config in a helper. This can lead to very verbose configurations if one wants - to use this in place of `include`. - -Ultimately, the verbosity that this implementation entails would encourage users to continue to use `include`. While -this is not a bad thing, it was preferred to reduce constructs instead of introducing new ones. Note that while `import` -introduces a new construct, the proposal includes deprecating the `include` block in favor of `import` such that -eventually `import` replaces `include`. - ## Open Question From 2b1039d2ae8ec1258b0409b3b1fb1f9bdca84990 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Tue, 11 Feb 2020 18:18:10 -0800 Subject: [PATCH 05/11] [skip ci] Add commentary on #759 --- _docs/rfc/imports.md | 117 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index a817646feb..3d1a471da6 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -588,6 +588,123 @@ blocks in this way. ### In consideration +#### Single terragrunt.hcl file per environment + +This implementation introduces a new block `module` that replaces `dependency`, `terraform`, and `inputs`. This approach +is documented in [#759](https://github.com/gruntwork-io/terragrunt/issues/759). + +In addition to the general analysis of that proposal, here are the list of Pros and Cons related to the problem of +sharing config: + +Pros: + +- Circumvent the reference problem by having all the references in a single shared scope. + +Cons: + +- If you need to share configurations across environments, you would still need one of the alternative approaches to + import the other config. This, however, is likely to be a rare occurrence. +- This is a drastic change to Terragrunt, completely changing the way it works. + + +Let's walk through how each of the import use cases look like with this implementation: + +**Hierarchical variables** + +In this approach, a hierarchy of variables is unnecessary because all the blocks are defined in a single scope. However, +depending on the scope of `locals`, certain things like reusing a repetitive variable becomes more challenging. + +To understand this, consider a use case where you are operating in two regions, `us-east-1` and `eu-west-1`. To simplify +this example, consider a limited application deployment where you have two modules: `vpc` and `app`. + +If we assume that `locals` are scoped within the file that they are defined, then you can implement this by separating +the environment `terragrunt.hcl` into two files, `us_east_1.hcl` and `eu_west_1.hcl`: + +us_east_1.hcl + +```hcl +locals { + region = "us-east-1" +} + +module "vpc" { + # additional args omitted for brevity + aws_region = local.region +} + +module "app" { + # additional args omitted for brevity + aws_region = local.region +} +``` + +eu_west_1.hcl + +```hcl +locals { + region = "eu-west-1" +} + +module "vpc" { + # additional args omitted for brevity + aws_region = local.region +} + +module "app" { + # additional args omitted for brevity + aws_region = local.region +} +``` + +However, if the namespace of `locals` was shared across the environment, then the above approach would not work and each +region file will need to define a different name for the region variable. + +Note that in this example, we assumed that the environment split is at the account level. An alternative split is to +have each environment be at the region level. The advantage of this approach is to define a different state bucket for +each region, which is a better posture for disaster recovery. In this approach, it doesn't matter what the scope of the +`locals` is. However, the downside of this approach is that there will be some repetition to pull in the AWS account ID +info across all the different regions. + +**Reusing common variables** + +Reusing common variables depends on the scope of `locals`. If the scope of `locals` is shared across the environment, +then you can define the common variable in `locals` blocks to share across the entier environment. If instead the scope +of `locals` is per file, we either: + +- Define all the code for the single environment in a single file. +- Implement [globals](#globals) in a way that can be shared across the environment. + +**Reusing dependencies** + +Reusing dependencies is not a problem in this approach because the namespace for the environment is shared. That is, you +can reference any of the other `module` blocks to hook up the dependency within a single environment. For example, if +you had two modules `app` and `mysql` which depend on the `vpc` module, you could define the config as follows: + +``` +module "vpc" { + # args omitted for brevity +} + +module "app" { + # args omitted for brevity + vpc_id = module.vpc.outputs.vpc_id +} + +module "mysql" { + # args omitted for brevity + vpc_id = module.vpc.outputs.vpc_id +} +``` + +This example reuses the outputs of `module.vpc` across the two modules, which is the equivalent of having the `vpc` +`dependency` block redefined in the two module configs. + +**Keeping remote state configuration DRY** + +This example is covered in [the original issue that proposed this +idea](https://github.com/gruntwork-io/terragrunt/issues/759). + + #### hcldecode or read_terragrunt_config helper function Instead of defining a dedicated block for `import`, we could define a helper function that parses the relevant config. From d1aa8f0f54838cefa22864ebd17ef806e6fb8a6b Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 12 Feb 2020 15:47:21 -0800 Subject: [PATCH 06/11] [skip ci] Fix formatting to make subsections clearer --- _docs/rfc/imports.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index 3d1a471da6..f7e889ec00 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -609,7 +609,7 @@ Cons: Let's walk through how each of the import use cases look like with this implementation: -**Hierarchical variables** +_Hierarchical variables_ In this approach, a hierarchy of variables is unnecessary because all the blocks are defined in a single scope. However, depending on the scope of `locals`, certain things like reusing a repetitive variable becomes more challenging. @@ -674,13 +674,13 @@ of `locals` is per file, we either: - Define all the code for the single environment in a single file. - Implement [globals](#globals) in a way that can be shared across the environment. -**Reusing dependencies** +_Reusing dependencies_ Reusing dependencies is not a problem in this approach because the namespace for the environment is shared. That is, you can reference any of the other `module` blocks to hook up the dependency within a single environment. For example, if you had two modules `app` and `mysql` which depend on the `vpc` module, you could define the config as follows: -``` +```hcl module "vpc" { # args omitted for brevity } @@ -699,7 +699,7 @@ module "mysql" { This example reuses the outputs of `module.vpc` across the two modules, which is the equivalent of having the `vpc` `dependency` block redefined in the two module configs. -**Keeping remote state configuration DRY** +_Keeping remote state configuration DRY_ This example is covered in [the original issue that proposed this idea](https://github.com/gruntwork-io/terragrunt/issues/759). @@ -711,7 +711,7 @@ Instead of defining a dedicated block for `import`, we could define a helper fun For example, the explicit triple import example in the [hierarchical variables use case](#hierarchical-variables-included-across-multiple-terragrunt-hcl-files) can be implemented as: -``` +```hcl locals { root_config = read_terragrunt_config("../../../root.hcl") region_config = read_terragrunt_config("../../region.hcl") @@ -745,7 +745,7 @@ so that we can use assignment to override them. Let's walk through a few more of the use cases: -**Keeping remote state configuration DRY** +_Keeping remote state configuration DRY_ parent @@ -790,7 +790,7 @@ remote_state = deep_merge(local.root_config.remote_state, { config = { key = rel due to the fact that `remote_state` is a block and not an attribute. -**Reusing dependencies** +_Reusing dependencies_ We can't reuse `dependency` blocks in this implementation because there is no way to auto merge the blocks. @@ -798,7 +798,7 @@ If `dependency` was instead an attribute, we could use the following alternative vpc_dependency_config.hcl -``` +```hcl dependency = { vpc = { config_path = "/path/to/app/vpc/module" @@ -806,7 +806,7 @@ dependency = { } ``` -``` +```hcl locals { common_deps = read_terragrunt_config("../vpc_dependency_config.hcl") } From e0e8f8f013e76b05df14acaa58c6e68116cf7036 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 12 Feb 2020 15:51:08 -0800 Subject: [PATCH 07/11] [skip ci] Use underline italics for sub headings --- _docs/rfc/imports.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index f7e889ec00..f51771ba86 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -609,7 +609,7 @@ Cons: Let's walk through how each of the import use cases look like with this implementation: -_Hierarchical variables_ +__*Hierarchical variables*__ In this approach, a hierarchy of variables is unnecessary because all the blocks are defined in a single scope. However, depending on the scope of `locals`, certain things like reusing a repetitive variable becomes more challenging. @@ -665,7 +665,7 @@ each region, which is a better posture for disaster recovery. In this approach, `locals` is. However, the downside of this approach is that there will be some repetition to pull in the AWS account ID info across all the different regions. -**Reusing common variables** +__*Reusing common variables*__ Reusing common variables depends on the scope of `locals`. If the scope of `locals` is shared across the environment, then you can define the common variable in `locals` blocks to share across the entier environment. If instead the scope @@ -699,7 +699,7 @@ module "mysql" { This example reuses the outputs of `module.vpc` across the two modules, which is the equivalent of having the `vpc` `dependency` block redefined in the two module configs. -_Keeping remote state configuration DRY_ +__*Keeping remote state configuration DRY*__ This example is covered in [the original issue that proposed this idea](https://github.com/gruntwork-io/terragrunt/issues/759). @@ -745,7 +745,7 @@ so that we can use assignment to override them. Let's walk through a few more of the use cases: -_Keeping remote state configuration DRY_ +__*Keeping remote state configuration DRY*__ parent @@ -790,7 +790,7 @@ remote_state = deep_merge(local.root_config.remote_state, { config = { key = rel due to the fact that `remote_state` is a block and not an attribute. -_Reusing dependencies_ +__*Reusing dependencies*__ We can't reuse `dependency` blocks in this implementation because there is no way to auto merge the blocks. From a1821626408d77c4fa7089c6894ac75c9e32ec16 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 12 Feb 2020 15:54:26 -0800 Subject: [PATCH 08/11] [skip ci] Just use real heading --- _docs/rfc/imports.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index f51771ba86..96a0f36e48 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -609,7 +609,7 @@ Cons: Let's walk through how each of the import use cases look like with this implementation: -__*Hierarchical variables*__ +##### Hierarchical variables In this approach, a hierarchy of variables is unnecessary because all the blocks are defined in a single scope. However, depending on the scope of `locals`, certain things like reusing a repetitive variable becomes more challenging. @@ -665,7 +665,7 @@ each region, which is a better posture for disaster recovery. In this approach, `locals` is. However, the downside of this approach is that there will be some repetition to pull in the AWS account ID info across all the different regions. -__*Reusing common variables*__ +##### Reusing common variables Reusing common variables depends on the scope of `locals`. If the scope of `locals` is shared across the environment, then you can define the common variable in `locals` blocks to share across the entier environment. If instead the scope @@ -699,7 +699,7 @@ module "mysql" { This example reuses the outputs of `module.vpc` across the two modules, which is the equivalent of having the `vpc` `dependency` block redefined in the two module configs. -__*Keeping remote state configuration DRY*__ +##### Keeping remote state configuration DRY This example is covered in [the original issue that proposed this idea](https://github.com/gruntwork-io/terragrunt/issues/759). @@ -745,7 +745,7 @@ so that we can use assignment to override them. Let's walk through a few more of the use cases: -__*Keeping remote state configuration DRY*__ +##### Keeping remote state configuration DRY parent @@ -790,7 +790,7 @@ remote_state = deep_merge(local.root_config.remote_state, { config = { key = rel due to the fact that `remote_state` is a block and not an attribute. -__*Reusing dependencies*__ +##### Reusing dependencies We can't reuse `dependency` blocks in this implementation because there is no way to auto merge the blocks. From bbbfcca5c0850712e1a7c649c273b26b8f41b424 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 26 Feb 2020 16:40:37 -0800 Subject: [PATCH 09/11] [skip ci] Final polish with conclusions --- _docs/rfc/imports.md | 300 +++++++++++++++++++++++-------------------- 1 file changed, 160 insertions(+), 140 deletions(-) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index 96a0f36e48..b1fd008a2a 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -1,6 +1,6 @@ # Imports -**STATUS**: In proposal +**STATUS**: In development ## Background @@ -99,7 +99,131 @@ Given the problem statement, this RFC aims to propose a solution that addresses ## Proposed solution -The proposed solution is to introduce a new block `import` which replaces the functionality of `include`. We use a new +The proposed solution is an incremental improvement to the situation by implementing a series of increasingly more +expensive solutions. The following is a summary of the solutions to be built: + +- [Short term: read_terragrunt_config helper function](#read_terragrunt_config_helper_function) +- [Medium term: import blocks](#import-blocks) +- [Long term: single terragrunt.hcl file](#single-terragrunt-hcl-file-per-environment) + +### read_terragrunt_config helper function + +For this approach, we define a helper function that parses the relevant config and exposes it for use in the terragrunt +config. For example, the explicit triple import example in the [hierarchical variables use +case](#hierarchical-variables-included-across-multiple-terragrunt-hcl-files) can be implemented as: + +```hcl +locals { + root_config = read_terragrunt_config("../../../root.hcl") + region_config = read_terragrunt_config("../../region.hcl") + env_config = read_terragrunt_config("../env.hcl") +} + +inputs = merge( + local.root_config.inputs, + local.region_config.inputs, + local.env_config.inputs, + { + # args to module + }, +) +``` + +Pros: + +- Relatively simple implementation. It is significantly easier to add helper functions than it is to introduce new + blocks in terragrunt. +- Supports all the use cases described above. + +Cons: + +- Everything has to be explicit in the config. We can not support automatic merging capabilities when using helper + functions, as you can't manipulate the config in a helper. This can lead to very verbose configurations if one wants + to use this in place of `include`. + +Note that to take full advantage of this approach, all the blocks in `terragrunt.hcl` need to be redefined as attributes +so that we can use assignment to override them. + +Let's walk through a few more of the use cases: + +#### (read_terragrunt_config) Keeping remote state configuration DRY + +parent + +```hcl +remote_state { + backend = "s3" + config = { + bucket = "my-terraform-state" + key = "${path_relative_to_include()}/terraform.tfstate" + region = "us-east-1" + encrypt = true + dynamodb_table = "my-lock-table" + } +} +``` + +child + +```hcl +locals { + root_config = read_terragrunt_config("../../../root.hcl") +} + +remote_state { + backend = local.root_config.remote_state.backend + config = merge( + local.root_config.remote_state.config, + { + # The relative path between the root terragrunt directory and the current terragrunt file directory. + key = relpath(import.root.terragrunt_dir, get_terragrunt_dir()) + }, + ) +} +``` + +Note that we can't do: + +```hcl +remote_state = deep_merge(local.root_config.remote_state, { config = { key = relpath } }) +``` + +due to the fact that `remote_state` is a block and not an attribute. + + +#### (read_terragrunt_config) Reusing dependencies + +We can't reuse `dependency` blocks in this implementation because there is no way to auto merge the blocks. + +If `dependency` was instead an attribute, we could use the following alternative syntax: + +vpc_dependency_config.hcl + +```hcl +dependency = { + vpc = { + config_path = "/path/to/app/vpc/module" + } +} +``` + +```hcl +locals { + common_deps = read_terragrunt_config("../vpc_dependency_config.hcl") +} + +dependency = local.common_deps.dependency + +inputs = { + name = "unique-name" + vpc_id = dependency.vpc.outputs.vpc_id +} +``` + + +### import blocks + +This approach is to introduce a new block `import` which replaces the functionality of `include`. We use a new block instead of reusing `include` for backwards compatibility. As part of the implementation, `include` and the relevant functions (`get_parent_terragrunt_dir`, `path_relative_to_include`, and `path_relative_from_include`) will be deprecated and will throw a warning whenever someone uses it. @@ -135,15 +259,25 @@ The `import` block works as follows: attribute pointing to the target config. - Imports are not compatible with `include`. Having both blocks will cause a terragrunt syntax error. +Pros: + +- Supports all the use cases described above. + +Cons: + +- Since we don't allow `path_relative` functions, reusability is limited when compared to `include` (e.g the remote + state use case). This can be resolved if we implement the relevant `path_relative` functions for `import` blocks. + + Let's take a look at a few common use cases and how we might use `import` to address them: - [Hierarchical variables included across multiple terragrunt.hcl - files](#hierarchical-variables-included-across-multiple-terragrunt-hcl-files) -- [Reusing common variables](#reusing-common-variables) -- [Reusing dependencies](#reusing-dependencies) -- [Keeping remote state configuration DRY](#keeping-remote-state-configuration-dry) + files](#import-block-hierarchical-variables-included-across-multiple-terragrunt-hcl-files) +- [Reusing common variables](#import-block-reusing-common-variables) +- [Reusing dependencies](#import-block-reusing-dependencies) +- [Keeping remote state configuration DRY](#import-block-keeping-remote-state-configuration-dry) -### Hierarchical variables included across multiple terragrunt.hcl files +#### (import block) Hierarchical variables included across multiple terragrunt.hcl files Consider the following folder structure from the canonical example: @@ -298,7 +432,7 @@ The `merge = true` option is a convenience feature to make the child config less overridable configuration in your hierarchy. -### Reusing common variables +#### (import block) Reusing common variables Many resources are named with the region or environment that they belong to. A canonical terragrunt live configuration uses hierarchical variables to pass in the region and environment settings to terraform. However, there is no way in the @@ -324,7 +458,7 @@ inputs = merge( ``` -### Reusing dependencies +#### (import block) Reusing dependencies In the problem statement we discussed a use case where we want to pass in and merge the `vpc_id`. This can be implemented with `import` blocks in two ways. @@ -332,7 +466,7 @@ implemented with `import` blocks in two ways. - [Auto merge](#auto-merge) - [Explicit reference](#explicit-reference) -#### Auto merge +##### Auto merge With auto merge, you can `import` the configuration that declares the dependency and the input directly and have nothing in the child config. For example: @@ -381,7 +515,7 @@ inputs = { } ``` -#### Explicit reference +##### Explicit reference As an alternative to auto merge, you can also be explicit in referencing the inputs block in the import: @@ -414,7 +548,7 @@ inputs = { -### Keeping remote state configuration DRY +#### (import block) Keeping remote state configuration DRY A canonical use case for terragrunt is to share the remote state configuration. The classic example is having a root `terragrunt.hcl` configuration that specifies the `remote_state`, which is then imported using `include` to all the @@ -562,7 +696,7 @@ The advantage of this configuration is that we're only pulling the `remote_state blocks and attributes (e.g `inputs`) is not inherited to the child config. This was something that was not possible to do with `include`. -#### Why can't we use the merge function? +##### Why can't we use the merge function? In the above example, we have to explicitly declare the block. A natural extension of this pattern is to use the `merge` function directly on the `remote_state`: @@ -582,13 +716,7 @@ Unfortunately, this is a syntax error because `remote_state` is a block, and not blocks in this way. - - -## Alternatives - -### In consideration - -#### Single terragrunt.hcl file per environment +### Single terragrunt.hcl file per environment This implementation introduces a new block `module` that replaces `dependency`, `terraform`, and `inputs`. This approach is documented in [#759](https://github.com/gruntwork-io/terragrunt/issues/759). @@ -609,7 +737,7 @@ Cons: Let's walk through how each of the import use cases look like with this implementation: -##### Hierarchical variables +#### (single file) Hierarchical variables In this approach, a hierarchy of variables is unnecessary because all the blocks are defined in a single scope. However, depending on the scope of `locals`, certain things like reusing a repetitive variable becomes more challenging. @@ -665,7 +793,7 @@ each region, which is a better posture for disaster recovery. In this approach, `locals` is. However, the downside of this approach is that there will be some repetition to pull in the AWS account ID info across all the different regions. -##### Reusing common variables +#### (single file) Reusing common variables Reusing common variables depends on the scope of `locals`. If the scope of `locals` is shared across the environment, then you can define the common variable in `locals` blocks to share across the entier environment. If instead the scope @@ -699,131 +827,17 @@ module "mysql" { This example reuses the outputs of `module.vpc` across the two modules, which is the equivalent of having the `vpc` `dependency` block redefined in the two module configs. -##### Keeping remote state configuration DRY +#### (single file) Keeping remote state configuration DRY This example is covered in [the original issue that proposed this idea](https://github.com/gruntwork-io/terragrunt/issues/759). -#### hcldecode or read_terragrunt_config helper function - -Instead of defining a dedicated block for `import`, we could define a helper function that parses the relevant config. -For example, the explicit triple import example in the [hierarchical variables use -case](#hierarchical-variables-included-across-multiple-terragrunt-hcl-files) can be implemented as: - -```hcl -locals { - root_config = read_terragrunt_config("../../../root.hcl") - region_config = read_terragrunt_config("../../region.hcl") - env_config = read_terragrunt_config("../env.hcl") -} - -inputs = merge( - local.root_config.inputs, - local.region_config.inputs, - local.env_config.inputs, - { - # args to module - }, -) -``` - -Pros: -- Relatively simple implementation. It is significantly easier to add helper functions than it is to introduce new - blocks in terragrunt. -- Can address all the same use cases as `import` blocks. -Cons: - -- Everything has to be explicit in the config. We can not support automatic merging capabilities when using helper - functions, as you can't manipulate the config in a helper. This can lead to very verbose configurations if one wants - to use this in place of `include`. - -Note that to take full advantage of this approach, all the blocks in `terragrunt.hcl` need to be redefined as attributes -so that we can use assignment to override them. - -Let's walk through a few more of the use cases: - -##### Keeping remote state configuration DRY - -parent - -```hcl -remote_state { - backend = "s3" - config = { - bucket = "my-terraform-state" - key = "${path_relative_to_include()}/terraform.tfstate" - region = "us-east-1" - encrypt = true - dynamodb_table = "my-lock-table" - } -} -``` - -child - -```hcl -locals { - root_config = read_terragrunt_config("../../../root.hcl") -} - -remote_state { - backend = local.root_config.remote_state.backend - config = merge( - local.root_config.remote_state.config, - { - # The relative path between the root terragrunt directory and the current terragrunt file directory. - key = relpath(import.root.terragrunt_dir, get_terragrunt_dir()) - }, - ) -} -``` - -Note that we can't do: - -```hcl -remote_state = deep_merge(local.root_config.remote_state, { config = { key = relpath } }) -``` - -due to the fact that `remote_state` is a block and not an attribute. - - -##### Reusing dependencies - -We can't reuse `dependency` blocks in this implementation because there is no way to auto merge the blocks. - -If `dependency` was instead an attribute, we could use the following alternative syntax: - -vpc_dependency_config.hcl - -```hcl -dependency = { - vpc = { - config_path = "/path/to/app/vpc/module" - } -} -``` - -```hcl -locals { - common_deps = read_terragrunt_config("../vpc_dependency_config.hcl") -} - -dependency = local.common_deps.dependency - -inputs = { - name = "unique-name" - vpc_id = dependency.vpc.outputs.vpc_id -} -``` - - - -### Rejected +## Alternatives -#### Enhancing include with import semantics +### Enhancing include with import semantics Instead of having a dedicated block with the new functionality, we could enhance `include` to have all the semantics of `import`. This reduces complexity of the configuration by being able to recycle a very similar construct that already @@ -860,7 +874,7 @@ Cons: Ultimately, the potential for badly shooting yourself in the foot was enough to warrant a new block to start clean. -#### globals +### globals Globals was a potential solution to the problem proposed by the community. `globals` work the same way as `locals`, except they support merging across `include`. For example, to address the use case of [reusing common @@ -961,3 +975,9 @@ similar concerns: - [Request to allow more than one level of include](https://github.com/gruntwork-io/terragrunt/issues/303) - [Request to reference inputs from another config](https://github.com/gruntwork-io/terragrunt/issues/967) - [Partially override components of an input](https://github.com/gruntwork-io/terragrunt/issues/1011) + +Relevant PRs and Releases: + +- [PR for RFC](https://github.com/gruntwork-io/terragrunt/pull/1025) +- [PR for read_terragrunt_config](https://github.com/gruntwork-io/terragrunt/pull/1051) (released in + [v0.22.3](https://github.com/gruntwork-io/terragrunt/releases/tag/v0.22.3)) From 0fcc5d564167cc82345d85fa7ed789031e3486a5 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 26 Feb 2020 16:41:35 -0800 Subject: [PATCH 10/11] [skip ci] fix broken link --- _docs/rfc/imports.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index b1fd008a2a..dddc29b8b5 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -102,7 +102,7 @@ Given the problem statement, this RFC aims to propose a solution that addresses The proposed solution is an incremental improvement to the situation by implementing a series of increasingly more expensive solutions. The following is a summary of the solutions to be built: -- [Short term: read_terragrunt_config helper function](#read_terragrunt_config_helper_function) +- [Short term: read_terragrunt_config helper function](#read_terragrunt_config-helper-function) - [Medium term: import blocks](#import-blocks) - [Long term: single terragrunt.hcl file](#single-terragrunt-hcl-file-per-environment) From 73e7edf06d43f710e460eb318677b88a6bc858e9 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Wed, 26 Feb 2020 16:42:41 -0800 Subject: [PATCH 11/11] [skip ci] fix more broken links --- _docs/rfc/imports.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_docs/rfc/imports.md b/_docs/rfc/imports.md index dddc29b8b5..6f508a2d50 100644 --- a/_docs/rfc/imports.md +++ b/_docs/rfc/imports.md @@ -104,7 +104,7 @@ expensive solutions. The following is a summary of the solutions to be built: - [Short term: read_terragrunt_config helper function](#read_terragrunt_config-helper-function) - [Medium term: import blocks](#import-blocks) -- [Long term: single terragrunt.hcl file](#single-terragrunt-hcl-file-per-environment) +- [Long term: single terragrunt.hcl file](#single-terragrunthcl-file-per-environment) ### read_terragrunt_config helper function @@ -272,7 +272,7 @@ Cons: Let's take a look at a few common use cases and how we might use `import` to address them: - [Hierarchical variables included across multiple terragrunt.hcl - files](#import-block-hierarchical-variables-included-across-multiple-terragrunt-hcl-files) + files](#import-block-hierarchical-variables-included-across-multiple-terragrunthcl-files) - [Reusing common variables](#import-block-reusing-common-variables) - [Reusing dependencies](#import-block-reusing-dependencies) - [Keeping remote state configuration DRY](#import-block-keeping-remote-state-configuration-dry)