[Karpenter] Minor cleanups (#1056)

modules/eks/karpenter-node-pool/README.md

@@ -203,7 +203,7 @@ components:
 | <a name="input_labels_as_tags"></a> [labels\_as\_tags](#input\_labels\_as\_tags) | Set of labels (ID elements) to include as tags in the `tags` output.<br>Default is to include all labels.<br>Tags with empty values will not be included in the `tags` output.<br>Set to `[]` to suppress all generated tags.<br>**Notes:**<br>  The value of the `name` tag, if included, will be the `id`, not the `name`.<br>  Unlike other `null-label` inputs, the initial setting of `labels_as_tags` cannot be<br>  changed in later chained modules. Attempts to change it will be silently ignored. | `set(string)` | <pre>[<br>  "default"<br>]</pre> | no |
 | <a name="input_name"></a> [name](#input\_name) | ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'.<br>This is the only ID element not also included as a `tag`.<br>The "name" tag is set to the full `id` string. There is no tag with the value of the `name` input. | `string` | `null` | no |
 | <a name="input_namespace"></a> [namespace](#input\_namespace) | ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique | `string` | `null` | no |
-| <a name="input_node_pools"></a> [node\_pools](#input\_node\_pools) | Configuration for node pools. See code for details. | <pre>map(object({<br>    # The name of the Karpenter provisioner. The map key is used if this is not set.<br>    name = optional(string)<br>    # Whether to place EC2 instances launched by Karpenter into VPC private subnets. Set it to `false` to use public subnets.<br>    private_subnets_enabled = bool<br>    # The Disruption spec controls how Karpenter scales down the node group.<br>    # See the example (sadly not the specific `spec.disruption` documentation) at https://karpenter.sh/docs/concepts/nodepools/ for details<br>    disruption = optional(object({<br>      # Describes which types of Nodes Karpenter should consider for consolidation.<br>      # If using 'WhenUnderutilized', Karpenter will consider all nodes for consolidation and attempt to remove or<br>      # replace Nodes when it discovers that the Node is underutilized and could be changed to reduce cost.<br>      # If using `WhenEmpty`, Karpenter will only consider nodes for consolidation that contain no workload pods.<br>      consolidation_policy = optional(string, "WhenUnderutilized")<br><br>      # The amount of time Karpenter should wait after discovering a consolidation decision (`go` duration string, s|m|h).<br>      # This value can currently (v0.36.0) only be set when the consolidationPolicy is 'WhenEmpty'.<br>      # You can choose to disable consolidation entirely by setting the string value 'Never' here.<br>      # Earlier versions of Karpenter called this field `ttl_seconds_after_empty`.<br>      consolidate_after = optional(string)<br><br>      # The amount of time a Node can live on the cluster before being removed (`go` duration string, s|m|h).<br>      # You can choose to disable expiration entirely by setting the string value 'Never' here.<br>      # This module sets a default of 336 hours (14 days), while the Karpenter default is 720 hours (30 days).<br>      # Note that Karpenter calls this field "expiresAfter", and earlier versions called it `ttl_seconds_until_expired`,<br>      # but we call it "max_instance_lifetime" to match the corresponding field in EC2 Auto Scaling Groups.<br>      max_instance_lifetime = optional(string, "336h")<br><br>      # Budgets control the the maximum number of NodeClaims owned by this NodePool that can be terminating at once.<br>      # See https://karpenter.sh/docs/concepts/disruption/#disruption-budgets for details.<br>      # A percentage is the percentage of the total number of active, ready nodes not being deleted, rounded up.<br>      # If there are multiple active budgets, Karpenter uses the most restrictive value.<br>      # If left undefined, this will default to one budget with a value of nodes: 10%.<br>      # Note that budgets do not prevent or limit involuntary terminations.<br>      # Example:<br>      #   On Weekdays during business hours, don't do any deprovisioning.<br>      #     budgets = {<br>      #       schedule = "0 9 * * mon-fri"<br>      #       duration = 8h<br>      #       nodes    = "0"<br>      #     }<br>      budgets = optional(list(object({<br>        # The schedule specifies when a budget begins being active, using extended cronjob syntax.<br>        # See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#schedule-syntax for syntax details.<br>        # Timezones are not supported. This field is required if Duration is set.<br>        schedule = optional(string)<br>        # Duration determines how long a Budget is active after each Scheduled start.<br>        # If omitted, the budget is always active. This is required if Schedule is set.<br>        # Must be a whole number of minutes and hours, as cron does not work in seconds,<br>        # but since Go's `duration.String()` always adds a "0s" at the end, that is allowed.<br>        duration = optional(string)<br>        # The percentage or number of nodes that Karpenter can scale down during the budget.<br>        nodes = string<br>      })), [])<br>    }), {})<br>    # Karpenter provisioner total CPU limit for all pods running on the EC2 instances launched by Karpenter<br>    total_cpu_limit = string<br>    # Karpenter provisioner total memory limit for all pods running on the EC2 instances launched by Karpenter<br>    total_memory_limit = string<br>    # Set a weight for this node pool.<br>    # See https://karpenter.sh/docs/concepts/scheduling/#weighted-nodepools<br>    weight = optional(number, 50)<br>    # Karpenter provisioner taints configuration. See https://aws.github.io/aws-eks-best-practices/karpenter/#create-provisioners-that-are-mutually-exclusive for more details<br>    taints = optional(list(object({<br>      key    = string<br>      effect = string<br>      value  = string<br>    })))<br>    startup_taints = optional(list(object({<br>      key    = string<br>      effect = string<br>      value  = string<br>    })))<br>    # Karpenter node metadata options. See https://karpenter.sh/docs/concepts/nodeclasses/#specmetadataoptions for more details<br>    metadata_options = optional(object({<br>      httpEndpoint            = optional(string, "enabled")<br>      httpProtocolIPv6        = optional(string, "disabled")<br>      httpPutResponseHopLimit = optional(number, 2)<br>      # httpTokens can be either "required" or "optional"<br>      httpTokens = optional(string, "required")<br>    }), {})<br>    # The AMI used by Karpenter provisioner when provisioning nodes. Based on the value set for amiFamily, Karpenter will automatically query for the appropriate EKS optimized AMI via AWS Systems Manager (SSM)<br>    ami_family = string<br>    # Karpenter nodes block device mappings. Controls the Elastic Block Storage volumes that Karpenter attaches to provisioned nodes.<br>    # Karpenter uses default block device mappings for the AMI Family specified.<br>    # For example, the Bottlerocket AMI Family defaults with two block device mappings,<br>    # and normally you only want to scale `/dev/xvdb` where Containers and there storage are stored.<br>    # Most other AMIs only have one device mapping at `/dev/xvda`.<br>    # See https://karpenter.sh/docs/concepts/nodeclasses/#specblockdevicemappings for more details<br>    block_device_mappings = list(object({<br>      deviceName = string<br>      ebs = optional(object({<br>        volumeSize          = string<br>        volumeType          = string<br>        deleteOnTermination = optional(bool, true)<br>        encrypted           = optional(bool, true)<br>        iops                = optional(number)<br>        kmsKeyID            = optional(string, "alias/aws/ebs")<br>        snapshotID          = optional(string)<br>        throughput          = optional(number)<br>      }))<br>    }))<br>    # Set acceptable (In) and unacceptable (Out) Kubernetes and Karpenter values for node provisioning based on Well-Known Labels and cloud-specific settings. These can include instance types, zones, computer architecture, and capacity type (such as AWS spot or on-demand). See https://karpenter.sh/v0.18.0/provisioner/#specrequirements for more details<br>    requirements = list(object({<br>      key      = string<br>      operator = string<br>      # Operators like "Exists" and "DoesNotExist" do not require a value<br>      values = optional(list(string))<br>    }))<br>  }))</pre> | n/a | yes |
+| <a name="input_node_pools"></a> [node\_pools](#input\_node\_pools) | Configuration for node pools. See code for details. | <pre>map(object({<br>    # The name of the Karpenter provisioner. The map key is used if this is not set.<br>    name = optional(string)<br>    # Whether to place EC2 instances launched by Karpenter into VPC private subnets. Set it to `false` to use public subnets.<br>    private_subnets_enabled = bool<br>    # The Disruption spec controls how Karpenter scales down the node group.<br>    # See the example (sadly not the specific `spec.disruption` documentation) at https://karpenter.sh/docs/concepts/nodepools/ for details<br>    disruption = optional(object({<br>      # Describes which types of Nodes Karpenter should consider for consolidation.<br>      # If using 'WhenUnderutilized', Karpenter will consider all nodes for consolidation and attempt to remove or<br>      # replace Nodes when it discovers that the Node is underutilized and could be changed to reduce cost.<br>      # If using `WhenEmpty`, Karpenter will only consider nodes for consolidation that contain no workload pods.<br>      consolidation_policy = optional(string, "WhenUnderutilized")<br><br>      # The amount of time Karpenter should wait after discovering a consolidation decision (`go` duration string, s|m|h).<br>      # This value can currently (v0.36.0) only be set when the consolidationPolicy is 'WhenEmpty'.<br>      # You can choose to disable consolidation entirely by setting the string value 'Never' here.<br>      # Earlier versions of Karpenter called this field `ttl_seconds_after_empty`.<br>      consolidate_after = optional(string)<br><br>      # The amount of time a Node can live on the cluster before being removed (`go` duration string, s|m|h).<br>      # You can choose to disable expiration entirely by setting the string value 'Never' here.<br>      # This module sets a default of 336 hours (14 days), while the Karpenter default is 720 hours (30 days).<br>      # Note that Karpenter calls this field "expiresAfter", and earlier versions called it `ttl_seconds_until_expired`,<br>      # but we call it "max_instance_lifetime" to match the corresponding field in EC2 Auto Scaling Groups.<br>      max_instance_lifetime = optional(string, "336h")<br><br>      # Budgets control the the maximum number of NodeClaims owned by this NodePool that can be terminating at once.<br>      # See https://karpenter.sh/docs/concepts/disruption/#disruption-budgets for details.<br>      # A percentage is the percentage of the total number of active, ready nodes not being deleted, rounded up.<br>      # If there are multiple active budgets, Karpenter uses the most restrictive value.<br>      # If left undefined, this will default to one budget with a value of nodes: 10%.<br>      # Note that budgets do not prevent or limit involuntary terminations.<br>      # Example:<br>      #   On Weekdays during business hours, don't do any deprovisioning.<br>      #     budgets = {<br>      #       schedule = "0 9 * * mon-fri"<br>      #       duration = 8h<br>      #       nodes    = "0"<br>      #     }<br>      budgets = optional(list(object({<br>        # The schedule specifies when a budget begins being active, using extended cronjob syntax.<br>        # See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#schedule-syntax for syntax details.<br>        # Timezones are not supported. This field is required if Duration is set.<br>        schedule = optional(string)<br>        # Duration determines how long a Budget is active after each Scheduled start.<br>        # If omitted, the budget is always active. This is required if Schedule is set.<br>        # Must be a whole number of minutes and hours, as cron does not work in seconds,<br>        # but since Go's `duration.String()` always adds a "0s" at the end, that is allowed.<br>        duration = optional(string)<br>        # The percentage or number of nodes that Karpenter can scale down during the budget.<br>        nodes = string<br>      })), [])<br>    }), {})<br>    # Karpenter provisioner total CPU limit for all pods running on the EC2 instances launched by Karpenter<br>    total_cpu_limit = string<br>    # Karpenter provisioner total memory limit for all pods running on the EC2 instances launched by Karpenter<br>    total_memory_limit = string<br>    # Set a weight for this node pool.<br>    # See https://karpenter.sh/docs/concepts/scheduling/#weighted-nodepools<br>    weight      = optional(number, 50)<br>    labels      = optional(map(string))<br>    annotations = optional(map(string))<br>    # Karpenter provisioner taints configuration. See https://aws.github.io/aws-eks-best-practices/karpenter/#create-provisioners-that-are-mutually-exclusive for more details<br>    taints = optional(list(object({<br>      key    = string<br>      effect = string<br>      value  = string<br>    })))<br>    startup_taints = optional(list(object({<br>      key    = string<br>      effect = string<br>      value  = string<br>    })))<br>    # Karpenter node metadata options. See https://karpenter.sh/docs/concepts/nodeclasses/#specmetadataoptions for more details<br>    metadata_options = optional(object({<br>      httpEndpoint            = optional(string, "enabled")<br>      httpProtocolIPv6        = optional(string, "disabled")<br>      httpPutResponseHopLimit = optional(number, 2)<br>      # httpTokens can be either "required" or "optional"<br>      httpTokens = optional(string, "required")<br>    }), {})<br>    # The AMI used by Karpenter provisioner when provisioning nodes. Based on the value set for amiFamily, Karpenter will automatically query for the appropriate EKS optimized AMI via AWS Systems Manager (SSM)<br>    ami_family = string<br>    # Karpenter nodes block device mappings. Controls the Elastic Block Storage volumes that Karpenter attaches to provisioned nodes.<br>    # Karpenter uses default block device mappings for the AMI Family specified.<br>    # For example, the Bottlerocket AMI Family defaults with two block device mappings,<br>    # and normally you only want to scale `/dev/xvdb` where Containers and there storage are stored.<br>    # Most other AMIs only have one device mapping at `/dev/xvda`.<br>    # See https://karpenter.sh/docs/concepts/nodeclasses/#specblockdevicemappings for more details<br>    block_device_mappings = list(object({<br>      deviceName = string<br>      ebs = optional(object({<br>        volumeSize          = string<br>        volumeType          = string<br>        deleteOnTermination = optional(bool, true)<br>        encrypted           = optional(bool, true)<br>        iops                = optional(number)<br>        kmsKeyID            = optional(string, "alias/aws/ebs")<br>        snapshotID          = optional(string)<br>        throughput          = optional(number)<br>      }))<br>    }))<br>    # Set acceptable (In) and unacceptable (Out) Kubernetes and Karpenter values for node provisioning based on Well-Known Labels and cloud-specific settings. These can include instance types, zones, computer architecture, and capacity type (such as AWS spot or on-demand). See https://karpenter.sh/v0.18.0/provisioner/#specrequirements for more details<br>    requirements = list(object({<br>      key      = string<br>      operator = string<br>      # Operators like "Exists" and "DoesNotExist" do not require a value<br>      values = optional(list(string))<br>    }))<br>  }))</pre> | n/a | yes |
 | <a name="input_regex_replace_chars"></a> [regex\_replace\_chars](#input\_regex\_replace\_chars) | Terraform regular expression (regex) string.<br>Characters matching the regex will be removed from the ID elements.<br>If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits. | `string` | `null` | no |
 | <a name="input_region"></a> [region](#input\_region) | AWS Region | `string` | n/a | yes |
 | <a name="input_stage"></a> [stage](#input\_stage) | ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release' | `string` | `null` | no |

modules/eks/karpenter-node-pool/main.tf

@@ -39,6 +39,10 @@ resource "kubernetes_manifest" "node_pool" {
         }
       )
       template = {
+        metadata = {
+          labels      = each.value.labels
+          annotations = each.value.annotations
+        }
         spec = merge({
           nodeClassRef = {
             apiVersion = "karpenter.k8s.aws/v1beta1"

modules/eks/karpenter-node-pool/variables.tf

@@ -70,7 +70,9 @@ variable "node_pools" {
     total_memory_limit = string
     # Set a weight for this node pool.
     # See https://karpenter.sh/docs/concepts/scheduling/#weighted-nodepools
-    weight = optional(number, 50)
+    weight      = optional(number, 50)
+    labels      = optional(map(string))
+    annotations = optional(map(string))
     # Karpenter provisioner taints configuration. See https://aws.github.io/aws-eks-best-practices/karpenter/#create-provisioners-that-are-mutually-exclusive for more details
     taints = optional(list(object({
       key    = string

modules/eks/karpenter/README.md

@@ -399,7 +399,8 @@ For more details on the CRDs, see:
 | <a name="input_kube_exec_auth_enabled"></a> [kube\_exec\_auth\_enabled](#input\_kube\_exec\_auth\_enabled) | If `true`, use the Kubernetes provider `exec` feature to execute `aws eks get-token` to authenticate to the EKS cluster.<br>Disabled by `kubeconfig_file_enabled`, overrides `kube_data_auth_enabled`. | `bool` | `true` | no |
 | <a name="input_kube_exec_auth_role_arn"></a> [kube\_exec\_auth\_role\_arn](#input\_kube\_exec\_auth\_role\_arn) | The role ARN for `aws eks get-token` to use | `string` | `""` | no |
 | <a name="input_kube_exec_auth_role_arn_enabled"></a> [kube\_exec\_auth\_role\_arn\_enabled](#input\_kube\_exec\_auth\_role\_arn\_enabled) | If `true`, pass `kube_exec_auth_role_arn` as the role ARN to `aws eks get-token` | `bool` | `true` | no |
-| <a name="input_kubeconfig_context"></a> [kubeconfig\_context](#input\_kubeconfig\_context) | Context to choose from the Kubernetes kube config file | `string` | `""` | no |
+| <a name="input_kubeconfig_context"></a> [kubeconfig\_context](#input\_kubeconfig\_context) | Context to choose from the Kubernetes config file.<br>If supplied, `kubeconfig_context_format` will be ignored. | `string` | `""` | no |
+| <a name="input_kubeconfig_context_format"></a> [kubeconfig\_context\_format](#input\_kubeconfig\_context\_format) | A format string to use for creating the `kubectl` context name when<br>`kubeconfig_file_enabled` is `true` and `kubeconfig_context` is not supplied.<br>Must include a single `%s` which will be replaced with the cluster name. | `string` | `""` | no |
 | <a name="input_kubeconfig_exec_auth_api_version"></a> [kubeconfig\_exec\_auth\_api\_version](#input\_kubeconfig\_exec\_auth\_api\_version) | The Kubernetes API version of the credentials returned by the `exec` auth plugin | `string` | `"client.authentication.k8s.io/v1beta1"` | no |
 | <a name="input_kubeconfig_file"></a> [kubeconfig\_file](#input\_kubeconfig\_file) | The Kubernetes provider `config_path` setting to use when `kubeconfig_file_enabled` is `true` | `string` | `""` | no |
 | <a name="input_kubeconfig_file_enabled"></a> [kubeconfig\_file\_enabled](#input\_kubeconfig\_file\_enabled) | If `true`, configure the Kubernetes provider with `kubeconfig_file` and use that kubeconfig file for authenticating to the EKS cluster | `bool` | `false` | no |