Skip to content

Latest commit

 

History

History
204 lines (169 loc) · 19.6 KB

README.md

File metadata and controls

204 lines (169 loc) · 19.6 KB

Description

This module creates a Google Kubernetes Engine (GKE) cluster.

NOTE: This is an experimental module and the functionality and documentation will likely be updated in the near future. This module has only been tested in limited capacity.

Example

The following example creates a GKE cluster and a VPC designed to work with GKE. See VPC Network section for more information about network requirements.

  - id: network1
    source: modules/network/vpc
    settings:
      subnetwork_name: gke-subnet
      secondary_ranges:
        gke-subnet:
        - range_name: pods
          ip_cidr_range: 10.4.0.0/14
        - range_name: services
          ip_cidr_range: 10.0.32.0/20

  - id: gke_cluster
    source: modules/scheduler/gke-cluster
    use: [network1]

Also see a full GKE example blueprint.

VPC Network

This module is configured to create a VPC-native cluster. This means that alias IPs are used and that the subnetwork requires secondary ranges for pods and services. In the example shown above these secondary ranges are created in the VPC module. By default the gke-cluster module will look for ranges with the names pods and services. These names can be configured using the pods_ip_range_name and services_ip_range_name settings.

Multi-networking

To enable Multi-networking, pass multivpc module to gke-cluster module as described in example below. Passing a multivpc module enables multi networking and Dataplane V2 on the cluster.

  - id: network
    source: modules/network/vpc
    settings:
      subnetwork_name: gke-subnet
      secondary_ranges:
        gke-subnet:
        - range_name: pods
          ip_cidr_range: 10.4.0.0/14
        - range_name: services
          ip_cidr_range: 10.0.32.0/20

  - id: multinetwork
    source: modules/network/multivpc
    settings:
      network_name_prefix: multivpc-net
      network_count: 8
      global_ip_address_range: 172.16.0.0/12
      subnetwork_cidr_suffix: 16

  - id: gke-cluster
    source: modules/scheduler/gke-cluster
    use: [network, multinetwork] ## enables multi networking and Dataplane V2 on cluster
    settings:
      cluster_name: $(vars.deployment_name)

Find an example of multi networking in GKE here.

Cluster Limitations

The current implementations has the following limitations:

  • Autopilot is disabled
  • Auto-provisioning of new node pools is disabled
  • Network policies are not supported
  • General addon configuration is not supported
  • Only regional cluster is supported

License

Copyright 2022 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Name Version
terraform >= 1.3
google > 5.0
google-beta > 5.0
kubernetes ~> 2.23

Providers

Name Version
google > 5.0
google-beta > 5.0

Modules

Name Source Version
kubectl_apply ../../management/kubectl-apply n/a
workload_identity terraform-google-modules/kubernetes-engine/google//modules/workload-identity ~> 34.0

Resources

Name Type
google-beta_google_container_cluster.gke_cluster resource
google-beta_google_container_node_pool.system_node_pools resource
google_client_config.default data source
google_project.project data source

Inputs

Name Description Type Default Required
additional_networks Additional network interface details for GKE, if any. Providing additional networks enables multi networking and creates relevat network objects on the cluster. 
list(object({
    network            = string
    subnetwork         = string
    subnetwork_project = string
    network_ip         = string
    nic_type           = string
    stack_type         = string
    queue_count        = number
    access_config = list(object({
      nat_ip       = string
      network_tier = string
    }))
    ipv6_access_config = list(object({
      network_tier = string
    }))
    alias_ip_range = list(object({
      ip_cidr_range         = string
      subnetwork_range_name = string
    }))
  }))
 [] no
authenticator_security_group The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format [email protected] string null no
autoscaling_profile (Beta) Optimize for utilization or availability when deciding to remove nodes. Can be BALANCED or OPTIMIZE_UTILIZATION. string "OPTIMIZE_UTILIZATION" no
cluster_availability_type Type of cluster availability. Possible values are: {REGIONAL, ZONAL} string "REGIONAL" no
cluster_reference_type How the google_container_node_pool.system_node_pools refers to the cluster. Possible values are: {SELF_LINK, NAME} string "SELF_LINK" no
configure_workload_identity_sa When true, a kubernetes service account will be created and bound using workload identity to the service account used to create the cluster. bool false no
default_max_pods_per_node The default maximum number of pods per node in this cluster. number null no
deployment_name Name of the HPC deployment. Used in the GKE cluster name by default and can be configured with prefix_with_deployment_name. string n/a yes
enable_dataplane_v2 Enables Dataplane v2. This setting is immutable on clusters. If null, will default to false unless using multi-networking, in which case it will default to true bool null no
enable_dcgm_monitoring Enable GKE to collect DCGM metrics bool false no
enable_filestore_csi The status of the Filestore Container Storage Interface (CSI) driver addon, which allows the usage of filestore instance as volumes. bool false no
enable_gcsfuse_csi The status of the GCSFuse Filestore Container Storage Interface (CSI) driver addon, which allows the usage of a gcs bucket as volumes. bool false no
enable_master_global_access Whether the cluster master is accessible globally (from any region) or only within the same region as the private endpoint. bool false no
enable_multi_networking Enables multi networking (Requires GKE Enterprise). This setting is immutable on clusters and enables Dataplane V2. If null, will determine state based on if additional_networks are passed in. bool null no
enable_node_local_dns_cache Enable GKE NodeLocal DNSCache addon to improve DNS lookup latency bool false no
enable_parallelstore_csi The status of the Google Compute Engine Parallelstore Container Storage Interface (CSI) driver addon, which allows the usage of a parallelstore as volumes. bool false no
enable_persistent_disk_csi The status of the Google Compute Engine Persistent Disk Container Storage Interface (CSI) driver addon, which allows the usage of a PD as volumes. bool true no
enable_private_endpoint (Beta) Whether the master's internal IP address is used as the cluster endpoint. bool true no
enable_private_ipv6_google_access The private IPv6 google access type for the VMs in this subnet. bool true no
enable_private_nodes (Beta) Whether nodes have internal IP addresses only. bool true no
gcp_public_cidrs_access_enabled Whether the cluster master is accessible via all the Google Compute Engine Public IPs. To view this list of IP addresses look here https://cloud.google.com/compute/docs/faq#find_ip_range bool false no
labels GCE resource labels to be applied to resources. Key-value pairs. map(string) n/a yes
maintenance_exclusions List of maintenance exclusions. A cluster can have up to three. 
list(object({
    name            = string
    start_time      = string
    end_time        = string
    exclusion_scope = string
  }))
 [] no
maintenance_start_time Start time for daily maintenance operations. Specified in GMT with HH:MM format. string "09:00" no
master_authorized_networks External network that can access Kubernetes master through HTTPS. Must be specified in CIDR notation. 
list(object({
    cidr_block   = string
    display_name = string
  }))
 [] no
master_ipv4_cidr_block (Beta) The IP range in CIDR notation to use for the hosted master network. string "172.16.0.32/28" no
min_master_version The minimum version of the master. If unset, the cluster's version will be set by GKE to the version of the most recent official release. string null no
name_suffix Custom cluster name postpended to the deployment_name. See prefix_with_deployment_name. string "" no
network_id The ID of the GCE VPC network to host the cluster given in the format: projects/<project_id>/global/networks/<network_name>. string n/a yes
networking_mode Determines whether alias IPs or routes will be used for pod IPs in the cluster. Options are VPC_NATIVE or ROUTES. VPC_NATIVE enables IP aliasing. The default is VPC_NATIVE. string "VPC_NATIVE" no
pods_ip_range_name The name of the secondary subnet ip range to use for pods. string "pods" no
prefix_with_deployment_name If true, cluster name will be prefixed by deployment_name (ex: <deployment_name>-<name_suffix>). bool true no
project_id The project ID to host the cluster in. string n/a yes
region The region to host the cluster in. string n/a yes
release_channel The release channel of this cluster. Accepted values are UNSPECIFIED, RAPID, REGULAR and STABLE. string "UNSPECIFIED" no
service_account DEPRECATED: use service_account_email and scopes. 
object({
    email  = string,
    scopes = set(string)
  })
 null no
service_account_email Service account e-mail address to use with the system node pool string null no
service_account_scopes Scopes to to use with the system node pool. set(string) 
[
  "https://www.googleapis.com/auth/cloud-platform"
]
 no
services_ip_range_name The name of the secondary subnet range to use for services. string "services" no
subnetwork_self_link The self link of the subnetwork to host the cluster in. string n/a yes
system_node_pool_disk_size_gb Size of disk for each node of the system node pool. number 100 no
system_node_pool_disk_type Disk type for each node of the system node pool. string null no
system_node_pool_enable_secure_boot Enable secure boot for the nodes. Keep enabled unless custom kernel modules need to be loaded. See here for more info. bool true no
system_node_pool_enabled Create a system node pool. bool true no
system_node_pool_image_type The default image type used by NAP once a new node pool is being created. Use either COS_CONTAINERD or UBUNTU_CONTAINERD. string "COS_CONTAINERD" no
system_node_pool_kubernetes_labels Kubernetes labels to be applied to each node in the node group. Key-value pairs.
(The kubernetes.io/ and k8s.io/ prefixes are reserved by Kubernetes Core components and cannot be specified)		 map(string) null no
system_node_pool_machine_type Machine type for the system node pool. string "e2-standard-4" no
system_node_pool_name Name of the system node pool. string "system" no
system_node_pool_node_count The total min and max nodes to be maintained in the system node pool. 
object({
    total_min_nodes = number
    total_max_nodes = number
  })
 
{
  "total_max_nodes": 10,
  "total_min_nodes": 2
}
 no
system_node_pool_taints Taints to be applied to the system node pool. 
list(object({
    key    = string
    value  = any
    effect = string
  }))
 
[
  {
    "effect": "NO_SCHEDULE",
    "key": "components.gke.io/gke-managed-components",
    "value": true
  }
]
 no
timeout_create Timeout for creating a node pool string null no
timeout_update Timeout for updating a node pool string null no
zone Zone for a zonal cluster. string null no

Outputs

Name Description
cluster_id An identifier for the resource with format projects/{{project_id}}/locations/{{region}}/clusters/{{name}}.
gke_cluster_exists A static flag that signals to downstream modules that a cluster has been created. Needed by community/modules/scripts/kubernetes-operations.
gke_version GKE cluster's version.
instructions Instructions on how to connect to the created cluster.
k8s_service_account_name Name of k8s service account.