terraform/deployments: add ci-pr workers

this commit is intended to allow the [`prs` pipeline] to co-exist with `nci` in the `hush-house` gke cluster. it does so by: - adding a node pool to the cluster (`ci-workers-pr`) whose size is the same as we had previously in our BOSH deployment - adding a deployment (`ci-pr`) that puts workers in such node pool. As such deployment is supposed to run untrusted workloads, we needed a way of restricting the network access that this deployment could have in order to avoid lateral movements in the internal net (in the case of our BOSH environment, we had a totally different network - in k8s, we'd need to be in a separate cluster if we wanted to go w/ the same approach of different nets). This led us to enabling the enforcement of network policies in the cluster, and creating a policy for `ci-pr` that would target the pods delpoyed by it, effectively blocking internal connectivity to anything we didn't want (i.e., anything that's not ci's TSA). ps.: these changes have already been applied. [`prs` pipeline]: https://nci.concourse-ci.org/teams/main/pipelines/prs Signed-off-by: Denise Yu <[email protected]> Signed-off-by: Ciro S. Costa <[email protected]>
concourse · Nov 22, 2019 · d462342 · d462342
1 parent c3cff2b
commit d462342
Show file tree

Hide file tree

Showing 9 changed files with 156 additions and 2 deletions.
diff --git a/deployments/with-creds/ci-pr/Chart.yaml b/deployments/with-creds/ci-pr/Chart.yaml
@@ -0,0 +1,12 @@
+name: ci
+apiVersion: v1
+version: 0.1.1
+appVersion: 0.0.139
+description: A Concourse deployment used by Concourse to ship Concourse
+maintainers:
+- name: cirocosta
+  email: [email protected]
+- name: kmannem
+  email: [email protected]
+- name: svohra
+  email: [email protected]
diff --git a/deployments/with-creds/ci-pr/README.md b/deployments/with-creds/ci-pr/README.md
@@ -0,0 +1,32 @@
+# ci-pr
+
+The `ci-pr` deployment deploys the workers used to run untrusted workloads that
+when someeone pushes a PR to `concourse/concourse`.
+
+It relies solely on the [Concourse chart](https://github.com/concourse/concourse-chart).
+
+
+## Restrictions
+
+Being a deployment that's supposed to accept untrusted workloads (from community
+PRs), network constraints are set to this deployment - see [`NetworkPolicy`].
+
+[`NetworkPolicy`]: ./templates/network-policy.yaml
+
+
+## Deploying
+
+To deploy these workers, run `make deploy-ci-pr` from `/deployments/with-creds`.
+
+If you want to force a rolling update (recreate all pods), say after updating
+secrets, increment the `rollingUpdate` annotation declared in [`values.yaml`].
+
+[`values.yaml`]: ./values.yaml
+
+
+## Debugging
+
+Metrics, logs, and debug endpoints work the same as for the [`ci`] deployment.
+Check that deployment's README to know more.
+
+[`ci`]: ../ci
diff --git a/deployments/with-creds/ci-pr/requirements.lock b/deployments/with-creds/ci-pr/requirements.lock
@@ -0,0 +1,6 @@
+dependencies:
+- name: concourse
+  repository: https://kubernetes-charts.storage.googleapis.com/
+  version: 8.2.7
+digest: sha256:cb825c3e6f195eb63ddf2401a73ce01bb9852d28cc8bb309b40e9f1393eaec00
+generated: "2019-10-16T08:08:05.20907-04:00"
diff --git a/deployments/with-creds/ci-pr/requirements.yaml b/deployments/with-creds/ci-pr/requirements.yaml
@@ -0,0 +1,5 @@
+---
+dependencies:
+- name: concourse
+  version: 8.2.7
+  repository: https://kubernetes-charts.storage.googleapis.com/
diff --git a/deployments/with-creds/ci-pr/templates/NOTES.txt b/deployments/with-creds/ci-pr/templates/NOTES.txt
@@ -0,0 +1 @@
+Take off!
diff --git a/deployments/with-creds/ci-pr/templates/network-policy.yaml b/deployments/with-creds/ci-pr/templates/network-policy.yaml
@@ -0,0 +1,30 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: only-external
+  namespace: {{ .Release.Name }}
+spec:
+  podSelector:
+    matchLabels:
+      release: {{ .Release.Name }}
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress: []                 # disallow any ingress
+  egress:
+    - ports:                  # allow dns
+        - port: 53
+          protocol: TCP
+        - port: 53
+          protocol: UDP
+    - to:
+        - namespaceSelector:  # allow pods in `ci` namespace laeblled `ci-web`
+            matchLabels:
+              release: ci
+          podSelector:
+            matchLabels:
+              app: ci-web
+        - ipBlock:
+            cidr: 0.0.0.0/0   # allow any out
+            except:
+              - 10.0.0.0/8    # except internal comms
diff --git a/deployments/with-creds/ci-pr/values.yaml b/deployments/with-creds/ci-pr/values.yaml
@@ -0,0 +1,46 @@
+postgresql:
+  enabled: false
+
+concourse:
+  image: concourse/concourse
+  imageDigest: sha256:e93a0149e3efe9186e403a188066c93a96ea2f223b24d49952275b63dc3e2c4d
+
+  postgresql:
+    enabled: false
+
+  web:
+    enabled: false
+
+  persistence:
+    worker:
+      storageClass: ssd
+      size: 750Gi
+
+  worker:
+    replicas: 3
+    terminationGracePeriodSeconds: 3600
+    livenessProbe:
+      periodSeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 45
+    nodeSelector: { cloud.google.com/gke-nodepool: ci-workers-pr }
+    hardAntiAffinity: true
+    env:
+    - name: CONCOURSE_GARDEN_NETWORK_POOL
+      value: "10.254.0.0/16"
+    - name: CONCOURSE_GARDEN_MAX_CONTAINERS
+      value: "500"
+    - name: CONCOURSE_GARDEN_DENY_NETWORK
+      value: "169.254.169.254/32"
+    resources:
+      limits:   { cpu: 7500m, memory: 14Gi }
+      requests: { cpu: 0m,    memory: 0Gi  }
+
+  concourse:
+    worker:
+      rebalanceInterval: 2h
+      baggageclaim: { driver: overlay }
+      tag: "pr"
+      healthcheckTimeout: 40s
+      tsa:
+        hosts: ['ci-web.ci.svc.cluster.local:2222']
diff --git a/terraform/cluster/main.tf b/terraform/cluster/main.tf
@@ -38,6 +38,10 @@ resource "google_container_cluster" "main" {
     horizontal_pod_autoscaling {
       disabled = false
     }
+
+    network_policy_config {
+      disabled = false
+    }
   }
 
   master_auth {
@@ -54,6 +58,11 @@ resource "google_container_cluster" "main" {
       start_time = "03:00"
     }
   }
+
+  network_policy {
+    provider = "CALICO"
+    enabled = true
+  }
 }
 
 resource "google_container_node_pool" "main" {

diff --git a/terraform/main.tf b/terraform/main.tf
@@ -47,7 +47,7 @@ module "cluster" {
       machine-type = "n1-standard-4"
       max          = 5
       min          = 1
-      node_count   = 3
+      node_count   = 4
       preemptible  = false
       version      = "1.12.5-gke.5"
     },
@@ -56,7 +56,7 @@ module "cluster" {
       auto-upgrade = false
       disk-size    = "50"
       disk-type    = "pd-ssd"
-      image        = "ubuntu"
+      image        = "UBUNTU"
       local-ssds   = 0
       machine-type = "custom-8-16384"
       max          = 20
@@ -80,6 +80,19 @@ module "cluster" {
       version      = "1.14.7-gke.14 "
     },
 
+     "ci-workers-pr" = {
+      auto-upgrade = false
+      disk-size    = "50"
+      disk-type    = "pd-ssd"
+      image        = "COS"
+      local-ssds   = 0
+      machine-type = "custom-8-16384"
+      max          = 10
+      min          = 1
+      node_count   = 3
+      preemptible  = false
+      version      = "1.14.7-gke.14 "
+    },
   }
 }