Merge pull request #59 from controlplane-com/rpk

Rpk

examples/mongodb/README.md

@@ -37,7 +37,7 @@ Syntax2: `<REPLICA_NAME>.<WORKLOAD_NAME>.<GVC_ALIAS>.svc.cluster.local`
 Example2: `mongodb1-1.mongodb1.j3i2ddut40d.svc.cluster.local`  
 Look for the GVC Alias on the info page of the GVC in the console, or query using the CLI.
 
-**Note**: All communication between workloads on Control Plane happens within an Istio-based service mesh with enforced mTLS and least privileged access managed using the [firewall](https://docs.controlplane.com/reference/workload#internal) feature. The method described in this example allows only internal communication, which is secured with mTLS and a firewall.
+**Note**: All communication between workloads on Control Plane happens within an Istio-based service mesh with enforced mTLS and least privileged access managed using the [firewall](https://docs.controlplane.com/reference/workload#internal) feature. The method described in this example allows only internal communication, secured with mTLS and a firewall, unless this Helm template is modified to support public access.
 
 ### Cleanup
 

examples/redpanda-kafka/Chart.yaml

@@ -0,0 +1,6 @@
+apiVersion: v2
+name: redpanda_kafka
+description: A Redpanda Kafka helm chart for Control Plane
+type: application
+version: 0.1.0
+appVersion: "1.0.0"

examples/redpanda-kafka/README.md

@@ -0,0 +1,50 @@
+## Redpanda Kafka cluster example
+
+Instructions on running a Redpanda Kafka cluster.
+
+### Steps to run this example:
+
+**HELM**
+
+The [Helm CLI](https://helm.sh/docs/intro/install/#through-package-managers) and [Control Plane CLI](https://docs.controlplane.com/reference/cli#install-npm) must be installed.
+
+1. Clone this repo and update the [values.yaml](./values.yaml) file as needed. Use the file to edit Kafka Cluster configurations by modifying the `redpanda.custom_configurations` value as required.
+
+2. If custom configurations to Redpanda Kafka cluster are required, make sure to set these for `redpanda.custom_configurations` in the [values.yaml](./values.yaml) file. Please refer to [RPK documentation](https://docs.redpanda.com/current/reference/properties/) for further instructions.
+
+3. To access the Redpanda Console from the Internet, make sure your IP is whitelisted by updating the `redpanda_console.firewall.external_inboundAllowCIDR` setting in the [values.yaml](./values.yaml) file.  
+   It is not recommended to expose this setting to `0.0.0.0/0` without authentication. Authentication and authorization of RedPanda console is a feature requires an Enterprise License with Redpanda. [click here to learn more](https://docs.redpanda.com/current/manage/security/console/authentication/).
+
+4. If the GVC does not exist, create it and select location(s).
+
+   ```bash
+   cpln gvc create --name redpanda --location aws-us-west-2
+   ```
+
+5. Run the command below from this directory.
+
+   ```bash
+   cpln helm install redpanda --gvc redpanda
+   ```
+
+### Accessing Redpanda cluster
+
+Workloads are allowed to access Redpanda Cluster based on the `firewall` configuration you specify. You can learn more about in our [documentation](https://docs.controlplane.com/reference/workload#internal).
+
+To resolve a specific Redpanda workload's replica, the client must be in the same GVC.  
+Syntax1: `<REPLICA_NAME>.<WORKLOAD_NAME>`  
+Example1: `redpanda1-0.redpanda1`  
+Or  
+Syntax2: `<REPLICA_NAME>.<WORKLOAD_NAME>.<GVC_ALIAS>.svc.cluster.local`  
+Example2: `redpanda1-1.redpanda1.j3i2ddut40d.svc.cluster.local`  
+Look for the GVC Alias on the info page of the GVC in the console, or query using the CLI.
+
+**Note**: All communication between workloads on Control Plane happens within an Istio-based service mesh with enforced mTLS and least privileged access managed using the [firewall](https://docs.controlplane.com/reference/workload#internal) feature. The method described in this example allows only internal communication, secured with mTLS and a firewall, unless this Helm template is modified to support public access.
+
+### Cleanup
+
+**HELM**
+
+```bash
+cpln helm uninstall redpanda
+```

examples/redpanda-kafka/scripts/console-start.sh

@@ -0,0 +1,51 @@
+#!/bin/sh
+
+set -x
+
+if [ -z "$CUSTOM_RPK_REPLICAS" ]; then
+  echo "CUSTOM_RPK_REPLICAS is not set"
+  exit 1
+fi
+
+CONSOLE_CONFIG_FILE="
+kafka:
+    brokers: ["
+i=0
+while [ $i -lt "$CUSTOM_RPK_REPLICAS" ]; do
+  CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE\"${CUSTOM_RPK_WORKLOAD_NAME}-${i}.${CUSTOM_RPK_WORKLOAD_NAME}.${CPLN_GVC_ALIAS}.svc.cluster.local:${CUSTOM_RPK_PORT}\""
+  if [ $i -lt $(($CUSTOM_RPK_REPLICAS-1)) ]; then
+    CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE,"
+  fi
+  i=$(($i + 1))
+done
+CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE]
+    schemaRegistry:
+        enabled: true
+        urls: ["
+i=0
+while [ $i -lt "$CUSTOM_RPK_REPLICAS" ]; do
+  CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE\"http://${CUSTOM_RPK_WORKLOAD_NAME}-${i}.${CUSTOM_RPK_WORKLOAD_NAME}.${CPLN_GVC_ALIAS}.svc.cluster.local:8081\""
+  if [ $i -lt $(($CUSTOM_RPK_REPLICAS-1)) ]; then
+    CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE,"
+  fi
+  i=$(($i + 1))
+done
+CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE]
+redpanda:
+    adminApi:
+        enabled: true
+        urls: ["
+i=0
+while [ $i -lt "$CUSTOM_RPK_REPLICAS" ]; do
+  CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE\"http://${CUSTOM_RPK_WORKLOAD_NAME}-${i}.${CUSTOM_RPK_WORKLOAD_NAME}.${CPLN_GVC_ALIAS}.svc.cluster.local:9644\""
+  if [ $i -lt $(($CUSTOM_RPK_REPLICAS-1)) ]; then
+    CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE,"
+  fi
+  i=$(($i + 1))
+done
+CONSOLE_CONFIG_FILE="$CONSOLE_CONFIG_FILE]"
+
+echo "$CONSOLE_CONFIG_FILE" > "$CONFIG_FILEPATH"
+
+# Execute the console application
+/app/console

examples/redpanda-kafka/scripts/redpanda-start.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+set -x
+
+MAX_SEEDS=3
+NUM_REPLICAS=${CUSTOM_RPK_REPLICAS:-3}
+NUM_SEEDS=$(( NUM_REPLICAS > MAX_SEEDS ? MAX_SEEDS : NUM_REPLICAS ))
+PET_ORDINAL=$(echo "$POD_NAME" | rev | cut -d'-' -f 1 | rev)
+WORKLOAD_NAME=$(echo $CPLN_WORKLOAD | sed 's|.*/workload/\([^/]*\)$|\1|')
+
+# Generate the seeds list according to the number of replicas provided and maximum value of 3
+SEEDS=""
+for i in $(seq 0 $((NUM_SEEDS - 1))); do
+  SEED="$WORKLOAD_NAME-$i.$WORKLOAD_NAME.$CPLN_GVC_ALIAS.svc.cluster.local:33145"
+  if [ -z "$SEEDS" ]; then
+    SEEDS="$SEED"
+  else
+    SEEDS="$SEEDS,$SEED"
+  fi
+done
+
+CUSTOM_CONFIGURATIONS=${CUSTOM_CONFIGURATIONS:-""}
+
+rpk redpanda start --kafka-addr internal://0.0.0.0:$CUSTOM_RPK_PORT \
+    --advertise-kafka-addr internal://$WORKLOAD_NAME-$PET_ORDINAL.$WORKLOAD_NAME.$CPLN_GVC_ALIAS.svc.cluster.local:$CUSTOM_RPK_PORT \
+    --pandaproxy-addr internal://0.0.0.0:8082 \
+    --advertise-pandaproxy-addr internal://$WORKLOAD_NAME-$PET_ORDINAL.$WORKLOAD_NAME.$CPLN_GVC_ALIAS.svc.cluster.local:8082 \
+    --schema-registry-addr internal://0.0.0.0:8081 \
+    --rpc-addr "0.0.0.0:33145" \
+    --advertise-rpc-addr $WORKLOAD_NAME-$PET_ORDINAL.$WORKLOAD_NAME.$CPLN_GVC_ALIAS.svc.cluster.local:33145 \
+    --smp 1 --default-log-level=$CUSTOM_RPK_LOGLVL \
+    --set redpanda.empty_seed_starts_cluster=false \
+    --seeds $SEEDS \
+    $CUSTOM_CONFIGURATIONS

examples/redpanda-kafka/templates/console-start.yaml

@@ -0,0 +1,9 @@
+kind: secret
+name: {{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }}-console-start
+description: {{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }} console start script
+tags: {}
+type: opaque
+data:
+  encoding: plain
+  payload: |-
+{{ .Files.Get "scripts/console-start.sh" | indent 4 }}

examples/redpanda-kafka/templates/redpanda-start.yaml

@@ -0,0 +1,9 @@
+kind: secret
+name: {{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }}-start
+description: {{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }} start script
+tags: {}
+type: opaque
+data:
+  encoding: plain
+  payload: |-
+{{ .Files.Get "scripts/redpanda-start.sh" | indent 4 }}

examples/redpanda-kafka/templates/redpanda.yaml

@@ -0,0 +1,177 @@
+---
+kind: identity
+name: {{ .Values.redpanda.name }}-identity
+gvc: {{ .Values.cpln.gvc }}
+description: {{ .Values.redpanda.name }}
+---
+kind: policy
+name: {{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }}
+tags: {}
+origin: default
+bindings:
+  - permissions:
+      - reveal
+    principalLinks:
+      - //gvc/{{ .Values.cpln.gvc }}/identity/{{ .Values.redpanda.name }}-identity
+targetKind: secret
+targetLinks:
+  - //secret/{{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }}-start
+  - //secret/{{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }}-console-start
+---
+kind: volumeset
+name: {{ .Values.redpanda.name }}-data
+description: {{ .Values.redpanda.name }}-data
+gvc: {{ .Values.cpln.gvc }}
+tags: {}
+spec:
+  fileSystemType: {{ .Values.redpanda.volume.fileSystemType }}
+  initialCapacity: {{ .Values.redpanda.volume.initialCapacity }}
+  performanceClass: {{ .Values.redpanda.volume.performanceClass }}
+  snapshots:
+    createFinalSnapshot: {{ .Values.redpanda.snapshots.createFinalSnapshot }}
+    retentionDuration: {{ .Values.redpanda.snapshots.retentionDuration }}
+    schedule: {{ .Values.redpanda.snapshots.schedule }}
+---
+kind: workload
+name: {{ .Values.redpanda.name }}
+gvc: {{ .Values.cpln.gvc }}
+description: {{ .Values.redpanda.name }}
+spec:
+  type: stateful
+  containers:
+    - name: redpanda
+      cpu: '{{ .Values.redpanda.cpu }}'
+      memory: {{ .Values.redpanda.memory }}
+      env:
+        - name: CUSTOM_RPK_LOGLVL
+          value: '{{ .Values.redpanda.loglevel }}'
+        - name: CUSTOM_RPK_PORT
+          value: '{{ .Values.redpanda.port }}'
+        {{- if .Values.redpanda.custom_configurations }}
+        - name: CUSTOM_CONFIGURATIONS
+          value: '{{ .Values.redpanda.custom_configurations }}'
+        {{- end}}
+      ports:
+        - number: 9644
+          protocol: http
+        - number: 8081
+          protocol: http
+        - number: 8082
+          protocol: http
+        - number: {{ .Values.redpanda.port }}
+          protocol: tcp
+        - number: 33145
+          protocol: tcp
+      livenessProbe:
+        exec:
+          command:
+            - /bin/bash
+            - '-c'
+            - curl --silent --fail -m 5 localhost:9644/v1/status/ready
+        failureThreshold: 5
+        initialDelaySeconds: 100
+        periodSeconds: 10
+        successThreshold: 1
+        timeoutSeconds: 2
+      command: /bin/bash
+      args:
+        - '-c'
+        - cp /scripts/redpanda-start.sh /tmp/redpanda-start.sh && chmod +x /tmp/redpanda-start.sh && /tmp/redpanda-start.sh
+      image: {{ .Values.redpanda.image }}
+      volumes:
+        - path: /var/lib/redpanda/data
+          recoveryPolicy: retain
+          uri: 'cpln://volumeset/{{ .Values.redpanda.name }}-data'
+        - path: /scripts/redpanda-start.sh
+          recoveryPolicy: retain
+          uri: 'cpln://secret/{{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }}-start'
+  defaultOptions:
+    autoscaling:
+      maxConcurrency: 0
+      maxScale: {{ .Values.redpanda.replicas }}
+      metric: disabled
+      minScale: {{ .Values.redpanda.replicas }}
+      scaleToZeroDelay: 300
+      target: 100
+    capacityAI: false
+    debug: false
+    suspend: false
+    timeoutSeconds: 15
+{{- if .Values.redpanda.firewall }}
+  firewallConfig:
+    {{- if or (hasKey .Values.redpanda.firewall "external_inboundAllowCIDR") (hasKey .Values.redpanda.firewall "external_outboundAllowCIDR") }}
+    external:
+      inboundAllowCIDR: {{- if .Values.redpanda.firewall.external_inboundAllowCIDR }}{{ .Values.redpanda.firewall.external_inboundAllowCIDR | splitList "," | toYaml | nindent 8 }}{{- else }} []{{- end }}
+      outboundAllowCIDR: {{- if .Values.redpanda.firewall.external_outboundAllowCIDR }}{{ .Values.redpanda.firewall.external_outboundAllowCIDR | splitList "," | toYaml | nindent 8 }}{{- else }} []{{- end }}
+    {{- end }}
+    {{- if hasKey .Values.redpanda.firewall "internal_inboundAllowType" }}
+    internal:
+      inboundAllowType: {{ default "[]" .Values.redpanda.firewall.internal_inboundAllowType }}
+    {{- end }}
+{{- end }}
+  identityLink: //identity/{{ .Values.redpanda.name }}-identity
+  localOptions: []
+  rolloutOptions:
+    maxSurgeReplicas: 25%
+    maxUnavailableReplicas: '1'
+    minReadySeconds: 0
+    scalingPolicy: Parallel
+  securityOptions:
+    filesystemGroupId: 101
+---
+kind: workload
+name: {{ .Values.redpanda.name }}-console
+gvc: {{ .Values.cpln.gvc }}
+description: {{ .Values.redpanda.name }}-console
+spec:
+  type: standard
+  containers:
+    - name: console
+      cpu: {{ .Values.redpanda_console.cpu }}
+      memory: {{ .Values.redpanda_console.memory }}
+      image: {{ .Values.redpanda_console.image }}
+      env:
+        - name: CONFIG_FILEPATH
+          value: '{{ .Values.redpanda_console.env.CONFIG_FILEPATH }}'
+        - name: CUSTOM_RPK_REPLICAS
+          value: '{{ .Values.redpanda.replicas }}'
+        - name: CUSTOM_RPK_WORKLOAD_NAME
+          value: '{{ .Values.redpanda.name }}'
+        - name: CUSTOM_RPK_PORT
+          value: '{{ .Values.redpanda.port }}'
+      ports:
+        - number: '{{ .Values.redpanda_console.port }}'
+          protocol: http
+      command: /bin/sh
+      args:
+        - '-c'
+        - cp /scripts/console-start.sh /tmp/console-start.sh && chmod +x /tmp/console-start.sh && sh /tmp/console-start.sh
+      volumes:
+        - path: /scripts/console-start.sh
+          recoveryPolicy: retain
+          uri: 'cpln://secret/{{ .Values.redpanda.name }}-{{ .Values.cpln.gvc }}-console-start'
+  defaultOptions:
+    autoscaling:
+      maxScale: {{ .Values.redpanda_console.replicas }}
+      metric: disabled
+      minScale: {{ .Values.redpanda_console.replicas }}
+      scaleToZeroDelay: 300
+      target: 100
+    capacityAI: false
+    debug: false
+    suspend: false
+    timeoutSeconds: 15
+{{- if .Values.redpanda_console.firewall }}
+  firewallConfig:
+    {{- if or (hasKey .Values.redpanda_console.firewall "external_inboundAllowCIDR") (hasKey .Values.redpanda_console.firewall "external_outboundAllowCIDR") }}
+    external:
+      inboundAllowCIDR: {{- if .Values.redpanda_console.firewall.external_inboundAllowCIDR }}{{ .Values.redpanda_console.firewall.external_inboundAllowCIDR | splitList "," | toYaml | nindent 8 }}{{- else }} []{{- end }}
+      outboundAllowCIDR: {{- if .Values.redpanda_console.firewall.external_outboundAllowCIDR }}{{ .Values.redpanda_console.firewall.external_outboundAllowCIDR | splitList "," | toYaml | nindent 8 }}{{- else }} []{{- end }}
+    {{- end }}
+    {{- if hasKey .Values.redpanda_console.firewall "internal_inboundAllowType" }}
+    internal:
+      inboundAllowType: {{ default "[]" .Values.redpanda_console.firewall.internal_inboundAllowType }}
+    {{- end }}
+{{- end }}
+  identityLink: //identity/{{ .Values.redpanda.name }}-identity
+

examples/redpanda-kafka/values.yaml

@@ -0,0 +1,43 @@
+# Default values for redpanda_kafka.
+# This is a YAML-formatted file.
+# Declare variables to be passed into your templates.
+
+redpanda:
+  name: redpanda
+  image: docker.redpanda.com/redpandadata/redpanda:v24.1.3
+  replicas: 3
+  port: 9200
+  loglevel: info
+  memory: 2Gi # EX. 3000Mi = 3Gi
+  cpu: 1 # use Integer values, 1/2/3/4 etc.. Can't be 1.5
+
+  #https://docs.redpanda.com/current/reference/properties/cluster-properties
+  # custom_configurations: "--set redpanda.alter_topic_cfg_timeout_ms=6000 --set redpanda.auto_create_topics_enabled=true"
+  custom_configurations: ""
+
+  volume:
+    initialCapacity: 50 # In Gigabytes. For high-throughput-ssd minimum is '1000'
+    fileSystemType: xfs # Redpanda recommends using XFS
+    performanceClass: general-purpose-ssd # high-throughput-ssd / general-purpose-ssd
+  snapshots:
+    createFinalSnapshot: 'true'
+    retentionDuration: 7d
+    schedule: '0 1 * * *'
+  # To disable all traffic, comment out the corresponding rule. Docs: https://docs.controlplane.com/concepts/security#firewall
+  firewall:
+    internal_inboundAllowType: "same-org" # Options: same-org / same-gvc(Recommended)
+    # external_inboundAllowCIDR: 0.0.0.0/0 # Provide a comma-separated list
+    external_outboundAllowCIDR: "0.0.0.0/0" # Provide a comma-separated list
+
+redpanda_console:
+  image: docker.redpanda.com/redpandadata/console:v2.5.2
+  replicas: 1
+  port: 8080
+  memory: 250Mi # EX. 3000Mi = 3Gi
+  cpu: 150m # vCPU or milicores with; EX. 1.5 = 1500m
+  env:
+    CONFIG_FILEPATH: /tmp/config.yml
+  firewall:
+    internal_inboundAllowType: "same-gvc" # Options: same-org / same-gvc(Recommended)
+    # external_inboundAllowCIDR: xx.xx.xx.xx/32 # Provide a comma-separated list
+    external_outboundAllowCIDR: "0.0.0.0/0" # Provide a comma-separated list