Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable repo #63

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Configurable repo #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5a38ff6
Configurable repo
jasonchung1871 Dec 30, 2024
24e53d3
Update values.yaml
jasonchung1871 Dec 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions charts/crunchy-postgres/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,21 @@ A chart to provision a [Crunchy Postgres](https://www.crunchydata.com/) cluster.
| Parameter | Description | Default |
| ---------------------------------------------------- | ------------------------------------------------------------- | ---------------------- |
| `pgBackRest.image` | Crunchy pgBackRest | |
| `pgBackRest.retention` | Number of backups/days to keep depending on retentionFullType | `2` |
| `pgBackRest.retentionFullType` | Either 'count' or 'time' | `count` |
| `pgBackRest.repos.schedules.full` | Full backup schedule | `0 8 * * *` |
| `pgBackRest.repos.schedules.incremental` | Incremental backup schedule | `0 0,4,12,16,20 * * *` |
| `pgBackRest.repos.schedules.volume.addessModes` | Access modes | `ReadWriteOnce` |
| `pgBackRest.repos.schedules.volume.storage` | Access modes | `64Mi` |
| `pgBackRest.repos.schedules.volume.storageClassName` | Storage class name modes | `netapp-file-backup` |
| `pgBackRest.repos.path` | The path inside the bucket where the backups will be saved to, set it to `/` to use the root of the bucket. | `/dbbackup` |
| `pgBackRest.repos.s3.uriStyle` | Style of URL to use for S3 communication. [More Info](https://pgbackrest.org/configuration.html#section-repository/option-repo-s3-uri-style) | `path` |
| `pgBackRest.repos.s3.bucket` | The bucket to use for backups | `bucketName` |
| `pgBackRest.repos.s3.endpoint` | The endpoint to use, for example s3.ca-central-1.amazonaws.com | `endpointName` |
| `pgBackRest.repos.s3.region` | The region to use, not necessary if your S3 system does not specify one | `ca-central-1` |
| `pgBackRest.repos.s3.key` | The key to use to access the bucket. MUST BE KEPT SECRET | `s3KeyValue` |
| `pgBackRest.repos.s3.keySecret` | The key secret for the key set above. MUST BE KEPT SECRET | `s3SecretValue` |
| `pgBackRest.repos.schedules.name` | Schedule name for pgbackrest | `full-weekly` |
| `pgBackRest.repos.schedules.type` | Schedule type for pgbackrest | `full` |
| `pgBackRest.repos.schedules.schedule` | Full backup schedule | `0 8 * * *` |
| `pgBackRest.repos.schedules.retentionLimit` | Number of backups/days to keep depending on retentionPolicy | `12` |
| `pgBackRest.repos.schedules.retentionPolicy` | Either 'count' or 'time' | `count` |
| `pgBackRest.repos.volume.accessModes` | Access modes | `ReadWriteOnce` |
| `pgBackRest.repos.volume.storage` | Access modes | `64Mi` |
| `pgBackRest.repos.volume.storageClassName` | Storage class name modes | `netapp-file-backup` |
| `pgBackRest.repoHost.requests.cpu` | CPU requests | `1m` |
| `pgBackRest.repoHost.requests.memory` | Memory requests | `64Mi` |
| `pgBackRest.repoHost.limits.cpu` | CPU limits | `50m` |
Expand All @@ -58,13 +66,6 @@ A chart to provision a [Crunchy Postgres](https://www.crunchydata.com/) cluster.
| `pgBackRest.s3.enabled` | Enables the s3 repo backups | `false` |
| `pgBackRest.s3.createS3Secret` | Creates the s3 secret based on key and keySecret | `true` |
| `pgBackRest.s3.s3Secret` | The secret name to be created or read from | `s3-pgbackrest` |
| `pgBackRest.s3.s3Path` | The path inside the bucket where the backups will be saved to, set it to `/` to use the root of the bucket. | `/dbbackup` |
| `pgBackRest.s3.s3UriStyle` | Style of URL to use for S3 communication. [More Info](https://pgbackrest.org/configuration.html#section-repository/option-repo-s3-uri-style) | `path` |
| `pgBackRest.s3.bucket` | The bucket to use for backups | `bucketName` |
| `pgBackRest.s3.endpoint` | The endpoint to use, for example s3.ca-central-1.amazonaws.com | `endpointName` |
| `pgBackRest.s3.region` | The region to use, not necessary if your S3 system does not specify one | `ca-central-1` |
| `pgBackRest.s3.key` | The key to use to access the bucket. MUST BE KEPT SECRET | `s3KeyValue` |
| `pgBackRest.s3.keySecret` | The key secret for the key set above. MUST BE KEPT SECRET | `s3SecretValue` |
---

### Patroni
Expand All @@ -75,6 +76,7 @@ A chart to provision a [Crunchy Postgres](https://www.crunchydata.com/) cluster.
| Parameter | Description | Default |
| ------------------------------------------- | ------------------------------------------------------------------- | --------------------------------- |
| `patroni.postgresql.pg_hba` | pg_hba permissions | `"host all all 0.0.0.0/0 md5"` |
| `patroni.postgresql.parameters.timezone` | timezone | `"America/Vancouver"` |
| `crunchyImage` | Crunchy Postgres image | `...crunchy-postgres:ubi8-14.7-0` |
| `patroni.parameters.shared_buffers` | The number of shared memory buffers used by the server | `16MB` |
| `patroni.parameters.wal_buffers` | The number of disk-page buffers in shared memory for WAL | `64KB` |
Expand Down
91 changes: 58 additions & 33 deletions charts/crunchy-postgres/templates/PostgresCluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,14 @@ metadata:
spec:
metadata:
labels: {{ include "crunchy-postgres.labels" . | nindent 6 }}
{{ if .Values.crunchyImage }}
{{ if .Values.crunchyImage }}
image: {{ .Values.crunchyImage }}
{{ end }}
imagePullPolicy: {{.Values.imagePullPolicy}}
postgresVersion: {{ .Values.postgresVersion }}
{{ if .Values.postGISVersion }}
postGISVersion: {{ .Values.postGISVersion | quote }}
{{ end }}
postgresVersion: {{ .Values.postgresVersion }}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate postgresVersion


{{ if .Values.pgmonitor.enabled }}

Expand Down Expand Up @@ -102,47 +101,72 @@ spec:
backups:
pgbackrest:
{{ if .Values.pgBackRest.image }}
image: {{ .Values.pgBackRest.image }}
image: {{ .Values.pgBackRest.image | quote }}
{{ end }}
{{- if .Values.pgBackRest.s3.enabled }}
{{ if eq $.Values.pgBackRest.s3.enabled true }}
configuration:
- secret:
name: {{ .Values.pgBackRest.s3.s3Secret }}
{{- end }}
name: {{ .Values.pgBackRest.s3.s3Secret | quote }}
{{ end }}
global:
# Support both PVC and s3 backups
repo1-retention-full: {{ .Values.pgBackRest.retention | quote }}
repo1-retention-full-type: {{ .Values.pgBackRest.retentionFullType }}
{{- if .Values.pgBackRest.s3.enabled }}
repo2-retention-full: {{ .Values.pgBackRest.retention | quote }}
repo2-retention-full-type: {{ .Values.pgBackRest.retentionFullType }}
repo2-path: {{ .Values.pgBackRest.s3.s3Path }}
repo2-s3-uri-style: {{ .Values.pgBackRest.s3.s3UriStyle }}
{{- end }}
{{ range $repo := .Values.pgBackRest.repos }}
{{ if eq $.Values.pgBackRest.s3.enabled true }}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could probably be changed to check if the s3 key exists, using this check for the bucket path, also did not want to change the other uses of s3.enabled

{{ $repo.name }}-s3-bucket: {{ $repo.s3.bucket }}
{{ $repo.name }}-s3-endpoint: {{ $repo.s3.endpoint }}
{{ $repo.name }}-s3-key: {{ $repo.s3.key }}
{{ $repo.name }}-s3-key-secret: {{ $repo.s3.keySecret }}
{{ if .s3.region }}
{{ $repo.name }}-s3-region: {{ $repo.s3.region }}
{{ end }}
{{ if $repo.s3.uriStyle }}
{{ $repo.name }}-s3-uri-style: {{ $repo.s3.uriStyle }}
{{ end }}
{{ if $repo.path }}
{{ $repo.name }}-path: {{ $repo.path }}
{{ end }}
{{ end }}
{{ range $schedule := $repo.schedules }}
{{ if eq $schedule.type "full" }}
{{ $repo.name }}-retention-full-type: {{ $schedule.retentionPolicy }}
{{ $repo.name }}-retention-full: {{ $schedule.retentionLimit | quote }}
{{ end }}
{{ if eq $schedule.type "differential" }}
{{ $repo.name }}-retention-diff: {{ $schedule.retentionLimit | quote }}
{{ end }}
{{ if eq $schedule.type "incremental" }}
{{ $repo.name }}-retention-archive: {{ $schedule.retentionLimit | quote }}
{{ end}}
{{ end }}
{{ end }}
repos:
# hardcoding repo1 until we solution allowing multiple repos
- name: repo1
schedules:
full: {{ .Values.pgBackRest.repos.schedules.full }}
incremental: {{ .Values.pgBackRest.repos.schedules.incremental }}
{{ range $repo := .Values.pgBackRest.repos }}
- name: {{ $repo.name }}
{{ if $repo.volume }}
volume:
volumeClaimSpec:
accessModes:
- {{ .Values.pgBackRest.repos.volume.accessModes }}
accessModes: {{ toYaml $repo.volume.accessModes | nindent 14 }}
resources:
requests:
storage: {{ .Values.pgBackRest.repos.volume.storage }}
storageClassName: {{ .Values.pgBackRest.repos.volume.storageClassName }}
{{- if .Values.pgBackRest.s3.enabled }}
- name: repo2
schedules:
full: {{ if .Values.pgBackRest.s3.fullSchedule }}{{ .Values.pgBackRest.s3.fullSchedule }}{{ else }}{{ .Values.pgBackRest.repos.schedules.full }}{{ end }}
incremental: {{ if .Values.pgBackRest.s3.incrementalSchedule }}{{ .Values.pgBackRest.s3.incrementalSchedule }}{{ else }}{{ .Values.pgBackRest.repos.schedules.incremental }}{{ end }}
storage: {{ $repo.volume.storage | quote }}
{{ if $repo.volume.storageClassName }}
storageClassName: {{ $repo.volume.storageClassName | quote }}
{{ end }}
{{ end }}
{{ if $repo.s3 }}
s3:
bucket: {{ .Values.pgBackRest.s3.bucket }}
endpoint: {{ .Values.pgBackRest.s3.endpoint }}
region: {{ .Values.pgBackRest.s3.region }}
{{- end }}
bucket: {{ $repo.s3.bucket | quote }}
endpoint: {{ $repo.s3.endpoint | quote }}
{{ if $repo.s3.region }}
region: {{ $repo.s3.region | quote }}
{{ end }}
{{ end }}
{{ if $repo.schedules }}
schedules:
{{ range $type, $schedule := .schedules }}
{{ $schedule.type }}: {{ $schedule.schedule }}
{{ end }}
{{ end }}
{{ end }}
# this stuff is for the "pgbackrest" container (the only non-init container) in the "postgres-crunchy-repo-host" pod
repoHost:
resources:
Expand Down Expand Up @@ -185,6 +209,7 @@ spec:
min_wal_size: {{ .Values.patroni.postgresql.parameters.min_wal_size }}
max_wal_size: {{ .Values.patroni.postgresql.parameters.max_wal_size }}
max_slot_wal_keep_size: {{ .Values.patroni.postgresql.parameters.max_slot_wal_keep_size }}
timezone: {{ .Values.patroni.postgresql.parameters.timezone}}

proxy:
pgBouncer:
Expand Down
56 changes: 27 additions & 29 deletions charts/crunchy-postgres/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ fullnameOverride: crunchy-postgres
crunchyImage: # it's not necessary to specify an image as the images specified in the Crunchy Postgres Operator will be pulled by default
#crunchyImage: artifacts.developer.gov.bc.ca/bcgov-docker-local/crunchy-postgres-gis:ubi8-15.2-3.3-0 # use this image for POSTGIS
postgresVersion: 15
#postGISVersion: '3.3' # use this version of POSTGIS.
#postGISVersion: '3.3' # use this version of POSTGIS. both crunchyImage and this property needs to have valid values for POSTGIS to be enabled.
imagePullPolicy: IfNotPresent

# enable to bootstrap a standby cluster from backup. Then disable to promote this standby to primary
Expand Down Expand Up @@ -54,13 +54,31 @@ pgBackRest:
# If retention-full-type set to 'time' then the number defined in retention will take that many days worth of full backups before expiration
retentionFullType: count
repos:
schedules:
full: 0 8 * * *
incremental: 0 0,4,12,16,20 * * *
volume:
accessModes: "ReadWriteOnce"
storage: 64Mi
storageClassName: netapp-file-backup
- name: repo1
volume:
accessModes:
- "ReadWriteOnce"
storage: 2Gi
storageClassName: netapp-file-backup
# s3:
# uriStyle: path
# bucket: "bucketName"
# endpoint: "endpointName"
# region: "ca-central-1"
schedules:
- name: full-weekly
type: full
schedule: '0 8 * * 0' # Every sunday at midnight
retentionLimit: "12"
retentionPolicy: count
- name: diff-daily
type: differential
schedule: '0 8 * * 1-6' # Every day except sunday at midnight
retentionLimit: "6"
- name: incremental
type: incremental
schedule: '0 0,4,12,16,20 * * *' # Every 4 hours, except midnight to prevent a lock conflict
retentionLimit: "90"
repoHost:
requests:
cpu: 1m
Expand All @@ -80,27 +98,6 @@ pgBackRest:
createS3Secret: true
# the s3 secret name
s3Secret: s3-pgbackrest
# the path start with /, it will be created under bucket if it doesn't exist
s3Path: "/habackup"
# s3UriStyle is host or path
s3UriStyle: path
# bucket specifies the S3 bucket to use,
bucket: "bucketName"
# endpoint specifies the S3 endpoint to use.
endpoint: "endpointName"
# region specifies the S3 region to use. If your S3 storage system does not
# use "region", fill this in with a random value.
region: "ca-central-1"
# key is the S3 key. This is stored in a Secret.
# Please DO NOT push this value to GitHub
key: "s3keyValue"
# keySecret is the S3 key secret. This is stored in a Secret.
# Please DO NOT push this value to GitHub
keySecret: "s3SecretValue"
# setting the below to be one plus of the default schedule
# to avoid conflicts
fullSchedule: "0 9 * * *"
incrementalSchedule: "0 1,5,13,17,21 * * *"

patroni:
postgresql:
Expand All @@ -111,6 +108,7 @@ patroni:
min_wal_size: 32MB
max_wal_size: 64MB # default is 1GB
max_slot_wal_keep_size: 128MB # default is -1, allowing unlimited wal growth when replicas fall behind
timezone: 'America/Vancouver'

proxy:
pgBouncer:
Expand Down
Loading