Skip to content
This repository has been archived by the owner on Oct 16, 2023. It is now read-only.

Postgresql Acornfile plus a blog post about writing acorn files #27

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
183 changes: 183 additions & 0 deletions postgresql/Acornfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
args: {
// Number of replicas
replicas: 1

// The user credentials to be set
postgresUser: "admin"

postgresDb: "acorn"

postgresReplicationUser: "replica"

// It can be used to define location for the database files
// pgData: "/var/lib/postgresql/data"
pgData: "/acorn/data"

// Backup Schedule
backupSchedule: "@hourly"

// Restore from Backup. Takes a backup file name
restoreFromBackup: ""
}

for i in std.range(args.replicas) {
containers: {
if i != 0 {
"postgresql-\(i)": {
image: "postgres:14.5-bullseye"
//image: "bitnami/postgresql:14"
ports: {
internal: [
//prometheus monitoring
"9187:9187"
]
expose: "5433:5433"
}
env: {
"POSTGRES_PASSWORD": "secret://root-credentials/password"
"POSTGRES_USER": "secret://root-credentials/username"
"POSTGRES_DB": "\(args.postgresDb)"
"PGDATA": "\(args.pgData)"
}
dirs: {
"\(args.pgData)": "volume://pgdata-\(i)"
"/backup": "volume://backup"
}
files: {
"/etc/postgresql/postgresql.conf": "secret://postgresstandby-conf/template"
}
}
}

if i == 0 {
"postgresql-master": {
image: "postgres:14.5-bullseye"
//image: "bitnami/postgresql:14"
ports: {
internal: [
//prometheus monitoring
"9187:9187"
]
expose: "5432:5432"
}
env: {
"POSTGRES_PASSWORD": "secret://root-credentials/password"
"POSTGRES_USER": "secret://root-credentials/username"
"POSTGRES_DB": "\(args.postgresDb)"
"PGDATA": "\(args.pgData)"
}
dirs: {
"\(args.pgData)": "volume://pgdata-0"
"/backup": "volume://backup"
"/acorn/scripts": "./scripts"
}
files: {
"/etc/postgresql/postgresql.conf": "secret://postgresmaster-conf/template"
}
}
}
}

// The volume for container 0 will always be present, even when scaled to 0 for restore.
if i != 0 {
volumes: {
"pgdata-\(i)": {}
}
}
}

secrets: {
"root-credentials": {
type: "basic"
data: {
username: "\(args.postgresUser)"
}
}
"postgresmaster-conf": {
type: "template"
data: {
template: """
### master configuration
listen_addresses = '*'
port = 5432
shared_buffers = 128MB
max_connections = 100
dynamic_shared_memory_type = posix
timezone = 'Etc/UTC'
lc_messages = 'en_US.utf8'
lc_monetary = 'en_US.utf8'
lc_numeric = 'en_US.utf8'
lc_time = 'en_US.utf8'
default_text_search_config = 'pg_catalog.english'
wal_level = replica
hot_standby = on
max_wal_senders = 10
max_replication_slots = 10
hot_standby_feedback = on
"""
}
}
"postgresstandby-conf": {
type: "template"
data: {
template: """
### master configuration
listen_addresses = '*'
port = 5432
shared_buffers = 128MB
max_connections = 100
dynamic_shared_memory_type = posix
timezone = 'Etc/UTC'
lc_messages = 'en_US.utf8'
lc_monetary = 'en_US.utf8'
lc_numeric = 'en_US.utf8'
lc_time = 'en_US.utf8'
default_text_search_config = 'pg_catalog.english'
wal_level = hot_standby
standby_mode = on
"""
}
}
}

volumes: {
"pgdata-0": {}
"backup": {}
}

if args.backupSchedule != "" {
jobs: {
"backup": {
image: "postgres:14.5-bullseye"
dirs: {
"/acorn/scripts": "./scripts"
"/backup": "volume://backup"
}
command: ["sh","/acorn/scripts/backup.sh"]
env: {
"POSTGRES_USER": "secret://root-credentials/username"
"POSTGRES_PASSWORD": "secret://root-credentials/password"
"POSTGRES_DB": "\(args.postgresDb)"
}
schedule: "@hourly"
}
}
}

if args.restoreFromBackup != "" {
jobs: {
"restore-from-backup": {
image: "postgres:14.5-bullseye"
dirs: {
"/acorn/scripts": "./scripts"
"/backup": "volume://backup"
}
env: {
"POSTGRES_USER": "secret://root-credentials/username"
"POSTGRES_PASSWORD": "secret://root-credentials/password"
"POSTGRES_DB": "\(args.postgresDb)"
}
command: ["sh","/acorn/scripts/restore.sh", args.restoreFromBackup]
}
}
}
72 changes: 72 additions & 0 deletions postgresql/POST.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Writing Acorn files


## 1. All-in-one experience

After creating the Acorn package for highly available (HA) PostgreSQL I have found a lot of interesting and useful features for dockerized applications developers. In general, I have found Acorn as an effective tool for docker based application deployments to Kubernetes clusters. It is cofortable to manage both for local application development (e.g. with Minikube) and either with managed clusters like EKS, AKS or GKE.

### Docker compose for Kubernetes

In two words Acorn file could be considered as a docker-compose file for deploying application into Kubernetes cluster (instead of local Docker utilization). It is quite easy to prepare fast installation package to deliver docker application to Kubernetes cluster and manage it with own CLI tool, like for example:

$ acorn app
NAME IMAGE HEALTHY UP-TO-DATE CREATED
rough-field 3435258ee811 3/3 3 12d ago

### Scripting language

The other powerfull feature is built-in scripting language which could be used in Acorn files. For example, in cycles and for variables.

for i in std.range(args.replicas)


## 2. Secrets management

Secret management is a quite often difficulity in containerized applications development. Acorn gives the special entity for secrets management and also it is supported by Acorn CLI like the following:

$ acorn secrets
ALIAS NAME TYPE KEYS CREATED
rough-field.postgresmaster-conf postgresmaster-conf-x6w7x template [template] 12d ago
rough-field.root-credentials root-credentials-nmq2b basic [password username] 12d ago

Simple but powerful function is password auto generation. It gives the possibility to generate passwords on the fly (and do not pay a lot of attention for the secured storage). After generation it could be easily revealed with the following example command:

$ acorn secret expose rough-field.root-credentials
NAME TYPE KEY VALUE
root-credentials-nmq2b basic password gzlxcfg8hx4cbpw6
root-credentials-nmq2b basic username admin

## 3. Disk mounts

Mounted disks are useful and often used for persistent data storage creation. But also it could be used for mounting initialisation and service scripts. For example:

volumes: {
"backup": {}
}

Also, separate local folder directories could be mapped as system path. For example, scripts from the folder __scripts__ could be mounted as follows:

dirs: {
"/acorn/scripts": "./scripts"
"/backup": "volume://backup"
}

## 4. Some issues for the further improvements

Acorn is the new technology and there are some challenges could be faced.

### Ingress controllers

If your application requires ingress controllers, then additional steps should be done. Otherwise the following error could be met:

$ acorn check
NAME PASSED MESSAGE
IngressCapability false Ingress not ready (test timed out)

Usually it is related with default ingress class. It could be added as a notation to the ingressClass:

ingressclass.kubernetes.io/is-default-class: „true“

### User context

Some issues could be faced if Docker container uses user context (for example UID: 1001). If this or another issue is met then the best way to resolve it is to ask your question in Acorn slack channel. Acorn has good community support and can provide you with actual solution.
23 changes: 23 additions & 0 deletions postgresql/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# PostgreSQL

This Acorn provides a single node PostgreSQL 14.5 (based on Debian Bullseye) instance.

## Pre-req

- storage class for PVs

## Installation

### Revealing generated password

The secret with admin credentials contains the admin user name and password. Password is generated automatically and could be revealed in the following way:

$ acorn secret expose appname.root-credentials

Where the _appname_ should be substituted with your application name. It will return something like:

NAME TYPE KEY VALUE
root-credentials-ccrh4 basic password 5h9bq4cl2gw5h5d2
root-credentials-ccrh4 basic username admin


17 changes: 17 additions & 0 deletions postgresql/scripts/backup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/bash

set -e

backup_host=${1}

backup_root_dir='/backup'
ts=`date +"%Y%m%d-%H%M%S"`
backup_dir_name="postgres-backup-${ts}"
this_backup_dir="${backup_root_dir}/${backup_dir_name}"

mkdir -p ${this_backup_dir}

PGPASSWORD=${POSTGRES_PASSWORD} pg_dump -h postgresql ${POSTGRES_DB} -U ${POSTGRES_USER} > "${this_backup_dir}/dump.sql"

cd ${backup_root_dir}
tar -zcvf ${backup_dir_name}.tgz ${this_backup_dir} && rm -rf ${this_backup_dir}
28 changes: 28 additions & 0 deletions postgresql/scripts/restore.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash

set -e

backup_filename=${1}
backup_database=${2}
#backup_dir_name="${1%.*}"

backup_root_dir='/backups'
backup_to_restore="${backup_root_dir}/${backup_filename}"

#touch ${backup_root_dir}/restore_in_progress

if [ ! -f "${backup_to_restore}" ]; then
echo "Backup file ${backup_to_restore} not found!"
exit 1
fi

echo "Untaring backup... ${backup_to_restore}"
tar -zxvf "${backup_to_restore}" -C /scratch/

#echo "Restoring..."
psql --set ON_ERROR_STOP=on "${backup_database}" < "${backup_filename}"

#echo "Cleaning up scratch..."
rm -rf /scratch/*

#echo "remove backup arg and scale to 1"