fix: Disallow concurrently-running Kubernetes CronJobs by default

Under normal circumstances, we never want two instances of the same backup or restore job to run concurrently. However, Kubernetes does default to allowing this, the `concurrencyPolicy` option of a `CronJob` spec being `Allow` by default. Define a new configuration option, `BACKUP_K8S_CRONJOB_CONCURRENCYPOLICY`, and have it default to `Forbid` instead. Reference: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#concurrency-policy
hastexo · Oct 10, 2023 · 80412b8 · 80412b8
1 parent 1192464
commit 80412b8
Show file tree

Hide file tree

Showing 4 changed files with 12 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,11 @@
+## Unreleased
+
+* [fix] Avoid running Kubernetes CronJob instances concurrently by
+  default: add the `BACKUP_K8S_CRONJOB_CONCURRENCYPOLICY` setting
+  and default it to `Forbid` (rather than
+  [the Kubernetes default](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#concurrency-policy),
+  `Allow`).
+
 ## Version 2.1.0 (2023-08-22)
 
 * [Enhancement] Support Tutor 16, Open edX Palm and Python 3.11.

diff --git a/README.md b/README.md
@@ -188,6 +188,7 @@ Configuration
 * `BACKUP_K8S_CRONJOB_RESTORE_ENABLE` (default: `false`, periodic restore is disabled.)
 * `BACKUP_K8S_CRONJOB_RESTORE_SCHEDULE` (default: `"30 0 * * *"`, once a day at 30 mins past
    midnight)
+* `BACKUP_K8S_CRONJOB_CONCURRENCYPOLICY` (default: `"Forbid"`, see [the Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#concurrency-policy) for other available options)
 * `BACKUP_K8S_USE_EPHEMERAL_VOLUMES` (default: `false`)
 * `BACKUP_K8S_EPHEMERAL_VOLUME_SIZE` (default: `10Gi`)
 

diff --git a/tutorbackup/patches/k8s-jobs b/tutorbackup/patches/k8s-jobs
@@ -111,6 +111,7 @@ spec:
   schedule: '{{ BACKUP_K8S_CRONJOB_BACKUP_SCHEDULE }}'
   startingDeadlineSeconds: {{ BACKUP_K8S_CRONJOB_STARTING_DEADLINE_SECONDS }}
   successfulJobsHistoryLimit: {{ BACKUP_K8S_CRONJOB_HISTORYLIMIT_SUCCESS }}
+  concurrencyPolicy: '{{ BACKUP_K8S_CRONJOB_CONCURRENCYPOLICY }}'
   jobTemplate:
     spec:
       template:
@@ -217,6 +218,7 @@ spec:
   schedule: '{{ BACKUP_K8S_CRONJOB_RESTORE_SCHEDULE }}'
   startingDeadlineSeconds: {{ BACKUP_K8S_CRONJOB_STARTING_DEADLINE_SECONDS }}
   successfulJobsHistoryLimit: {{ BACKUP_K8S_CRONJOB_HISTORYLIMIT_SUCCESS }}
+  concurrencyPolicy: '{{ BACKUP_K8S_CRONJOB_CONCURRENCYPOLICY }}'
   jobTemplate:
     spec:
       template:

diff --git a/tutorbackup/plugin.py b/tutorbackup/plugin.py
@@ -21,6 +21,7 @@
         "K8S_CRONJOB_BACKUP_SCHEDULE": "0 0 * * *",
         "K8S_CRONJOB_RESTORE_ENABLE": False,
         "K8S_CRONJOB_RESTORE_SCHEDULE": "30 0 * * *",
+        "K8S_CRONJOB_CONCURRENCYPOLICY": "Forbid",
         "K8S_USE_EPHEMERAL_VOLUMES": False,
         "K8S_EPHEMERAL_VOLUME_SIZE": "10Gi",
         "S3_HOST": "{{ S3_HOST | default('') }}",