OpenSearch Snapshot repo config using Operator (#370)

#### Related issue: #278 #### Key Points: - Added logic to configure the snapshot repo settings for the OpenSearch cluster. - Multiple snapshot repo's can be configured at the same time. - Added logic to run a k8s job to call OpenSearch API to add the user configured snapshot repo settings. - The setup expects the following prerequisites are met: 1) The related plugins (ex repository-s3) are installed using [add plugins](https://github.com/Opster/opensearch-k8s-operator/blob/main/docs/userguide/main.md#add-plugins) method. 2) The required roles/permissions for the backend cloud are pre-created. 3) Since add a snapshot repo configuration should be done after all the cluster nodes are up and ready, ensure the cluster is fully healthy before adding the snapshot settings. - Once the Snapshot setting is added and applied, user can create the right policies from OpenSearch Dashboard (or via snapshot management API) to run the snapshots to the configured repos (which is done by the operator) #### Sample configuration: NOTE: Add the `snapshot` setting and apply the config file, only after the cluster is fully functional. ``` apiVersion: opensearch.opster.io/v1 kind: OpenSearchCluster metadata: name: my-first-cluster spec: security: tls: http: generate: true transport: generate: true perNode: true general: snapshotRepositories: - name: my_s3_repository_3 type: s3 settings: bucket: opensearch-s3-snapshot region: us-east-1 base_path: os-snapshot_3 - name: my_s3_repository_4 type: s3 settings: bucket: opensearch-s3-snapshot region: us-east-1 base_path: os-snapshot_1 httpPort: 9400 serviceName: my-first-cluster version: 2.6.0 pluginsList: ["repository-s3"] drainDataNodes: true dashboards: version: 2.6.0 enable: true replicas: 1 nodePools: - component: masters replicas: 3 persistence: emptyDir: {} roles: - "data" - "cluster_manager" ```
opensearch-project · Apr 12, 2023 · 4f346e5 · 4f346e5
2 parents 0bea273 + 18f8d0a
commit 4f346e5
Show file tree

Hide file tree

Showing 13 changed files with 377 additions and 9 deletions.
diff --git a/.gitignore b/.gitignore
@@ -18,5 +18,6 @@ testbin/
 *.bin
 
 .vscode
+.idea/
 
-opensearch-operator/functionaltests/kubeconfig
+opensearch-operator/functionaltests/kubeconfig
diff --git a/charts/opensearch-operator/templates/opensearchclusters.opensearch.opster.io-crd.yaml b/charts/opensearch-operator/templates/opensearchclusters.opensearch.opster.io-crd.yaml
@@ -3216,6 +3216,22 @@ spec:
                     type: string
                   setVMMaxMapCount:
                     type: boolean
+                  snapshotRepositories:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                        settings:
+                          additionalProperties:
+                            type: string
+                          type: object
+                        type:
+                          type: string
+                      required:
+                      - name
+                      - type
+                      type: object
+                    type: array
                   vendor:
                     enum:
                     - Opensearch
@@ -3249,6 +3265,33 @@ spec:
                       type: object
                       x-kubernetes-map-type: atomic
                     type: array
+                  resources:
+                    description: ResourceRequirements describes the compute resource
+                      requirements.
+                    properties:
+                      limits:
+                        additionalProperties:
+                          anyOf:
+                          - type: integer
+                          - type: string
+                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                          x-kubernetes-int-or-string: true
+                        description: 'Limits describes the maximum amount of compute
+                          resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
+                        type: object
+                      requests:
+                        additionalProperties:
+                          anyOf:
+                          - type: integer
+                          - type: string
+                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                          x-kubernetes-int-or-string: true
+                        description: 'Requests describes the minimum amount of compute
+                          resources required. If Requests is omitted for a container,
+                          it defaults to Limits if that is explicitly specified, otherwise
+                          to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
+                        type: object
+                    type: object
                   version:
                     type: string
                 type: object

diff --git a/docs/designs/crd.md b/docs/designs/crd.md
@@ -210,6 +210,12 @@ GeneralConfig defines global Opensearch cluster configuration
         <td>Set the security context for the cluster pods' containers.</td>
         <td>false</td>
         <td> - </td>
+      </tr><tr>
+        <td><b>snapshotRepositories</b></td>
+        <td>[]SnapshotRepoConfig</td>
+        <td>Snapshot Repo settings</td>
+        <td>false</td>
+        <td> - </td>
       </tr>
 </table>
 

diff --git a/docs/userguide/main.md b/docs/userguide/main.md
@@ -246,7 +246,7 @@ Please note:
 * Updating the list for an already installed cluster will lead to a rolling restart of all opensearch nodes to install the new plugin.
 * If your plugin requires additional configuration you must provide that either through `additionalConfig` (see section [Configuring opensearch.yml](#configuring-opensearchyml)) or as secrets in the opensearch keystore (see section [Add secrets to keystore](#add-secrets-to-keystore)).
 
-## Add secrets to keystore
+### Add secrets to keystore
 
 Some OpenSearch features (e.g. snapshot repository plugins) require sensitive configuration. This is handled via the opensearch keystore. The operator allows you to populate this keystore using Kubernetes secrets.
 To do so add the secrets under the `general.keystore` section:
@@ -307,7 +307,78 @@ spec:
         roles:
           - "data"
 ```
+### Configuring Snapshot Repo (BETA):
 
+This feature is Currently in BETA, you can configure the snapshot repo settings for the OpenSearch cluster through the operator. Using `snapshotRepositories` settings you can configure multiple snapshot repos. Once the snapshot repo is configured a user can create custom `_ism` policies through dashboard to backup the in indexes.
+
+Note: BETA flagged Features in a release are experimental. Therefore, we do not recommend the use of configuring snapshot repo in a production environment. For updates on the progress of snapshot/restore, or if you want leave feedback/contribute that could help improve the feature, please refer to the issue on [GitHub](https://github.com/Opster/opensearch-k8s-operator/issues/278).
+
+```yaml
+spec:
+  general:
+    snapshotRepositories: 
+        - name: my_s3_repository_1
+          type: s3
+          settings:
+            bucket: opensearch-s3-snapshot
+            region: us-east-1
+            base_path: os-snapshot
+        - name: my_s3_repository_3
+          type: s3
+          settings:
+            bucket: opensearch-s3-snapshot
+            region: us-east-1
+            base_path: os-snapshot_1
+```
+#### Prerequisites for Configuring Snapshot Repo:
+
+Before applying the setting `snapshotRepositories` to the operator, please ensure the following prerequisites are met.
+
+1. The right cloud provider native plugins are installed.
+Example:
+```yaml
+spec:
+  general:
+    pluginsList: ["repository-s3"]
+```
+
+2. Ensure the cluster is fully healthy before applying the `snapshotRepositories` settings to the custom resource. 
+Note: For the BETA you cannot add the settings if the cluster is not yet provisioned and healthy, otherwise the configuration of the repositories will fail.
+
+3. The required roles/permissions for the backend cloud are pre-created.
+Example: Following is the AWS IAM role added for kubernetes nodes so that snapshots can be published to `opensearch-s3-snapshot` s3 bucket.
+```json
+{
+    "Statement": [
+        {
+            "Action": [
+                "s3:ListBucket",
+                "s3:GetBucketLocation",
+                "s3:ListBucketMultipartUploads",
+                "s3:ListBucketVersions"
+            ],
+            "Effect": "Allow",
+            "Resource": [
+                "arn:aws:s3:::opensearch-s3-snapshot"
+            ]
+        },
+        {
+            "Action": [
+                "s3:GetObject",
+                "s3:PutObject",
+                "s3:DeleteObject",
+                "s3:AbortMultipartUpload",
+                "s3:ListMultipartUploadParts"
+            ],
+            "Effect": "Allow",
+            "Resource": [
+                "arn:aws:s3:::opensearch-s3-snapshot/*"
+            ]
+        }
+    ],
+    "Version": "2012-10-17"
+}
+```
 ## Configuring Dashboards
 
 The operator can automatically deploy and manage a OpenSearch Dashboards instance. To do so add the following section to your cluster spec:

diff --git a/go.work.sum b/go.work.sum
@@ -1,29 +1,48 @@
 github.com/asaskevich/govalidator v0.0.0-20200428143746-21a406dcc535/go.mod h1:oGkLhpf+kjZl6xBf758TQhh5XrAeiJv/7FRz/2spLIg=
 github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
 github.com/emicklei/go-restful v2.9.5+incompatible h1:spTtZBk5DYEvbxMVutUuTyh1Ao2r4iyvLdACqsl/Ljk=
+github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84=
 github.com/form3tech-oss/jwt-go v3.2.3+incompatible h1:7ZaBxOI7TMoYBfyA3cQHErNNyAWIKUMIwqxEtgHOs5c=
+github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g=
 github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
 github.com/googleapis/gnostic v0.5.5 h1:9fHAtK0uDfpveeqqo1hkEZJcFvYXAiCN3UutL8F9xHw=
 github.com/mitchellh/mapstructure v1.4.1/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
 github.com/onsi/ginkgo/v2 v2.1.4/go.mod h1:um6tUpWM/cxCK3/FK8BXqEiUMUwRgSM4JXG47RKZmLU=
 github.com/onsi/ginkgo/v2 v2.3.0/go.mod h1:Eew0uilEqZmIEZr8JrvYlvOM7Rr6xzTmMV8AyFNU9d0=
+github.com/onsi/ginkgo/v2 v2.3.1 h1:8SbseP7qM32WcvE6VaN6vfXxv698izmsJ1UQX9ve7T8=
+github.com/onsi/ginkgo/v2 v2.3.1/go.mod h1:Sv4yQXwG5VmF7tm3Q5Z+RWUpPo24LF1mpnz2crUb8Ys=
 github.com/onsi/gomega v1.19.0/go.mod h1:LY+I3pBVzYsTBU1AnDwOSxaYi9WoWiqgwooUqq9yPro=
 github.com/onsi/gomega v1.22.0/go.mod h1:iYAIXgPSaDHak0LCMA+AWBpIKBr8WZicMxnE8luStNc=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
 github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
+golang.org/x/crypto v0.0.0-20220315160706-3147a52a75dd h1:XcWmESyNjXJMLahc3mqVQJcgSTDxFxhETVlfk9uGc38=
 golang.org/x/crypto v0.1.0 h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU=
 golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw=
 golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/mod v0.6.0/go.mod h1:4mET923SAdbXp2ki8ey+zGs1SLqsuM2Y0uvdZR/fUNI=
 golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20220319134239-a9b59b0215f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/tools v0.1.10/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.2.0/go.mod h1:y4OqIKeOV/fWJetJ8bXPU1sEVniLMIyDAZWeHdV+NTA=
 k8s.io/gengo v0.0.0-20210813121822-485abfe95c7c/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E=
 k8s.io/gengo v0.0.0-20211129171323-c02415ce4185/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E=
+k8s.io/klog/v2 v2.70.1 h1:7aaoSdahviPmR+XkS7FyxlkkXs6tHISSG03RxleQAVQ=
+k8s.io/klog/v2 v2.70.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0=
+k8s.io/kube-openapi v0.0.0-20220803162953-67bda5d908f1 h1:MQ8BAZPZlWk3S9K4a9NCkIFQtZShWqoha7snGixVgEA=
+k8s.io/kube-openapi v0.0.0-20220803162953-67bda5d908f1/go.mod h1:C/N6wCaBHeBHkHUesQOQy2/MZqGgMAFPqGsGQLdbZBU=
+k8s.io/utils v0.0.0-20220728103510-ee6ede2d64ed h1:jAne/RjBTyawwAy0utX5eqigAwz/lQhTmy+Hr/Cpue4=
+k8s.io/utils v0.0.0-20220728103510-ee6ede2d64ed/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
diff --git a/opensearch-operator/api/v1/opensearch_types.go b/opensearch-operator/api/v1/opensearch_types.go
@@ -49,7 +49,8 @@ type GeneralConfig struct {
 	// Additional volumes to mount to all pods in the cluster
 	AdditionalVolumes []AdditionalVolume `json:"additionalVolumes,omitempty"`
 	// Populate opensearch keystore before startup
-	Keystore []KeystoreValue `json:"keystore,omitempty"`
+	Keystore             []KeystoreValue      `json:"keystore,omitempty"`
+	SnapshotRepositories []SnapshotRepoConfig `json:"snapshotRepositories,omitempty"`
 	// Set security context for the cluster pods
 	PodSecurityContext *corev1.PodSecurityContext `json:"podSecurityContext,omitempty"`
 	// Set security context for the cluster pods' container
@@ -237,6 +238,12 @@ type KeystoreValue struct {
 	KeyMappings map[string]string `json:"keyMappings,omitempty"`
 }
 
+type SnapshotRepoConfig struct {
+	Name     string            `json:"name"`
+	Type     string            `json:"type"`
+	Settings map[string]string `json:"settings,omitempty"`
+}
+
 // ClusterSpec defines the desired state of OpenSearchCluster
 type ClusterSpec struct {
 	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster

diff --git a/opensearch-operator/api/v1/zz_generated.deepcopy.go b/opensearch-operator/api/v1/zz_generated.deepcopy.go
diff --git a/opensearch-operator/config/crd/bases/opensearch.opster.io_opensearchclusters.yaml b/opensearch-operator/config/crd/bases/opensearch.opster.io_opensearchclusters.yaml
@@ -3216,6 +3216,22 @@ spec:
                     type: string
                   setVMMaxMapCount:
                     type: boolean
+                  snapshotRepositories:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                        settings:
+                          additionalProperties:
+                            type: string
+                          type: object
+                        type:
+                          type: string
+                      required:
+                      - name
+                      - type
+                      type: object
+                    type: array
                   vendor:
                     enum:
                     - Opensearch
@@ -3249,6 +3265,33 @@ spec:
                       type: object
                       x-kubernetes-map-type: atomic
                     type: array
+                  resources:
+                    description: ResourceRequirements describes the compute resource
+                      requirements.
+                    properties:
+                      limits:
+                        additionalProperties:
+                          anyOf:
+                          - type: integer
+                          - type: string
+                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                          x-kubernetes-int-or-string: true
+                        description: 'Limits describes the maximum amount of compute
+                          resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
+                        type: object
+                      requests:
+                        additionalProperties:
+                          anyOf:
+                          - type: integer
+                          - type: string
+                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                          x-kubernetes-int-or-string: true
+                        description: 'Requests describes the minimum amount of compute
+                          resources required. If Requests is omitted for a container,
+                          it defaults to Limits if that is explicitly specified, otherwise
+                          to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
+                        type: object
+                    type: object
                   version:
                     type: string
                 type: object

diff --git a/opensearch-operator/go.mod b/opensearch-operator/go.mod
@@ -5,7 +5,7 @@ go 1.19
 require (
 	github.com/Masterminds/semver v1.5.0
 	github.com/banzaicloud/k8s-objectmatcher v1.8.0
-	github.com/banzaicloud/operator-tools v0.28.8
+	github.com/banzaicloud/operator-tools v0.28.10
 	github.com/go-logr/logr v1.2.3
 	github.com/hashicorp/go-version v1.6.0
 	github.com/jarcoal/httpmock v1.2.0

diff --git a/opensearch-operator/go.sum b/opensearch-operator/go.sum
@@ -95,6 +95,8 @@ github.com/banzaicloud/k8s-objectmatcher v1.8.0 h1:Nugn25elKtPMTA2br+JgHNeSQ04sc
 github.com/banzaicloud/k8s-objectmatcher v1.8.0/go.mod h1:p2LSNAjlECf07fbhDyebTkPUIYnU05G+WfGgkTmgeMg=
 github.com/banzaicloud/operator-tools v0.28.8 h1:juokC+0E7EnkdlmGR9Yn5UasRS6a5sNV4NTe+kFtpMA=
 github.com/banzaicloud/operator-tools v0.28.8/go.mod h1:WhWEQk/6F951K55DX6/cyGKqo6ymLYy1SEN54tCWnqo=
+github.com/banzaicloud/operator-tools v0.28.10 h1:mm+LvL31GV9BL//zs/CTKVcFdHHyZ+KIbuNoXrOo3gA=
+github.com/banzaicloud/operator-tools v0.28.10/go.mod h1:PhwQ6bn1blX+SuH5ALj3Fwvvmsi1KdAwke8NufILY8I=
 github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8=
 github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
-Original file line number
+Diff line change
@@ Expand Up / @@ -18,5 +18,6 @@ testbin/ @@
     *.bin
     .vscode
+    .idea/
-    opensearch-operator/functionaltests/kubeconfig
+    opensearch-operator/functionaltests/kubeconfig