[BUG] Opensearch operator does not properly append the opensearch.yml file when general.additionalConfig is specified

[digitalray]

What is the bug?

When you specify key pairs to general.additionalConfig, the operator adds those to the env instead of the opensearch.yml file. This is not the case when specifying dashboard.additionalConfig. In the Dashboard case, the additional values are appended to the dashboard config file.

How can one reproduce the bug?

1. Specify general.additionalConfig and dashboard.additionalConfig in OpensearchCluster manifest and deploy the cluster.

general:
    additionalConfig:
      s3.client.default.endpoint: "s3.test.svc.cluster.local:9020"
      s3.client.default.protocol: "https"
      s3.client.default.max_retries: "3"
      s3.client.default.path_style_access: "false"
      s3.client.default.read_timeout: "50s"
      s3.client.default.use_throttle_retries: "true"

dashboard:
    additionalConfig:
      logging.verbose: "true"
      opensearch.requestHeadersWhitelist: |
        ["authorization", "securitytenant"]
      opensearch_security.auth.type: |
        ["basicauth", "openid"]
      opensearch_security.auth.multiple_auth_enabled: "true"
      opensearch_security.multitenancy.enabled: "true"
      opensearch_security.multitenancy.tenants.preferred: |
        [ "Private", "Global" ]
      opensearch_security.readonly_mode.roles: |
        ["kibana_read_only"]

2. Check config map for opensearch cluster vs dashboard:

kubectl get cm opensearch-cluster-config -n opensearch -o yaml

apiVersion: v1
data:
  opensearch.yml: |
    plugins.security.audit.type: internal_opensearch
    plugins.security.authcz.admin_dn: ["CN=admin,OU=opensearch-cluster"]
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.nodes_dn: ["CN=opensearch-cluster,OU=opensearch-cluster"]
    plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
    plugins.security.ssl.http.enabled: true
    plugins.security.ssl.http.pemcert_filepath: tls-http/tls.crt
    plugins.security.ssl.http.pemkey_filepath: tls-http/tls.key
    plugins.security.ssl.http.pemtrustedcas_filepath: tls-http/ca.crt
    plugins.security.ssl.transport.enforce_hostname_verification: false
    plugins.security.ssl.transport.pemcert_filepath: tls-transport/tls.crt
    plugins.security.ssl.transport.pemkey_filepath: tls-transport/tls.key
    plugins.security.ssl.transport.pemtrustedcas_filepath: tls-transport/ca.crt
    plugins.security.system_indices.enabled: true
    plugins.security.system_indices.indices: [".opendistro-alerting-config",".opendistro-alerting-alert*",".opendistro-anomaly-results*",".opendistro-anomaly-detector*",".opendistro-anomaly-checkpoints",".opendistro-anomaly-detection-state",".opendistro-reports-*",".opendistro-notifications-*",".opendistro-notebooks",".opensearch-observability",".opendistro-asynchronous-search-response*",".replication-metadata-store"]
kind: ConfigMap

kubectl get cm opensearch-cluster-dashboards-config -n opensearch -o yaml

apiVersion: v1
data:
  opensearch_dashboards.yml: |
    logging.verbose: true
    opensearch.requestHeadersWhitelist: ["authorization", "securitytenant"]

    opensearch.ssl.verificationMode: none
    opensearch_security.auth.multiple_auth_enabled: true
    opensearch_security.auth.type: ["basicauth", "openid"]
    opensearch_security.multitenancy.enabled: true
    opensearch_security.multitenancy.tenants.enable_global: true
    opensearch_security.multitenancy.tenants.enable_private: true
    opensearch_security.multitenancy.tenants.preferred: [ "Private", "Global" ]
    opensearch_security.readonly_mode.roles: ["kibana_read_only"]

    server.name: opensearch-cluster-dashboards
kind: ConfigMap

2. Check env vars of opensearch cluster vs dashboard:

kubectl describe sts opensearch-cluster-masters | grep -v "<none>" | grep Env -A 14
    Environment:
      cluster.initial_master_nodes:            opensearch-cluster-bootstrap-0
      discovery.seed_hosts:                    opensearch-cluster-discovery
      cluster.name:                            opensearch-cluster
      network.bind_host:                       0.0.0.0
      network.publish_host:                     (v1:metadata.name)
      OPENSEARCH_JAVA_OPTS:                    -Xmx1024M -Xms1024M -Dopensearch.transport.cname_in_publish_address=true
      node.roles:                              cluster_manager,data
      http.port:                               9200
      s3.client.default.endpoint:              s3.test.svc.cluster.local:9020
      s3.client.default.max_retries:           3
      s3.client.default.path_style_access:     false
      s3.client.default.protocol:              http
      s3.client.default.read_timeout:          50s
      s3.client.default.use_throttle_retries:  true

❯ kubectl describe deployment opensearch-cluster-dashboards | grep -v "<none>" | grep Env -A 5
   Environment:
     OPENSEARCH_HOSTS:     https://opensearch-cluster.opensearch.svc.cluster.local:9200
     SERVER_HOST:          0.0.0.0
     OPENSEARCH_USERNAME:  <set to the key 'username' in secret 'admin-credentials-secret'>  Optional: false
     OPENSEARCH_PASSWORD:  <set to the key 'password' in secret 'admin-credentials-secret'>  Optional: false

What is the expected behavior?

Expected behavior is that when you specify additionalConfig, those key/values need to get appended to the config-map as it is the case with the dashboard. Then after that, the opensearch.yml file in the pod will get updated with those new values. However, what happens is that those values get added to the Environment variables instead while the config-map remains unchanged. Hence, the opensearch.yml file does not get updated as expected.

What is your host/environment?

Kubernetes Version: 1.29.9
Kubernetes Node OS: Alma Linux 8

❯ helm ls -n opensearch-operator
NAME               	NAMESPACE          	REVISION	UPDATED                             	STATUS  	CHART                    	APP VERSION
opensearch-operator	opensearch-operator	5       	2024-12-21 20:41:06.619698 -0800 PST	deployed	opensearch-operator-2.7.0	2.7.0


❯ helm ls -n opensearch
NAME                       	NAMESPACE 	REVISION	UPDATED                             	STATUS  	CHART                   	APP VERSION
opensearch-cluster         	opensearch	22      	2024-12-21 19:13:04.555023 -0800 PST	deployed	opensearch-cluster-2.7.0	2.7.0

opensearchCluster
general.version: 2.14.0
general.image: opensearchproject/opensearch:2.18.0

dashboard.version: 2.14.0
dashboard.image: opensearchproject/opensearch-dashboards:2.18.0
initHelper.image: busybox:latest

Do you have any screenshots?

n/a

Do you have any additional context?

I am trying to add S3 related configuration to the opensearch.yml file via general.additionalConfig section of OpenSearchCluster manifest.

[rameshar16]

Hi Team,

I am using the OpenSearch Operator 2.7.0. I am also facing the similar issue.

Thanks,

[digitalray]

Hi Team,

I am using the OpenSearch Operator 2.7.0. I am also facing the similar issue.

Thanks,

This is what I use as well. The problem is that env vars do not seem to have an effect on opensearch config changes. Unless maybe you need to set them with a certain prefix. But one thing is for sure, that the dashboard config changes are taking effect properly and the yaml file gets updated OK. Logically, the opensearch.yml would also need to get updated the same way.

[OlafRadicke]

I have a similar observation. I am trying to enable mTLS client authentication.

I am using Operator Chart 2.7.0 and the container images in version 2.16.0

In the CRD of the operator I create
plugins.security.authcz.admin_dn: '["CN=client01.opensearch.datenhaus.dev"]'

Logs from pod securityconfig-update I see errors for:

“CN=admin,OU=opensearch-cluster-02”

The content in the generated ConfigMap is:
“CN=admin,OU=opensearch-cluster-02”

In the generated configuration file of the pod (master) I find:
“CN=admin,OU=opensearch-cluster-02”

in the (auto-generated) certificate is again to be found:
OU=opensearch-cluster-02,CN=opensearch-cluster-02

I'm a bit surprised that the “Best practices for OpenSearch security” explicitly recommends client certificate authentication (https://opensearch.org/docs/latest/security/configuration/best-practices/) and then it doesn't work. Apart from that, the documentation is very confusing on the subject.

[digitalray]

I think I now see where I went wrong. It seems like the general.additionalConfig section is meant for opensearch env vars; however, the bootstrap.additionalConfig referenced in: https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/charts/opensearch-cluster/values.yaml#L117-L119 is meant for opensearch.yml changes. I will try the bootstrap value and will report back if that works.

[digitalray]

Just added like so

bootstrap:
    additionalConfig:
      s3.client.default.endpoint: "s3.test.svc.cluster.local:9020"
      s3.client.default.protocol: "https"
      s3.client.default.max_retries: "3"
      s3.client.default.path_style_access: "false"
      s3.client.default.read_timeout: "50s"
      s3.client.default.use_throttle_retries: "true"

Here is my helm diff

  metadata:
   name: opensearch-cluster
   namespace: opensearch
 spec:
+   bootstrap:
+     additionalConfig:
+
+       s3.client.default.endpoint: s3.test.svc.cluster.local:9020
+       s3.client.default.max_retries: "3"
+       s3.client.default.path_style_access: "false"
+       s3.client.default.protocol: http
+       s3.client.default.read_timeout: 50s
+       s3.client.default.use_throttle_retries: "true"
   initHelper:

All 3 masters do restart after the apply. However, when I exec to the pod and cat the config, I do not see the additional configs there.

[opensearch@opensearch-cluster-masters-0 ~]$ cat config/opensearch.yml
plugins.security.audit.type: internal_opensearch
plugins.security.authcz.admin_dn: ["CN=admin,OU=opensearch-cluster"]
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn: ["CN=opensearch-cluster,OU=opensearch-cluster"]
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: tls-http/tls.crt
plugins.security.ssl.http.pemkey_filepath: tls-http/tls.key
plugins.security.ssl.http.pemtrustedcas_filepath: tls-http/ca.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.pemcert_filepath: tls-transport/tls.crt
plugins.security.ssl.transport.pemkey_filepath: tls-transport/tls.key
plugins.security.ssl.transport.pemtrustedcas_filepath: tls-transport/ca.crt
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config",".opendistro-alerting-alert*",".opendistro-anomaly-results*",".opendistro-anomaly-detector*",".opendistro-anomaly-checkpoints",".opendistro-anomaly-detection-state",".opendistro-reports-*",".opendistro-notifications-*",".opendistro-notebooks",".opensearch-observability",".opendistro-asynchronous-search-response*",".replication-metadata-store"]

[digitalray]

In the operator code, there seems to be a function that will add the additional config to the opensearch config here:

opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/reconcilers.go

Lines 86 to 93 in 2e78967

	func (c *ReconcilerContext) AddConfig(key string, value string) {
	_, exists := c.OpenSearchConfig[key]
	if exists {
	fmt.Printf("Warning: Config key '%s' already exists. Will be overwritten\n", key)
	c.recorder.Eventf(c.instance, "Warning", "ConfigDuplicateKey", "Config key '%s' already exists in opensearch config. Will be overwritten", key)
	}
	c.OpenSearchConfig[key] = value
	}

But it is not being used for the general.additionalConfig or bootstrap.additionalconfig. Both of those are being used only to set environment variables:

First here:

opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/cluster.go

Lines 143 to 153 in 2e78967

	extraConfig := helpers.MergeConfigs(r.instance.Spec.General.AdditionalConfig, nodePool.AdditionalConfig)
	sts := builders.NewSTSForNodePool(
	username,
	r.instance,
	nodePool,
	nodePoolConfig.ConfigHash,
	r.reconcilerContext.Volumes,
	r.reconcilerContext.VolumeMounts,
	extraConfig,
	)

Then here:
https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/opensearch-operator/pkg/builders/cluster.go#L549-L556

IMO, I think that there should be separate helm value for env where it should only change env vars and in case of additionalConfig, the AddConfig reconciler should be used to actually modify the opensearch.yml.

[OlafRadicke]

@digitalray I am ready to test a patch.

[swoehrl-mw]

Hi @digitalray
The bootstrap.additionalConfig is only used for the bootstrap pod during inital cluster provisioning, so changing that will have no impact on the config of the nodepools.
general.additionalConfig and nodePools.x.additionalConfig are the right places to configure settings.
Both are not meant as env vars. Instead the operator uses a mechanism from the opensearch docker image that uses the env vars to generate its opensearch.yaml. The configmap is currently only used for some base and security plugin configuration. Everything else happens via the env var mechanism.

It is a known issue that using the env var mechanism leads to problems (see also #883) and that the whole mechanism should be reworked to directly generate the opensearch.yaml in the operator logic.

[swoehrl-mw]

@OlafRadicke Regarding your issue: The operator does not allow setting plugins.security.authcz.admin_dn and similar directly as these are set internally based on the tls configuration.

[OlafRadicke]

@swoehrl-mw does this mean that it is not possible to implement the 'Best practices for OpenSearch security' ( https://opensearch.org/docs/latest/security/configuration/best-practices/ ) and follow the instructions at https://opensearch.org/docs/latest/security/authentication-backends/client-auth/? Because apparently I need the entry plugins.security.ssl.http.clientauth_mode: OPTIONAL or better REQUIRE.