[BUG] securityadmin.sh -backup gives ERR: Seems audit from cluster is not in legacy format: java.io.IOException: A version of 1

[asfoorial]

What is the bug?
I am getting the below error when try to take a backup of the security yml files.

ERR: Seems audit from cluster is not in legacy format: java.io.IOException: A version of 1 can not have a _meta key for AUDIT

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Untar opensearch 2.0.0 and run opensearch-tar-install.sh
2. Ran the below

./securityadmin.sh -backup my-backup-directory
-icl
-nhnv
-cacert …/…/…/config/root-ca.pem
-cert …/…/…/config/node.pem
-key …/…/…/config/node.key

3. Got the below error

ERR: Seems audit from cluster is not in legacy format: java.io.IOException: A version of 1 can not have a _meta key for AUDIT

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: Rocky Linux 8.4
• Version : Opensearch 2.0.0

[cliu123]

[Triage] Thanks for reporting the issue!

[amalgamm]

Same for me. Configs are correctly validated for security admin v7, but the backup detects all configs as version 6
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!

Dropping and recreating .opendistro_security index does not solved the problem.

[cliu123]

Same for me. Configs are correctly validated for security admin v7, but the backup detects all configs as version 6 Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!

Dropping and recreating .opendistro_security index does not solved the problem.

OpenSearch 1.0.0 is a fork of Elasticsearch 7.10.2. It supports migration from ES 7.10.2 to OpenSearch 1.0.0, but doesn't support migration from ES 6 to OpenSearch.

[amalgamm]

OpenSearch 1.0.0 is a fork of Elasticsearch 7.10.2. It supports migration from ES 7.10.2 to OpenSearch 1.0.0, but doesn't support migration from ES 6 to OpenSearch.

There was no migration in my case. My opensearch cluster is quite fresh. It started from scratch on v1.2.4. Week ago I bumped it to 2.0.0 so .opensearch_security index was originally created with securityadmin v7 on opensearch 1.2.4.

Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!

This error appeared after upgrading to 2.0.0

[pawelw1]

I'm getting exactly the same error on a freshly created cluster v 2.0.0.

[j0hnth0m]

Same after upgrading v 1.3.2 to V 2.0.1.

[mousemaxx]

I have some problem after migration ODFE 1.13.1.0 to OpenSearch 2.0.0
Before migration ODFE was checked for the need to update system indexes:
/securityadmin.sh -migrate ....

...
Elasticsearch Version: 7.10.2
Open Distro Security Version: 1.13.1.0
...
Clusterstate: GREEN
Number of nodes: 74
Number of data nodes: 50
.opendistro_security index already exists, so we do not need to create one.
ERR: Seems cluster is already migrated

But after migrating to OpenSearch I still get the error:
securityadmin.sh -backup ....
...
OpenSearch Version: 2.0.0
...
Clusterstate: GREEN
Number of nodes: 74
Number of data nodes: 50
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Will retrieve '/config' into /---/config.yml (legacy mode)
ERR: Seems config from cluster is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "description" (class org.opensearch.security.securityconf.impl.v6.ConfigV6$AuthcDomain), not marked as ignorable (6 known properties: "enabled", "http_enabled", "transport_enabled", "http_authenticator", "authentication_backend", "order"]) at [Source: (String)"{"_meta":{"type":"config","config_version":2},"config":{"dynamic":{"filtered_alias_mode":"warn","disable_rest_auth":false,"disable_intertransport_auth":false,"respect_request_indices_options":false,"license":null,"kibana":{"multitenancy_enabled":true,"server_username":"kibanaserver","index":".kibana"},"http":{"anonymous_auth_enabled":false,"xff":{"enabled":false,"internalProxies":"192\\.168\\.0\\.10|192\\.168\\.0\\.11","remoteIpHeader":"x-forwarded-for"}},"authc":{"basic_internal_auth_domain":{""[truncated 3490 chars]; line: 1, column: 692] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["config"]->org.opensearch.security.securityconf.impl.v6.ConfigV6["dynamic"]->org.opensearch.security.securityconf.impl.v6.ConfigV6$Dynamic["authc"]->org.opensearch.security.securityconf.impl.v6.ConfigV6$Authc["basic_internal_auth_domain"]->org.opensearch.security.securityconf.impl.v6.ConfigV6$AuthcDomain["description"])
Will retrieve '/roles' into /---/roles.yml (legacy mode)
ERR: Seems roles from cluster is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "cluster_permissions" (class org.opensearch.security.securityconf.impl.v6.RoleV6), not marked as ignorable (5 known properties: "tenants", "readonly", "indices", "hidden", "cluster"]) at [Source: (String)"{[truncated 12365 chars]; line: 1, column: 89] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["ITi-Platforms-ITiDefence-OWN"]->org.opensearch.security.securityconf.impl.v6.RoleV6["cluster_permissions"])
Will retrieve '/rolesmapping' into /---/roles_mapping.yml (legacy mode)
ERR: Seems rolesmapping from cluster is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "backend_roles" (class org.opensearch.security.securityconf.impl.v6.RoleMappingsV6), not marked as ignorable (6 known properties: "and_backendroles", "readonly", "users", "backendroles", "hidden", "hosts"]) at [Source: (String)"{[truncated 5877 chars]; line: 1, column: 122] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["---"]->org.opensearch.security.securityconf.impl.v6.RoleMappingsV6["backend_roles"])
Will retrieve '/internalusers' into /---/internal_users.yml (legacy mode)
ERR: Seems internalusers from cluster is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "backend_roles" (class org.opensearch.security.securityconf.impl.v6.InternalUserV6), not marked as ignorable (7 known properties: "readonly", "username", "attributes", "hidden", "password", "roles", "hash"]) at [Source: (String)"{[truncated 8607 chars]; line: 1, column: 132] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["---"]->org.opensearch.security.securityconf.impl.v6.InternalUserV6["backend_roles"])
Will retrieve '/actiongroups' into /---/action_groups.yml (legacy mode)
ERR: Seems actiongroups from cluster is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "allowed_actions" (class org.opensearch.security.securityconf.impl.v6.ActionGroupsV6), not marked as ignorable (3 known properties: "readonly", "permissions", "hidden"]) at [Source: (String)"{; line: 1, column: 63] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["---"]->org.opensearch.security.securityconf.impl.v6.ActionGroupsV6["allowed_actions"])
Will retrieve '/nodesdn' into /---/nodes_dn.yml (legacy mode)
ERR: Seems nodesdn from cluster is not in legacy format: java.io.IOException: A version of 1 can not have a _meta key for NODESDN
Will retrieve '/whitelist' into /data/elk/back/sa1/whitelist.yml (legacy mode)
ERR: Seems whitelist from cluster is not in legacy format: java.io.IOException: A version of 1 can not have a _meta key for WHITELIST
Will retrieve '/audit' into /data/elk/back/sa1/audit.yml (legacy mode)
ERR: Seems audit from cluster is not in legacy format: java.io.IOException: A version of 1 can not have a _meta key for AUDIT

Аt the same time, it is not possible to carry out the migration, when I start
/securityadmin.sh -migrate ....
I get the same error :(

[ggt]

Same issue Here, everything was ok before 2.0.0

[pawelw1]

I'm getting exactly the same error on a freshly created cluster v 2.0.0.

I still have the same issue in 2.0.1

[asfoorial]

I suggest to deprecate the securityadmin tools in favor of the Security REST API. Let the admin user, or users with the right permissions, to manage security configuration through the REST API.

[Jakob3xD]

I suggest to deprecate the securityadmin tools in favor of the Security REST API. Let the admin user, or users with the right permissions, to manage security configuration through the REST API.

How should the config/opensearch-security/config.yml file be managed? I am not aware of an REST API for that.
Moreover I dont think the file should be managed via an API. There are also reserved users which currently can only be managed by file.

[asfoorial]

Below is a call to update the security config. It is all documented here https://opensearch.org/docs/latest/security-plugin/access-control/api/#update-configuration

PUT _plugins/_security/api/securityconfig/config
{
"dynamic": {
"filtered_alias_mode": "warn",
"disable_rest_auth": false,
"disable_intertransport_auth": false,
"respect_request_indices_options": false,
"opensearch-dashboards": {
"multitenancy_enabled": true,
"server_username": "kibanaserver",
"index": ".opensearch-dashboards"
},
"http": {
"anonymous_auth_enabled": false
},
"authc": {
"basic_internal_auth_domain": {
"http_enabled": true,
"transport_enabled": true,
"order": 0,
"http_authenticator": {
"challenge": true,
"type": "basic",
"config": {}
},
"authentication_backend": {
"type": "intern",
"config": {}
},
"description": "Authenticate via HTTP Basic against internal users database"
}
},
"auth_failure_listeners": {},
"do_not_fail_on_forbidden": false,
"multi_rolespan_enabled": true,
"hosts_resolver_mode": "ip-only",
"do_not_fail_on_forbidden_empty": false
}
}

[asfoorial]

Moreover, the current securityadmin utility also calls a (hidden) REST API. I think that API should be visible to the admin user. As for the reserved users such as the admin, you can set it as a regular user first time you start the cluster. In fact, I still wonder why the "reserved" admin is designed not to be able to change his own password!

As far as I have seen, all securityadmin functionality is possible through the REST API except for changing security index replication settings. There is no visible REST API to do it.

[pawelw1]

@asfoorial In the same link you have a note that suggests using securityadmin.sh as it is far safer thank REST API call.
Also to enable PUT API for config update, you need to set plugins.security.unsupported.restapi.allow_securityconfig_modification: option which already suggests that is unspported.

[hume-github]

Existing documentation tells the user to use securityadmin.sh, specifically recommending it over using API calls (which are barely documented). At the same time securityadmin.sh spews errors and 0-byte files when trying to protect existing configurations (as the documentation recommends).

This leaves the user in limbo.

[peternied]

The documentation issues are tracked with opensearch-project/documentation-website#530 If there are more things that you'd like to see please file issues on the documentation website and we can see about providing better details, or if you'd like to improve the docs themselves we would be happy to review pull requests.

[mousemaxx]

Updating to 2.1.0 didn't solve the issue with securityadmin.sh

[TheMeier]

Same problem with a completely fresh install of opensearch 2.1.0

[df-cgdm]

+1

[df-cgdm]

For users and roles we can use the API but I think it is important that we can update the security config in a secure manner (not with API mainly when it's not "supported" => plugins.security.unsupported.restapi.allow_securityconfig_modification )

[asfoorial]

I am in favor of replacing securityadmin tool with the API. I still wonder why it still not supported. It is more consistant to have everything done through the API.

[df-cgdm]

if it's done through API, it would be good to work like the "restricted" user and roles => Only admin user authentified with SSL Certificate should be able to call this API.

[pawelw1]

2.2.0 has still the same issue.

[mysinmyc]

I had the same error by running securityadmin.sh in a cluster v2.0.0 after upgrade from version 1.2.2

In my case it seems that the issue was caused by te following check inside the src/main/java/org/opensearch/security/tools/SecurirtyAdmin.java

         final boolean legacy = createLegacyMode || (indexExists
                    && securityIndex.getMappings() != null
                    && securityIndex.getMappings().get(index) != null);

as workaround i've tried to recompile a copy src/main/java/org/opensearch/security/tools/SecurirtyAdmin.java by forcing legacy to false and in my case it seems to works.

I've compared differences between the two version and previously there was an additional check inside the index mappings

            final boolean legacy = createLegacyMode || (indexExists
                    && securityIndex.getMappings() != null
                    && securityIndex.getMappings().get(index) != null
                    && securityIndex.getMappings().get(index).containsKey("security"));
            

could it be the cause of my issue?

if could help i attach the output of GET /.opendistro_security

{
  ".opendistro_security" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "actiongroups" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "audit" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "config" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "internalusers" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "nodesdn" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "roles" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "rolesmapping" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tenants" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "whitelist" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-all",
        "provided_name" : ".opendistro_security",
        "creation_date" : "1613745476824",
        "number_of_replicas" : "2",
        "uuid" : "hGC3JnmmQ1KIvsmEWVBK5w",
        "version" : {
          "created" : "7100099",
          "upgraded" : "136217827"
        }
      }
    }
  }
}

here part of the output of my securityadmin

./securityadmin.sh -backup /tmp/b -icl -nhnv -cacert ../../../config/root-ca.pem -cert ../../../config/kirk.pem -key ../../../config/kirk-key.pem

---

** This tool will be deprecated in the next major release of OpenSearch **
** #1755 **

---

Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.0.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: xxxxxxxxxxxxxxxxx
Clusterstate: YELLOW
Number of nodes: 6
Number of data nodes: 3
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Will retrieve '/config' into /tmp/b/config.yml (legacy mode)
ERR: Seems config from cluster is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "description" (class org.opensearch.security.securityconf.impl.v6.ConfigV6$AuthcDomain), not marked as ignorable (6 known properties: "enabled", "http_enabled", "transport_enabled", "http_authenticator", "authentication_backend", "order"])
at [Source: (String)"{"_meta":{"type":"config","config_version":2},"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx "[truncated 2394 chars]; line: 1, column: 488] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["config"]->org.opensearch.security.securityconf.impl.v6.ConfigV6["dynamic"]->org.opensearch.security.securityconf.impl.v6.ConfigV6$Dynamic["authc"]->org.opensearch.security.securityconf.impl.v6.ConfigV6$Authc["basic_internal_auth_domain"]->org.opensearch.security.securityconf.impl.v6.ConfigV6$AuthcDomain["description"])
Will retrieve '/roles' into /tmp/b/roles.yml (legacy mode)
ERR: Seems roles from cluster is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "cluster_permissions" (class org.opensearch.security.securityconf.impl.v6.RoleV6), not marked as ignorable (5 known properties: "tenants", "readonly", "indices", "hidden", "cluster"])
at [Source: (String)"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"[truncated 5344 chars]; line: 1, column: 91] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["asynchronous_search_read_access"]->org.opensearch.security.securityconf.impl.v6.RoleV6["cluster_permissions"])
Will retrieve '/rolesmapping' into /tmp/b/roles_mapping.yml (legacy mode)
ERR: Seems rolesmapping from cluster is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "backend_roles" (class org.opensearch.security.securityconf.impl.v6.RoleMappingsV6), not marked as ignorable (6 known properties: "and_backendroles", "readonly", "users", "backendroles", "hidden", "hosts"])

....

[sabil05]

Hello @cwperks / All,

I am also trying to integrate the OpenID connect with Opensearch and I am executing the securityadmin.sh and getting the same error message as mentioned above..

bash-4.2# ./securityadmin.sh -cd /usr/share/opensearch/config/opensearch-security -icl -nhnv \
>  -key /usr/share/opensearch/config/kirk-key.pem \
>  -cert /usr/share/opensearch/config/kirk.pem \
>  -cacert /usr/share/opensearch/config/root-ca.pem
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.2.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
**Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Populate config from /usr/share/opensearch/config/opensearch-security/
ERR: Seems /usr/share/opensearch/config/opensearch-security/config.yml is not in legacy format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "http_enabled" (class org.opensearch.security.securityconf.impl.v7.ConfigV7), not marked as ignorable** (one known property: "dynamic"])
 at [Source: (String)"{"openid_auth_domain":{"http_enabled":true,"transport_enabled":true,"order":0,"http_authenticator":{"type":"openid","challenge":false,"config":{"subject_key":"preferred_username","roles_key":"admin_role","openid_connect_idp":{"enable_ssl":true,"verify_hostnames":false,"openid_connect_url":"https://xyz.com/kums/.well-known/openid-configuration"}}}},"authentication_backend":{"type":"noop"}}"; line: 1, column: 43] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["openid_auth_domain"]->org.opensearch.security.securityconf.impl.v7.ConfigV7["http_enabled"])
Will update '/roles' with /usr/share/opensearch/config/opensearch-security/roles.yml (legacy mode)
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml (legacy mode)
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml (legacy mode)
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml (legacy mode)
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml (legacy mode)
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml (legacy mode)
   SUCC: Configuration for 'whitelist' created or updated
ERR: cannot upload configuration, see errors above

[cwperks]

Hey @sabil05, the fix is scheduled to be included in the 2.3.0 release at the end of this week (9/15/22). This bug impacted 2.0, 2.1 and 2.2.

[cwperks]

@mysinmyc Thank you for the detailed explanation of the incorrect legacy check in the security admin script! A fix for the issue has been scheduled for the 2.3.0 release.

[sabil05]

Hello @cwperks,

Thank you so much for the information.

[davidlago]

We are doing some "spring cleaning in the fall", and to make sure we focus our energies on the right issues and we get a better picture of the state of the repo, we are closing all issues that we are carrying over from the ODFE era (ODFE is no longer supported/maintained, see post here).

If you believe this issue should still be considered for current versions of OpenSearch, apologies! Please let us know by re-opening it.

Thanks!

[asfoorial]

This issue is related to OpenSearch 2.x and not ODFE. Some of us talked about an upcoming fix in 2.3.0 but it is not released yet for us to verify.

[peternied]

I think @davidlago closed out this issue by accident, I'm reopening since there is active discussion and issues that we should address.

[davidlago]

Indeed I did, sorry!

[sabil05]

Hello @cwperks,

I have used 2.3.0 version and executed the securityadmin.sh.. Still getting the errors

The Legacy index error has disappeared.

**Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!

However, still getting the following error message.

Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/config/opensearch-security/
ERR: Seems /usr/share/opensearch/config/opensearch-security/config.yml is not in OpenSearch Security 7 format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "http_enabled" (class org.opensearch.security.securityconf.impl.v7.ConfigV7), not marked as ignorable (one known property: "dynamic"])
 at [Source: (String)"{"openid_auth_domain":{"http_enabled":true,"transport_enabled":true,"order":0,"http_authenticator":{"type":"openid","challenge":false,"config":{"subject_key":"preferred_username","roles_key":"admin_role","openid_connect_idp":{"enable_ssl":true,"verify_hostnames":false,"openid_connect_url":"https://xyz.com/auth/realms/kums/.well-known/openid-configuration"}}}},"authentication_backend":{"type":"noop"}}"; line: 1, column: 43] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["openid_auth_domain"]->org.opensearch.security.securityconf.impl.v7.ConfigV7["http_enabled"])

[cwperks]

@sabil05 Let's move this conversation to the forum here: https://forum.opensearch.org/t/openid-connect-integration-with-opensearch/10876/9

It sounds like you encountered this bug when running security admin, but there is a separate configuration issue preventing the nodes from starting up that is unrelated to this bug.

[stephen-crawford]

[CLOSED] Closed because original issue was addressed. Any further issues are encouraged to file a new issue ticket.