Data Stream Settings expects manage_template to be false (not true) to avoid creating the ecs-logstash template

[herrBez]

Logstash information:

Please include the following information:

1. Logstash version: logstash 8.4.3
2. Logstash installation source: tar
3. How is Logstash being run: command line
4. How was the Logstash Plugin installed: the plugin was shipped with the product

JVM (e.g. java -version): Builtin JVM

openjdk 17.0.4 2022-07-19
OpenJDK Runtime Environment Temurin-17.0.4+8 (build 17.0.4+8)

OS version (uname -a if on a Unix-like system): Linux LAPTOP-LFCO0LFS 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

The manage_template setting is not compatible with the data_stream settings. When data_stream settings are active and the Logstash Output Plugin is configured with a user without the manage_index_template cluster privilege it encounters 403 errors, because Logstash cannot load the Index Template ecs-logstash.

In my expectation when data_stream is used this template should not be loaded.

I dig a little bit in the source code and I think this line may explain the unexpected behavior: the comment state that manage_template is false by default, but the documentation (and also my experience) shows that manage_template is true by default.

Steps to reproduce:

1. Create a user in Elasticsearch with the monitor cluster privilege and the auto_configure,create index privileges for logs-*
2. Make sure there is no ecs-logstash index template in the Elasticsearch Cluster (the monitor cluster privilege allows to read the index_template, thus if there is already an index_template named ecs-logstash, the plugin will not try to load it again)
3. Configure the following pipeline:

input { generator {} }

filter { sleep { time => 1 } }

output {
      elasticsearch {
        user => "logstash_writer"
        password => "XXXX"
        data_stream => "true"
        data_stream_dataset => "apache.access"
        data_stream_namespace => "default"
        data_stream_type => "logs"
        hosts => "https://XXXXX:9200"
      }
}

4. It should immediately fail with the error reported in the next section of the issue

Provide logs (if relevant):

[2022-10-19T07:38:51,515][ERROR][logstash.outputs.elasticsearch][another_test] Failed to install template {:message=>"Got response code '403' contacting Elasticsearch at URL 'https://test-fleet-12d4a4.es.us-central1.gcp.cloud.es.io:443/_index_template/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:408:in `template_put'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:494:in `install_template'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:318:in `finish_register'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:283:in `block in register'", "/home/mirko/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:154:in `block in after_successful_connection'"]}

Possible Related Issues

• Logstash throws a "Failed to install template" on events sent from Elastic Agent #1071
• 'action' doesn't default to 'create' (as datastreams is default) #1070