Is it possible to support indexing of dynamic variables using rollover?

[zcola]

output {
elasticsearch {
hosts => [ 'ccc.om:9200' ]
ilm_enabled => true
ilm_rollover_alias => "cbg_%{product}_%{log}_loghub"
}
}

def setup_ilm
      return unless ilm_enabled?
      if default_index?(@index) || !default_rollover_alias?(@ilm_rollover_alias)
        logger.warn("Overwriting supplied index #{@index} with rollover alias #{@ilm_rollover_alias}") unless default_index?(@index)
        @index = @ilm_rollover_alias
        maybe_create_rollover_alias
        maybe_create_ilm_policy
      end
    end

Now it is time to determine if rolloverneeds to create rollover when it starts.

@robbavey

[zcola]

If ES bulk can determine that the target index is not a rollover index and returns an error to logstash , then should it be able to create it automatically?

[ppf2]

This does not appear to work today.

Without ILM, Logstash variable substitution works as expected:

index => "mytest-%{[@metadata][index]}-%{+YYYY.MM.dd}" 

But the equivalent for ILM does not work. Both ilm_rollover_alias and ilm_pattern settings throw errors when attempting to use Logstash substitution syntax:

ilm_rollover_alias => "logstash-%{[@metadata][index]}"
ilm_pattern => "{now/d}-000001"

[2019-05-30T13:36:58,042][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Malformed escape pair at index 18: /_template/logstash-%{[@metadata][index]}", :class=>"Java::JavaNet::URISyntaxException", 

ilm_rollover_alias => "logstash" 
ilm_pattern => "%{[@metadata][index]}-{now/d}-000001"

LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://node1:9200/%253C%2525%257B%255B%2540metadata%255D%255Bindex%255D%257D-%257Bnow%252Fd%257D-000001%253E'
                    perform_request at /Users/ppf2/Elastic/ElasticStack_6_0/6.7.0/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80

[ppf2]

Until this is addressed, it will be helpful to set expectations in the documentation in these sections:

• Should we tell Beats -> LS users to avoid using LS/ILM integration until the enhancement is made?
  • https://www.elastic.co/guide/en/beats/libbeat/6.8/breaking-changes-6.0.html#breaking-changes-ls-index (we actually state here that it is a requirement to substitute in beat + beat version into the index setting so ILM users will be looking for its equivalent)
  • https://www.elastic.co/guide/en/beats/metricbeat/current/logstash-output.html#logstash-output (same for each beat type)
• Add a note here to indicate that the ILM settings do not support variable substitution notation in Logstash: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm

[Battleroid]

This is unfortunate; it's possible to use the substitutions on ilm_policy (I confused this with something else), but nothing else. This makes setting up multiple RO aliases a bit difficult.

[Battleroid]

I managed to cobble something terrible together that appears to work (see here). It's not great, but it is functional. I still have some things to tinker with but at least this works for now.

Sample run/config as an example here:
https://gist.github.com/Battleroid/beec2b88c9b59fd3665defa08162c4fb

[Battleroid]

I've made some additional changes to this to allow it to do the creation of the rollover index with the specified pattern, but without setting the settings, etc. When I tried this initially without the modifications the cluster came to a complete stop after a few hours, filled with ilm-set-step-info tasks.

Letting logstash just create the aliases seems to be a good compromise. We are able to have something external do the process of rollover. We don't have to worry about creating the index patterns or about ILM rollover related tasks clogging things up.

[jeffrysleddens]

We also would love to use ILM within logstash but were hit by this issue. We would like to use dynamic naming in the ilm_rollover_alias setting like we used to in the index setting.

A short search learns that quite a few people run into this limitation with ilm in the elasticsearch output:
https://discuss.elastic.co/t/es-output-using-ilm-settings-malformed-escape-pair-at-index/178117
https://discuss.elastic.co/t/use-of-metadata-in-ilm-configuration/182918

[AndrewMcQuerry]

Same here. We would love to leverage the ilm_* settings within logstash to help manage this but were stopped dead-in-the-water due to this limitation.

[jeffrey-e]

Same issue here, +1 for adding this feature

[astraios]

+1 here, being able to dynamically set ilm_rollover_alias/ilm_policy using the same syntax as the index field would be huge.

[spencergilbert]

I don't see how ilm without dynamic setting like index is even usable if you're not looking to hand curate all of your indices, or wanting to work outside of the cluster to provide the functionality

[msvechla]

I've opened an issue about this scenario when ILM initially was introduced. Having the ability to use variables inside these settings is essential for us to make dynamic rollover patterns work.

This feature would be a huge benefit for our large elasticsearch infrastructure.

EDIT: Here the request from 2018 about this feature: #805 (comment)

[nicolas-123]

+1, also looking to set ilm_rollover_alias dynamically