Add require_data_stream feature

[eyalkoren]

Closes #97032

Adding the ability to set require_data_stream parameter (boolean) on several APIs.
For document indexing and update, this flag affects also auto creation of underlying index - if set to true, an index will be created only if a matching index template is found and it contains a data stream template.

[elasticsearchmachine]

Hi @eyalkoren, I've created a changelog YAML for you.

[dakrone]

This looks like a good general direction to me. I think we should check whether writing to an alias that points to a data stream will still work with the require_data_stream flag (and potentially whether it should or should not.) I left a few minor comments since this is still a draft PR

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

Alright, I've pulled out the update support, and made this flag mean "if require_data_stream is set, the indexing operation needs to be targeting a data stream, or a data stream that will be created by a template". I've also added more tests for this. Hopefully it's clearer and not too bad to review.

[jbaiera]

LGTM! 🚢

[dakrone]

@elasticmachine update branch

[axw]

🎉 thank you @dakrone and @eyalkoren!

[leandrojmp]

Hello @dakrone and @eyalkoren

Does this change have any impact when using Logstash elasticsearch output to index data on data streams, but on data streams that have a custom naming pattern?

We are planning the upgrade to 8.13 and checking the release notes, we have Logstash writing to a custom data stream and since logstash does not support custom data stream names, we use data_stream => false and on the index option we point to the data stream alias name and use the action as a create.

Our logstash outputs are like this:

output {
    elasticsearch {
        hosts => ["HOSTS"]
        index => "data-stream-name"
        action => "create"
        http_compression => true
        data_stream => false
        manage_template => false
        ilm_enabled => false
        cacert => 'ca.crt'
        user => 'USER'
        password => 'PASSWORD'
    }
}

Not sure if this change here will break this.

[axw]

@leandrojmp this is an opt-in feature, so it will have no effect on your use case. You would need to reconfigure Logstash to pass ?require_data_stream=true in the _bulk API requests for it to have any effect.

[dmytro-freger-mesalvo]

Hi,

I have in my exporter:

  elasticsearch:
     endpoints: ["https://elasticsearch-es-http:9200"]
     logs_index: "logs-test"
     logs_dynamic_index:
       enabled: false
     user: "elastic"
     password: "ILJ0lVi91D91z4gfe1202YZO"
     tls:
       insecure: false
       ca_file: "/etc/otel/certs/tls.crt"
       insecure_skip_verify: false

How can I set require_data_stream to false?

thank you

[eyalkoren]

@dmytro-freger-mesalvo I am not sure this is supported. Why are you trying to set it to false?
@carsonip @lahsivjar - can you assist with this?

[carsonip]

Hi @dmytro-freger-mesalvo , do you mind either

• creating an issue in https://github.com/open-telemetry/opentelemetry-collector-contrib under component exporter/elasticsearch; or
• pinging me on CNCF community slack in channel #opentelemetry

I'll be able to assist you further.