Introduce aliases for data streams

[martijnvg]

Currently aliases can only refer to indices. This issue is about extending that ability to data streams.

In order to reduce complexity aliases for data streams will behave differently than aliases for indices. Data stream aliases should only be able to refer to data streams. A data stream alias should not be able to refer to a backing index or any other regular index.

Aliases pointing to data streams will implementation wise be different than aliases pointing to indices. Data stream aliases will refer to data stream names. This should automatically allow resolvability to all backing indices of a data stream, even when rollovers have occurred. Data stream aliases are separately stored next to data streams in the cluster state.

Authorization shouldn't be defined on the alias level, but be determined by the authorization defined on the data stream level.

Ideally we should reuse the current aliases api in order to allow users to define aliases for data streams.

Data stream aliases should at least support the following functionality:

• Being able to refer to one or more data streams by a common name. For example logs alias that point to logs-http and logs-my-application data stream.
• Being able to define which data stream is the write data stream, so ingesting can happen via a data stream alias. Write requests are then resolved to the write index of the data stream designated as write data stream. This to support fail over scenario in a bi-direction ccr setup (as is described here).

I currently do not see use cases where we need to be able to support index and search routing on aliases for data streams.
Being able to define a filter on a data stream alias seems to make sense to me, but I think when needed can be added as a follow up.

Tasks:

• Basic alias support for aliases. This includes being able to create aliases to data streams via update aliases API and view these aliases via get aliases API. Add basic alias support for data streams #72613
• Add support for adding a query to a data stream aliases that always get executed as a filter whenever the corresponding data stream alias gets used (filtered aliases). Add filter support to data stream aliases #74784
• Add support for creating data stream aliases as part of data stream creation from a template. Create data stream aliases from template #73867
• Add support for hidden data stream aliases.
• Add write alias support to data stream aliases. This to support bi-directional replication of data streams with ccr. Add support for is_write_index flag to data stream aliases. #73462
• Adjust CCR to replicate data stream aliases to follower cluster. Replicate data stream aliases via CCR #73988
• Support aliases that point to both regular indices and data streams

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[danhermann]

• Authorization shouldn't be defined on the alias level, but be determined by the authorization defined on the data stream level.

Big +1 on this one. That will simplify lots of things.

In order to reduce complexity aliases for data streams will behave differently than aliases for indices. Data stream aliases should only be able to refer to data streams. A data stream alias should not be able to refer to a backing index or any other regular index.

Not allowing aliases to refer to regular indices and data streams at the same time seems like a big limitation. I suppose that limitation could be relaxed later? I suppose authorization would be tricky to determine for aliases pointing to data streams and regular indices.

[martijnvg]

Not allowing aliases to refer to regular indices and data streams at the same time seems like a big limitation. I suppose that limitation could be relaxed later?

We could think about relaxing it later, because essentially even a data stream will resolve to indices. However like you mention there can be complications, especially if we endup having different authorization schemes for aliases pointing to indices and aliases pointing to data streams.