Support migrating indexes to data streams.

[anders-swanson]

Is your feature request related to a problem? Please describe.
OpenSearch supports both indexes and data streams. It would be good for users of an OpenSearch cluster to be able to seamlessly migrate their existing indexes to data stream.

Describe the solution you'd like
An API under _data_stream (.e.g., _data_stream/_migrate), that accepts a POST of the indexes to migrate to data streams.

Describe alternatives you've considered
I have been looking for a seamless way to migrate existing indexes to data streams (I want to take full advantage of data streams), without impacting the end-user experience. Not being able to migrate causes a poor user experience, as users may need to look at multiple data streams/indexes to get all their data.

[dblock]

What's the current workaround using client-side tools? Does that have downside other than performance (and how much)?

[anders-swanson]

@dblock you may use the reindex API to migrate data from an existing index to a data stream, however the caveat is that existing index patterns may or may not match the new data stream name.
Any existing read/write clients must be updated to reference the data stream.

[dblock]

I don't think I fully understand this, help me out with an example?

[mgentry612]

@dblock , here is the equivalent functionality in Elasticsearch:

• Migrate existing indices to a data stream elastic/elasticsearch#61046
• REST endpoint and tests for data stream migration elastic/elasticsearch#64415

[dblock]

Thanks. If anyone wants to contribute, please feel free to implement this feature in OpenSearch without looking at any non-APLv2 (ES) code.

[anubisg1]

i'm stuck with this and i am not sure i understand how to migrate within opensearch

[arianitu]

Can this be added? Right now if you use Amazon OpenSearch Service and migrate a data stream (backing indexes) to cold storage via an ISM policy, you can never add it back to the data stream.

This is problematic since that means it breaks everything you rely on when you use data streams (index patterns, etc) when migrating back from cold storage to warm/hot. You would basically need to add rules for the data stream its self and then the indexes that are migrated back from cold storage.

In my opinion, this makes data streams not very appealing at all. Also, is there any logic to not allowing an API that simply puts an index into an existing data stream, what's preventing that from being possible?

[arianitu]

Also, for Amazon OpenSearch, ISM policies don't even work on the data stream name directly (you must match the underlying Indices such as .ds-data-stream*) overall, I honestly feel like data stream integration is absolutely awful in OpenSearch. How can it even be released like this?