Allow restoring data streams from snapshot that already exist in a cluster?

[martijnvg]

Currently it isn't possible to restore a data stream in a cluster that already has a data stream with the same name.
We do support restoring indices with the same name in a cluster. In order for that to work the indices to be restored
need to be closed. This is currently an issue for data streams, since the write index of a data stream is not allowed to be closed.

When restoring an entire snapshot (with all indices, global state) into an existing cluster this is problematic, since that entire operation will fail if there are data streams part of the cluster. So I think we should find a way to support restoring data streams in place in a similar way indices can be restored in place.

The reason closing a write index is not allowed, is to ensure that a data stream always can accept writes. For the same reason deleting a write index is not allowed (the write index can only be deleted when deleting the entire data stream).
The question is here is that maybe we should allow a write index to be closed. If that happens then the data stream wouldn't be accepting writes, but it can easily be restored by executing an open index request. So isn't destructive to a data stream and it temporarily disables writing into a data stream. If we would allow closing/opening a data stream then the restoring a data stream in place is possible. Also other administrative operations that can only be done on a closed index are then also possible.

I'm not sure if there are other restrictions or reasons to not allow replacing a data stream in place. Currently I only see the validation that prevents closing a write index as a reason why currently we can't support this. However if there are other reasons then let's discuss these here too.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[martijnvg]

We discussed this and see two solutions which should allow backing indices of data streams be restored in place:

1. If an index that gets restored is a write index of a data stream then the associated data stream can be rolled over in order to avoid the validation that prevents closing a write index to be triggered. This way the index to restore in place is no longer a write index, so that it can be closed and the restore in place can be performed. A side effect is that an empty new write index is created, but that is only a downside if no more documents will be ingested via the data stream in question.
2. Remove that validation from the close index api that prevents closing a write index of a data stream. Closing the write index isn't really destructive like deleting (open index api call would fix it). Also there is no guarantee that indexing into a data stream is always possible, primary shards may be unavailable too. Closing an index has the same effect (shards not being available).

The first solution should already work, but needs to be validated and then documented. However this is not the desired solution, because this leaks implementation details. It requires users of the restore api to know about whether an index is a write index of a data stream and then perform a rollover before closing the index. This implementations details shouldn't be required to know when restoring indices in an existing cluster. So the preference is to implement solution 2.

After being validated, the first solution can be used as a work around. If no concerns are raised then we will probably move forward with the second solution in the next few weeks.

[martijnvg]

After some investigating it turns out that the first option (which is a workaround) isn't working. If the a data stream has been rolled over and all backing indices (except new write index) are closed then the restore fails with another validation error.

The write index that was created during the rollover, will not be part of the restored data stream and then the validation that checks whether there are conflicting indices in the namespace of a data stream fails:

data stream [ds] could create backing indices that conflict with 1 existing index(s) or alias(s) including '.ds-ds-2021.03.26-000002'
java.lang.IllegalStateException: data stream [ds] could create backing indices that conflict with 1 existing index(s) or alias(s) including '.ds-ds-2021.03.26-000002'
	at __randomizedtesting.SeedInfo.seed([1948C16CB4388D42:CF70460C3956D1D0]:0)
	at org.elasticsearch.cluster.metadata.Metadata$Builder.validateDataStreams(Metadata.java:1479)
	at org.elasticsearch.cluster.metadata.Metadata$Builder.build(Metadata.java:1356)
	at org.elasticsearch.cluster.ClusterState$Builder.metadata(ClusterState.java:520)
	at org.elasticsearch.snapshots.RestoreService$1.execute(RestoreService.java:555)
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:48)

The workaround described in option 1 is not possible, so we should focus on get the fix described in option 2.
In addition to that we should also adjust the data stream validation checks for conflicting indices. We are
going to fix this soon, which came out of a closed pr (#69669) that was opened for a different reason.
(Avoiding the need for the check for conflicting indices in the namespace of a data stream validation)