Skip to content

Conversation

@ywelsch
Copy link
Contributor

@ywelsch ywelsch commented Mar 4, 2019

This PR adds the capability to snapshot replicated closed indices.

It also changes snapshot requests in v8.0.0 to automatically expand wildcards to closed indices and hence start snapshotting closed indices by default. For v7.2.0 and above, wildcards are by default only expanded to open indices, which can be changed by explicitly setting the expand_wildcards option either to all or closed (see https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-index.html).

Note that indices are always restored as open indices, even if they have been snapshotted as closed replicated indices.

Relates to #33888

@ywelsch ywelsch added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 v7.2.0 labels Mar 4, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

private String[] indices = EMPTY_ARRAY;

private IndicesOptions indicesOptions = IndicesOptions.strictExpandOpen();
private IndicesOptions indicesOptions = IndicesOptions.strictExpand();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will be changed back to IndicesOptions.strictExpandOpen for the 7.x backport.

This comment was marked as off-topic.

This comment was marked as off-topic.

@original-brownbear
Copy link
Contributor

@original-brownbear
Copy link
Contributor

original-brownbear commented Mar 4, 2019

@ywelsch looks like we have a possible related test failure:

6:40:10 FAILURE 4.92s J4 | SnapshotBlocksIT.testCreateSnapshotWithIndexBlocks <<< FAILURES!
16:40:10    > Throwable #1: java.lang.AssertionError: Request executed with success but a ClusterBlockException was expected
16:40:10    > 	at __randomizedtesting.SeedInfo.seed([117D2CBB1420960:B693C3083F0F2E94]:0)
16:40:10    > 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertBlocked(ElasticsearchAssertions.java:166)
16:40:10    > 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertBlocked(ElasticsearchAssertions.java:191)
16:40:10    > 	at org.elasticsearch.action.admin.cluster.snapshots.SnapshotBlocksIT.testCreateSnapshotWithIndexBlocks(SnapshotBlocksIT.java:117)
16:40:10    > 	at java.lang.Thread.run(Thread.java:748)

edit: timing ... :)

@original-brownbear
Copy link
Contributor

@ywelsch seems that same test is still unstable:

SnapshotInProgressException[Cannot delete indices that are being snapshotted: [[test-blocks-1/plmJu35yRciEdkKdldmIGQ]]. Try again after snapshot finishes or cancel the currently running snapshot.]
	at __randomizedtesting.SeedInfo.seed([6B7E4401E4599C16:DCFA55C26A14BBE2]:0)
	at org.elasticsearch.snapshots.SnapshotsService.checkIndexDeletion(SnapshotsService.java:1423)
	at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService.deleteIndices(MetaDataDeleteIndexService.java:95)
	at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService$1.execute(MetaDataDeleteIndexService.java:83)
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687)
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310)
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210)
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142)
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Copy link
Contributor

@original-brownbear original-brownbear left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one question about a test :)

@ywelsch ywelsch merged commit de2a296 into elastic:master Mar 6, 2019
ywelsch added a commit that referenced this pull request Mar 6, 2019
This adds the capability to snapshot replicated closed indices.

It also changes snapshot requests in v8.0.0 to automatically expand wildcards to closed indices and hence start snapshotting closed indices by default. For v7.1.0 and above, wildcards are by default only expanded to open indices, which can be changed by explicitly setting the expand_wildcards option either to all or closed.

Note that indices are always restored as open indices, even if they have been snapshotted as closed replicated indices.

Relates to #33888
ywelsch added a commit that referenced this pull request Mar 6, 2019
@tlrx tlrx mentioned this pull request Apr 1, 2019
50 tasks
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request May 14, 2020
The code path for closed indices is dead code here ever since elastic#39644
because `shards(currentState, indexIds, ...)` does not set
`MISSING` on a closed index's shard any longer.
=> simplified the code accordingly.
original-brownbear added a commit that referenced this pull request May 14, 2020
The code path for closed indices is dead code here ever since #39644
because `shards(currentState, indexIds, ...)` does not set
`MISSING` on a closed index's shard that is assigned any longer. Before that change it would always set `MISSING` for a closed index's shard even it was assigned. 
=> simplified the code accordingly.
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request Jul 13, 2020
The code path for closed indices is dead code here ever since elastic#39644
because `shards(currentState, indexIds, ...)` does not set
`MISSING` on a closed index's shard that is assigned any longer. Before that change it would always set `MISSING` for a closed index's shard even it was assigned.
=> simplified the code accordingly.
original-brownbear added a commit that referenced this pull request Jul 13, 2020
The code path for closed indices is dead code here ever since #39644
because `shards(currentState, indexIds, ...)` does not set
`MISSING` on a closed index's shard that is assigned any longer. Before that change it would always set `MISSING` for a closed index's shard even it was assigned.
=> simplified the code accordingly.
@jakelandis jakelandis removed the v8.0.0 label Jul 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants