Skip to content

Fix Bug with Concurrent Snapshot and Index Delete #73456

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
May 27, 2021
Merged

Fix Bug with Concurrent Snapshot and Index Delete #73456

merged 6 commits into from
May 27, 2021

Conversation

@original-brownbear
Copy link
Contributor

@original-brownbear original-brownbear commented May 27, 2021

Fix state machine bug that fixes the incorrect assumption a finished snapshot delete
could only start shard snapshots when in fact it can also move snapshots to a
completed state.

@original-brownbear
Fix Bug with Concurrent Snapshot and Index Delete
e34407e 
Fix state machine bug that fixes the incorrect assumption a finished snapshot delete
could only start shard snapshots when in fact it can also move snapshots to a
completed state.
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label May 27, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 27, 2021

Pinging @elastic/es-distributed (Team:Distributed)

DaveCTurner
DaveCTurner approved these changes May 27, 2021
View reviewed changes
Copy link
Contributor

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested a couple of small tidy-ups but LGTM.

tlrx
tlrx reviewed May 27, 2021
View reviewed changes
server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java
final SnapshotsInProgress.Entry updatedEntry = entry.withShardStates(updatedAssignmentsBuilder.build());
snapshotEntries.add(updatedEntry);
changed = true;
if (updatedEntry.state().completed()) {
Copy link
Member

@tlrx tlrx May 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry if this is a stupid remark but should we do the same thing for the cloning path?

Copy link
Contributor Author

@original-brownbear original-brownbear May 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries :) The clone path isn't affected by this, the logic there is different and always sets the snapshot overall state correctly. Also, there cannot be a concurrent removal of shards for clones anyway :)

Copy link
Member

@tlrx tlrx May 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦🏻 makes sense, thanks!

@DaveCTurner DaveCTurner merged commit 14a31b9 into elastic:master May 27, 2021
DaveCTurner pushed a commit that referenced this pull request May 27, 2021
@original-brownbear @DaveCTurner
Fix Bug with Concurrent Snapshot and Index Delete (#73456)
ab00c71 
Fixes the incorrect assumption in the snapshot state machine that a finished
snapshot delete could only start shard snapshots: in fact it can also move
snapshots to a completed state.
DaveCTurner pushed a commit that referenced this pull request May 27, 2021
@original-brownbear @DaveCTurner
Fix Bug with Concurrent Snapshot and Index Delete (#73456)
2efca17 
Fixes the incorrect assumption in the snapshot state machine that a finished
snapshot delete could only start shard snapshots: in fact it can also move
snapshots to a completed state.
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request May 27, 2021
@DaveCTurner
Document elastic#73456 as a known issue
f42fe7a 
Adds docs to the release notes for 7.9-7.12 recording elastic#73456 as a known
issue.
@DaveCTurner DaveCTurner mentioned this pull request May 27, 2021
Merged
@original-brownbear original-brownbear deleted the fix-snapshot-bug branch May 27, 2021 13:13
@original-brownbear
Copy link
Contributor Author

original-brownbear commented May 27, 2021

Thanks David!!

DaveCTurner added a commit that referenced this pull request May 27, 2021
@DaveCTurner
Document #73456 as a known issue (#73471)
32b3511 
Adds docs to the release notes for 7.9.0-7.13.0 recording #73456 as a
known issue.
DaveCTurner added a commit that referenced this pull request May 27, 2021
@DaveCTurner
Document #73456 as a known issue (#73471)
c86e447 
Adds docs to the release notes for 7.9.0-7.13.0 recording #73456 as a
known issue.
DaveCTurner added a commit that referenced this pull request May 27, 2021
@DaveCTurner
Document #73456 as a known issue (#73471)
8c7f97f 
Adds docs to the release notes for 7.9.0-7.13.0 recording #73456 as a
known issue.
jrodewig pushed a commit that referenced this pull request May 27, 2021
@DaveCTurner @jrodewig
Document #73456 as a known issue (#73471)
416e0cb 
Adds docs to the release notes for 7.9.0-7.13.0 recording #73456 as a
known issue.
jrodewig pushed a commit that referenced this pull request May 27, 2021
@DaveCTurner @jrodewig
Document #73456 as a known issue (#73471)
b57b789 
Adds docs to the release notes for 7.9.0-7.13.0 recording #73456 as a
known issue.
jrodewig pushed a commit that referenced this pull request May 27, 2021
@DaveCTurner @jrodewig
Document #73456 as a known issue (#73471)
ef88ffa 
Adds docs to the release notes for 7.9.0-7.13.0 recording #73456 as a
known issue.
limingnihao added a commit to limingnihao/elasticsearch that referenced this pull request May 28, 2021
@limingnihao
Merge branch 'master' into custom
209e049 
* master: (1643 commits)
  Make DataStreamsSnapshotsIT resilient to failures because of local time. (elastic#73516)
  Upgrade netty to 4.1.63 (elastic#73011)
  [DOCS] Create a new page for dissect content in scripting docs (elastic#73437)
  Deprecate freeze index API (elastic#72618)
  [DOCS] Remove 'closed data stream' reference
  [DOCS] Update alias references (elastic#73427)
  [DOCS]  Create a new page for grok content in scripting docs (elastic#73118)
  Remove dependency on azure shadowjar since it's no longer required
  [DOCS] Update backport policy for known issues (elastic#73489)
  Shadowed dependencies should be hidden from pom dependencies (elastic#73467)
  Disable transitive dependencies when resolving bwc JDBC driver artifact (elastic#73448)
  Print full JVM implementation version at start of build (elastic#73439)
  [DOCS] Update snapshot/restore for data stream aliases (elastic#73438)
  Upgrade Azure SDK and Jackson (elastic#72833) (elastic#72995)
  [DOCS] Fix typo (elastic#73337) (elastic#73474)
  [DOCS] Fix typo (elastic#73444) (elastic#73472)
  [DOCS] Update alias security for data stream aliases (elastic#73436)
  Fix Bug with Concurrent Snapshot and Index Delete (elastic#73456)
  [DOCS] Move common scripting use cases up a level (elastic#73445)
  Add more validation for data stream aliases. (elastic#73416)
  ...
@nickofthyme nickofthyme mentioned this pull request May 31, 2021
Closed
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request Nov 11, 2021
@DaveCTurner
Add docs about repair of repo affected by corruption bug
48fedfa 
The known-issue docs give the impression that an upgrade will restore
the lost data in the repository. This isn't the case, so this commit
clarifies this in the docs.

Relates elastic#73456
Relates elastic#75598
Relates elastic#79221
@DaveCTurner DaveCTurner mentioned this pull request Nov 11, 2021
Merged
DaveCTurner added a commit that referenced this pull request Nov 15, 2021
@DaveCTurner
Add docs about repair of repo affected by corruption bug (#80662)
b9ce698 
The known-issue docs give the impression that an upgrade will restore
the lost data in the repository. This isn't the case, so this commit
clarifies this in the docs.

Relates #73456
Relates #75598
Relates #79221
DaveCTurner added a commit that referenced this pull request Nov 15, 2021
@DaveCTurner
Add docs about repair of repo affected by corruption bug (#80662)
964761b 
The known-issue docs give the impression that an upgrade will restore
the lost data in the repository. This isn't the case, so this commit
clarifies this in the docs.

Relates #73456
Relates #75598
Relates #79221
DaveCTurner added a commit that referenced this pull request Nov 15, 2021
@DaveCTurner
Add docs about repair of repo affected by corruption bug (#80662)
f97bbe5 
The known-issue docs give the impression that an upgrade will restore
the lost data in the repository. This isn't the case, so this commit
clarifies this in the docs.

Relates #73456
Relates #75598
Relates #79221
DaveCTurner added a commit that referenced this pull request Nov 15, 2021
@DaveCTurner
Add docs about repair of repo affected by corruption bug (#80662)
6a906ea 
The known-issue docs give the impression that an upgrade will restore
the lost data in the repository. This isn't the case, so this commit
clarifies this in the docs.

Relates #73456
Relates #75598
Relates #79221
@original-brownbear original-brownbear restored the fix-snapshot-bug branch April 18, 2023 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tlrx tlrx tlrx left review comments

@DaveCTurner DaveCTurner DaveCTurner approved these changes

Assignees

No one assigned

Labels

blocker >bug :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. v7.13.1 v7.14.0 v8.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@original-brownbear @elasticmachine @tlrx @DaveCTurner @jakelandis