Adding repository index generational files #19002
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
abeyad
added
>enhancement
review
:Distributed Coordination/Snapshot/Restore
abeyad
force-pushed
the
snapshot-index-generational-files
branch
from
fbc09ae to
d0b767b
Compare
Author
|
@imotov FYI Follow up question: In this PR, we are removing listing the snap-* files to determine the snapshots in favor of the index generational files. Do you think we need to keep the snap-* way of listing snapshots as a last resort in case we have an old repository with only the "index" file and that index file happens to be corrupted or missing? Or is this unnecessary? |
Contributor
|
@abeyad it's probably unnecessary. LGTM. |
Before, a repository would maintain an index file (named 'index') per repository, that contained the current snapshots in the repository. This file was not atomically written, so repositories had to depend on listing the blobs in the repository to determine what the current snapshots are, and only rely on the index file if the repository does not support the listBlobs operation. This could cause an incorrect view of the current snapshots in the repository if any prior snapshot delete operations failed to delete snapshot metadata files. This commit introduces the atomic writing of the index file, and because atomic writes are not guaranteed if the file already exists, we write to a generational index file (index-N, where N is the current generation). We also maintain an index-latest file that contains the current generation, for those repositories that cannot list blobs. Closes elastic#19002 Relates elastic#18156
abeyad
force-pushed
the
snapshot-index-generational-files
branch
from
d0b767b to
85d9871
Compare
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before, a repository would maintain an index file (named 'index') per
repository, that contained the current snapshots in the repository.
This file was not atomically written, so repositories had to depend on
listing the blobs in the repository to determine what the current
snapshots are, and only rely on the index file if the repository does
not support the listBlobs operation. This could cause an incorrect view
of the current snapshots in the repository if any prior snapshot delete
operations failed to delete snapshot metadata files.
This commit introduces the atomic writing of the index file, and because
atomic writes are not guaranteed if the file already exists, we write to
a generational index file (index-N, where N is the current generation).
We also maintain an index-latest file that contains the current
generation, for those repositories that cannot list blobs.
Relates #18156