Store Pending Deletions Fix #40345
Merged
Store Pending Deletions Fix #40345
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FilterDirectory.getPendingDeletions does not delegate, fixed temporarily by overriding in StoreDirectory. This in turn caused duplicate file name use after a trimUnsafeCommits had been done, since a new IndexWriter would not consider the pending deletes in IndexFileDeleter. This should only happen on windows (AFAIK). Reenabled doing index updates for all tests using IndexShardTests.indexOnReplicaWithGaps (which could fail due to above when using mocked WindowsFS).
henningandersen
added
>bug
:Distributed Indexing/CRUD
Collaborator
|
Pinging @elastic/es-distributed |
Member
|
@henningandersen I also dug into a recent test failure of testRestoreLocalHistoryFromTranslogOnPromotion (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=centos-6/312/console). I think that test failure has the same cause as the one you are fixing here. I proposed https://issues.apache.org/jira/browse/LUCENE-8734 to record nextWriteDVGen. |
Contributor
Author
|
@elasticmachine run elasticsearch-ci/packaging-sample |
s1monw
reviewed
Mar 22, 2019
| public Set<String> getPendingDeletions() throws IOException { | ||
| // FilterDirectory.getPendingDeletions does not delegate, working around it here. | ||
| // to be removed once fixed in FilterDirectory. | ||
| return unwrap(this).getPendingDeletions(); |
Contributor
There was a problem hiding this comment.
I think this fix is trappy, but I am ok with it for now. I would like us to delegate all our subclasses of FilterDirectory as well. We have a few more. I am torn on the unwrap to be honest.
Contributor
Author
There was a problem hiding this comment.
The problem with that route is that most of the subclasses of this are in lucene, including the MockDirectoryWrapper created by LuceneTestCase and for instance HardlinkCopyDirectoryWrapper used by StoreRecovery.
I agree that using unwrap is questionable, but in the given context this should be safe here until fixed in lucene.
Contributor
There was a problem hiding this comment.
I agree, we should still pro-actively implement the right fix in all subclasses and remove the unwrap once lucene is upgraded
Contributor
Author
Contributor
Author
|
@elasticmachine run elasticsearch-ci/packaging-sample |
1 similar comment
Contributor
Author
|
@elasticmachine run elasticsearch-ci/packaging-sample |
Added getPendingDeletions delegation to all elasticsearch FilterDirectory subclasses that were not trivial test-only overrides to minimize the risk of hitting this issue in another case.
s1monw
approved these changes
Mar 26, 2019
henningandersen
added a commit
that referenced
this pull request
Mar 26, 2019
FilterDirectory.getPendingDeletions does not delegate, fixed temporarily by overriding in StoreDirectory. This in turn caused duplicate file name use after a trimUnsafeCommits had been done, since a new IndexWriter would not consider the pending deletes in IndexFileDeleter. This should only happen on windows (AFAIK). Reenabled doing index updates for all tests using IndexShardTests.indexOnReplicaWithGaps (which could fail due to above when using mocked WindowsFS). Added getPendingDeletions delegation to all elasticsearch FilterDirectory subclasses that were not trivial test-only overrides to minimize the risk of hitting this issue in another case.
henningandersen
added a commit
that referenced
this pull request
Mar 26, 2019
FilterDirectory.getPendingDeletions does not delegate, fixed temporarily by overriding in StoreDirectory. This in turn caused duplicate file name use after a trimUnsafeCommits had been done, since a new IndexWriter would not consider the pending deletes in IndexFileDeleter. This should only happen on windows (AFAIK). Reenabled doing index updates for all tests using IndexShardTests.indexOnReplicaWithGaps (which could fail due to above when using mocked WindowsFS). Added getPendingDeletions delegation to all elasticsearch FilterDirectory subclasses that were not trivial test-only overrides to minimize the risk of hitting this issue in another case.
henningandersen
added a commit
that referenced
this pull request
Mar 26, 2019
FilterDirectory.getPendingDeletions does not delegate, fixed temporarily by overriding in StoreDirectory. This in turn caused duplicate file name use after a trimUnsafeCommits had been done, since a new IndexWriter would not consider the pending deletes in IndexFileDeleter. This should only happen on windows (AFAIK). Reenabled doing index updates for all tests using IndexShardTests.indexOnReplicaWithGaps (which could fail due to above when using mocked WindowsFS). Added getPendingDeletions delegation to all elasticsearch FilterDirectory subclasses that were not trivial test-only overrides to minimize the risk of hitting this issue in another case.
henningandersen
added a commit
that referenced
this pull request
Mar 26, 2019
Added missing file/commit.
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Apr 16, 2019
The recent upgrade to lucene 8.1.0 included the fix for LUCENE-8735 (pending deletions did not delegate). Removed the temporary workaround for this. Follow up to elastic#40345
henningandersen
added a commit
that referenced
this pull request
Apr 17, 2019
The recent upgrade to lucene 8.1.0 included the fix for LUCENE-8735 (pending deletions did not delegate). Removed the temporary workaround that was previously added for this issue. Follow up to #40345
gurkankaymak
pushed a commit
to gurkankaymak/elasticsearch
that referenced
this pull request
May 27, 2019
The recent upgrade to lucene 8.1.0 included the fix for LUCENE-8735 (pending deletions did not delegate). Removed the temporary workaround that was previously added for this issue. Follow up to elastic#40345
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
FilterDirectory.getPendingDeletions does not delegate, fixed
temporarily by overriding in StoreDirectory.
This in turn caused duplicate file name use after a trimUnsafeCommits
had been done, since a new IndexWriter would not consider the pending
deletes in IndexFileDeleter. This should only happen on windows (AFAIK).
Reenabled doing index updates for all tests using
IndexShardTests.indexOnReplicaWithGaps (which could fail due to above
when using mocked WindowsFS).
Follow-up to #40249
A follow-up should fix this in FilterDirectory.