Skip to content

Conversation

@dnhatn
Copy link
Member

@dnhatn dnhatn commented Jul 4, 2018

Since #29679 we started adding stale operations to Lucene to have a complete history in Lucene. As the stale docs are rare, we accepted to have duplicate copies of them to keep an engine simple.

However, we now need to make sure that we have a single copy per stale operation in Lucene because the Lucene rollback requires a single document for each sequence number.

Relates #31637

Since elastic#29679 we started adding stale operations to Lucene to have a complete history in Lucene. As the stale docs are rare, we accepted to have duplicate copies of them to keep an engine simple.

However, we now need to make sure that we have a single copy per stale operation in Lucene because the Lucene rollback requires a single document for each sequence number.
@dnhatn dnhatn added >feature :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Jul 4, 2018
@dnhatn dnhatn requested a review from bleskes July 4, 2018 20:32
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn dnhatn mentioned this pull request Jul 4, 2018
11 tasks
@dnhatn
Copy link
Member Author

dnhatn commented Jul 4, 2018

@bleskes This PR is to deal with the duplicated - nested documents. The rollback won't be correct without this deduplication logic in the following scenario.

  1. Indexing a nested document
  2. That nested document arrives again (might happen during recovery + replication), and will be added to Lucene as stale
  3. Delete that nested documents
  4. Rollback the deletion in step 3. With the current implementation, we don't know how to rollback precisely a single copy of that nested document.

@bleskes
Copy link
Contributor

bleskes commented Jul 8, 2018

we discussed this via another channel and agreed we need to spend some thought on whether this is needed.

@jasontedor
Copy link
Member

@dnhatn I think that we can close this PR for now?

@dnhatn
Copy link
Member Author

dnhatn commented Sep 2, 2018

@jasontedor Thanks for the ping. Yes, I should have closed it.

I am closing this as we are going to implement a commit-based rollback.

@dnhatn dnhatn closed this Sep 2, 2018
@dnhatn dnhatn deleted the single-stale-op branch October 3, 2018 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants