Skip to content

Conversation

@dnhatn
Copy link
Member

@dnhatn dnhatn commented Jun 5, 2018

This PR adapts/utilizes recent enhancements in Lucene-7.4:

  1. Replaces exactNumDocs by the soft-deletes count in SegmentCommitInfo. This enhancement allows us to back out changes introduced in Use exact numDocs in synced-flush and metadata snapshot #30228.

  2. Always configure the soft-deletes field in IWC

@dnhatn dnhatn added >feature :Distributed Indexing/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Jun 5, 2018
@dnhatn dnhatn requested review from bleskes and s1monw June 5, 2018 01:11
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn dnhatn changed the title Replace exactNumDocs by soft-del count in SegmentCommitInfo Replace exact numDocs by soft-del count in SegmentCommitInfo Jun 5, 2018
@dnhatn
Copy link
Member Author

dnhatn commented Jun 5, 2018

run sample packaging tests

@s1monw
Copy link
Contributor

s1monw commented Jun 5, 2018

@elasticmachine run sample packaging tests

Copy link
Contributor

@s1monw s1monw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM I am glad we dont' have to jump through hoops anymore for the doc stats.

import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.notNullValue;

@LuceneTestCase.AwaitsFix(bugUrl = "should-be-fixed-by-upstream")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you elaborate what caused this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are failing on the master branch.

Copy link
Contributor

@bleskes bleskes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice stats :)

dnhatn added 4 commits June 5, 2018 08:58
This snapshot includes:
- LUCENE-8341: Record soft deletes in SegmentCommitInfo which will resolve elastic#30851
- LUCENE-8335: Enforce soft-deletes field up-front
@dnhatn
Copy link
Member Author

dnhatn commented Jun 5, 2018

Thanks @s1monw and @bleskes

@dnhatn dnhatn merged commit 755a25a into elastic:ccr Jun 5, 2018
@dnhatn dnhatn deleted the soft-deletes-stats branch June 5, 2018 18:11
dnhatn added a commit that referenced this pull request Jun 5, 2018
This PR adapts/utilizes recent enhancements in Lucene-7.4:

- Replaces exactNumDocs by the soft-deletes count in SegmentCommitInfo.
This enhancement allows us to back out changes introduced in #30228.

- Always configure the soft-deletes field in IWC
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Jun 5, 2018
* ccr:
  [DOCS] Creates rest-api folder in docs
  [Rollup] Disallow index patterns that match the rollup index (elastic#30491)
  Replace exact numDocs by soft-del count in SegmentInfo (elastic#31086)
  Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (elastic#31073)
  Add cors support to NioHttpServerTransport (elastic#30827)
  [DOCS] Fixes security example (elastic#31082)
  Allow terms query in _rollup_search (elastic#30973)
  Removing erroneous repeat
  Adapt bwc versions after backporting elastic#30983 to 6.4
  [Tests] Muting RatedRequestsTests#testXContentParsingIsNotLenient
  TEST:  Retry synced-flush if ongoing ops on primary (elastic#30978)
  Fix docs build.
  Only auto-update license signature if all nodes ready (elastic#30859)
  Add BlobContainer.writeBlobAtomic() (elastic#30902)
  Add a doc value format to binary fields. (elastic#30860)
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Jun 5, 2018
* ccr:
  [DOCS] Creates rest-api folder in docs
  [Rollup] Disallow index patterns that match the rollup index (elastic#30491)
  Replace exact numDocs by soft-del count in SegmentInfo (elastic#31086)
  Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (elastic#31073)
  Add cors support to NioHttpServerTransport (elastic#30827)
  [DOCS] Fixes security example (elastic#31082)
  Allow terms query in _rollup_search (elastic#30973)
  Removing erroneous repeat
  Adapt bwc versions after backporting elastic#30983 to 6.4
  [Tests] Muting RatedRequestsTests#testXContentParsingIsNotLenient
  TEST:  Retry synced-flush if ongoing ops on primary (elastic#30978)
  Fix docs build.
  Only auto-update license signature if all nodes ready (elastic#30859)
  Add BlobContainer.writeBlobAtomic() (elastic#30902)
  Add a doc value format to binary fields. (elastic#30860)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. :Distributed Indexing/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. >feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants