Skip to content

sync before trimUnreferencedReaders to improve index preformance #47790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Oct 9, 2019
Merged

sync before trimUnreferencedReaders to improve index preformance #47790

merged 12 commits into from
Oct 9, 2019

Conversation

@dengweisysu
Copy link
Contributor

@dengweisysu dengweisysu commented Oct 9, 2019

close #46201 and see more discussion in #46203
65770379-8c683a00-e168-11e9-836f-1f87d6c10f5c

dengweisysu and others added 12 commits August 21, 2019 12:39
@dengweisysu
Merge pull request #1 from elastic/master
624729e 
merge the lastest commits from author
@dengweisysu
do sync before closeIntoReader to make sure we move most of the data …
9986233 
…to disk outside of the lock
@dengweisysu
modify comments in Translog.rollGeneration
96e9741
@elasticmachine
Merge branch 'master' into master
113625b
@dengweisysu
Merge pull request #2 from elastic/master
5097ed2 
merge latest code from author
@dengweisysu
Merge pull request #3 from elastic/master
fbbf8b5 
merge lastest code from author
@dengweisysu
trimUnreferencedReaders: move sync translog operation outside writeLock
448682a
@dengweisysu
trimUnreferencedReaders: move try-catch inside if condition
e6744c2
@elasticmachine
Merge branch 'master' into master
e951021
@dengweisysu
Merge pull request #4 from elastic/master
d485026 
merge latest code from author
@dengweisysu
Merge pull request #5 from elastic/master
50a4f37 
merge from author
@dengweisysu
trimUnreferencedReaders: sync before write lock
c2dbddc
@ywelsch ywelsch added :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >enhancement v7.4.1 v7.5.0 v8.0.0 labels Oct 9, 2019
@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 9, 2019

Pinging @elastic/es-distributed (:Distributed/Engine)

@ywelsch
Copy link
Contributor

ywelsch commented Oct 9, 2019

@elasticmachine ok to test

henningandersen
henningandersen approved these changes Oct 9, 2019
View reviewed changes
Copy link
Contributor

@henningandersen henningandersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Thanks @dengweisysu , a lot of effort went behind this change, highly appreciated.

dnhatn
dnhatn approved these changes Oct 9, 2019
View reviewed changes
Copy link
Member

@dnhatn dnhatn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @dengweisysu :)

@dnhatn dnhatn merged commit f749bac into elastic:master Oct 9, 2019
dnhatn pushed a commit that referenced this pull request Oct 9, 2019
@dengweisysu @dnhatn
Sync translog without lock before trim unreferenced readers (#47790)
dc4224f 
This commit is similar to the optimization made in #45765. With this
change, we fsync most of the data of the current generation without
holding writeLock when trimming unreferenced readers.

Relates #45765
dnhatn pushed a commit that referenced this pull request Oct 9, 2019
@dengweisysu @dnhatn
Sync translog without lock before trim unreferenced readers (#47790)
c6253d2 
This commit is similar to the optimization made in #45765. With this
change, we fsync most of the data of the current generation without
holding writeLock when trimming unreferenced readers.

Relates #45765
@codebrain codebrain mentioned this pull request Oct 25, 2019
Closed
39 tasks
@jakelandis jakelandis removed the v8.0.0 label Jul 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dnhatn dnhatn dnhatn approved these changes

@henningandersen henningandersen henningandersen approved these changes

Assignees

No one assigned

Labels

:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >enhancement v7.4.1 v7.5.0 v8.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Optimize translog writing by move sync outside writeLock in trimUnreferencedReaders

6 participants

@dengweisysu @elasticmachine @ywelsch @dnhatn @henningandersen @jakelandis