Skip to content

Split cluster state update tasks into roles #14899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Nov 30, 2015
Merged

Split cluster state update tasks into roles #14899

merged 9 commits into from
Nov 30, 2015

Conversation

@jasontedor
Copy link
Member

@jasontedor jasontedor commented Nov 20, 2015

This commit splits cluster state update tasks into roles. Those roles
are:

  • task info
  • task configuration
  • task executor
  • task listener

All tasks that have the same executor will be executed in batches. This
removes the need for local batching as was previously in
MetaDataMappingService.

Additionally, this commit reintroduces batching on mapping update calls.

Relates #13627

This was referenced Nov 24, 2015
Closed
Merged
Merged
bleskes
bleskes reviewed Nov 25, 2015
View reviewed changes
core/src/main/java/org/elasticsearch/cluster/ClusterService.java Outdated
Copy link
Contributor

@bleskes bleskes Nov 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add note about the executor semantics?

Copy link
Member Author

@jasontedor jasontedor Nov 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docs in cab59785023f6ba31a955421bd5f3a66f7799a96.

@jasontedor jasontedor mentioned this pull request Nov 25, 2015
Merged
bleskes
bleskes reviewed Nov 25, 2015
View reviewed changes
core/src/main/java/org/elasticsearch/cluster/service/InternalClusterService.java Outdated
Copy link
Contributor

@bleskes bleskes Nov 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since it seem we are going all gang-ho on java8 , rename the runnable to lambdas? :)

Copy link
Member Author

@jasontedor jasontedor Nov 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a7a07c77c50220dd69b60bc232cd799f33490fe3. :)

@ywelsch
Copy link
Contributor

ywelsch commented Nov 25, 2015

Oops, I got confused with the Github interface (commenting on commit instead of PR). I like the change of associating failures with their originating task using a map instead of position 👍.

@jasontedor
Copy link
Member Author

jasontedor commented Nov 25, 2015

@ywelsch I liked your suggestion a lot so I pushed 66af98ea5b0a925e9c3dc55465c8aceb91813b6f.

bleskes
bleskes reviewed Nov 30, 2015
View reviewed changes
core/src/main/java/org/elasticsearch/cluster/ClusterStateTaskConfig.java Outdated
Copy link
Contributor

@bleskes bleskes Nov 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left over?

Copy link
Member Author

@jasontedor jasontedor Nov 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ff01dc904a13870bec4d11e7c9260940cbc6e6c4.

bleskes
bleskes reviewed Nov 30, 2015
View reviewed changes
core/src/test/java/org/elasticsearch/cluster/ClusterServiceIT.java
Copy link
Contributor

@bleskes bleskes Nov 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

randomize?

Copy link
Member Author

@jasontedor jasontedor Nov 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7172e972e4d821048c4278117a8425f357fccc4c.

Copy link
Member Author

@jasontedor jasontedor Dec 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was lost on squash but has been integrated into master now in bbef8ac.

@bleskes
Copy link
Contributor

bleskes commented Nov 30, 2015

Good job. LGTM. Left some minor comments. no need for another review.

bleskes and others added 9 commits November 30, 2015 08:59
@bleskes @jasontedor
Split cluster state update tasks into roles
f0f89e7 
This commit splits cluster state update tasks into roles. Those roles
are:
 - task info
 - task configuration
 - task executor
 - task listener

All tasks that have the same executor will be executed in batches. This
removes the need for local batching as was previously in
MetaDataMappingService.

Additionally, this commit reintroduces batching on mapping update calls.

Relates #13627
@jasontedor
Add test for cluster state batch updates
eaba3d5
@jasontedor
Simplify grouping of cluster state update tasks
1fb6a1f
@jasontedor
Simplify InternalClusterService#submitStateUpdateTask with lambdas
c3f97e7
@jasontedor
Explicitly correspond cluster state tasks and execution results
7caee2f
@jasontedor
Simplify loop in InternalClusterService#runTasksForExecutor
9aa9447
@jasontedor
Add builder to create cluster state executor results
72e18ec
@jasontedor
ClusterStateTaskListener#onNoLongerMaster now throws NotMasterException
ffb3e0a
@jasontedor
Add docs for cluster state update task batching
fba74c9
@jasontedor jasontedor removed the review label Nov 30, 2015
jasontedor added a commit that referenced this pull request Nov 30, 2015
@jasontedor
Merge pull request #14899 from jasontedor/cluster-state-batch
c4a2298 
Split cluster state update tasks into roles
@jasontedor jasontedor merged commit c4a2298 into elastic:master Nov 30, 2015
@jasontedor jasontedor deleted the cluster-state-batch branch November 30, 2015 14:19
This was referenced Dec 1, 2015
Closed
Merged
Merged
Merged
jasontedor added a commit that referenced this pull request Dec 4, 2015
@jasontedor
Cleanup ClusterServiceIT#testClusterStateBatchedUpdates
bbef8ac 
This commit addresses some issues that arose during the review of #14899
but were lost during squash while integrating into master.
 - the number of test threads is dropped to at most eight
 - a local variable is renamed for clarity
 - task priorities are randomized
@jasontedor jasontedor mentioned this pull request Dec 14, 2015
Merged
@ywelsch ywelsch mentioned this pull request Jan 10, 2017
Merged
ywelsch added a commit that referenced this pull request Jan 10, 2017
@ywelsch
Use general cluster state batching mechanism for snapshot state updat…
1cbb97d 
…es (#22528)

Relates to #14899
@clintongormley clintongormley added :Distributed Indexing/Distributed A catch all label for anything in the Distributed Indexing Area. Please avoid if you can. and removed :Cluster labels Feb 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@bleskes bleskes

Labels

:Distributed Indexing/Distributed A catch all label for anything in the Distributed Indexing Area. Please avoid if you can. >enhancement v5.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jasontedor @ywelsch @bleskes @clintongormley