Tribe nodes should apply cluster state updates in batches

[jasontedor]

This commit applies the general mechanism for applying cluster state updates in batches to tribe nodes.

Relates #14899, relates #14725

[jasontedor]

@bleskes I'll rebase this pull request on master when #14899 is reintegrated there. The salient commit for this review is thus 476ab3c91b038f5a328bf9360c87e4ee792643d0 pending #14899 (all the changes for that commit are in TribeService.java).

[jasontedor]

@bleskes I've rebased this pull request on the latest changes from #14899.

[jasontedor]

@bleskes This pull request has been rebased on master since #14899 has been integrated there.

[bleskes]

I think we can go further with this one. Now it just saves on cluster state update task, but the work done is the same. I think each executor can pick the latest task (double check that batching maintains order, it should :)) and only apply that.

We could in theory also have a global executor (i.e., for all clients) which takes the tasks, groups them by source cluster and applies them to a shared builder but then we run the risk of exceptions in one client blocking updates from all client. I don't think it's worth it.

[jasontedor]

I think we can go further with this one.

@bleskes Addressed in 74adaca4cebd9c0fde32a5080d7cfbbf2dbe9746.

[jasontedor]

@bleskes In addition to 74adaca4cebd9c0fde32a5080d7cfbbf2dbe9746, I also pushed 479e50adb26b2f5453841273c05cdb17c530065e to only return a new cluster state instance in the tribe service if there were changes to the cluster state.

[bleskes]

LGTM. Double checking - do we have a test that makes sure tasks are passed in order?

[jasontedor]

LGTM. Double checking - do we have a test that makes sure tasks are passed in order?

I don't think so; should we address that in a separate issue?

[bleskes]

I don't think so; should we address that in a separate issue?

++ on another issue.. It's an important semantics. We should test it under heavy concurrency.

[jasontedor]

++ on another issue.. It's an important semantics. We should test it under heavy concurrency.

I opened #15483 to track this.

[jasontedor]

Integrated to master in 709740e and backported to 2.x in 8fa5c68.