TribeNode: batch processing of cluster states

[imotov]

The cluster state updates from underlying clusters are now updated in batches, which should improve the cluster change propagation performance. Related to #12814.

[bleskes]

@imotov talked this one through I feel the code will be much simpler (with less added components) if we use the recently added PendingClusterStatesQueue per incoming cluster. An update would then need to poll all the queues for their next states, process that and that's it. The queue already manages batching.

[imotov]

@bleskes I refactored the batch processing. It's still work in progress, more tests are needed and I need to convert other batch tasks into the new mode. But I just want to make sure that I understood you correctly before I jump too deep into it.

[bleskes]

@imotov thx! What I had in mind is extract the executor part out of the cluster state update task, so you’d have a task to do and another executor to apply multiple of those tasks. The semantics would be that all tasks with the same executor can be run in batch. This is very similar to this PR.

To see whether this would work I took your PR and started playing and I ended breaking the update task into 4 roles:

• The info needed for the task (arbitrary object, like your param)
• Configuration (priority, timeout etc.)
• Executor
• Listener

I made all of those interfaces so they can be combined at will (and not having to worry about multiple inheritance) and made the ClusterSTateUpdateTask implement all 3 and have Void as the param of task. This mean that all existing code just compiled :)

Then I took MetaDataMappingService as a use case and removed all local batching and implemented it using the new stuff. It worked pretty well, I think. I even re-introduce batching on the mapping update calls needed for indexing (which removed in the work towards 2.0, see #10720).

It’s all here: https://github.com/elastic/elasticsearch/compare/master...bleskes:cluster_state_batch?w=1

I made it so it will be very easy to port to 2.1. I think it’s important because of the mapping update batching. Let me know what you think

[clintongormley]

Closing in favour of #14993