Hierarchical Tribe Nodes

[sherry-ger]

We are hoping hierarchical tribe nodes to solve the following issues:

1. Tribe start up is slow, especially for the "top level" tribe, which talk with all 30 clusters cross a number of data centers. It would take up to 10 minutes before a tribe node is fully up and ready to answer queries. The idea is each data center would form its own tribe in order to buffer some of the slowness.
2. Query slowness. Queries are slow when talking to the "top level" tribe nodes but may be faster to each data center based tribe.
3. When one or more downstream clusters have issues, it affects the whole tribe. With hierarchical tribe nodes, it will affect some of the tribes but not others.

[TinLe]

Adding more information. The "clusters" consist of independent ELK cluster per data center. Currently we have three main data centers (that's 30 x 3 = 90 ELK clusters).

A cluster is an ELK cluster that consumed log data for a specific set of services/applications. These services/application run in all data centers. Per security policy, log data generated in each data center must stay within that data center. For each cluster, there is tribe setup to aggregate data across all data centers for that particular cluster.

Separate from each cluster having their own tribe, there is a top level set of tribe nodes (1 per data center) that talk to all 3 DCs and all 30 clusters (x the 3 DCs).

The model is distributed, loosely coupled. Each cluster is independently managed/own by the team that own the service/application. Only the top level tribe cluster that talk to all clusters is own by me.

I would like to avoid Single Point of Failure as much as possible. Downstream clusters having issue should not affect the performance, availability of the top level Tribe cluster. The Tribe node should be robust enough to deal with downstream clusters dropping out, rejoining, having issues.

Another issue that we've seen and reported here: #12804

[TinLe]

This is becoming very urgent. Our tribe nodes are crashing on an hourly basis now due to GC. Essentially they become unresponsive, nothing in log until we bounce them, the while dying, there will be a few lines in log that say they are hitting GC.

We are already using maxheap of 31GB on 64GB dedicated tribe nodes. I have also tested on 256GB nodes, with various maxheap from 64GB, 96GB, 100GB, 120GB, 128GB. Best time the node stay alive/responsive was ~5hrs. I've also tested with CMS vs G1GC. None of them make much of a difference.

[clintongormley]

Honestly, I don't see a clear path to making this work. The tribe nodes need to have the cluster state from all clusters in order to be able to do their job. The tribe joins each cluster as a client node. Query slowness is basically down to network latency and the slowest node across all clusters.

Sorry, but I can't give you a magic solution here.

[TinLe]

I agree that it is a hard problem. The Galene team solved similar problem,and it took some effort.

https://engineering.linkedin.com/search/did-you-mean-galene

Federation/aggregation of tribe is a hard problem and require internal re-architecting.

[jpountz]

Maybe there are ways we can improve the tribe nodes to improve the situation, but I don't think hierarchical tribe nodes are a solution. It's also not as easy as it seems since some parts of the code are assuming that the reduce phase is happening only once (eg. for pipeline aggs or the ways terms aggs can over-request shards).

[clintongormley]

#13627 will also help with this extreme use of tribe nodes.