Detecting hidden indices takes nontrivial time

[DaveCTurner]

Metadata$Builder#build is called on every node in every cluster state update that changes the cluster metadata. In a busy cluster with many shards, this means it's on a fairly hot path. Profiling cluster state updates in such a cluster indicates that this method spends about 23% of its time calling Setting#get:

The only such call is this one:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/metadata/Metadata.java

Line 1436 in 411d7e7

final boolean visible = IndexMetadata.INDEX_HIDDEN_SETTING.get(indexMetadata.getSettings()) == false;

Frequently-accessed settings in IndexMetadata are generally made into fields to avoid having to do the lookup work on each access. We can make a substantial time saving in this method by moving the index.hidden setting to a field.

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[grcevski]

Hi @DaveCTurner, is there a way to share an expanded image (or the profile itself) for the last section of the flame graph, just what's under Setting.get? I'm curious if we can simply improve the performance of Setting.get?

[DaveCTurner]

Sure, here it is, although beware that there's only 27 samples in this method so there will be some quantisation error.

Some of that is computing the keyset for the index settings (happens once per instance, slightly puzzled why we didn't already compute it in IndexMetadata$Builder.build()) and the rest is looking up a string in a map (including hashing the setting name).