[HealthAPI] Warn when max shards per node setting is reached #91119
Description
The Health API should signal when a deployment is closing the max_shards_per_node limit.
Due to this does not relate to the shard_availability information, it should be a new indicator.
The settings keys are cluster.max_shards_per_node & cluster.max_shards_per_node.frozen
Thresholds:
- Yellow: room for < 10 shards
- Red: room for < 5 shards
Tasks
- In order to maintain feature parity with the old
checkShards, it should use the configurations provided by the master node. The Health API information is fetched from any node, so there could be a mismatch between the setting values by hence an incorrect indicator status. It's necessary to create a newHealthMetadataentry to use the values from the master node. (PR Add to HealthMetadata information about ShardLimits #94116) - Create a new Health Indicator that makes use of
ShardLimitValidator.canAddShardsToClusterto validate the status of the cluster (PR Add new Shards Capacity Health Indicator #94552) - Add mappings for new impacts and diagnoses (PR https://github.com/elastic/telemetry/pull/2123)
- Add documentation for the new indicator https://github.com/elastic/elasticsearch/blob/main/docs/reference/health/health.asciidoc
- Validate that the new indicator it's correctly displayed in the Deployment > monitoring > health pages
Open questions:
- What capabilities will be affected by this indicator?
- Should it be checked only for hot and content tiers?
- We'll use the current
checkShardsmethod, which it's already considering the frozen or regular tiers.
- We'll use the current