Add new Shards Capacity Health Indicator

[HiDAl]

Introduces a new Health Indicator to check the cluster's health from the shards' capacity perspective.

It calculates the amount of available room for data and frozen groups, according to the following rules:

if data or frozen nodes have less than 5 shards -> RED
if data or frozen nodes have less than 10 shards -> YELLOW
otherwise -> GREEN

This is the output in case the cluster is unhealthy:

GET _health_report/shards_capacity
{
  "cluster_name": "runTask",
  "indicators": {
    "shards_capacity": {
      "status": "red",
      "symptom": "Cluster is close to reaching the configured maximum number of shards for data nodes.",
      "details": {
        "data": {
          "max_shards_in_cluster": 14,
          "current_used_shards": 10
        },
        "frozen": {
          "max_shards_in_cluster": 10,
          "current_used_shards": 0
        }
      },
      "impacts": [
        {
          "id": "elasticsearch:health:shards_capacity:impact:upgrade_blocked",
          "severity": 1,
          "description": "The cluster has too many used shards to be able to upgrade.",
          "impact_areas": [
            "deployment_management"
          ]
        },
        {
          "id": "elasticsearch:health:shards_capacity:impact:creation_of_new_indices_blocked",
          "severity": 1,
          "description": "The cluster is running low on room to add new shards. Adding data to new indices is at risk",
          "impact_areas": [
            "ingest"
          ]
        }
      ],
      "diagnosis": [
        {
          "id": "elasticsearch:health:shards_capacity:diagnosis:increase_max_shards_per_node",
          "cause": "Elasticsearch is about to reach the maximum number of shards it can host, based on your current settings.",
          "action": "Increase the value of [cluster.max_shards_per_node] cluster setting or remove data indices to clear up resources.",
          "help_url": "https://ela.st/fix-shards-capacity"
        }
      ]
    }
  }
}

relates #94079 and #91119

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @HiDAl, I've created a changelog YAML for you.

[HiDAl]

@elasticsearchmachine run elasticsearch-ci/part-3

[andreidan]

Thanks for working on this Pablo

This generally looks great, I left a few rather minor comments

@tylerperk can you please go through the copy (as most of the output of the API is presented in the UI)
@shubhaat would you like to have a go through the copy?

[andreidan]

@HiDAl the docs needs an update too https://github.com/elastic/elasticsearch/blob/main/docs/reference/health/health.asciidoc

[HiDAl]

@andreidan I did rename the indicator to ShardsCapacity

1. I didn't rename the class ShardLimitsValidator because > 30 files are using the class, hence this PR will easily become a mess. I'll rename it in a separate PR.
2. didn't rename the record ShardLimitsMetadata because it actually contains the configured limits.

[HiDAl]

@elasticsearchmachine run elasticsearch-ci/part-3

[andreidan]

@HiDAl the ShardLimitsValidator and ShardLimitsMetadata can stay named as they are IMO (they're not user facing and are extensively documented )

Can you please update the PR description to reflect the latest state?

[andreidan]

Thanks for iterating on this Pablo. This LGTM 🚀 - left a few very minor suggestions

[HiDAl]

@andreidan I've applied all the recommended changes :)

[HiDAl]

@elasticmachine update branch

[HiDAl]

@elasticsearchmachine run elasticsearch-ci/part-1