Indices stats API should provide external refresh stats

[clandry94]

Describe the feature:

Currently, refresh stats are contained at the index level are an average of all shard refreshes within that index and exposed through the _stats api. Likewise, individual shards do collect refresh statistics, but only for INTERNAL refresh events and these statistics are not publicly accessible through the rest API.

I'm proposing the following two features to be added to elasticsearch:

1. A mechanism or data structure to distinguish between EXTERNAL and INTERNAL refreshes within shards
   2) Extending the _stats to provide shard level refresh statistics. this is already implemented e.g.

GET /my_index/_stats/refresh

would respond with

{
  "_shards" : {
    "total" : 800,
    "successful" : 800,
    "failed" : 0
  },
  "_all" : {
    "primaries" : {
      "refresh" : {
        "total" : 3952969,
        "total_time_in_millis" : 325583210,
        "listeners" : 0
      }
    },
    "total" : {
      "refresh" : {
        "total" : 7898248,
        "total_time_in_millis" : 649856023,
        "listeners" : 0
      }
    }
  },
  "indices" : {
    "products.3" : {
      "uuid" : "rFXl6ARRXRIiXVjKlh4qLEw",
      "primaries" : {
        "refresh" : {
          "total" : 3952969,
          "total_time_in_millis" : 325583210,
          "listeners" : 0
        }
      },
      "total" : {
        "refresh" : {
          "total" : 7898248,
          "total_time_in_millis" : 649856023,
          "listeners" : 0
        }
      }
     "shards": [
       { 
         "id": 1,
         "refresh" : {
          "total" : 9050918,
          "total_time_in_millis" : 948194819,
          "listeners" : 0
        }
      },
      { 
         "id": 2,
         "refresh" : {
          "total" : 412740,
          "total_time_in_millis" : 4921479849,
          "listeners" : 0
        }
      },
     ]
    }
  }
}

Related issue and prior discussion with @s1monw #36541

I can take on this and start working on it if all looks good 👍

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

@clandry94 could you clarify how the change to the stats API differs from the statistics returned by GET /my_index/_stats/refresh?level=shards? I don't think we should return shard-level statistics without ?level=shards enabled, but if that is enabled we already seem to return what you propose.

(I'm ok with adding more stats to distinguish the two kinds of refresh as you describe, but that doesn't seem to be the main proposal here)

[clandry94]

Hey @DaveCTurner. I wasn't aware of the level flag or that the scope of stats could be restricted to the shards. Thank you for that info. Are these refresh statistics for shards internal or external refreshes?

[DaveCTurner]

I do not know this code intimately, but tracing through rather naïvely it seems they are internal refreshes:

elasticsearch/server/src/main/java/org/elasticsearch/index/shard/IndexShard.java

Line 2308 in 733a6d3

Collections.singletonList(new RefreshMetricUpdater(refreshMetric)),

[s1monw]

@clandry94 @DaveCTurner these are internal refreshes.

[clandry94]

Just a quick update to this, I've finally gotten some time to work on collecting external refresh metrics and have something that I think is good. I'm planning on opening up a PR this coming week or next week. Sorry for the long delay 😄

[dnhatn]

Close in favor of #38643.