Expose estimated disk usage and watermark information in nodes stats API

[dakrone]

We currently log whether a node is over the high and low watermark once every 30 seconds. To be more efficient and reduce the amount of logs generated, we should log it only once (using a latch) when the watermark is exceeded, and once when the disk goes back under the watermark.

We should also expose whether a node is above the watermarks in the nodes stats.

[synhershko]

+1 for exposing this in node stats, way better than nagging in the logs

[kimchy]

brainstorming here, but maybe we should have an allocation status API, and each decider can return an explanation + structured flags (allow_all, allow_primary, allow_replica) on a cluster view, node view, or index view (i.e. when concurrent recoveries is breached, ...). If its provided with a specific shard, then maybe we can give details on that specific shard?

[dakrone]

@kimchy +1 on an allocation status API, I think we should separate the two (do both, but separately I mean), I think the disk usage percentage and watermark passed/not-passed should be exposed via the nodes stats API as part of the FsStats as a first step, then we can add the allocation status API as an additional step.

[kimchy]

@dakrone ++

[clintongormley]

@dakrone you planning on returning to this one at some stage?

[dakrone]

@clintongormley yes, I've updated the title for this as well for its actual work.