Reduce severity of log messages when hitting disk watermarks #48038
Description
Today if an Elasticsearch node reaches a disk watermark then it will repeatedly emit logging about it, which implies that some action needs to be taken by the administrator. This is misleading. Elasticsearch strives to keep nodes under the high watermark, but it is normal to have a few nodes occasionally exceed this level. Nodes may be over the low watermark for an extended period without any ill effects.
On 2019-08-28 the distributed team discussed this and decided to reduce the severity of these messages. Messages about the low watermark should not be warnings at all, and ideally should not be emitted repeatedly since we expect nodes to be over this watermark for an extended period. Messages about the high watermark could reasonably be warnings iff the cluster cannot relocate shards to bring nodes under the watermark again; if the cluster is actively relocating shards that will balance the disk usage then there is no need for a warning.