Improve Elasticsearch network monitoring

[Tim-Brooks]

Network Monitoring

TcpChannel

The TcpChannel represents tcp connection.

• Counter - Number of bytes sent
• Counter - Number of bytes received

Transport.Connection

The Transport.Connection represents a collection of channels to another node.

• Counter - Number of requests sent
• Counter - Number of request timeouts

ConnectionManager

The ConnectionManager represents a manager for outgoing cluster connections. We have one for the local cluster and one for each remote cluster.

• Gauge - Number of open Transport.Connection
• Gauge - Number of open outbound TcpChannel
• Counter - Number of failed requests due to flush failure
• Counter - Number of request timeouts
• Counter - Number of requests sent
• Counter - Transport.Connections closed
• Counter - TcpChannels closed

TcpTransport

The TcpTransport holds the state of all the incoming tcp connections.

• Gauge - Number of open inbound TcpChannel
• Counter - Number of requests received
• Counter - Number of responses sent
• Counter - Number of failed responses due to flush failure
• Counter - Number of inbound TcpChannels closed
• Counter - Keepalives sent
• Counter - Keepalives received
• Gauge - Recycled bytes consumed from PageCacheRecycler
• Gauge - Un-recycled bytes consumed from PageCacheRecycler
• Gauge - Byte size of netty’s heap buffer pool
• Gauge - Byte size of netty’s direct buffer pool

[elasticmachine]

Pinging @elastic/es-distributed

[Tim-Brooks]

@dliappis - @ywelsch told me that you might have some thoughts on this meta issue?

[dliappis]

Thanks for the ping @tbrooks8 . Thinking of this at a higher level, and aiming at using these metrics to reduce the time spent to identify real network issues, first of all I have a few general observations.

In most cases the source/dest IP:port pairs are important to help with diagnosis. With the suggested Metric Types Gauge/Meter/EWMA, I am not sure how such things would work, e.g. would the metric name be something like "<SRC_IP>:<SRC_PORT>-<DST_IP>:<DST_PORT>`? If yes, this would need to also be periodically reaped from the list over time to reflect the lifecycle of connections.

At any rate, assuming we have solved the naming issue with sockets, I've added some stats that I think can be valuable. (Possibly not all may be implementable and some may adversely impact performance)

• Meter: throughput (bytes/s) per interface used and generated by Elasticsearch.

ConnectionManager:

• List of Metric with top N connections based on total network volume. e.g.:
  ["<SRC_IP1:SRC_PORT1-DST_IP1:PORT1>": "100MB", "<SRC_IP2:SRC_PORT2-DST_IP2:PORT2>": "93MB", ...] over period of time say 10s.
• To help identify SSL misconfigurations: Gauge - # of requests where SSL negotiation failed? (this would be even more useful again if if sockets can be identified)

TcpTransport:

• Gauge or Metric- number of open (established) connections per source port
• Gauge or Metric- number of open connections per IP-port pair
• Would it be possible to have a periodic TCP based RTT check among nodes and store the result a a list of EWMA (like uptime) over a period?
• Same as for ConnectionManager for SSL: Gauge # of requests where SSL negotiation failed
• Meter - size of netty send queue per time period
• Meter - size of netty recv queue per time period

[jasontedor]

We typically avoid rates (metric and EWMA) within Elasticsearch. Instead, we expose counters and expect that our monitoring products will expose the rates based on counters collected over a time period. How many connections were opened in the last time period? Do a derivative on the number of connections opened counter.

[Tim-Brooks]

I'll make some adjustments based the the comments.

We typically avoid rates (metric and EWMA) within Elasticsearch

I noticed that. I'll make everything counters and I guess our strategy here is that this monitoring has to be enabled for these metrics to provide for the most value.

I also think that this removes most of the purpose of histogram/latency values. But I will think about it.

[yaronp68]

Would it be possible to meter response and request latency per all connections between 2 specific nodes (so the user can find out which network connection has lower throughput)

[ywelsch]

Also relates to #19335

[ywelsch]

I think it might also be interesting to get some of these stats on a per-connection-type level (e.g. BULK vs RECOVERY vs REG vs STATE)

[jasontedor]

I noticed that. I'll make everything counters and I guess our strategy here is that this monitoring has to be enabled for these metrics to provide for the most value.

Thanks @tbrooks8.

I want to add, since it wasn't clear from my initial comment: I am fully in favor of the ideas here to expose more metrics about networking within Elasticsearch. My comment was only an iteration on what is presented here to be consistent with our we expose metrics in general and think about them in the context of exposing them through monitoring higher up in the stack. Thanks for taking this on.