Send to the telemetry how each connection is used

[tomaka]

As additional fields to the sc_network::network_state::NetworkState struct, we should pass basic statistics as to how this connection has been used, for example the recent queries, or anything that can help understand why the connection is necessary.

[bkchr]

Isn't this more something for prometheus?

[tomaka]

I still don't know exactly what prometheus is or how to use it.
On the other hand I know that if we send this to the telemetry, then we can examine this data in 5 seconds.

[bkchr]

I also don't really know, however, I have the feeling that we should not send all the data to the telemetry. Given the amount of nodes, we may start to overload it with this data. If there is no need to have the data public available it should be exposed with prometheus.

The telemetry macros are also very expensive, we needed to disable them in grandpa as they killed the performance.

Cc @mxinden to help us with prometheus.

[mxinden]

#5034 introduces a Prometheus registry to client/networking and thus the possibility to expose metrics from within client/networking.

Prometheus trades data resolution for efficiency. It does not store data about distinct events, but only aggregates. This sounds like a good fit for a Prometheus metric.

We have substrate_peers_count as a metric. What we could do is add a label substrate_peers_count{souce="xxx"} to the metric indicating why a set of peers was added. What do you think?

(I am currently out sick, sorry for the delay.)

[tomaka]

What do you think?

The idea was rather to have a way to detect connections that stay open even though they aren't required by neither Kademlia nor the PSM.

However your idea is also a good one, to at least have some idea how things work.

[tomaka]

The general direction seems to be towards Prometheus.
Closing in favour of paritytech/polkadot-sdk#560