Add EventSource/EventCounter tracing and metrics for Kestrel

[analogrelay]

Some of the events/counters to add (some may already exist, others may be valuable too, discuss below):

• Connections - These should probably be emitted by default, when the source is enabled (Informational level).
  • ConnectionStart event & counter - Triggered on a new connection.
  • ConnectionStop event & counter - Triggered on graceful termination of a connection.
  • ConnectionAbort event & counter- Triggered when a connection is terminated abnormally.
  • ConnectionReject event & counter - Triggered when a connection is terminated because a limit was exhausted.
  • ConnectionTimeOut event & counter- Triggered when a connection is terminated by the server because it timed out.
  • ConnectionDuration counter - Measures the length of time a connection is established for
  • ConcurrentConnections counter - Measures the number of concurrent connections (recorded each time a connection is established/terminated).
• Requests - These should probably be emitted by default, when the source is enabled (Information Level).
  • RequestStart event & counter - Triggered when a request is started
  • RequestStop event & counter - Triggered when a request is completed (includes response status code)
  • RequestAbort event & counter - Triggered when a request is terminated abnormally
  • RequestTimeOut event & counter - Triggered when a request is times out
  • RequestFailure event & counter - Triggered when an exception escapes from the middleware pipeline.
  • RequestsSucceeded counter - Counts the number of requests that ended with 2xx-class status codes
  • RequestsRedirected counter - Counts the number of requests that ended with 3xx-class status codes (this does include 304 Not Modified, but the 3xx class is called "Redirection" in the spec, and isn't a 304 just a redirection to the cache? :P)
  • RequestsRejected counter - Counts the number of requests that ended with 4xx-class status codes
  • RequestsFailed counter - Counts the number of requests that ended with 5xx-class status codes
  • RequestsUpgraded counter - Counts the number of requests that ended with the 101 Switching Protocols status code.
  • RequestDuration counter - Measures the length of time a request takes to process. (May need to be measured in Kestrel and servers because of flushing)
• Network Details - These should definitely be behind a Keyword or Verbose trace level.
  • RequestBytesRead counter - Counts the number of bytes read for HTTP requests.
  • ResponseBytesWritten counter - Counts the number of bytes written for HTTP responses.

It might be worth coordinating across HttpSysServer and IISIntegration to share code. Also, benchmarking the impact of disabled EventSources will be important. Obviously, avoiding regression of benchmarking scenarios is an extremely high priority, we may need to have some tricks ready for that.

See https://gist.github.com/anurse/af1859663ac91c6cf69c820cebe92303 for some guidance on adding EventSources and EventCounters to ASP.NET projects.

[muratg]

Backlogging based on the discussion with @DamianEdwards the other day.

[MolallaComm]

FWIW, I'd like to see this incorporated into a app.UseMetrics("/metrics") helper similar to how app.UseHealthChecks() works. Then it would be easy to build a grafana dashboard for all our kestrel based apps.

[benaadams]

For the Requests counters, aren't they better done by Hosting and add to the other hosting counters (which already cover some of them) .

Number of open WebSockets also would be handy