Diagnostics Channel - Next steps

[mhdawson]

The concept of a diagnostic channel was discussed in the Diagnostics summit as well as here: #134

In a nutshell the concept is to provide an API that modules can use to publish events, and an API that consumers (such as APM vendors) can consume. The key differences between the channel provided by these APIs and trace events are:

• operation is synchronous, the consumer has the ability to take an action before the publish event completes on the emitter side.
• data is passed (and any other data available indirectly) by the publish and this data can be mutated by the consumer.

This approach requires both implementation of the diagnostic channel as well as updates to modules in order to use the API. An interim step where monkey patching is used to add calls to the API is seen as a good way to bootstrap.

Microsoft already has an experimental implementation which we should evaluate to see what parts (if any) that we can re-use.

An initial cut at next steps include:

• Create a new repo where we can experiment with a new implementation under the nodejs org - diagnostic-channel
• review the existing experimental implementation
• create performance regression suite
• define the diagnostic publish API
• define the diagnostic consumer API
• pull what parts we can from existing implementation into publish API
• pull what parts we can from existing implementation into consumer API
• Iterate until we hit a 0.1.0 release that is ready for broader review

[mcollina]

Overhead and performance should be a first level concern about this API. It should result in no-ops if it is not enabled, and get smart if it is enabled. I think we should trade usability for performance whenever possible, as long as it still satisfy our use case.

[jkrems]

@mcollina - [ ] Create performance regression suite?

[mcollina]

@jkrems 👍 !

[jkrems]

Added to the top post

[Qard]

I brought it up in the diag meeting just now, but I worked a bit on a proof-of-concept universal diagnostics channel type thing recently, after I got back from the Ottawa summit and then Elastic{ON}. It's a very rough demo and definitely needs more perf focus before it could be actually usable, but you can have a look at it here: https://github.com/Qard/universal-monitoring-bus

It uses subscription queries to express interest in certain messages in the channel, similar to trace_events, but in a currently more flexible way (sacrificing performance for flexibility while trying to identify interesting filter data). It has no direct graph-forming, instead opting to direct the message bus into aggregators, which can live in-process or out-of-process, and makes no assumptions about transport mechanism. It is simply a callback that receives JSON-serialization-ready objects that you can do whatever you like with.

It also expresses both spans and metrics updates uniformly in a single message format, with purpose-specific extension, allowing for an easy shared communication channel or aggregator re-use and even correlation between the spans and metrics data.