|
| 1 | +# New Tracing Quickstart |
| 2 | + |
| 3 | +From now on old-tracing (based on iohk-monitoring-framework) exists |
| 4 | +together with new-tracing (based on trace-dispatcher and cardano-tracer) for a while. |
| 5 | + |
| 6 | +In this transition time new-tracing can be tested and improved. Since we have several hundred trace messages |
| 7 | +it is expected that you will find regressions and bugs in the port, please help to find |
| 8 | +and fix them. |
| 9 | + |
| 10 | +In this transition time new tracing will for technical reason have a restricted functionality, |
| 11 | +as e.g. the reconfiguration of a running node is not available. |
| 12 | + |
| 13 | +To switch to new tracing set the value `UseTraceDispatcher` to true. If you do this, the |
| 14 | +config file needs to contain several values for the configuration of new-tracing, as we |
| 15 | +describe next. |
| 16 | + |
| 17 | +The current tracing system has two ways to identify the message, a hierarchical name we |
| 18 | +call it's `Namespace` and the `Kind field` in machine representation. We base our implementation |
| 19 | +on the namespace, and require a one-to-one correspondence between namespaces and messages (bijective mapping). |
| 20 | + |
| 21 | +As we have two mechanisms for the same purpose for historic reasons, we will soon |
| 22 | +__deprecate the Kind field__, and it will disappear in the near future. So we strongly |
| 23 | +advice to use namespaces for any analysis tools of traces! |
| 24 | + |
| 25 | +### Configuration of new tracing |
| 26 | + |
| 27 | +In Cardano a default configuration is given in the module [Cardano.Node.Tracing.DefaultTraceConfig](https://github.com/input-output-hk/cardano-node/blob/master/cardano-node/src/Cardano/Node/Tracing/DefaultTraceConfig.hs). In the config file all entries of the default configuration can be overridden. To remove a frequency limiter, define a limiter with maxFrequency 0.0. |
| 28 | + |
| 29 | +1. Specify a filter for the severity of the messages you want to see, e.g.: |
| 30 | + |
| 31 | + ~~~yaml |
| 32 | + # Show messages of Severity Notice or higher as default |
| 33 | + "": |
| 34 | + severity: Notice |
| 35 | + |
| 36 | + # But show ChainDB messages starting from Info |
| 37 | + ChainDB: |
| 38 | + severity: Info |
| 39 | + ~~~ |
| 40 | + |
| 41 | + |
| 42 | + So the namespaces are used for configuration values, which works |
| 43 | + down to individual messages, and the more specialized value overwrites the more general. |
| 44 | + |
| 45 | + If you don't want to see any messages from tracers the new severity `Silence` |
| 46 | + exists, which suppresses all messages. |
| 47 | + |
| 48 | + |
| 49 | +2. Specify in which detail level, the messages get shown. |
| 50 | + |
| 51 | + ~~~yaml |
| 52 | + "": |
| 53 | + # Keep this |
| 54 | + severity: Notice |
| 55 | + # All messages are shown with normal detail level |
| 56 | + detail: DNormal |
| 57 | + ~~~ |
| 58 | + |
| 59 | + Other options would be DMinimal, DDetailed and DMaximum. This has only an effect on messages which support the representation in different ways. |
| 60 | + |
| 61 | +3. Specify limiters for the frequency of messages |
| 62 | + |
| 63 | + Eliding tracers are not supported in new-tracing, instead you can limit the |
| 64 | + frequency in which messages get shown. |
| 65 | + |
| 66 | + ~~~yaml |
| 67 | + ChainDB.AddBlockEvent.AddedBlockToQueue: |
| 68 | + # Only show a maximum of 2 of these messages per second |
| 69 | + maxFrequency: 2.0 |
| 70 | + ~~~ |
| 71 | + |
| 72 | + The activity of limiters will be written in the traces as well. |
| 73 | + |
| 74 | +4. Specify the backends the messages are routed to. |
| 75 | + |
| 76 | + ~~~yaml |
| 77 | + "": |
| 78 | + # Keep this |
| 79 | + severity: Notice |
| 80 | + # And this |
| 81 | + detail: DNormal |
| 82 | + # And specify a list of backends to use |
| 83 | + backends: |
| 84 | + - Stdout MachineFormat |
| 85 | + - EKGBackend |
| 86 | + - Forwarder |
| 87 | + ~~~ |
| 88 | + |
| 89 | + These are all the backends currently supported. With Stdout you have the |
| 90 | + options MachineFormat or HumanFormatColoured/HumanFormatUncoloured. |
| 91 | + If messages don't support representation in HumanFormat* they are shown in MachineFormat anyway. |
| 92 | + |
| 93 | + Forwarder means that messages are send to cardano-tracer |
| 94 | + |
| 95 | +Configuration can be written in JSON and YAML, we have shown the examples in YAML. |
| 96 | + |
| 97 | +### Configuration and use of cardano-tracer |
| 98 | + |
| 99 | +`cardano-tracer` is a part of the new tracing infrastructure. It is a separate service that accepts different messages from the node and handles them. |
| 100 | +So it is assumed that if you want to use the new tracing infrastructure - you will use `cardano-tracer`. Please read its [documentation](https://github.com/input-output-hk/cardano-node/blob/master/cardano-tracer/docs/cardano-tracer.md) for more details. |
| 101 | + |
| 102 | +This example describes the simplest case, when the node and `cardano-tracer` on the same machine. |
| 103 | + |
| 104 | +First of all, add `--tracer-socket-path-connect /tmp/forwarder.sock` option to the node's CLI options, asking it to connect to `cardano-tracer`. |
| 105 | + |
| 106 | +Then build and run `cardano-tracer`: |
| 107 | + |
| 108 | +~~~shell |
| 109 | +$ cabal build cardano-tracer && cabal install cardano-tracer --installdir=PATH_TO_DIR --overwrite-policy=always |
| 110 | +$ cd PATH_TO_DIR |
| 111 | +$ ./cardano-tracer --config PATH_TO_CONFIG |
| 112 | +~~~ |
| 113 | + |
| 114 | +where `PATH_TO_CONFIG` is a path to tracer's configuration file. This is an example of such a configuration: |
| 115 | + |
| 116 | +~~~yaml |
| 117 | +--- |
| 118 | +network: |
| 119 | + tag: AcceptAt |
| 120 | + contents: "/tmp/forwarder.sock" |
| 121 | +logging: |
| 122 | +- logRoot: "/tmp/cardano-tracer-logs" |
| 123 | + logMode: FileMode |
| 124 | + logFormat: ForMachine |
| 125 | +~~~ |
| 126 | + |
| 127 | +That's it. After you run the node, it will establish the connection with `cardano-tracer` and will start to forward messages to it. |
| 128 | +As a result, you will find log files, in JSON format, in `/tmp/cardano-tracer-logs` directory. |
| 129 | + |
| 130 | +### Development during transition-time |
| 131 | + |
| 132 | +For developing tracers in the transition time we suggest: |
| 133 | + |
| 134 | +1. Don't use strictness annotations for trace types. Trace messages are either |
| 135 | +discarded immediately (which happens frequently) or instantly converted to another format |
| 136 | +but never stored. So strictness annotations make the code less efficient without any benefit. |
| 137 | +As well it doesn't play well together with the required prototypes of messages in the |
| 138 | +new framework. |
| 139 | + |
| 140 | +2. If you develop new tracers we suggest that you develop the new tracers first, |
| 141 | +and then map to old tracers, as the new tracers will survive. You will find plenty of |
| 142 | +examples in cardano-node under Cardano.Node.Tracing.Tracers. |
| 143 | + |
| 144 | +3. Contact the benchmarking team or node-logging channel for any questions and reviews. |
| 145 | + |
| 146 | +### Documentation of trace messages and further documentation |
| 147 | + |
| 148 | +This is a document which is regenerated periodically and documents all trace-messages, metrics and data-points in cardano-node. It as well displays the handling of these |
| 149 | +messages with the current default configuration: |
| 150 | + |
| 151 | +[Cardano Trace Documentation](https://github.com/input-output-hk/cardano-node/blob/master/doc/new-tracing/tracers_doc_generated.md) |
| 152 | + |
| 153 | +This document describes the underlying library trace-dispatcher: |
| 154 | + |
| 155 | +[trace-dispatcher: efficient, simple and flexible program tracing](https://github.com/input-output-hk/cardano-node/blob/master/trace-dispatcher/doc/trace-dispatcher.md) |
| 156 | + |
| 157 | +This document describes a seperate service for logging and monitoring Cardano nodes: |
| 158 | + |
| 159 | +[Cardano Tracer](https://github.com/input-output-hk/cardano-node/blob/master/cardano-tracer/docs/cardano-tracer.md) |
0 commit comments