Top-level tags/labels for metrics

[felixbarny]

For elastic/apm#70, we have the need for top-level tags/labels for metrics. In particular, these are transaction.name, transaction.type, span.type and span.subtype. This is needed for a unified query bar experience. For example, filtering for a particular transaction.name should filter the related metrics as well as they should be used on the service detail view, powering the breakdown graphs.

Some ideas on how to enable top-level labels:

• APM Server knows about "well-known" tags like transaction_name and transaction_type and brings them to top-level
• Extend the metricset spec (https://github.com/elastic/apm-server/blob/master/docs/spec/metricsets/metricset.json) to allow for top-level key/value pairs (dots in the keys have to be allowed)

[graphaelli]

Did you consider modifying the metrics intake instead to allow transaction and other information in a strictly defined way, similar to how errors and spans take transaction_id? This would allow validation on the values, for example ensuring that transaction_id is a string and not a boolean or number as labels can be. This should also address the good point in elastic/apm#70 (comment), where it's surprising that labels end up in different places depending on the version of apm-server.

To be concrete, this means a metric payload could look like:

{
  "metricset": {
    "samples": {
      "self_time.sum": {
        "value": 200
      },
      "self_time.count": {
        "value": 10
      },
      "duration.sum": {
        "value": 300
      },
      "duration.count": {
        "value": 10
      }
    },
    "tags": {
      "some": "abc",
      "code": 200,
      "success": true
    },
    "timestamp": 1496170422281000,
    "transaction": {
      "name": "APIRestController#orders",
      "type": "request"
    },
    "span": {
      "type": "transaction"
    }
  }
}

[felixbarny]

I like that suggestion a lot!