[META 691] Reporter for OTel Metrics

[elastic-apm-tech]

See meta issue and spec for the description and details:

• Meta issue: Reporter for OTel Metrics apm#691
• Spec issue: [META 691] Spec: Reporter for OTel Metrics apm#692

[trentm]

I'll edit this comment to give a summary of this PoC work. I'll also use elastic/apm#691 for updates/Qs/discussions that aren't necessarily specific to the Node.js agent.

This is a placeholder for now.

[trentm]

vanilla OTel JS Metrics usage

Basic OTel JS Metrics usage looks like this (adapted from this example):

const api = require("@opentelemetry/api-metrics");
// const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc'); // OTLP/gRPC
// const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-http'); // OTLP/HTTP+JSON
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-proto'); // OTLP/HTTP+protobuf
const { MeterProvider, PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');


// Setup a MeterProvider.
const metricExporter = new OTLPMetricExporter({ 
    // url: <the collector or APM server>
});
const meterProvider = new MeterProvider({
  // metadata
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'basic-metric-service',
  }),
});
meterProvider.addMetricReader(new PeriodicExportingMetricReader({
  exporter: metricExporter,
  exportIntervalMillis: 1000,
}));
api.metrics.setGlobalMeterProvider(meterProvider);


// User code to create metrics (called "Instruments" in OTel-land).
const meter = api.metrics.getMeter('my-metrics-play');
const requestCounter = meter.createCounter('my_requests', {
  description: 'Example of a Counter',
});
const upDownCounter = meter.createUpDownCounter('my_up_down_counter', {
  description: 'Example of a UpDownCounter',
});
const histogram = meter.createHistogram('my_histogram', {
  description: 'Example of a Histogram',
});

// Then some artifical code showing updating of those metrics.
setInterval(() => {
  requestCounter.add(1, attributes);
  upDownCounter.add(Math.random() > 0.5 ? 1 : -1, attributes);
  histogram.record(Math.random(), attributes);
}, 1000);

The block to "Setup a MeterProvider" is mostly boilerplate that most users will tuck away in a setup module, or that will be wrapped in higher-level SDK wrappers.

Metrics in OTel JS will be GA shortly. When it is, that const api = require("@opentelemetry/api-metrics") will become const api = require("@opentelemetry/api") -- the same import as a user does to access api.trace.* for tracing.

The above script works to send OTel Metrics to APM server when configured like this:

// ...
const metricExporter = new OTLPMetricExporter({ 
    url: apmServerUrl + '/v1/metrics',
    headers: {
        Authorization: `Bearer $(secretToken)`
    }
});
// ...

or via [the OTEL_EXPORTER_...] environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT=<APM server url>
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer <APM secret token>'

The OTLP/gRPC exporter (@opentelemetry/exporter-metrics-otlp-grpc) works with APM Server as well.
The OTLP/HTTP+JSON exporter does not work with APM Server, because APM server doesn't support that protocol yet.

[trentm]

Possible APM agent API changes

Here is some musing on what the end result of this work could be, from the point of view of a user of the APM agent. Starting from the smallest changes and working up.

We could start with a ElasticAPMMetricExporter that could be used in the boilerplate above something like the following. This handles sending collected OTel metrics to the configured APM server -- whether that is via the intake v2 API, or via separate OTLP requests.

const apm = require('elastic-apm-node').start()

// ...

// Setup a MeterProvider.
const meterProvider = new MeterProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'basic-metric-service',
  }),
});
meterProvider.addMetricReader(new PeriodicExportingMetricReader({
  exporter: apm.getOTelMetricExporter(),                 // <-- HERE
  exportIntervalMillis: 1000,
}));
api.metrics.setGlobalMeterProvider(meterProvider);

// ...

---

Next we might want to customize the MetricReader -- to use the same metricsInterval as the APM agent, to possibly flush metrics on shutdown.

// Setup a MeterProvider.
const meterProvider = new MeterProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'basic-metric-service',
  }),
});
meterProvider.addMetricReader(apm.getOTelMetricReader())                // <-- HERE
api.metrics.setGlobalMeterProvider(meterProvider);

// ...

---

A limitation so far is how our APM metadata is handled. The intended place to provide this info in the OTel API is in that Resource passed to MeterProvider. Options here are:

1. Provide a custom MeterProvider that knows how to use the APM agent's gather metadata automatically.
2. Hack into or customize the MetricExporter class to (mostly) ignore the given Resource and handle the APM agent's metadata at the lower exporting level.
3. Provide a utility API to get an OTel Resource matching the APM agent's metadata.

So far, I've been playing with (1). Option (2) may be necessary to try the idea of a multipart request to APM server with the metadata JSON as one part, and the protobuf as the second. Option (3) is problematic because we gather metadata (e.g. cloud metadata) asynchronously. My read of OTel JS is that it does not support this. The user is expected to wait for resource detection before creating a MeterProvider.

// Setup a MeterProvider.
const meterProvider = apm.getOTelMeterProvider()
api.metrics.setGlobalMeterProvider(meterProvider);

// ...

---

Finally, the APM agent could optionally .setGlobalMeterProvider(...). Then, the simple/golden path is as follows, and it just works.

const apm = require('elastic-apm-node').start()
const otel = require('@opentelemetry/api')

const meter = otel.metrics.getMeter('my-meter')
const counter = meter.createCounter('my_counter')
counter.add(1)

The advanced path is to get the APM agent-provided MeterProvider and add a reader using our exporter. The user could then add other readers as well -- like the Kibana example using a Prometheus pull exporter. And it would be up to the user to set .setGlobalMeterProvider() if they would like.

---

What does this look like with the OTel Bridge on?

const apm = require('elastic-apm-node').start({
    opentelemetryBridgeEnabled: true
})
const otel = require('@opentelemetry/api')

// Tracing
const tracer = otel.trace.getTracer('hello-world')
tracer.startActiveSpan('hi', span => {
    console.log('hello')
    span.end()
})

// Metrics
const meter = otel.metrics.getMeter('my-example-meter')
const hiCounter = meter.createCounter('his', { description: 'num hello worlds' })
hiCounter.add(1)

[trentm]

Use a single 'elastic_apm_metadata' resource attribute?

See elastic/apm#691 (comment) for an idea to pass our metadata JSON as an OTel Resource Attribute, and let APM server use that.

One issue, at least for the Node.js agent, is that our metadata isn't complete until sometime after the APM agent has started -- by which time the user may already have called the OTel API to get a Meter or MeterProvider.

The OTel equivalent -- resource detectors -- are, of course, async, but the experimental "NodeSDK" documentation shows waiting for that detection before using the OTel API:

// ...

// You can optionally detect resources asynchronously from the environment.
// Detected resources are merged with the resources provided in the SDK configuration.
sdk.start().then(() => {
  // Resources have been detected and SDK is started
});

// ...

The sdk.start() will await this.detectResources(). So, basically, OTel JS doesn't support late-added resources.

Digging through the OTel Metrics code, the resource provided to the MeterProvider is only used when metrics are first sent. And it can be updated late via:

apm._transport.on('cloud-metadata', encodedMetadata => {
  meterProvider._sharedState.resource = meterProvider._sharedState.resource.merge(
    new Resource({ elastic_apm_metadata: encodedMetadata })
  )
})

Some constraints/TODOs:

• The 'cloud-metadata' event from the http-client isn't reliable for this, because of the other expectExtraMetadata code path. We could impl a different "metadata is ready" event that covers both cases.
• New constraint: avoid collecting (and sending) metrics until after metadata is ready.

[trentm]

Sharing the APM agent's connection pool for OTLP/HTTP requests

I looked a bit into whether the OTel JS Metrics SDK's OTLP/HTTP requests could share the APM agent's http-client's connection pool. (Note: This is limited to OTLP/HTTP. I haven't looked for OTLP/gRPC.)

I am able to get require('@opentelemetry/exporter-metrics-otlp-proto').OTLPMetricExporter
to use the APM agent's HTTP agent. The "HTTP agent" is a Node.js term for the object that handles the connection pool and other lower-level HTTP client details. We use an HTTP agent from the "agentkeepalive" module.

The code to share the connection pool is:

metricExporter._otlpExporter.agent = apm._transport._agent

I can observe the agentkeepalive behaviour via (where metrics-otlphttp-exporter.js is the script I'm playing with):

DEBUG=agentkeepalive NODE_DEBUG=http node metrics-otlphttp-exporter.js

However, some caveats:

• A separate connection is created for the (a) intake v2 requests and (b) the OTLP/HTTP requests. This is because the socket for (a) is almost never free -- because we do long-lived chunked requests, by design. So when (b) comes along, it creates a second socket. Thereafter those two sockets are reused.
• That re-use for (b) was when I was testing with metricsInterval: '3s'. I believe with the default 30s (or anything higher than 4s), the Node.js agent will timeout this free socket, resulting in a new connection being made (no keep-alive) for each OTLP/HTTP metrics upload request. A free socket is timed out after 4s (_conf.freeSocketTimeout = 4000) because to avoid a possible race with the server-side keep-alive timeout. See occasional "socket hang up" errors in APM server intake requests #2594 for details on why the Node.js APM agent does this. It is possible this could be mitigated with a server Keep-Alive: timeout=30 response header or similar (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive#directives).

I believe that if we really need/want to maintain a single APM server socket connection per APM agent, then we'd have to do one of:

• send metrics via the same intake v2 API requests;
• coordinate the sending of intake-v2 and OTLP/HTTP requests in the same queue, i.e. pipeline those requests; or
• use HTTP/2 or HTTP/3.

Perhaps the Go agent is already using HTTP/2 to talk to APM server's intake-v2?
The OTLP/HTTP transport in the JS OTel Metrics SDK is currently using HTTP/1.1.

---

I see that Andrew basically already said all this succinctly. :)

For HTTP/2 connections there'll be multiplexing over the same connection, but there may still be multiple connections due to the long-lived nature of our streaming requests.

[trentm]

From Felix's possible "Cons" for using OTLP to send metrics:

Potentially adds additional dependencies (such as gRPC and the OTel metrics SDK) to the core of the agent

How large of a dep is the OTel Metrics SDK and the various OTLP exporters?

tl;dr

Adding the OTel Metrics SDK dependency to the Node.js APM agent is a ~25% increase in install size (from 19M to 25M).
Further, adding the OTLP exporter packages adds more: OTLP/HTTP+JSON (to 28M), OTLP/Protobuf (to 37M), OTLP/gRPC (to 39M).
OTLP/HTTP+JSON isn't currently supported by APM server, so we are looking at adding ~14MB to the Node.js agent's install size to use OTLP for sending metrics.

In general, this may be fine for users.
For Lambda, that throws us over the "30MB" overhead we document here: https://www.elastic.co/guide/en/apm/lambda/current/aws-lambda-overhead.html#_impact_on_the_deployment_package_size
Note that there is some opportunity in the current Node.js APM agent's deps to recover a lot of space and get the install size down.

details

Currently Node.js agent sizes:

% ./dev-utils/make-distribution.sh
% du -sh build/dist/*
3.8M	build/dist/elastic-apm-node-lambda-layer.zip
 19M	build/dist/nodejs

Upgrading to @opentelemetry/api@1.3.0 which will be required by the metrics SDK, adds ~100k compressed:

% du -sh build/dist/*
3.9M	build/dist/elastic-apm-node-lambda-layer.zip
 19M	build/dist/nodejs

Adding the metrics SDK adds ~6M (25%).
Note that this may throw us over the "30MB" Lambda layer impact mentioned here:
https://www.elastic.co/guide/en/apm/lambda/current/aws-lambda-overhead.html#_impact_on_the_deployment_package_size

% npm ls | rg opente
├── @opentelemetry/api@1.3.0
├── @opentelemetry/sdk-metrics@1.8.0
% ./dev-utils/make-distribution.sh
% du -sh build/dist/*
5.4M	build/dist/elastic-apm-node-lambda-layer.zip
 25M	build/dist/nodejs

Adding each of the OTLP/{foo} exporters.

• the OTLP/HTTP+JSON exporter:

% npm ls | rg opentelemetry
├── @opentelemetry/api@1.3.0
├── @opentelemetry/exporter-metrics-otlp-http@0.34.0
├── @opentelemetry/sdk-metrics@1.8.0
% du -sh build/dist/*
6.1M	build/dist/elastic-apm-node-lambda-layer.zip
 28M	build/dist/nodejs

• the OTLP/HTTP+Protobuf exporter:

% npm ls | rg opentelemetry
├── @opentelemetry/api@1.3.0
├── @opentelemetry/exporter-metrics-otlp-proto@0.34.0
├── @opentelemetry/sdk-metrics@1.8.0
% du -sh build/dist/*
7.6M	build/dist/elastic-apm-node-lambda-layer.zip
 37M	build/dist/nodejs

• the OTLP/gRPC exporter:

% npm ls | rg opentelemetry
├── @opentelemetry/api@1.3.0
├── @opentelemetry/exporter-metrics-otlp-grpc@0.34.0
├── @opentelemetry/sdk-metrics@1.8.0
% du -sh build/dist/*
8.4M	build/dist/elastic-apm-node-lambda-layer.zip
 39M	build/dist/nodejs

[trentm]

My work is here: https://github.com/elastic/apm-agent-nodejs/compare/trentm/otel-metrics-poc
Best example usage is here: https://github.com/elastic/apm-agent-nodejs/blob/trentm/otel-metrics-poc/examples/otel-metrics-poc/use-otel-metrics-via-elastic-apm.js
Main exporter conversion impl is here: https://github.com/elastic/apm-agent-nodejs/blob/trentm/otel-metrics-poc/lib/otel-metrics-poc/ElasticApmMetricExporter.js#L102-L155

Putting this on the back burner for a little bit.