System for managing own telemetry attributes within pipeline components

[djaglowski]

Some types of components are fully or partially shared across multiple instances. (e.g. otlp receiver / memory limiter)

I have previously pursued some optimistic models in which the collector would define certain ways in which sharing is "supported", but the reality is that sharing is an unbounded problem. e.g. Someone could write a processor that shares a data structure between logs and traces pipelines only if it's instantiated on a Wednesday and the component name is "foo". I am convinced that there are too many novel cases to model ahead of time, so here is a proposal to normalize attributes as much as possible, while still allowing component authors to both add or subtract attributes from their telemetry when appropriate.

I will use Loggers as the example here, but I believe the same general ideas can be applied to TracerProviders and MeterProviders.

---

General Idea:

Create a "Logger" struct specifically for use within components. This logger satisfies the same interface and behaves essentially the same as what we have today. However, it has the following important differences:

1. The attributes are preset for the component as if it uses no sharing of parts with other instances of the component. (As defined in this RFC)
2. A Without(key ...string) method is provided, which allows component authors to remove attributes as appropriate for portions of the code that are shared. This is opposite the typical pattern used by loggers, but necessary if we want our default set of attributes to be correct.

For example:

• OTLP receiver: Since the entire component struct is shared, the factory will use Without("otelcol.signal") during construction to create a logger appropriate for the shared instance.
• Memory limiter processor: For any code outside of the shared part, it can use the logger which it was given. For the shared part, is can create a new logger by calling Without("otelcol.signal", "otelcol.component.id", "otelcol.pipeline.id").
• Custom component mentioned above: Typically uses the given logger, but if creating a logs or traces instance on a Wednesday and the component name is "foo", then it calls Without("otelcol.signal", "otelcol.component.id") to adjust the scope.

---

This implementation looks something like this:

import (
	"go.opentelemetry.io/otel/attribute"
	"go.uber.org/zap"
)

type ComponentLogger struct {
	*zap.Logger
	root *zap.Logger
	diff []attribute.KeyValue
}

func NewComponentLogger(root *zap.Logger, defaultAttrs map[string]string) *ComponentLogger {
	defaultLogger := root
	for k, v := range defaultAttrs {
		defaultLogger = defaultLogger.With(zap.String(k, v))
	}

	return &ComponentLogger{
		Logger: defaultLogger,
		root:   root,
		diff:   defaultAttrs,
	}
}

func (cl *ComponentLogger) Without(keys ...string) *zap.Logger {
	excludeKeys := make(map[string]struct{})
	for _, key := range keys {
		excludeKeys[key] = struct{}{}
	}

	logger := cl.root
	for _, attr := range cl.diff {
		if _, excluded := excludeKeys[string(attr.Key)]; !excluded {
			logger = logger.With(zap.String(k, v))
		}
	}
	return logger
}

func (cl *ComponentLogger) With(fields ...zap.Field) *ComponentLogger {
	// Create new diff by appending the new fields as attributes
	newDiff := make([]attribute.KeyValue, len(cl.diff))
	copy(newDiff, cl.diff)
	for _, f := range fields {
		newDiff = append(newDiff, attribute.Any(f.Key, f.Interface))
	}

	return &ComponentLogger{
		Logger: cl.Logger.With(fields...),
		root:   cl.root,
		diff:   newDiff,
	}
}

---

TracerProvider and MeterProvider can expose a similar With/Without interface, but the implementation will be different.

[dmitryax]

I like the general idea. A couple of questions:

• Is there a need to have With method at this point?
• Requiring string arguments for Without might be error-prone. Yes, we can provide public constants. But maybe we can have more specific methods, e.g., WithSharedSignal(), instead?

[djaglowski]

Is there a need to have With method at this point?

We can leave it out. There may be some uses out there already, but they would be able to access it on the embedded zap logger.

Requiring string arguments for Without might be error-prone. Yes, we can provide public constants. But maybe we can have more specific methods, e.g., WithSharedSignal(), instead?

We have 5 different attributes defined in the RFC, which differ based on component kind. Additionally, I would hope that in practice we see the need for removing attributes only a few specific cases. So my opinion is that such methods would be either inapplicable (e.g. receiver doesn't have a otelcol.pipeline.id to remove) or unadvisable in most cases. That said, I think limiting the values to those we enumerate makes things less error prone, so that would be my preference.

[djaglowski]

I think we need a new module which eventually contains:

• Attribute keys agreed upon in the RFC
• The logger described above
• Similar meter & tracer providers

Constraints:

• Component implementations will use the package, so cannot be in service.
• Should be able to construct appropriate attribute sets from e.g. pipeline.Signal and component.ID, so cannot be in component. Seems out of place in pipeline too though.

The best I can come up with is component/componentattribute. Does this seem reasonable @open-telemetry/collector-maintainers, or do you have other suggestions?

[djaglowski]

Draft solution: #12259

[mx-psi]

The best I can come up with is component/componentattribute. Does this seem reasonable @open-telemetry/collector-maintainers, or do you have other suggestions?

Sounds reasonable to me

[dmitryax]

We have 5 different attributes defined in the RFC, which differ based on component kind. Additionally, I would hope that in practice we see the need for removing attributes only a few specific cases. So my opinion is that such methods would be either inapplicable (e.g. receiver doesn't have a otelcol.pipeline.id to remove) or unadvisable in most cases. That said, I think limiting the values to those we enumerate makes things less error prone, so that would be my preference.

We have the solution for the logger now. However, we will likely need to apply a similar approach to the meter and tracer in the future. If we keep adding Without to them, it'll be hard for API users to catch up and keep it consistent. Especially when we need to replace data_type metrics attribute with otelcol.signal (probably via a feature gate). Having something like WithSharedSIgnal would help to consolidate it across the signals and centralize such migrations if needed. I don't believe we need dropping any other attributes at this point. Once we have such a need, a new method could be added, even if it's not useful in some scenarios.

Also, I think it could be one mutation call to TelemetrySettings instead of changing logger, meter and tracer separately. Something like TelemetrySettings.SharedSignal would return another instance of TelemetrySettings with logger, meter, and tracer not having that attribute.