Description
Component(s)
exporter/kafka
Is your feature request related to a problem? Please describe.
Inspired by #32735 because it is a related problem:
When the setting "topic" is not specified, the same kafka exporter config can be used in all three pipelines if the topic names match the default values:
exporter:
kafka:
pipelines:
metrics:
exporters: [kafka] # publishes topic otlp_metrics
logs:
exporters: [kafka] # publishes topic otlp_logs
traces:
exporters: [kafka] # publishes topic otlp_spans
If the topic is set to any value, this structure will work in exporter perspective.
exporter:
kafka:
topic: custom_traces_topic
pipelines:
metrics:
exporter: [kafka] # publishes topic custom_traces_topic
logs:
exporter: [kafka] # publishes topic custom_traces_topic
traces:
exporter: [kafka] # publishes topic custom_traces_topic
What happens in this case is that the three exporters will send to the same topic. This is a race condition that will succeed in 1/3 of scenarios at the receiving end (see #32735).
To avoid this problem, the user must create three different exporters for each pipeline to set custom topic names. This is error prone and inconsistent with the default behavior that allows having one exporter for all three pipelines with the default topic names.
Describe the solution you'd like
Having three different topic names by default but being able to override it with only a single one is a strange feature.
Just create three topic properties of the kafka exporter:
exporter:
kafka:
traces_topic: custom_traces_topic # default otlp_spans
metrics_topic: custom_metrics_topic # default otlp_metrics
logs_topic: custom_logs_topic # default otlp_logs
pipelines:
metrics:
exporters: [kafka] # publishes topic custom_metrics_topic
logs:
exporters: [kafka] # publishes topic custom_logs_topic
traces:
exporters: [kafka] # publishes topic custom_traces_topic
Alternative definition (solution should match for exporter and receiver):
exporter:
kafka:
topic:
traces: custom_traces_topic # default otlp_spans
metrics: custom_metrics_topic # default otlp_metrics
logs: custom_logs_topic # default otlp_logs
Describe alternatives you've considered
The documentation gives some hints about determining the actual topic:
- The client application sending telemetry data to OpenTelemetry should not be concerned with setting topic names in attributes that are used internally to transport OpenTelemetry information using Kafka.
- The context could be configured with topic names.
- found no example how to configure that
- need to have some logic to determine from telemetry data to configure the topic to use
- evaluated for every message
- more complex setup than simply defining three static properties
- Feature enhancement of this ticket.
Current workaround: define three exporters with all properties redundant except the topic property and use them individually in three pipelines.
Additional context
No response