Description
Component(s)
receiver/kafka
Is your feature request related to a problem? Please describe.
When the setting "topic" is not specified, the same kafka receiver config can be used in all three pipelines if the topic names match the default values:
receivers:
kafka:
pipelines:
metrics:
receivers: [kafka] # consumes topic otlp_metrics
logs:
receivers: [kafka] # consumes topic otlp_logs
traces:
receivers: [kafka] # consumes topic otlp_spans
If the topic is set to any value, this structure will not work:
receivers:
kafka:
topic: custom_traces_topic
pipelines:
metrics:
receivers: [kafka] # consumes topic custom_traces_topic => THIS WON'T WORK
logs:
receivers: [kafka] # consumes topic custom_traces_topic => THIS WON'T WORK
traces:
receivers: [kafka] # consumes topic custom_traces_topic
What happens in this case is that the three receivers will try to claim the same topic. This is a race condition that will succeed in 1/3 of scenarios.
To avoid this problem, the user must create three different configs for each pipeline. This is inconsistent with the default behavior that allows having one config for all three pipelines if the topic names are matching.
Describe the solution you'd like
The otlp receiver provides traces_url_path, metrics_url_path, and logs_url_path configuration to allow the URL paths that signal data needs to be sent to be modified per signal type.
The same principle can be applied to the topics of the kafka receiver:
receivers:
kafka:
traces_topic: custom_traces_topic # default otlp_spans
metrics_topic: custom_metrics_topic # default otlp_metrics
logs_topic: custom_logs_topic # default otlp_logs
pipelines:
metrics:
receivers: [kafka] # consumes topic custom_metrics_topic
logs:
receivers: [kafka] # consumes topic custom_logs_topic
traces:
receivers: [kafka] # consumes topic custom_traces_topic
Describe alternatives you've considered
No response
Additional context
No response