Description
Component(s)
exporter/kafka, extension/observer/kafkatopicsobserver, internal/kafka, receiver/kafka, receiver/kafkametrics
Describe the issue you're reporting
There are currently four components that use Kafka in some way:
- exporter/kafkaexporter
- receiver/kafkareceiver
- receiver/kafkametricsreceiver
- extension/observer/kafkatopicsobserver
These all depend on internal/kafka, but they each independently define a common subset of configuration. There are some subtle differences between them, and some subtle issues, e.g.
- inconsistent default for client ID:
- kafkaexporter defaults to "sarama"
- kafkareceiver defaults to "otel-collector"
- kafkametricsreceiver defaults to "otel-metrics-receiver"
- kafkatopicsobserver hardcodes it to "sarama" (it doesn't configure it; sarama defaults to this), and does not even provide configuration
- configuration is apparently being cargo-culted, e.g. SessionTimeout and HeartbeatTimeout exist in kafkatopicsobserver, but those configuration attributes are only relevant to Kafka consumers -- which the observer is not.
I propose we move common configuration to a new internal/kafka/configkafka package, modelled after confighttp, etc. Configuration will be split by purpose, i.e. general client config, consumer group config, producer config.
At the same time I propose we make the defaults for this common config consistent across each of the above components. This would be a breaking change, since it implies changing the default client ID. Users who depend on the client ID can explicitly configure it before/during upgrade to the new collector version.
I have a WIP branch at https://github.com/axw/opentelemetry-collector-contrib/tree/configkafka for an example of how this could look. There's also some unrelated cleanups/improvements in there that I intend to create PR(s) out of. Probably the most interesting parts would be:
- https://github.com/axw/opentelemetry-collector-contrib/blob/configkafka/internal/kafka/configkafka/config.go
- https://github.com/axw/opentelemetry-collector-contrib/blob/configkafka/exporter/kafkaexporter/config.go
- https://github.com/axw/opentelemetry-collector-contrib/blob/configkafka/receiver/kafkareceiver/config.go
- https://github.com/axw/opentelemetry-collector-contrib/blob/configkafka/receiver/kafkametricsreceiver/config.go
- https://github.com/axw/opentelemetry-collector-contrib/blob/configkafka/extension/observer/kafkatopicsobserver/config.go
If there's agreement, I will clean up my branch and create a PR.