You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we only allow a single producerConf for both Kafka Producers for the raw and bad output streams. This means we end up dynamically inferring things like client.id which are then set to producer-1 and producer-2 (and which renders the Stream Lineage view in Confluent Cloud effectively useless.
It also means permissions for multiple topics need to be shared rather than being individual.
All other Kafka applications enforce segmented configuration for each producer + consumer and the Collector should follow suit.
The text was updated successfully, but these errors were encountered:
We had a discussion about this when starting to work on the refactoring of the collector.
The question was : should we take the opportunity to align the configuration of the collector with the other apps, and have :
input {
}
output {
good {
}
bad {
}
}
It was decided to keep the existing format for now, so that we don't need to worry about changing Terraform yet (both for OSS and our infra).
Once we have made the switch to http4s collector we can see to make these breaking changes and unify the configuration format of the collector with the other apps, which will make possible to configure good and bad streams differently.
Currently we only allow a single
producerConf
for both Kafka Producers for the raw and bad output streams. This means we end up dynamically inferring things likeclient.id
which are then set toproducer-1
andproducer-2
(and which renders the Stream Lineage view in Confluent Cloud effectively useless.It also means permissions for multiple topics need to be shared rather than being individual.
All other Kafka applications enforce segmented configuration for each producer + consumer and the Collector should follow suit.
The text was updated successfully, but these errors were encountered: