Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[data] Kafka GMK v2 config adds #7

Merged
merged 3 commits into from
Jan 9, 2025

Conversation

avinashak
Copy link

Note, zstd compression is still not viable but toggling all the necessary depdendency options for it anyway to avoid this song and dance.

@@ -321,7 +321,7 @@ postgres-openssl = { version = "0.5.0", default-features = false, features = ["r
pulsar = { version = "6.3.0", default-features = false, features = ["tokio-runtime", "auth-oauth2", "flate2", "lz4", "snap", "zstd"], optional = true }
rand = { version = "0.8.5", default-features = false, features = ["small_rng"] }
rand_distr = { version = "0.4.3", default-features = false }
rdkafka = { version = "0.35.0", default-features = false, features = ["tokio", "libz", "ssl", "zstd"], optional = true }
rdkafka = { version = "0.36.2", default-features = false, features = ["tokio", "libz", "ssl", "zstd", "sasl", "curl", "zstd-pkg-config"], optional = true }
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0.37 of rdkafka has breaking changes so 36.2 is the highest we can go to with simpler upgrades

@youknowjack0 youknowjack0 self-requested a review January 9, 2025 19:01
@youknowjack0
Copy link

I approve of this commit

Copy link

@alexander-jiang alexander-jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - did you test in staging or coder already, Avi?

@avinashak
Copy link
Author

yes this was powering the vector x gmk kafka tests in staging i was doing. will also test "in aggregate" after we merge this and your changes before we move forward

you can see the analytics.raw.<> topics in https://console.cloud.google.com/managedkafka/us-east1/clusters/discord-data-stg-kafka?invt=AbmYDQ&project=discord-data-stg which are the ones from vector

@avinashak avinashak merged commit 639ed3f into data_infra_vector_stable Jan 9, 2025
45 of 49 checks passed
@avinashak avinashak deleted the data/kafka-sasl branch January 9, 2025 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants