IBM/sarama Go client library for Apache Kafka support #68

matskramer · 2024-06-24T16:25:14Z

No description provided.

matskramer · 2024-06-24T16:27:40Z

Depends on branch mk_upload (another pull request)

masv3971

Kafka needs to be a separate package, see docs/diagrams for inspiration.

cmd/apigw/main.go

pkg/messagebrokers/event_consumer.go

internal/apigw/apiv1/event_publisher.go

.gitignore

cmd/apigw/main.go

cmd/ui/main.go

docker-compose-goland-debug.yaml

internal/mockas/apiv1/handlers.go

internal/ui/apiv1/handlers.go

internal/ui/httpserver/bind.go

internal/ui/httpserver/endpoints.go

pkg/messagebrokers/kafka/kafka.go

…ixed

… using docker-compose (the only service currently supported is UI, but more services can be added later)

…e to make it work locally)

…message

…and consumer in same service. Two different consumer groups for same topic in different services.

…ile. The ".env" file is excluded from git but there is a ".env.example" file commited

… FILE

…-compose file for debug in Goland.

… (implementation can then be different frameworks: Kafka, Pulsar, etc)

…(implementation can then be different frameworks: Kafka, Pulsar, etc)

…mpose.yaml

… 2.8 for ui and kafka message handling.

… some minor tests

…g, se docker-compose.yaml and "ui" for what ip to use (172.16.50.50:40000 is currently used)

internal/apigw/inbound/kafka_message_handler.go

pkg/messagebroker/kafka/kafka_generic.go

cmd/mockas/main.go

masv3971 · 2024-10-04T13:13:38Z

internal/mockas/inbound/kafka_message_handlers.go

+	"vc/pkg/trace"
+)
+
+func New(ctx context.Context, cfg *model.Cfg, log *logger.Log, apiv1Client *apiv1.Client, tracer *trace.Tracer) (messagebroker.EventConsumer, error) {


Comment above all exported functions/types (easier to understand later when using them)

I prefer writing clear, concise, and readable code. If the names of types, variables, and functions are meaningful, they often make the code's intent obvious. In such cases, additional comments explaining the obvious can clutter the codebase and make it harder to focus on what matters.

Do you find the code in the file hard to read and understand?

No, but https://tip.golang.org/doc/comment

pkg/messagebroker/kafka/consumer.go

pkg/messagebroker/kafka/producer.go

matskramer self-assigned this Jun 24, 2024

matskramer force-pushed the mk_sarama branch from 92494f9 to 939f336 Compare September 26, 2024 08:37

matskramer marked this pull request as ready for review September 30, 2024 16:01

matskramer requested a review from masv3971 September 30, 2024 16:01

masv3971 reviewed Oct 1, 2024

View reviewed changes

matskramer commented Oct 1, 2024

View reviewed changes

matskramer marked this pull request as draft October 1, 2024 15:41

matskramer marked this pull request as ready for review October 2, 2024 17:31

matskramer requested a review from masv3971 October 2, 2024 17:31

matskramer added 20 commits October 2, 2024 19:49

UI: Added UI html and js for upload. Endpoint is not yet there.

442d385

UI: First version of backend to upload using UI

c3e4797

UI: Using gin to bind to JSON

c70404a

UI: Added a TODO to remove code if custom bind functions don't gets f…

32fbfc8

…ixed

Added debug ability for a service in GoLand running in docker started…

cf4863d

… using docker-compose (the only service currently supported is UI, but more services can be added later)

UI: renamed an import alias

3126db5

Kafka: first simple config of Kafka with kRaft for docker-compose

8ad4ae7

Kafka: git ignores files in volumes kraft-data

b15a8f2

Kafka: First sending of message to topic-mock-next in kafka (prototyp…

7bc4ed9

…e to make it work locally)

Kafka: added som debug printf values

d750bb3

Kafka: improved kafka config

216dab8

Kafka: gitignore

e56bd47

Kafka: gitignore

06adad3

Kafka: finally a correct gitignore pattern

db580be

Kafka: First working prototype of consuming from a Kafka topic

f559efb

Kafka: Improved prototype of consuming from a Kafka topic

b66d4cd

Kafka: First working prototype flow: web ->UI->Kafka->Mockas->Upload

31c8053

Kafka: Two kafka brokers for same topic (first working prototype)

9e6a6f1

Kafka: added some prototype usage of headers and metadata in a Kafka …

5c760a5

…message

Kafka: improved some logging and error messages

84d9277

matskramer and others added 20 commits October 2, 2024 19:49

Kafka: Concept for: Produce and consume messages in chains. Producer …

a69c227

…and consumer in same service. Two different consumer groups for same topic in different services.

Kafka: Improved logging and error handling

3ca2220

Kafka: Improved logging and error handling

7c20371

Kafka: kafka client created in main for ui

9377b6d

Kafka: user UID and GID that is running Kafka is set using a ".env" f…

de93d70

…ile. The ".env" file is excluded from git but there is a ".env.example" file commited

Removed .env from version controll - NO SENSITIVE DATA EXISTED IN THE…

6f89477

… FILE

Kafka: More .env properties possible to set for kafka. Updated docker…

a9c86e8

…-compose file for debug in Goland.

Kafka: EventPublisher is the generic interface used to publish events…

a1690b1

… (implementation can then be different frameworks: Kafka, Pulsar, etc)

Kafka: EventConsumer is the generic interface used to consume events …

03811a9

…(implementation can then be different frameworks: Kafka, Pulsar, etc)

Kafka: Newer versions of sarama and grpc and their dependencies

56720d7

Kafka: removed .env file, all config values are now back in docker-co…

788b378

…mpose.yaml

Kafka: commit after rebase, services starts but api not refactored to…

92722ce

… 2.8 for ui and kafka message handling.

UI: apiv28 portal->document/list

f0db23e

Kafka and UI: supports apiv28

1454bb7

Kafka: fixed some minor TODOs

905e68f

Diagram over sync and async upload flow.

b4adf9d

fmt.

8070017

Fixed some of the review comments -starting in docker and works after…

e32337d

… some minor tests

Fixed some of the review comments - inbound/outbound for kafka

497f72e

Go Remote debugging of service ui in JetBrains Goland now works again…

0aa0056

…g, se docker-compose.yaml and "ui" for what ip to use (172.16.50.50:40000 is currently used)

matskramer force-pushed the mk_sarama branch from a8f69b0 to 0aa0056 Compare October 2, 2024 17:50

masv3971 reviewed Oct 3, 2024

View reviewed changes

Refactorings after review nr 2

5d43b6a

matskramer requested a review from masv3971 October 4, 2024 12:37

masv3971 reviewed Oct 4, 2024

View reviewed changes

matskramer added the enhancement New feature or request label Oct 7, 2024

Refactorings after review nr 3

5b3aa38

matskramer merged commit 76cd279 into main Oct 8, 2024
2 checks passed

matskramer deleted the mk_sarama branch October 8, 2024 15:48

matskramer mentioned this pull request Oct 17, 2024

MK upload custom json using ui and support for debugging a service running in Docker #66

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IBM/sarama Go client library for Apache Kafka support #68

IBM/sarama Go client library for Apache Kafka support #68

matskramer commented Jun 24, 2024

matskramer commented Jun 24, 2024

masv3971 left a comment

masv3971 Oct 4, 2024

matskramer Oct 7, 2024

masv3971 Oct 8, 2024

IBM/sarama Go client library for Apache Kafka support #68

IBM/sarama Go client library for Apache Kafka support #68

Conversation

matskramer commented Jun 24, 2024

matskramer commented Jun 24, 2024

masv3971 left a comment

Choose a reason for hiding this comment

masv3971 Oct 4, 2024

Choose a reason for hiding this comment

matskramer Oct 7, 2024

Choose a reason for hiding this comment

masv3971 Oct 8, 2024

Choose a reason for hiding this comment