43581 expose host.docker.internal with 19092 port

[dcdh]

…cation with a kafka connect (likes Debezium) devservice

[quarkus-bot]

Thanks for your pull request!

Your pull request does not follow our editorial rules. Could you have a look?

• title should not end up with ellipsis (make sure the title is complete)

_{This message is automatically generated by a bot.}

[dcdh]

Needed for debezium outbox devservice

https://issues.redhat.com/browse/DBZ-8281

debezium/debezium#5903

[geoand]

Can you please update the title and description as it very unclear what the intent is here.

Thanks

[dcdh]

Hello @geoand

Thanks for your comment, I am sorry, I lake of explanations, here is the full story.

Currently, redpanda and other kafka devservice are using the localhost + host port as boostrap server and advertised listener.
This works fine when all containers dependencies (likes postgres) are available inside localhost + host port and with the application in dev mode or under tests because they can all access containers using the localhost + host mapped ip.

But when I want to use a debezium connect running inside a container it is not working because the debezium connect container cannot access it from inside using host localhost + host port (because localhost referred to the debezium connect container aka 127.0.0.1 and not the host's localhost ip).

So I have tested to do it using devservices in shared network. But it is not possible because the postgresql jdbc url will used the docker network which is not accessible for my application in dev mode or under tests. Moreover, a bug is present: reusing a postgres container is not working because the key to determine if a container is already up is using the network id which is changing each time a test suite is launched or dev mode launched.

So the only way to make it works is to use host.docker.internal bind with the host-gateway inside the debezium connect container and next use it to reference all other containers dependencies likes redpanda and postgresql inside there url and debezium connect configuration.

Regarding postgresql nothing add to be done: my debezium connect can access it using jdbc:postgresql://host.docker.internal:5432 mapped to host port/hibernate_orm_test

But regarding redpanda and others I need it to understand what is host.docker.internal:host-gateway by exposing it using the 19092 port this way HOST_DOCKER_INTERNAL://0.0.0.0:19092 and advertise the dbezium connect with host.docker.internal:host mapped 19092 port likes this sample HOST_DOCKER_INTERNAL://host.docker.internal:19093

My change concerns only RedPanda and Kafka native. Strimzi is much complex to add it so I will not changed it - keep unsupported.

Here is a docker compose snippet to help understand it. This snippet can successfully run redpanda with debezium connect in a containers way

---
services:
  redpanda:
    image: vectorized/redpanda:v24.1.2
    container_name: redpanda
    command:
      - redpanda start
      - --mode dev-container
      - --smp 1
      - --kafka-addr INSIDE://0.0.0.0:9092,HOST_DOCKER_INTERNAL://0.0.0.0:19092
      - --advertise-kafka-addr INSIDE://redpanda:9092,HOST_DOCKER_INTERNAL://host.docker.internal:19093
    ports:
      - 19093:19092
  debezium:
    image: biz-flow-ext/connect:1.0.0-SNAPSHOT
    container_name: debezium
    environment:
      BOOTSTRAP_SERVERS: host.docker.internal:19093
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: connect_configs
      OFFSET_STORAGE_TOPIC: connect_offsets
    ports:
      - 8083:8083
    extra_hosts:
      - host.docker.internal:host-gateway
  busybox:
    image: yauritux/busybox-curl:latest
    container_name: busybox
    command: tail -f /dev/null
    extra_hosts:
      - host.docker.internal:host-gateway

Regarding debezium a draft PR has been made with a RedPanda hack here to test that it is working.
https://github.com/debezium/debezium/blob/9b981f8dd8d0ea78d36ce61d1403838d193b6cd0/debezium-quarkus-outbox-common/deployment/src/main/java/io/quarkus/kafka/client/deployment/RedpandaKafkaContainer.java#L90
https://github.com/debezium/debezium/blob/9b981f8dd8d0ea78d36ce61d1403838d193b6cd0/debezium-quarkus-outbox-common/deployment/src/main/java/io/quarkus/kafka/client/deployment/RedpandaKafkaContainer.java#L104

And it is working :) My tests are ok. I can consume message from kafka following a push by debezium :)
https://github.com/debezium/debezium/blob/9b981f8dd8d0ea78d36ce61d1403838d193b6cd0/debezium-quarkus-outbox/integration-tests/src/test/java/io/debezium/outbox/quarkus/it/OutboxTest.java#L89

So I just need to release a version of Quarkus with this fix next I will be able to finish my work on debezium side.

If you want some changes to be made, do not hesitate to contact me.

Regards,
Damien