fix: change advertised listeners

[yasinlachiny]

DOCKER_HOST_IP has a default environment variable.

${DOCKER_HOST_IP:-127.0.0.1}

It will create problems in multiple Kafka if you didn't specify DOCKER_HOST_IP.
KAFKA_ADVERTISED_LISTENERS will be 127.0.0.1:9092 , 127.0.0.1:9093, 127.0.0.1:9094
and if you want to connect for example from kafka1 or outside the cluster the Kafka will return for example 127.0.0.1:9094
and for sure it is not accessible from kafka1 or outside of the cluster.

If we specify DOCKER_HOST_IP everything would work fine but if we didn't specify DOCKER_HOST_IP the cluster wouldn't work.

Signed-off-by: yasin [email protected]

[charlescd]

Can you please provide more infos about your setup?

1. Which OS are you using?
2. What is your version of docker?
3. Are you using docker toolbox or docker for Windows/Mac or Linux?
4. Do you have a small scenario to reproduce? For example, I think you're not talking about connecting from the host (127.0.0.1 works fine from the host), but from another container, am I right?

[yasinlachiny]

Thank you for your response @charlescd
1-

root@yasin:~# lsb_release -d
Description:    Ubuntu 22.04 LTS

2-

root@yasin:~# docker -v
Docker version 20.10.17, build 100c701
root@yasin:~# docker compose version
Docker Compose version v2.6.0

3- I am using Docker for Linux.

4- For reproducing the error:

root@yasin:~# git clone https://github.com/conduktor/kafka-stack-docker-compose.git
root@yasin:~# cd kafka-stack-docker-compose/
root@yasin:~/kafka-stack-docker-compose# docker compose -f zk-multiple-kafka-multiple.yml up -d
root@yasin:~/kafka-stack-docker-compose# docker exec -it kafka1 bash
[appuser@kafka1 ~]$ kafka-topics --create --bootstrap-server localhost:9092 --topic test --partitions 3 --replication-factor 3
[2022-07-10 20:06:27,044] WARN [AdminClient clientId=adminclient-1] Connection to node 3 (/127.0.0.1:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-07-10 20:06:27,088] WARN [AdminClient clientId=adminclient-1] Connection to node 3 (/127.0.0.1:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-07-10 20:06:27,188] WARN [AdminClient clientId=adminclient-1] Connection to node 3 (/127.0.0.1:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

root@yasin:~/kafka-stack-docker-compose# docker exec -it kafka2 bash
[appuser@kafka2 ~]$ kafka-topics --create --bootstrap-server localhost:9093 --topic test --partitions 3 --replication-factor 3
[2022-07-10 20:11:25,188] WARN [AdminClient clientId=adminclient-1] Connection to node 3 (/127.0.0.1:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-07-10 20:11:25,224] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-07-10 20:11:25,247] WARN [Admi

If we export DOCKER_HOST_IP everything will be right from outside of the container(I mean using my host IP 203.0.113.168) but If we don't export DOCKER_HOST_IP It won't work.

I tested it from all containers and also from outside of the container and it didn't work. It makes sense because It will advertise 127.0.0.1:9092, 127.0.0.1:9093, and 127.0.0.1:9094 for localhost and also for networks outside of the docker network.
If I test it by Docker network IP it will work.

[appuser@kafka2 ~]$ kafka-topics --create --bootstrap-server kafka2:19093 --topic test --partitions 3 --replication-factor 3
Created topic test.

My host network IP is 203.0.113.168 and If I test it by this IP it won't work

[appuser@kafka2 ~]$ kafka-topics --create --bootstrap-server 203.0.113.168:9093 --topic test2 --partitions 3 --replication-factor 3
[2022-07-10 20:17:45,879] WARN [AdminClient clientId=adminclient-1] Connection to node 3 (/127.0.0.1:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-07-10 20:17:45,948] WARN [AdminClient clientId=adminclient-1] Connection to node 3 (/127.0.0.1:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-07-10 20:17:46,034] WARN [AdminClient clientId=adminclient-1] Connection to node 3 (/127.0.0.1:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

[yasinlachiny]

There are 2 options:
We want to use Kafka by docker network IP
We want to use Kafka from outside of the docker network

In both options, 127.0.0.1 is useless because in the second option we will override it and in the first option it won't work.

[yasinlachiny]

Hi @shounakchandra007 ,
Thanks for checking my changes.
Could you please reapprove? I fixed the conflicts and it's ok to merge.

[shounakchandra007]

I approved the changes @yasinlachiny

[yasinlachiny]

@shounakchandra007 Thank you a lot.

[shounakchandra007]

@shounakchandra007 Thank you a lot.

Can you please check your mail and revert me back, if possible

[TbotaPhantA]

I had the same error, because I didn't specify DOCKER_HOST_IP in .env. I couldn't create topics, because brokers couldn't talk to each other, because they tried to access localhost of the container, instead of the localhost of the host machine.

[chrisammon3000]

For anyone working on MacOS this worked for me:

export DOCKER_HOST_IP=$(ifconfig en0 | grep inet | grep -v inet6 | awk '{print $2}')
docker compose up