Adding clickhouse hostname to no_proxy

[drigolin]

Self-Hosted Version

24.8.0

CPU Architecture

x86_64

Docker Version

27.1.2

Docker Compose Version

2.29.1

Steps to Reproduce

1 Fresh install of 24.8.0 on a system behind a proxy
2 containers connecting to clickhouse are unable to connect

I had to add clickhouse on the no_proxy variable
my actual noproxy is relay,web,sentry,kafka,clickhouse,172.18.0.0/16,127.0.0.0/8
I added also kafka and the 172.18 Docker LAN during my tests.

I had to reinstall all sentry to have containers update no_proxy variable, I presume is copied from the no_proxy env variable only during this phase.

Expected Result

All containers connecting to clickhouse are not trying to do it via global proxy but stay in the docker lan

Actual Result

After doing that all seems working fine.

Event ID

No response

[hubertdeng123]

Are you saying that when no_proxy env variable is set, clickhouse is not being built properly with that setting while everything is? I'm glad you ended up getting things working though

[drigolin]

In the online documentation the suggested value for no_proxy is
"noProxy": "relay,web,sentry,127.0.0.0/8"
But I had to change it to
"noProxy": "relay,web,sentry,clickhouse,kafka,127.0.0.0/8"
to have my sentry working.
After first installation I didn't receive any errors, but I was not able to see any events into dashboards. Into log files I have found errors related to clickhouse query and kafka connections. Subscription* containers.

As I have found in my central proxy server logs requests to "https://clickhouse:8124..." from a container inserting data into clickhouse via http protocol I have decided to change the no_proxy env variables and docker config.json to have also kafka and clickhouse. I had to rebuild everything to have containers getting new config values. Stop and restart is not enough .

The server where sentry is running is behind a proxy and I followed the configuration proposed. Ah... by the way I had also to change proxy config from hostname based to IP based as the containers seems unable to resolve the hostname saved into /etc/hosts of my server.

And to have containers getting changes to no_proxy I had to destroy and recreate everything it seems that proxy env variables are copied at installation time and not updated anymore.

Maybe I did some mistake on my Docker environment, I'm new to Docker.

[aldy505]

Hello! I realized this is wrong and the docs is incomplete just now. Your no-proxy config should be:

{
  ...
  "proxies": {
    "no-proxy": "smtp,memcached,redis,postgres,kafka,clickhouse,snuba-api,symbolicator,web,worker,nginx,relay,vroom,172.17.0.0/16,127.0.0.0/8", // add the rest of your no-proxy config
  }
}

Please change 172.17.0.0/16 to the IP pool being used by Docker. I made an issue about putting this up on the docs site here: #3300

Let me know if the config above helped you.

[drigolin]

@aldy505 What I have found is that changing this docker config value doesn't work with an existing insatllation. I had to delete all containers and volumes and rebuild. It seems that compose scripts copy the noproxy value from env variables and not from docker config.
I had to change also system no_proxy value not only the docker config.
I see in compose scripts many time static copy of "no_proxy" env variable into config no_proxy of single container because is used by apt update and other http requests made during installation process.

Ad I told before I'm new to Docker maybe there are a better way to do it.

[aldy505]

@aldy505 What I have found is that changing this docker config value doesn't work with an existing insatllation. I had to delete all containers and volumes and rebuild.

Yeah after changing the Docker config value, you should also restart Docker and tear down the Sentry installation.

It seems that compose scripts copy the noproxy value from env variables and not from docker config.

The compose script does not read any noproxy, even if it is, it would need these: smtp,memcached,redis,postgres,kafka,clickhouse,snuba-api,symbolicator,web,worker,nginx,relay,vroom since they are hostnames that are only accessible inside the Docker network. The install script does not touch that one.