Temporary failure in name resolution for googleapis.com

[jerome2710]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

We are currently installing a clean Sentry setup when migrating from CentOS 7 to Debian. When enabling the Google Auth login, the callback to our application fails with a 504 Gateway Timeout.

Expected Result

We would expect a successful login.

Actual Result

The browser returns a 504 Gateway Timeout. When following the logs, it seems that googleapis.com cannot be resolved;

urllib3.exceptions.MaxRetryError: SafeHTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /oauth2/v4/token (Caused by NewConnectionError('<sentry.net.http.SafeHTTPSConnection object at 0x7efe23c266d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

I tried a few things and noticed that sentry-self-hosted-nginx-1 doesn't actually resolve googleapis.com. On our previous CentOS 7 setup, this actually does work:

root@server:/opt/sentry-self-hosted/sentry# docker exec -it sentry-self-hosted-nginx-1 sh
/ # ping googleapis.com
ping: bad address 'googleapis.com'

I just followed https://develop.sentry.dev/self-hosted/#getting-started, ran ./install.sh and docker compose up -d. Do I need additional configuration to allow the Sentry containers to access the internet for the Google Auth to work?

Product Area

Settings - Auth

Link

No response

DSN

No response

Version

24.8.0

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[hubertdeng123]

What version of self-hosted Sentry were you running before? It's likely that this is an issue with your VM and that there is some networking issue going on here.

[jerome2710]

@hubertdeng123 We were running Sentry 24.4.4 on CentOS 7 and are now running 24.8.0 on Debian 11.

I tried finding differences between the instances, as the old one is still running. But I can't seem to spot any. The new VM seems to be working fine, also in combination with other Docker services. It's just that Sentry can't reach Google or our Slack integration for instance.

[jerome2710]

After hours and hours, I found the problem.

Firstly, I checked if the internet issue was limited to Sentry or all of Docker. I did this by using the lightweight Busybox image:

> docker pull busybox
> docker run -it busybox ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
13 packets transmitted, 0 packets received, 100% packet loss

> docker run -it busybox wget google.com
wget: bad address 'google.com'

As you can see, Busybox is also unable to access the internet. So it isn't a Sentry issue.

Using the host network:

> docker run --network host -it busybox wget google.com
Connecting to google.com (142.251.39.110:80)
Connecting to www.google.com (172.217.23.196:80)
saving to 'index.html'
index.html           100% |****************************| 21568  0:00:00 ETA
'index.html' saved

So the issue is in the Docker network. I inspected the network:

> docker network inspect bridge

Which looked all good.

Then ChatGPT suggested stopping Docker, deleting the ip link and restarting Docker:

> systemctl stop docker
> ip link delete docker0
> systemctl start docker

And magically.. we now have internet.

I have no clue what the ip link did, but this seemed to be the solution to my problem. I will close with issue now.