[Bug]: Multiple false "Your server is unreachable" mails through the day

[alxhu-dev]

Error Message and Logs

I recieve multiple mails through the day with "Coolify cannot connect to your server (example.com). Please check your server and make sure it is running. All automations & integrations are turned off!".

Most time, exactly (!) one minute later I recieve a mail with "Your server (example.com) was offline for a while, but it is back online now. All automations & integrations are turned on again." (but sometimes it takes hours up to days until I recieve the reachable mail).

It always happens around on full hour (for example 3:01, 4:00, 5:02). It happens to servers in the local network as well as remote servers. Yesterday it even reported itself (localhost) as unreachable.

I have an external monitoring and I don't see any down time there, and I'm able to connect to any services on those systems.

Additionally: I removed any ticks (except Server Disk Usage) from the Notifications tab and I still recieve those "unreachable" messages.

Steps to Reproduce

Honestly I have no clue on how to reproduce this since all servers are reachable all the time.

1. Have servers running
2. ???

Example Repository URL

No response

Coolify Version

v4.0.0-beta.370

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Debian 12

Additional Information

It started about two weeks ago.

[4xx22]

I'm experiencing the same issue on 4.0.0-beta.373 and Debian 12. My setup is one server hosting Coolify and two additional servers connected to it, with all three servers hosted with different providers.

The "Your server is unreachable" notification only appears for the most recently added server, which was configured this week. The other two servers work without any issues.

I have external monitoring in place that confirms all servers are operational and can communicate with each other consistently. Usually, when I get the unreachable notification, I receive another notification a minute later indicating that the server is available again.

How does Coolify determine that the server is not reachable?

[M1ck0]

Same here. I added Discord notifications and get a little heart attack every time I see it

[alxhu-dev]

The issue has been resolved for me with the latest updates (v4.0.0-beta.376).

@M1ck0 @4xx22 Do you still experience this error or can I close the issue?

[algora-pbc]

💎 $10 bounty • 4xx22

Steps to solve:

1. Start working: Comment /attempt #4407 with your implementation plan
2. Submit work: Create a pull request including /claim #4407 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to coollabsio/coolify!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @peaklabs-dev	Dec 10, 2024, 2:15:08 PM	#4586

[4xx22]

The issue has been resolved for me with the latest updates (v4.0.0-beta.376).

@M1ck0 @4xx22 Do you still experience this error or can I close the issue?

I'm still having the issue, just had one occurrence now

[peaklabs-dev]

/attempt #4407 I will add a few fixes. What external monitoring solution do you use and how many times per minute does it check the server?

Options

• Cancel my attempt

[4xx22]

/attempt #4407 I will add a few fixes. What external monitoring solution do you use and how many times per minute does it check the server?

Options

I currently have Zabbix and Hetrixtools
Zabbix is checking the server every 10 seconds, HetrixTools every minutes

[M1ck0]

Issue still happens

[peaklabs-dev]

Yes the fix is not released yet.

[peaklabs-dev]

@4xx22 The fix for this issue has now been released, please let me know if it fixed your issues and if it is fixed, please award me the bounty.

[4xx22]

@peaklabs-dev The update have been deployed on our server, we'll test it and report back in a few days.

[algora-pbc]

@peaklabs-dev: You've been awarded a $10 bounty by 4xx22! 👉 Complete your Algora onboarding to collect the bounty.

[algora-pbc]

🎉🎈 @peaklabs-dev has been awarded $10! 🎈🎊

• Share on socials
• Give feedback

[Aldekein]

I'm on v4.0.0-beta.380 for a week or two, yesterday & today I started getting the error after the update. Again :(

"Coolify: Your server (localhost) is unreachable." today on email only.
"Coolify: Your server 'localhost' is unreachable. All automations & integrations are turned off! Please check your server! IMPORTANT: We automatically try to revive your server and turn on all automations & integrations." yesterday on a webhook notification only.

Needless to say I jumped to computer just to see it's up and running without problems.