"Confirmed A record with IP, but HTTP validation failed" error #4463

MatejKovacic · 2022-02-10T17:54:17Z

Prior to placing the issue, please check following: (fill out each checkbox with an X once done)

I understand that not following or deleting the below instructions will result in immediate closure and/or deletion of my issue.
I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
I have understood that answers are voluntary and community-driven, and not commercial support.
I have verified that my issue has not been already answered in the past. I also checked previous issues.

Summary

I have a fresh MailCow installation. I am trying to get my certificates signed by Let's Encrypt. Since I do not have IPv5, I disabled IPv6 according to documentation.

Then I run these commands:

cd /opt/mailcow-dockerized
touch data/assets/ssl/force_renew
docker-compose restart acme-mailcow

Logs

When I run docker-compose logs --tail=200 -f acme-mailcow I get this error:

acme-mailcow_1       | OK
acme-mailcow_1       | Thu Feb 10 16:45:23 CET 2022 - Initializing, please wait...
acme-mailcow_1       | Thu Feb 10 16:45:23 CET 2022 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow_1       | Thu Feb 10 16:45:23 CET 2022 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow_1       | Thu Feb 10 16:45:23 CET 2022 - Detecting IP addresses...
acme-mailcow_1       | Thu Feb 10 16:45:32 CET 2022 - OK: 185.58.xxx.xxx, 0000:0000:0000:0000:0000:0000:0000:0000
acme-mailcow_1       | Thu Feb 10 16:45:33 CET 2022 - Found A record for autodiscover.mydomain.mk: 185.58.xxx.xxx

acme-mailcow_1       | Thu Feb 10 16:47:42 CET 2022 - Confirmed A record with IP 185.58.xxx.xxx, but HTTP validation failed
acme-mailcow_1       | Thu Feb 10 16:47:42 CET 2022 - Found A record for autoconfig.mydomain.mk: 185.58.xxx.xxx
acme-mailcow_1       | Thu Feb 10 16:49:53 CET 2022 - Confirmed A record with IP 185.58.xxx.xxx, but HTTP validation failed
acme-mailcow_1       | Thu Feb 10 16:49:53 CET 2022 - Found A record for mail.mydomain.mk: 185.58.180.221
acme-mailcow_1       | Thu Feb 10 16:52:04 CET 2022 - Confirmed A record with IP 185.58.xxx.xxx, but HTTP validation failed
acme-mailcow_1       | Thu Feb 10 16:52:04 CET 2022 - Cannot validate any hostnames, skipping Let's Encrypt for 1 hour.
acme-mailcow_1       | Thu Feb 10 16:52:04 CET 2022 - Use SKIP_LETS_ENCRYPT=y in mailcow.conf to skip it permanently.
acme-mailcow_1       | OK

Reproduction

I tried that several times, always the same result. However, other parts of the system are working (receiving mails, sending mails,...).

System information

Question	Answer
My operating system	Debian 11.2
Is Apparmor, SELinux or similar active?	Yes
Virtualization technlogy (KVM, VMware, Xen, etc - LXC and OpenVZ are not supported	KVM
Server/VM specifications (Memory, CPU Cores)	10240 MB, 4 CPU cores
Docker Version (`docker version`)	20.10.12
Docker-Compose Version (`docker-compose version`)	1.29.2
Reverse proxy (custom solution)	don't know

Regarding AppArmor, I get this:

apparmor module is loaded.
7 profiles are loaded.
7 profiles are in enforce mode.
   /usr/bin/man
   docker-default
   lsb_release
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
0 profiles are in complain mode.
...
...

DNS problems? Please run docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254 (set the IP accordingly, if you changed the internal mailcow network) and post the output:

151.101.1.69
151.101.65.69
151.101.129.69
151.101.193.69

The text was updated successfully, but these errors were encountered:

MatejKovacic · 2022-02-10T19:13:34Z

Sorry, just another info.

Port 80 is accessible from outside, I can verify it with curl http://185.58.xxx.xxx or curl http://mail.mydomain.mk:

<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.21.6</center>
</body>
</html>

If I open HTTPS version of the site (curl https://185.58.xxx.xxx --insecure), I get the login page. So web is working normally.

MatejKovacic · 2022-02-11T15:45:02Z

I tried a solution suggested by issue #2323. So, I found out that http://185.58.xxx.xxx/.well-known/acme-challenge/1 returns 404 Not Found

Anyway, solution suggested to remove nextcloud.conf. However I go to /opt/mailcow-dockerized/data/conf/nginx/, I can see these files:

drwxr-xr-x  4 root root 4,0K feb 11 16:34 .
drwxr-xr-x 11 root root 4,0K feb 11 14:21 ..
-rw-r--r--  1 root root   50 feb 11 14:21 000-map-size.conf
-rw-r--r--  1 root root  492 feb 11 14:21 dynmaps.conf
drwxr-xr-x  2 root root 4,0K feb 11 14:21 includes
-rw-r--r--  1 root root   27 feb 11 15:21 listen_plain.active
-rw-r--r--  1 root root   49 feb 11 15:21 listen_ssl.active
-rw-r--r--  1 root root  534 feb 11 14:21 meta_exporter.conf
-rw-r--r--  1 root root   66 feb 11 15:21 server_name.active
-rw-r--r--  1 root root  246 feb 11 14:21 site.conf
-rw-r--r--  1 root root  330 feb 11 15:21 sites.active
-rw-r--r--  1 root root   38 feb 11 15:21 sogo.active
-rw-r--r--  1 root root   71 feb 11 15:21 sogo_eas.active
drwxr-xr-x  2 root root 4,0K feb 11 14:21 templates

MatejKovacic · 2022-02-11T17:45:49Z

Also, I can see a file: /opt/mailcow-dockerized/data/web/.well-known/acme-challenge/2294426XXXXXXXX and on a webserver (from outside, with browser), I can see: http://185.58.xxx.xxx/.well-known/acme-challenge/2294426XXXXXXXX.

MatejKovacic · 2022-02-11T18:21:01Z

OK, this is weird, but I managed to solve the problems.

I opened nano mailcow.conf and changed:

SKIP_IP_CHECK=y 
SKIP_HTTP_VERIFICATION=y

Then I said:

docker-compose down
service docker restart
docker-compose up -d
docker-compose logs --tail=200 -f acme-mailcow

...and verification went through.

abbzer0 · 2022-02-14T13:40:08Z

So I created a GitHub account JUST so I could reply to you. I was dealing with the same issue, even did a bunch of DNS changes, whatever I did I COULD NOT GET IT TO WORK.... Failed in the exact same way you referenced. Followed the last 4 things you said and now I'm in business!!

This was the key piece:
docker-compose down
service docker restart
docker-compose up -d
docker-compose logs --tail=200 -f acme-mailcow

acme-mailcow_1 | Mon Feb 14 08:32:17 EST 2022 - Initializing, please wait...
acme-mailcow_1 | unable to load certificate
acme-mailcow_1 | 140256805415752:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
acme-mailcow_1 | unable to load certificate
acme-mailcow_1 | 140275453987656:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
acme-mailcow_1 | unable to load certificate
acme-mailcow_1 | 139977021107016:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
acme-mailcow_1 | unable to load certificate
acme-mailcow_1 | 139940199373640:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
acme-mailcow_1 | Mon Feb 14 08:32:17 EST 2022 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow_1 | Mon Feb 14 08:32:17 EST 2022 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow_1 | Mon Feb 14 08:32:17 EST 2022 - Detecting IP addresses...
acme-mailcow_1 | Mon Feb 14 08:32:44 EST 2022 - OK: X.X.X.X, 0000:0000:0000:0000:0000:0000:0000:0000
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Found A record for autodiscover.mydomain.com: X.X.X.X
acme-mailcow_1 | (skipping check, returning 0)
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Confirmed A record X.X.X.X
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Found A record for autoconfig.mydomain.com: X.X.X.X
acme-mailcow_1 | (skipping check, returning 0)
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Confirmed A record X.X.X.X
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Found A record for mail.mydomain.com: X.X.X.X
acme-mailcow_1 | (skipping check, returning 0)
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Confirmed A record X.X.X.X
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Certificate /var/lib/acme/mail.mydomain.com/cert.pem missing or changed domains 'mail.mydomain.com autoconfig.mydomain.com autodiscover.mydomain.com' - start obtaining
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Copying shared private key for this certificate...
acme-mailcow_1 | Mon Feb 14 08:32:46 EST 2022 - Checking resolver...
acme-mailcow_1 | Mon Feb 14 08:32:48 EST 2022 - Resolver OK
acme-mailcow_1 | Mon Feb 14 08:32:48 EST 2022 - Using command acme-tiny --account-key /var/lib/acme/acme/account.pem --disable-check --csr /var/lib/acme/mail.mydomain.com/acme.csr --acme-dir /var/www/acme/
acme-mailcow_1 | Parsing account key...
acme-mailcow_1 | Parsing CSR...
acme-mailcow_1 | Found domains: autoconfig.mydomain.com, autodiscover.mydomain.com, mail.mydomain.com
acme-mailcow_1 | Getting directory...
acme-mailcow_1 | Directory found!
acme-mailcow_1 | Registering account...
acme-mailcow_1 | Registered! Account ID: https://acme-v02.api.letsencrypt.org/acme/acct/XXXXXXXXXXXXXX
acme-mailcow_1 | Creating new order...
acme-mailcow_1 | Order created!
acme-mailcow_1 | Verifying autoconfig.mydomain.com...
acme-mailcow_1 | autoconfig.mydomain.com verified!
acme-mailcow_1 | Verifying autodiscover.mydomain.com...
acme-mailcow_1 | autodiscover.mydomain.com verified!
acme-mailcow_1 | Verifying mail.mydomain.com...
acme-mailcow_1 | mail.mydomain.com verified!
acme-mailcow_1 | Signing certificate...
acme-mailcow_1 | Certificate signed!
acme-mailcow_1 | Mon Feb 14 08:33:35 EST 2022 - Deploying certificate /var/lib/acme/mail.mydomain.com/cert.pem...
acme-mailcow_1 | Mon Feb 14 08:33:35 EST 2022 - Verified hashes.
acme-mailcow_1 | Mon Feb 14 08:33:35 EST 2022 - Certificate successfully obtained
acme-mailcow_1 | Mon Feb 14 08:33:35 EST 2022 - Reloading or restarting services... (1)
acme-mailcow_1 | Restarting XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
acme-mailcow_1 | command completed successfully
acme-mailcow_1 | Restarting XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
acme-mailcow_1 | command completed successfully
acme-mailcow_1 | Restarting XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
acme-mailcow_1 | command completed successfully
acme-mailcow_1 | Mon Feb 14 08:33:39 EST 2022 - Waiting for containers to settle...
acme-mailcow_1 | Mon Feb 14 08:33:50 EST 2022 - Certificates successfully requested and renewed where required, sleeping one day

MatejKovacic · 2022-02-14T14:15:34Z

Interesting. Anyway, this seems a bug for me.

Also, it would be nice if Certbot would be implemented, not just acme-tiny and other methods of authentication would be supported (namely DNS authentication).

03c · 2022-04-06T18:33:02Z

This worked for me as well - thanks!

djxistenz · 2022-05-02T07:31:01Z

I had the exact same issue, and after using using @MatejKovacic's workaround it seems to work. I don't know if this is a long-term fix to the issue however, and I'm also of the opinion that this is a bug.

lriley2020 · 2022-05-31T10:14:55Z

Tried workaround from @MatejKovacic but now I have this error:

acme-mailcow_1       | Verifying autoconfig.mydomain.com...
acme-mailcow_1       | Traceback (most recent call last):
acme-mailcow_1       |   File "/usr/bin/acme-tiny", line 8, in <module>
acme-mailcow_1       |     sys.exit(main())
acme-mailcow_1       |   File "/usr/lib/python3.9/site-packages/acme_tiny.py", line 195, in main
acme-mailcow_1       |     signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact, check_port=args.check_port)
acme-mailcow_1       |   File "/usr/lib/python3.9/site-packages/acme_tiny.py", line 153, in get_crt
acme-mailcow_1       |     raise ValueError("Challenge did not pass for {0}: {1}".format(domain, authorization))
acme-mailcow_1       | ValueError: Challenge did not pass for autoconfig.mydomain.com: {'identifier': {'type': 'dns', 'value': 'autoconfig.mydomain.com'}, 'status': 'invalid', 'expires': '2022-06-07T09:33:16Z', 'challenges': [{'type': 'http-01', 'status': 'invalid', 'error': {'type': 'urn:ietf:params:acme:error:unauthorized', 'detail': 'xx.xx.xx.xx: Invalid response from http://autoconfig.mydomain.com/.well-known/acme-challenge/ibwMbqb5j3-WsRgZl7-eyBjoxv_epibC-9rDqcCYJAo: 404', 'status': 403}, 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/114491942986/Pw5H-w', 'token': 'ibwMbqb5j3-WsRgZl7-eyBjoxv_epibC-9rDqcCYJAo', 'validationRecord': [{'url': 'http://autoconfig.mydomain.com/.well-known/acme-challenge/ibwMbqb5j3-WsRgZl7-eyBjoxv_epibC-9rDqcCYJAo', 'hostname': 'autoconfig.mydomain.com', 'port': '80', 'addressesResolved': ['xx.xx.xx.xx'], 'addressUsed': 'xx.xx.xx.xx'}], 'validated': '2022-05-31T10:07:11Z'}]}
acme-mailcow_1       | Tue May 31 11:07:12 BST 2022 - Failed to obtain certificate /var/lib/acme/mailserver.mydomain.com/cert.pem for domains 'mailserver.mydomain.com autoconfig.mydomain.com autodiscover.mydomain.com'
acme-mailcow_1       | OK
acme-mailcow_1       | Tue May 31 11:07:12 BST 2022 - Some errors occurred, retrying in 30 minutes...
acme-mailcow_1       | OK

fromage9747 · 2022-06-06T15:41:52Z

Also, I can see a file: /opt/mailcow-dockerized/data/web/.well-known/acme-challenge/2294426XXXXXXXX and on a webserver (from outside, with browser), I can see: http://185.58.xxx.xxx/.well-known/acme-challenge/2294426XXXXXXXX.

This fixed it for me too. Fresh install of Mailcow done today and the same issue.

abbzer0 · 2022-06-06T23:49:56Z

So I'm a TOTAL n00b on GitHub, and only originally signed up so I could LITERALLY comment on this thread.

However, is there an effective way to notify the developers of this issue going forward? I would do it myself, I'm just not sure how.

lriley2020 · 2022-06-07T00:22:57Z

In the end, my error actually turned out to be down to my reverse proxy (nginx proxy manager) trying to handle acme requests itself rather than passing them on

milkmaker · 2022-08-06T20:07:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

nzsambo · 2022-09-08T05:45:27Z

Same deal here, fresh install of mailcow.

I can confirm that just SKIP_HTTP_VERIFICATION=y and docker-compose restart acme-mailcow && docker-compose logs --tail=200 -f acme-mailcow worked for me.

BlueSkillz · 2022-10-05T13:01:00Z

Thank you it worked for me.

hopkins385 · 2022-11-25T16:24:11Z

same here ... what is HTTP_VERIFICATION all about?

SLongus · 2023-01-14T03:15:18Z

Same here, fresh installation on debian 11

JJProplay · 2023-01-15T07:58:11Z

The same on Ubuntu 20.04 LTS

boba1975 · 2023-01-17T11:20:22Z

The same here, fresh installation on Ubuntu 22.04 LTS

vakartel · 2023-03-02T14:35:11Z

This is not a bug, but misconfiguration with NAT reflection. Read docs:

_

You can also skip this validation method by setting SKIP_HTTP_VERIFICATION=y in "mailcow.conf". Be warned that this is discouraged. In most cases, the HTTP verification is skipped to workaround unknown NAT reflection issues, which are not resolved by ignoring this specific network misconfiguration. If you encounter problems generating TLSA records in the DNS overview within mailcow, you are most likely having issues with NAT reflection you should fix.

_

and-ri · 2023-03-30T18:02:20Z

OK, this is weird, but I managed to solve the problems.

I opened nano mailcow.conf and changed:
SKIP_IP_CHECK=y 
SKIP_HTTP_VERIFICATION=y
Then I said:
docker-compose down
service docker restart
docker-compose up -d
docker-compose logs --tail=200 -f acme-mailcow
...and verification went through.

Thanks a lot bro! I already thought I would never solve this problem. Moreover, everything is fine on the Hetzner server, but there was such a problem on Contabo

Houndie · 2023-07-24T13:47:46Z

Giving a maybe better solution to people here:

Looking at the documentation, they talk about this problem here: https://docs.mailcow.email/post_installation/firststeps-ssl/?h=ufw#validation-errors-and-how-to-skip-validation

Upon reading that, I discovered I had ufw running on my system. I added an allow rule: ufw allow 80 and restarted acme docker compose restart acme-mailcow and was able to get a cert no problem.

whyauthentic · 2024-02-17T06:26:00Z

For me the Fix "works" but after the "order" is placed i get a error .-.

Traceback (most recent call last): File "/usr/bin/acme-tiny", line 8, in <module> sys.exit(main()) ^^^^^^ File "/usr/lib/python3.11/site-packages/acme_tiny.py", line 195, in main signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact, check_port=args.check_port)

user90210 · 2024-08-19T10:37:52Z

This was the key piece: docker-compose down service docker restart docker-compose up -d docker-compose logs --tail=200 -f acme-mailcow

this worked for me

jabuxas · 2024-08-31T17:17:42Z

In the end, my error actually turned out to be down to my reverse proxy (nginx proxy manager) trying to handle acme requests itself rather than passing them on

what config did you change for that? i'm having the same problem where I can't access the /.well-known/acme-challenge with mail.domain.com, but i can with https://localip and i'm not really sure what's wrong

thematrixdev · 2024-09-12T08:24:23Z

still seeing this issue in September 2024

extremMiSt · 2024-10-03T21:38:42Z

just saw this now (october 2024) on a fresh install, workaround still worked.

MatejKovacic added the bug label Feb 10, 2022

milkmaker added the stale Please update the issue with current status, unclear if it's still open/needed. label Aug 6, 2022

milkmaker closed this as not planned Won't fix, can't repro, duplicate, stale Aug 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Confirmed A record with IP, but HTTP validation failed" error #4463

"Confirmed A record with IP, but HTTP validation failed" error #4463

MatejKovacic commented Feb 10, 2022

MatejKovacic commented Feb 10, 2022

MatejKovacic commented Feb 11, 2022

MatejKovacic commented Feb 11, 2022

MatejKovacic commented Feb 11, 2022

abbzer0 commented Feb 14, 2022

MatejKovacic commented Feb 14, 2022

03c commented Apr 6, 2022

djxistenz commented May 2, 2022

lriley2020 commented May 31, 2022

fromage9747 commented Jun 6, 2022

abbzer0 commented Jun 6, 2022

lriley2020 commented Jun 7, 2022

milkmaker commented Aug 6, 2022

nzsambo commented Sep 8, 2022

BlueSkillz commented Oct 5, 2022

hopkins385 commented Nov 25, 2022

SLongus commented Jan 14, 2023

JJProplay commented Jan 15, 2023

boba1975 commented Jan 17, 2023

vakartel commented Mar 2, 2023

and-ri commented Mar 30, 2023

Houndie commented Jul 24, 2023

whyauthentic commented Feb 17, 2024

user90210 commented Aug 19, 2024

jabuxas commented Aug 31, 2024

thematrixdev commented Sep 12, 2024

extremMiSt commented Oct 3, 2024

"Confirmed A record with IP, but HTTP validation failed" error #4463

"Confirmed A record with IP, but HTTP validation failed" error #4463

Comments

MatejKovacic commented Feb 10, 2022

Summary

Logs

Reproduction

System information

MatejKovacic commented Feb 10, 2022

MatejKovacic commented Feb 11, 2022

MatejKovacic commented Feb 11, 2022

MatejKovacic commented Feb 11, 2022

abbzer0 commented Feb 14, 2022

MatejKovacic commented Feb 14, 2022

03c commented Apr 6, 2022

djxistenz commented May 2, 2022

lriley2020 commented May 31, 2022

fromage9747 commented Jun 6, 2022

abbzer0 commented Jun 6, 2022

lriley2020 commented Jun 7, 2022

milkmaker commented Aug 6, 2022

nzsambo commented Sep 8, 2022

BlueSkillz commented Oct 5, 2022

hopkins385 commented Nov 25, 2022

SLongus commented Jan 14, 2023

JJProplay commented Jan 15, 2023

boba1975 commented Jan 17, 2023

vakartel commented Mar 2, 2023

and-ri commented Mar 30, 2023

Houndie commented Jul 24, 2023

whyauthentic commented Feb 17, 2024

user90210 commented Aug 19, 2024

jabuxas commented Aug 31, 2024

thematrixdev commented Sep 12, 2024

extremMiSt commented Oct 3, 2024