Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNS issue #3735

Closed
3 tasks done
guyguy333 opened this issue Sep 4, 2020 · 33 comments
Closed
3 tasks done

DNS issue #3735

guyguy333 opened this issue Sep 4, 2020 · 33 comments
Labels
stale Please update the issue with current status, unclear if it's still open/needed.

Comments

@guyguy333
Copy link

Prior to placing the issue, please check following: (fill out each checkbox with an X once done)

  • I understand, that not following or deleting the below instructions, will result in immediate closing and deletion of my issue.
  • I have understood that answers are voluntary and community-driven, and not commercial support.
  • I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description of the bug:

After a mailcow update, containers have many DNS issues. If I understand right, unbound is the DNS server/forwarder for others containers. However, unbound itself have DNS issues as we can see at the beginning of the log : curl: (6) Could not resolve host: www.internic.net.

Running a docker container not related to mailcow has no DNS issues. I did try to disable all firewalls but it didn't fix the issue.

Docker container logs of affected containers:

Setting console permissions...
Receiving anchor key...
Receiving root hints...
curl: (6) Could not resolve host: www.internic.net                            

setup in directory /etc/unbound
generating unbound_server.key
Generating RSA private key, 3072 bit long modulus (2 primes)
......++++
.....++++
e is 65537 (0x010001)
generating unbound_control.key
Generating RSA private key, 3072 bit long modulus (2 primes)
..........................................................................................................................................++++
......................................++++
e is 65537 (0x010001)
create unbound_server.pem (self signed certificate)
create unbound_control.pem (signed client certificate)
Signature ok
subject=CN = unbound-control
Getting CA Private Key
Setup success. Certificates created. Enable in unbound.conf file to use
[1599208286] unbound[1:0] notice: init module 0: validator
[1599208286] unbound[1:0] notice: init module 1: iterator
[1599208286] unbound[1:0] info: start of service (unbound 1.9.6).
[1599208286] unbound[1:0] info: generate keytag query _ta-4f66. NULL IN
[1599208474] unbound[1:0] info: service stopped (unbound 1.9.6).
[1599208474] unbound[1:0] info: server stats for thread 0: 165 queries, 78 answers from cache, 87 recursions, 0 prefetch, 0 rejected by ip ratelimiting
[1599208474] unbound[1:0] info: server stats for thread 0: requestlist max 83 avg 20.6552 exceeded 0 jostled 0
[1599208474] unbound[1:0] info: average recursion processing time 6.085979 sec
[1599208474] unbound[1:0] info: histogram of recursion processing times
[1599208474] unbound[1:0] info: [25%]=0.0448779 median[50%]=0.090112 [75%]=2.71429
[1599208474] unbound[1:0] info: lower(secs) upper(secs) recursions
[1599208474] unbound[1:0] info:    0.000000    0.000001 4
[1599208474] unbound[1:0] info:    0.001024    0.002048 1
[1599208474] unbound[1:0] info:    0.008192    0.016384 2
[1599208474] unbound[1:0] info:    0.016384    0.032768 2
[1599208474] unbound[1:0] info:    0.032768    0.065536 23
[1599208474] unbound[1:0] info:    0.065536    0.131072 8
[1599208474] unbound[1:0] info:    0.131072    0.262144 1
[1599208474] unbound[1:0] info:    0.524288    1.000000 5
[1599208474] unbound[1:0] info:    1.000000    2.000000 4
[1599208474] unbound[1:0] info:    2.000000    4.000000 7
[1599208474] unbound[1:0] info:    4.000000    8.000000 3
[1599208474] unbound[1:0] info:   16.000000   32.000000 4
[1599208474] unbound[1:0] info:   32.000000   64.000000 6
Setting console permissions...
Receiving anchor key...
Receiving root hints...
######################################################################## 100.0%
setup in directory /etc/unbound
unbound_server.key exists
unbound_control.key exists
create unbound_server.pem (self signed certificate)
create unbound_control.pem (signed client certificate)
Signature ok
subject=CN = unbound-control
Getting CA Private Key
Setup success. Certificates created. Enable in unbound.conf file to use
[1599208519] unbound[1:0] notice: init module 0: validator
[1599208519] unbound[1:0] notice: init module 1: iterator
[1599208519] unbound[1:0] info: start of service (unbound 1.9.6).
[1599208519] unbound[1:0] info: generate keytag query _ta-4f66. NULL IN

Also in postfix:


warning: dnsblog_query: lookup error for DNS query 4.149.70.212.wl.mailspike.net: Host or domain name not found. Name service error for name=4.149.70.212.wl.mailspike.net type=A: Host not found, try again
--
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.hostkarma.junkemailfilter.com: Host or domain name not found. Name service error for name=4.149.70.212.hostkarma.junkemailfilter.com type=A: Host not found, try again
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.dnsbl.sorbs.net: Host or domain name not found. Name service error for name=4.149.70.212.dnsbl.sorbs.net type=A: Host not found, try again
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.bl.ipv6.spameatingmonkey.net: Host or domain name not found. Name service error for name=4.149.70.212.bl.ipv6.spameatingmonkey.net type=A: Host not found, try again
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.b.barracudacentral.org: Host or domain name not found. Name service error for name=4.149.70.212.b.barracudacentral.org type=A: Host not found, try again
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.bl.spamcop.net: Host or domain name not found. Name service error for name=4.149.70.212.bl.spamcop.net type=A: Host not found, try again
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.backscatter.spameatingmonkey.net: Host or domain name not found. Name service error for name=4.149.70.212.backscatter.spameatingmonkey.net type=A: Host not found, try again
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.ix.dnsbl.manitu.net: Host or domain name not found. Name service error for name=4.149.70.212.ix.dnsbl.manitu.net type=A: Host not found, try again
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.bl.mailspike.net: Host or domain name not found. Name service error for name=4.149.70.212.bl.mailspike.net type=A: Host not found, try again
9/4/2020, 12:25:26 PM | warning | warning: dnsblog_query: lookup error for DNS query 4.149.70.212.bl.spameatingmonkey.net: Host or domain name not found. Name service error for name=4.149.70.212.bl.spameatingmonkey.net type=A: Host not found, try again


Reproduction of said bug:

Nothing special, just an up to date mailcow.

System information:

Question Answer
My operating system Debian 10
Is Apparmor, SELinux or similar active? No
Virtualization technlogy (KVM, VMware, Xen, etc - LXC and OpenVZ are not supported KVM
Server/VM specifications (Memory, CPU Cores) 4CPUs, 8Go RAM
Docker Version (docker version) 19.03.12
Docker-Compose Version (docker-compose version) 1.26.2
Reverse proxy (custom solution) No
  • Output of git diff origin/master, any other changes to the code? If so, please post them.
diff --git a/data/conf/rspamd/local.d/actions.conf b/data/conf/rspamd/local.d/actions.conf
index 3de63a54..fa43014f 100644
--- a/data/conf/rspamd/local.d/actions.conf
+++ b/data/conf/rspamd/local.d/actions.conf
@@ -1,3 +1,3 @@
-reject = 15;
+reject = 50;
 add_header = 8;
 greylist = 7;
diff --git a/data/conf/sogo/sogo.conf b/data/conf/sogo/sogo.conf
index 9d04f0b5..7a450d70 100644
--- a/data/conf/sogo/sogo.conf
+++ b/data/conf/sogo/sogo.conf
@@ -59,7 +59,7 @@
     SOGoFirstDayOfWeek = "1";
 
     SOGoSieveFolderEncoding = "UTF-8";
-    SOGoPasswordChangeEnabled = YES;
+    SOGoPasswordChangeEnabled = NO;
     SOGoSentFolderName = "Sent";
     SOGoMailShowSubscribedFoldersOnly = NO;
     NGImap4ConnectionStringSeparator = "/";
  • All third-party firewalls and custom iptables rules are unsupported. Please check the Docker docs about how to use Docker with your own ruleset. Nevertheless, iptabels output can help us to help you: iptables -L -vn, ip6tables -L -vn, iptables -L -vn -t nat and ip6tables -L -vn -t nat.
  • DNS problems? Please run docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254 (set the IP accordingly, if you changed the internal mailcow network) and post the output.

docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254
151.101.1.69
151.101.129.69
151.101.193.69
151.101.65.69

This docker has no issues. A ping to stackoverflow.com in unbound container doesn't work.

@andryyy
Copy link
Contributor

andryyy commented Sep 4, 2020

Does your other Docker container also use its own recursor? ;) You cannot compare that.

docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254
151.101.1.69
151.101.129.69
151.101.193.69
151.101.65.69

This looks good.

@guyguy333
Copy link
Author

@172.22.1.254

No they don't. Here are others results:

 docker run --network=mailcowdockerized_mailcow-network -it tutum/dnsutils dig +short stackoverflow.com
;; connection timed out; no servers could be reached
docker run -it tutum/dnsutils dig +short stackoverflow.com
151.101.1.69
151.101.65.69
151.101.129.69
151.101.193.69
docker run --network=mailcowdockerized_mailcow-network -it tutum/dnsutils dig +short stackoverflow.com @172.22.1.254
151.101.129.69
151.101.1.69
151.101.65.69
151.101.193.69

@guyguy333
Copy link
Author

Here is iptable output:

root@mail:/opt/mailcow-dockerized# iptables -L -vn
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
10572 4484K f2b-recidive  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 153K   86M DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 153K   86M DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    2   276 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    2   148 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
 106K   81M ACCEPT     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
11630  805K DOCKER     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0           
34602 3939K ACCEPT     all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0           
11107  781K ACCEPT     all  --  br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain f2b-recidive (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    6   304 REJECT     all  --  *      *       195.54.160.183       0.0.0.0/0            reject-with icmp-port-unreachable
10566 4484K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  265 10724 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:443
  101  4248 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:80
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.9           tcp dpt:3306
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.249         tcp dpt:6379
    3   160 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.10          tcp dpt:587
   35  2084 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.10          tcp dpt:465
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.11          tcp dpt:8983
   78  4464 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.10          tcp dpt:25
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:12345
    1    60 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:4190
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:995
   11   672 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:993
   29  1732 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:143
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:110

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   148 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
34602 3939K DOCKER-ISOLATION-STAGE-2  all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0           
 153K   86M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0           
34604 3940K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 153K   86M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
# Warning: iptables-legacy tables present, use iptables-legacy to see them
root@mail:/opt/mailcow-dockerized# ip6tables -L -vn
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 119K  118M DOCKER-USER  all      *      *       ::/0                 ::/0                
 119K  118M DOCKER-ISOLATION-STAGE-1  all      *      *       ::/0                 ::/0                
79281  102M DOCKER     all      *      br-mailcow  ::/0                 ::/0                
42066   94M ACCEPT     all      *      br-mailcow  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
38796   15M ACCEPT     all      br-mailcow !br-mailcow  ::/0                 ::/0                
 8319  568K ACCEPT     all      br-mailcow br-mailcow  ::/0                 ::/0                

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 119K  118M RETURN     all      *      *       ::/0                 ::/0                

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  177  473K ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:25
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:465
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:587
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::11  tcp dpt:110
 7093 1570K ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::11  tcp dpt:143
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::11  tcp dpt:4190
20662 5960K ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::11  tcp dpt:993
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::11  tcp dpt:995
  949 82100 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::5  tcp dpt:443
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::5  tcp dpt:80

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
38796   15M DOCKER-ISOLATION-STAGE-2  all      br-mailcow !br-mailcow  ::/0                 ::/0                
 119K  118M RETURN     all      *      *       ::/0                 ::/0                

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all      *      br-mailcow  ::/0                 ::/0                
38796   15M RETURN     all      *      *       ::/0                 ::/0                
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
root@mail:/opt/mailcow-dockerized# iptables -L -vn -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  582 27616 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    2   148 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
   75  5222 MASQUERADE  all  --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0           
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.5           172.22.1.5           tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.5           172.22.1.5           tcp dpt:80
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.9           172.22.1.9           tcp dpt:3306
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.10          172.22.1.10          tcp dpt:587
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.10          172.22.1.10          tcp dpt:465
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.11          172.22.1.11          tcp dpt:8983
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.10          172.22.1.10          tcp dpt:25
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  br-mailcow *       0.0.0.0/0            0.0.0.0/0           
  265 10724 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.22.1.5:443
  102  4288 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.22.1.5:80
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.9:3306
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:7654 to:172.22.1.249:6379
    3   160 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.10:587
   84  5024 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.10:465
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:18983 to:172.22.1.11:8983
   79  4524 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.10:25
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.250:12345
    1    60 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4190 to:172.22.1.250:4190
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 to:172.22.1.250:995
   11   672 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.22.1.250:993
   29  1732 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.22.1.250:143
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110 to:172.22.1.250:110
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  224 18822 DOCKER     all      *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all      *      br-mailcow  ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL
   40  3646 MASQUERADE  all      *      !br-mailcow  fd4d:6169:6c63:6f77::/64  ::/0                
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:25
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:465
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:587
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::11  fd4d:6169:6c63:6f77::11  tcp dpt:110
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::11  fd4d:6169:6c63:6f77::11  tcp dpt:143
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::11  fd4d:6169:6c63:6f77::11  tcp dpt:4190
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::11  fd4d:6169:6c63:6f77::11  tcp dpt:993
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::11  fd4d:6169:6c63:6f77::11  tcp dpt:995
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::5  fd4d:6169:6c63:6f77::5  tcp dpt:443
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::5  fd4d:6169:6c63:6f77::5  tcp dpt:80

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all      *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    3   240 RETURN     all      br-mailcow *       ::/0                 ::/0                
    4   320 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::d]:25
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::d]:465
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::d]:587
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:110 to:[fd4d:6169:6c63:6f77::11]:110
   79  6662 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:143 to:[fd4d:6169:6c63:6f77::11]:143
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:4190 to:[fd4d:6169:6c63:6f77::11]:4190
  131 10988 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:993 to:[fd4d:6169:6c63:6f77::11]:993
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:995 to:[fd4d:6169:6c63:6f77::11]:995
    4   336 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:443 to:[fd4d:6169:6c63:6f77::5]:443
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:80 to:[fd4d:6169:6c63:6f77::5]:80
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them

@guyguy333
Copy link
Author

root@mail:/opt/mailcow-dockerized# iptables-legacy -L -vn
Chain INPUT (policy ACCEPT 305 packets, 130K bytes)
 pkts bytes target     prot opt in     out     source               destination         
12172 5062K MAILCOW    all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 3831 packets, 2008K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 173K   99M MAILCOW    all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 423 packets, 29531 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain MAILCOW (2 references)
 pkts bytes target     prot opt in     out     source               destination 
root@mail:/opt/mailcow-dockerized# ip6tables-legacy -L -vn
Chain INPUT (policy ACCEPT 5139 packets, 426K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 5139  426K MAILCOW    all      *      *       ::/0                 ::/0                

Chain FORWARD (policy ACCEPT 128K packets, 129M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 128K  129M MAILCOW    all      *      *       ::/0                 ::/0                

Chain OUTPUT (policy ACCEPT 3777 packets, 478K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain MAILCOW (2 references)
 pkts bytes target     prot opt in     out     source               destination 

@guyguy333
Copy link
Author

guyguy333 commented Sep 4, 2020

@andryyy I found the issue and it is related to netfilter container.

If I change IPV4_NETWORK in conf to something "unused" before and that I comment netfilter service, everything works fine. As long as I enable netfilter service, I've packets issue, egress is "broken" on mailcow network (including DNS, ...). If I comment netfilter service again without changing IPV4_NETWORK, it's then broken. It's like net filter is "blocking" traffic. I need to change IPV4_NETWORK again and disable netflix to get something working.

How can I help you to find the issue in netfilter container ?

@andryyy
Copy link
Contributor

andryyy commented Sep 5, 2020

Hm? Which issue did you find exactly?

If there was a general issue with netfilter, shouldn't everyone have that problem? :)

Maybe there is a broken rule in your fail2ban blacklist or something like that? But your iptables output seemed fine.

Perhaps someone can chime in and help you debug the network. It can be a bug in a constellation that happens on your system, try to compare tcpdumps etc.

I don't know what you changed IPV4_NETWORK to, but perhaps the default network conflicts with anything in your internal network.

I don't know how I can help you debug this, sorry.

@g0rbe
Copy link

g0rbe commented Sep 18, 2020

My logs are full of DNS errors too.

Like:

warning: dnsblog_query: lookup error for DNS query [IP].zen.spamhaus.org: Host or domain name not found. Name service error for name=[IP].zen.spamhaus.org type=A: Host not found, try again

and

(Host or domain name not found. Name service error for name=[DOMAIN] type=MX: Host not found, try again)

Restarting the whole server seems to solved the DNS issue.

@nouhouari
Copy link

nouhouari commented Oct 30, 2020

I'm running the latest version and I'm also facing issues with postfix. I can't send and receive emails.

Name service error for name=gmail.com type=MX: Host not found, try again
And
<[email protected]>: Sender address rejected: Domain not found

If I try to see the DNS record inside the postfix Docker container, I get this:

root@mail:/# dig mx gmail.com

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> mx gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 29948
;; flags: qr rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Fri Oct 30 01:32:56 GMT 2020
;; MSG SIZE  rcvd: 12

Same with nslookup command

root@mail:/#nslookup  gmail.com
Server:		127.0.0.11
Address:	127.0.0.11#53

Using a different DNS server (8.8.8.8) inside the container is working.

dig @8.8.8.8 gmail.com MX

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> @8.8.8.8 gmail.com MX
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31171
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;gmail.com.			IN	MX

;; ANSWER SECTION:
gmail.com.		3021	IN	MX	30 alt3.gmail-smtp-in.l.google.com.
gmail.com.		3021	IN	MX	20 alt2.gmail-smtp-in.l.google.com.
gmail.com.		3021	IN	MX	40 alt4.gmail-smtp-in.l.google.com.
gmail.com.		3021	IN	MX	5 gmail-smtp-in.l.google.com.
gmail.com.		3021	IN	MX	10 alt1.gmail-smtp-in.l.google.com.

;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Oct 30 01:38:59 GMT 2020
;; MSG SIZE  rcvd: 161

** server can't find gmail.com: REFUSED

I think this is the root cause of the sending/receiving issue but didn't figure out how to solve it yet.
Any help will be welcome.

@nouhouari
Copy link

I manage to make it work with a workaround by stopping the unbound service.

docker-compose stop unbound-mailcow
Stopping mailcowdockerized_unbound-mailcow_1 ... done

I suspect a conflict with the host DNS server running on port 53 (TCP/UDP) but not sure at all.

@guyguy333
Copy link
Author

I manage to make it work with a workaround by stopping the unbound service.

docker-compose stop unbound-mailcow
Stopping mailcowdockerized_unbound-mailcow_1 ... done

I suspect a conflict with the host DNS server running on port 53 (TCP/UDP) but not sure at all.

On my side, I required to disable netfilter-mailcow

@nouhouari
Copy link

nouhouari commented Nov 5, 2020

Hi @guyguy333,
how did you disable netfilter ? Did you comment it in the docker-compose file? or something else ?
Thanks

@guyguy333
Copy link
Author

Hi @guyguy333,
how did you disable netfilter ? Did you comment it in the docker-compose file? or something else ?
Thanks

I commented netfilter part in docker-compose.yml

@andryyy
Copy link
Contributor

andryyy commented Nov 5, 2020

You should not disable netfilter. Don't. That's not a valid fix, is unsupported and will break things. Thanks.

@shiz0
Copy link
Member

shiz0 commented Nov 5, 2020

You should not disable netfilter. Don't. That's not a valid fix, is unsupported and will break things. Thanks.

I second this, PLEASE DON'T DO THAT

@g0rbe
Copy link

g0rbe commented Nov 5, 2020

I used dig to query from the docker container, without success.

From the unbound doc:

Notice that the forward-zone is commented out. You can use the forward-zone directive if you do _NOT_ wish to query the root DNS server and you want to use other resolving DNS servers. For example, we have OpenDNS.org and Google Public DNS configured here. You can replace those ips with the DNS servers of your ISP if you wanted to.
#forward-zone:
#      name: "."
#      forward-addr: 8.8.8.8        # Google Public DNS
#      forward-addr: 216.87.84.211  # OpenNIC Colorado, US
#      forward-addr: 66.244.95.20   # OpenNIC Indiana, US
#      forward-addr: 72.14.189.120  # OpenNIC Texas, US
#      forward-addr: 4.2.2.4        # Level3 Verizon

I added the forward-zone to ./data/conf/unbound/unbound.conf and rebuilt with docker-compose.

I verified that the forward-zone is exist in the container. (OK)

I tried dig again: SUCCESS!

@andryyy you should add some upstream DNS to the config file, like 1.1.1.1 / 1.0.0.1.

I created a PR #3841

@nouhouari
Copy link

Commenting netfilter doesn't solve anything. I'm still not able to send/receive email.

postfix-mailcow_1    | Nov  5 13:33:48 mail postfix/smtp[959]: E63C76FA14AD: to=<[email protected]>, relay=none, delay=18, delays=8.3/0.02/10/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=gmail.com type=MX: Host not found, try again)

NOQUEUE: reject: RCPT from unknown[209.85.210.45]: 450 4.1.8 <[email protected]>: Sender address rejected: Domain not found; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mail-ot1-f45.google.com 

@andryyy
Copy link
Contributor

andryyy commented Nov 5, 2020

DONT do that. DO NOT use a public forwarder. Read the docs. That's a no-go...

@g0rbe
Copy link

g0rbe commented Nov 5, 2020

Can't send an email to the accountant, which is not so good...
Error: MX: Host not found

Can you suggest something to resolve this issue?

Edit:

From Andre:

If your network is not able to talk to the root DNS servers, setup a DNS recursor whereever you like and use that server as forwarder. But DON'T use a public DNS as forwarder in that "recursor" (it would not be a recursor then anymore anyway).

I set my own recursive DNS in my unbound.conf, 1.1.1.1 was just an example.
It is OK?

@nouhouari
Copy link

nouhouari commented Nov 5, 2020

I've added the following in unbound.conf

forward-zone:
      name: "."
      forward-addr: 8.8.8.8 

But dig command still show

dig mx gmail.com

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> mx gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 2805
;; flags: qr rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Thu Nov 05 14:07:20 GMT 2020
;; MSG SIZE  rcvd: 12

I've revert everything back. Still not working

@g0rbe
Copy link

g0rbe commented Nov 5, 2020

@nouhouari My comment is NOT the solution, so DONT use it.

Other: dig is not exist in the unbound container and the OS is not Debian but Ubuntu (at least, for me) so i assume, you used dig on the server which is use the servers DNS setting in /etc/resolv.conf. You should specify the containers IP in dig:

dig @172.22.1.254 gmail.com mx

The doc clearly write down how DONT set unbound.

@nouhouari
Copy link

@nouhouari My comment is NOT the solution, so DONT use it.

Other: dig is not exist in the unbound container and the OS is not Debian but Ubuntu (at least, for me) so i assume, you used dig on the server which is use the servers DNS setting in /etc/resolv.conf. You should specify the containers IP in dig:

dig @172.22.1.254 gmail.com mx

The doc clearly write down how DONT set unbound.

@g0rbe I'm using dig in postfix container, not in unbound container or the host.

@g0rbe
Copy link

g0rbe commented Nov 5, 2020

@nouhouari Do you uncomment the netfilter in docker-compose.yml?

@nouhouari
Copy link

nouhouari commented Nov 5, 2020

Yes, I'm aligned with master branch. No change.
So now, I came back to workaround by stopping the unbound service.

@vnukhr
Copy link

vnukhr commented Nov 20, 2020

Having the same issue as @guyguy333 on Ubuntu 20.04 (unbound not being able to resolve anything) I'd like to record my cause for posterity.

Running mailcow on a Hetzner dedicated server with firewall enabled in Hetzner Robot breaks unbound DNS resolving completely, while resolving from host works fine.

My guess is they have some source ip filtering in place that drops DNS packets from unbound container.

@andryyy
Copy link
Contributor

andryyy commented Nov 20, 2020

That's a very good hint, thank you for that. Should probably be posted in the docs.

@nouhouari
Copy link

Having the same issue as @guyguy333 on Ubuntu 20.04 (unbound not being able to resolve anything) I'd like to record my cause for posterity.

Running mailcow on a Hetzner dedicated server with firewall enabled in Hetzner Robot breaks unbound DNS resolving completely, while resolving from host works fine.

My guess is they have some source ip filtering in place that drops DNS packets from unbound container.

What do you recommend then? Still blocked with the same error and upgrade everything today.

@vnukhr
Copy link

vnukhr commented Dec 24, 2020

@nouhouari check with your hosting provider (or network equipment that you control) that they don't do any source (ip and/or mac) filtering for outgoing UDP traffic.

This was my case, but it's entirely possible that something is borked on your OS (ie firewall) that prevents unbound from resolving.

@nouhouari
Copy link

@nouhouari check with your hosting provider (or network equipment that you control) that they don't do any source (ip and/or mac) filtering for outgoing UDP traffic.

This was my case, but it's entirely possible that something is borked on your OS (ie firewall) that prevents unbound from resolving.

If it were a UDP outbound traffic issue, setting a DNS to 8.8.8.8 (/etc/resolv.conf) in postfix container won't work. What do you think?

@andryyy
Copy link
Contributor

andryyy commented Jan 4, 2021

No. That's not the same. Setting the DNS to 8.8.8.8 is completely different from using 127.0.0.11.
127.0.0.11 is the DNS proxy of Docker which includes NAT. Furthermore: Do not change the resolver in a container. If you want to test something, dig against the resolver you want to check.

Broken NAT can also lead to unmasqueraded packages, which are eventually dropped by the next gw. Just one of many things that can go wrong.

@nouhouari
Copy link

Hi @andryyy , I've changed the dns inside the container only to validate that it can not be an outgoing UDP traffic issue.
Is 127.0.0.11 a local DNS? And should I check with dig why this DNS is not working? Is it what you are suggesting?
Could you please share how I can check this DNS? Thank you

@github-actions
Copy link

github-actions bot commented Jun 2, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added stale Please update the issue with current status, unclear if it's still open/needed. and removed stale Please update the issue with current status, unclear if it's still open/needed. labels Jun 2, 2021
@github-actions
Copy link

github-actions bot commented Aug 6, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale Please update the issue with current status, unclear if it's still open/needed. label Aug 6, 2021
@jiapei100
Copy link

@guyguy333 @nouhouari
Did you guys come to a conclusion?
I met the same issue today: https://mail.longervision.com . Can you please give me a hand?? Thank you ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale Please update the issue with current status, unclear if it's still open/needed.
Projects
None yet
Development

No branches or pull requests

7 participants