How to use a private DNS with k3s

[EugenMayer]

Trying to configure k3s to use my private dns server 10.10.0.243 as the forward/upstream default to resolve. I fail to do so, my pods cannot resolve the private domains, only public ones.

Using ubuntu 22.04 with systemd based DNS.

The only way i could get it running is defining a coredns-custom custom-map like this

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  custom.server: |
     myprivatedomain.tld:53 {
      forward . 10.10.0.243
    }

Is there a better way to deal with this issue?

[brandond]

Since k3s does not use the OS DNS server to forward DNS requests

It does, unless the host's resolv.conf includes an invalid upstream, in which case it uses 8.8.8.8. If you want to point k3s at an alternative upstream, the easiest way is to create a custom resolv.conf, and point k3s at it with the --resolv-conf flag.

[EugenMayer]

@brandond you have answered this way several times, but there is more to it:

• we have a valid upstream - i mean it is 127.0.0.53 since it is ubuntu / systemd based DNS, but the configuration is perfectly valid. Still k3s picks 8.8.8.8 (but the way, this does work in rke2). I assume k3s determines that 127.0.0.53 is problematic form within the coredns container and skips it. It should instead just use the node ip - i mean this is normal for all new systemd / network-manager setups and will not go away.
• using --resolv-conf will only help until the restart tight, AFAICU the systemd unit will not include that param and one has to add the param to the unit file on every upgrade right?

Right now, the only way i could "fix" it is using

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom # this is the name of the configmap you can overwrite with your changes
  namespace: kube-system
data:
    custom.server: | 
    prviatedomain.tld:53 {
       forward . <provate DNS IP>
    }

which i would need to do for every private-domain i need to follow.

Happy to understand your POV better

[brandond]

we have a valid upstream - i mean it is 127.0.0.53 since it is ubuntu / systemd based DNS

That is not a valid upstream; 127.x.x.x in a pod reaches that pod, not the host.

It should instead just use the node ip

Its not any safer of an assumption to expect the node IP to work as a DNS server, in the case of systemd-resolved which only listens on that loopback address.

using --resolv-conf will only help until the restart tight, AFAICU the systemd unit will not include that param and one has to add the param to the unit file on every upgrade right?

Only if you are upgrading by re-running the install script, and don't have some way to ensure that you consistently pass the same arguments every time. You could use the config file, which isn't managed by the install script.

Right now, the only way i could "fix" it is using

Or you could just create a custom resolv.conf that points at the real upstream that is hosting your internal domains, which will presumably also act as a recursive resolver for public zones.

[EugenMayer]

Thank you for elaborating and thank you for helping!

Considering that ubuntu LTS is one of the official k8s distros and k3s not supporting the default cloud-init based installation is something that i really consider is a issue. I understood why you cannot use the default resolv.conf it and i also understand that most probably less people use a private DNS (thus e.g. like throwing an error on start is not desired).

To reduce the amount of issues for that questions (there are a couple already), tell people why the syst. Not just a generic "if it is invalid" - this would help reducing the amount of issues opened in k3s (probably)

i really would wish there would be a more elegant way to work with that issue. The only one i can think of technically is either

• loopback device alias like 172.31.31.254 to the loopback and address that as the DNS resolver
• habe a k3s config way of setting the resolv.conf or just the forward dns (probably already possible?)

[brandond]

k3s not supporting the default cloud-init based installation is something that i really consider is a issue.

I'm not sure what any of this has to do with cloud-init?

when systemd-resolved is in use, there is normally an unaltered resolv.conf at /run/systemd/resolve/resolv.conf that points to the actual upstream resolver. K3s will check both /etc/resolv.conf, and /run/systemd/resolve/resolv.conf, and use whichever one contains a viable upstream resolver. Only when neither file is viable does k3s use 8.8.8.8.

habe a k3s config way of setting the resolv.conf

There is such a way, I mentioned it above: use the --resolv-conf flag.

[dragonfleas]

when systemd-resolved is in use, there is normally an unaltered resolv.conf at /run/systemd/resolve/resolv.conf that points to the actual upstream resolver. K3s will check both /etc/resolv.conf, and /run/systemd/resolve/resolv.conf, and use whichever one contains a viable upstream resolver. Only when neither file is viable does k3s use 8.8.8.8.

I know this is an old thread but could this be added to the docs anywhere? Maybe inside of https://docs.k3s.io/networking/basic-network-options? It would be really useful because it took me ages to dig for what K3S looks for in particular. This is a hard requirement that is seemingly undocumented, besides in obscure github PRs and discussions.

Also where in the CoreFile does it specify that it searches /run/systemd/resolve/resolv.conf? I am really confused on where that is coming from. If there's explicitly only /etc/resolv.conf in the CoreFile, is CoreDNS the service taking care of the fallback systemd run conf?

[dragonfleas]

To expand on this, this is taken from the loop plugin readme for CoreDNS

There are many ways to work around this issue, some are listed here:

Add the following to your kubelet config yaml: resolvConf: (or via command line flag --resolv-conf deprecated in 1.10). Your "real" resolv.conf is the one that contains the actual IPs of your upstream servers, and no local/loopback address. This flag tells kubelet to pass an alternate resolv.conf to Pods. For systems using systemd-resolved, /run/systemd/resolve/resolv.conf is typically the location of the "real" resolv.conf, although this can be different depending on your distribution.

Disable the local DNS cache on host nodes, and restore /etc/resolv.conf to the original.
A quick and dirty fix is to edit your Corefile, replacing forward . /etc/resolv.conf with the IP address of your upstream DNS, for example forward . 8.8.8.8. But this only fixes the issue for CoreDNS, kubelet will continue to forward the invalid resolv.conf to all default dnsPolicy Pods, leaving them unable to resolve DNS.

It doesn't state that it will check /run/systemd/resolve/resolv.conf so does k3s provide the abstraction for this somewhere in the kubelet definition?

[EugenMayer]

This is interesting, this is my /run/systemd/resolve/resolv.conf

nameserver 10.10.0.243
nameserver 1.1.1.1
search .

It is never used by k3s. Is the search . the actual issue then?

k3s not supporting the default cloud-init based installation is something that i really consider is a issue.

Ubuntu 20.04/22.04 based setups which are e.g. booted using cloud-init (kind of common for setting up k3s/rke2) come with the describe systemd based dns setup. The statement you made about /run/systemd/resolve/resolv.conf made this irrelevant, since this would support those setups too.

[EugenMayer]

Just checked, in fact the CoreDNS pod uses dnsPolicy: Default.

The pod that is problematic uses dnsPolicy: ClusterFirst - looking at https://github.com/stevehipwell/helm-charts/blob/main/charts/nexus3/templates/deployment.yaml this will be the default since it is not set here. Checking the deployment on the cluster and it is as expected dnsPolicy: ClusterFirst

The pod will not be able to resolve private DNS entries unitl i deploy a coredns-custom with

myprivatdomain.tld:53 {
  forward . 10.10.0.243
}

This means:

• coreDNS has network access to the expted upstream 10.10.0.243 and would be able to forward, if asked
• are you expecting the pods to be created with dnsPolicy: Default or is this just right for the CoreDNS pod? If it is the former, i assume that can be defined as a default somewhere?

[brandond]

Generally you want pods to use ClusterFirst, which means they'll use coredns. Coredns should in turn forward everything to the upstreams specified in /run/systemd/resolve/resolv.conf, as I mentioned above. coredns randomly balances queries across all listed upstreams - which means that half of queries from your pods will be forwarded to 10.10.0.243 and will resolve your private domain properly, while half will be sent to 1.1.1.1 and will not. Have you tried just taking the public upstream out of /run/systemd/resolve/resolv.conf?

[EugenMayer]

Balancing?!? Since when that became a standard for nameservers? https://linux.die.net/man/5/resolv.conf does say as expected, they are tried in order and the first one that resolves, is used. Strictly sequential.

So when did CoreDNS decide to skip that and just load balance it? I mean in my case it is just "when the internal is down, try at least the public one" - it is a fallback and rather hard to change for me since the DNS config is pushed by the OpenStack DHCP / network init process.

Checking https://coredns.io/plugins/forward/ i cannot find that CoreDNS uses such a load-balancing behavior - could you reference it somehow? I understand, if that is how CoreDNS behaves, it has nothing to do with k3s.
But somehow i'am yet "reluctant" to believe that CoreDNS changed such a fundamental, especially when specifically asked to use the resolv.conf and it simply uses the nameservers in a different meaning as they are defined in sysops..

Thanks for all your help!

[brandond]

i cannot find that CoreDNS uses such a load-balancing behavior - could you reference it somehow

It's right there in the page you linked...

TO… are the destination endpoints to forward to. The TO syntax allows you to specify a protocol, tls://9.9.9.9 or dns:// (or no protocol) for plain DNS. The number of upstreams is limited to 15.
Multiple upstreams are randomized (see policy) on first use. When a healthy proxy returns an error during the exchange the next upstream in the list is tried.

policy specifies the policy to use for selecting upstream servers. The default is random.
random is a policy that implements random upstream selection.

The doc doesn't call this out, but it you point it at a resolv.conf instead of listing the upstreams in the config file itself, it will read the resolv.conf to determine what upstreams to use, just as if you had listed them all explicitly in the config file.

I guess it's not true load-balancing, but simply selecting a random upstream, and then using that one until it gets an error (and nxdomain is not an error), at which point the policy determines how it picks the next one to try. We use similar behavior for the k3s agent load-balancer tunnel to servers, so I have a bad habit of referring to this as load-balancing when it is really more of a failover pool.

The behavior stands though; coredns has a 50/50 chance of picking your private upstream when it start up.

when did CoreDNS decide to skip that and just load balance it

Always, as long as coredns has been around

[EugenMayer]

Sorry! Well i understood that

forward . 8.8.8.8 1.1.1.1

is randomized but

forward . /etc/resolve.conf

is telling it to use the resolve.conf and thus go by it's rules. I did not take the latter as "pick all the nameservers from that file and add them as simple upstreams".

But as i see the policy can be adjusted to policy: sequential.

IMHO if resolve.conf is used, it should become policy: sequential since anything else is just redefining the common expectations - but maybe it is just me sticking to the linux defaults, esp. on sysops. I guess there is no consideration in changing that. Kind of sad that it all came down to this in the end.

in the end

• /etc/resolv.conf is not used, since it is detected to be a systemd-dns environment
• instead /run/systemd/resolve/resolv.conf is used, which includes what i defined as DNS server, including my private DNS
• the setting is properly propagated by k3s to CoreDNS
• CoreDNS uses the dnsPolicy: default which does instruct it to use the OS
• but the CoreDNS policy setting in coreDNS defaults to "random" and thus my privateDNS server was not used half of the time

I understand that this is the CoreDNS default, surprising for me or not. Thank you for teaching me that!

Expecting k3s to have an extra-souce to fix that issue if /etc/resolv.conf is used is maybe not only too much to ask, it's probably not everyone's expectation either.

So for me defining there are 2 options

• defining a custom resolv.conf for k3s with only one server and use it with --resolv-conf
• try to override the policy in coreDNS :53. AFAICS there is no easy way to add that value only (and persist that even after k3s restarts). Current overrides coredns-custom only allow to redefine the entire section AFAIU.

Again, thank you for your patience and energy to share all those details. Very appritiated!

[brandond]

The releases that just came out this week allow you to override entries in the default server block (see #7583), you should be able to do something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  forward.override: |
    forward . /etc/resolv.conf {
      policy sequential
    }

I am not sure what happens if your internal upstream fails though; I suspect that it will switch over to the public upstream and stay there until it returns an error. You'd probably want to just have it forward to only your private upstream.

[EugenMayer]

One last question: Should i rephrase the intial question to something like : "How to use a private DNS with k3s" and change my initial question a bit, then mark the summary i gave above as answer?

The summary would be something like

Trying to configure k3s to use my private dns server `10.10.0.243` as the forward/upstream default to resolve. I fail to do so, my pods cannot resolve the private domains, only public ones.

Using ubuntu 22.04 with systemd based DNS.

The only way i could get it running is defining a `coredns-custom` custom-map like this

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  custom.server: |
     myprivatedomain.tld:53 {
      forward . 10.10.0.243
    }
---

Is there a better way to deal with this issue?

The reason i would like to do that is to value all the effort you put into answering all the qeuestions, details and teach. I think this thread could help someone else too but most of the issues i have read about this in k3s are not elaborating as much at all like you did here IMHO.

I let you decide of course. You helped me already (by a lot) - all i would like is that this might help others too (if anybody is actually as stubborn as iam)

[EugenMayer]

The releases that just came out this week allow you to override entries in the default server block (see #7583), you should be able to do something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  forward.override: |
    forward . /etc/resolv.conf {
      policy sequential
    }

I am not sure what happens if your internal upstream fails though; I suspect that it will switch over to the public upstream and stay there until it returns an error. You'd probably want to just have it forward to only your private upstream.

Did that

kubectl get cm -n kube-system coredns-custom -o yaml
apiVersion: v1
data:
  forward.override: |
    forward . /etc/resolv.conf {
      policy sequential
    }
immutable: false
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system

one rather odd thing is, it really takes a while after a server reboot for the DNS to actually work.
So after a server reboot, my pods all are started for already quiet some time, but the DNS resolving to private does only work after about 30-60s. Mayeb something quirky on my system - buy for everyone trying out, you might need to be a little patient.

Again, thank you for this really elegant solution!

[brandond]

Should i rephrase the intial question to something like : "How to use a private DNS with k3s" and change my initial question a bit, then mark the summary i gave above as answer?

You're welcome to do whatever you think would help people find it best!

[EugenMayer]

thank you, did so!

[EugenMayer]

If anybody is in need, i created a little helm chart to fix this issue, see https://github.com/EugenMayer/helm-charts/tree/main/charts/coredns-private-dns-fix

[zalmane]

I wanted to add a case that isn't covered by the customer /etc/resolv.conf which is using a local resolver such as dnsmasq. in that case we are forced to put in the private DNS since the /etc/resolv.conf shows 127.0.0.1.
Is there any solution to this other than manually putting in the DNS servers?

[mreiche]

I'm currently asking the same question here: #10951