Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.31] - Support for NodeExternalDNS #10854

Open
brandond opened this issue Sep 6, 2024 · 1 comment
Open

[Release-1.31] - Support for NodeExternalDNS #10854

brandond opened this issue Sep 6, 2024 · 1 comment
Assignees
@brandond @VestigeJ
Milestone
v1.31.2+k3s1

Comments

@brandond
Copy link
Contributor

brandond commented Sep 6, 2024

Backport fix for Support for NodeExternalDNS
@brandond brandond self-assigned this Sep 6, 2024
@brandond brandond added this to the v1.31.2+k3s1 milestone Sep 6, 2024
@VestigeJ
Copy link

VestigeJ commented Sep 16, 2024 •
edited
Loading

New Feature

##Environment Details
Validated using VERSION=v1.31.1-rc1+k3s1

Infrastructure

  • Cloud

Node(s) CPU architecture, OS, and version:

Linux 6.4.0-150600.23.17-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

Cluster Configuration:

NAME               STATUS   ROLES                       AGE   VERSION
ip-1-1-1-14        Ready    control-plane,etcd,master   20m   v1.31.1-rc1+k3s1

Config.yaml:

node-external-ip: 1.1.1.14
node-external-dns:
- cloudflare.com
- arc.k3s.com
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
cluster-init: true
embedded-registry: true

Validation

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ VERSION=v1.31.1-rc1+k3s1
$ sudo INSTALL_K3S_VERSION=$VERSION INSTALL_K3S_EXEC=server ./install-k3s.sh
$ set_kubefig
$ kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
$ k exec -it dnsutils -- nslookup cloudflare.com
$ k exec -it dnsutils -- nslookup -debug cloudflare.com
$ kgn -o yaml
$ k exec -it dnsutils -- nslookup -debug arc.k3s.com
$ k exec -it dnsutils -- nslookup -debug google.com

Results:

$ kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

pod/dnsutils created

$ kgp dnsutils

NAME       READY   STATUS    RESTARTS   AGE
dnsutils   1/1     Running   0          6s

$ k exec -it dnsutils -- nslookup cloudflare.com

Server:		10.43.0.10
Address:	10.43.0.10#53

Non-authoritative answer:
Name:	cloudflare.com
Address: 104.16.133.229
Name:	cloudflare.com
Address: 104.16.132.229
Name:	cloudflare.com
Address: 2611111:1111::::
Name:	cloudflare.com
Address: 2611:1111::::

$ k exec -it dnsutils -- nslookup -debug cloudflare.com

Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	cloudflare.com.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
	origin = ns.dns.cluster.local
	mail addr = hostmaster.cluster.local
	serial = 1726525239
	refresh = 7200
	retry = 1800
	expire = 86400
	minimum = 5
	ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find cloudflare.com.default.svc.cluster.local: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	cloudflare.com.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
	origin = ns.dns.cluster.local
	mail addr = hostmaster.cluster.local
	serial = 1726525239
	refresh = 7200
	retry = 1800
	expire = 86400
	minimum = 5
	ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find cloudflare.com.svc.cluster.local: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	cloudflare.com.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
	origin = ns.dns.cluster.local
	mail addr = hostmaster.cluster.local
	serial = 1726525239
	refresh = 7200
	retry = 1800
	expire = 86400
	minimum = 5
	ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find cloudflare.com.cluster.local: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	cloudflare.com.us-east-2.compute.internal, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  us-east-2.compute.internal
	origin = ns0.us-east-2.compute.internal
	mail addr = hostmaster.amazon.com
	serial = 2012103100
	refresh = 3600
	retry = 3600
	expire = 3600
	minimum = 60
	ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find cloudflare.com.us-east-2.compute.internal: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	cloudflare.com, type = A, class = IN
    ANSWERS:
    ->  cloudflare.com
	internet address = 104.16.132.229
	ttl = 30
    ->  cloudflare.com
	internet address = 104.16.133.229
	ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name:	cloudflare.com
Address: 104.16.132.229
Name:	cloudflare.com
Address: 104.16.133.229
------------
    QUESTIONS:
	cloudflare.com, type = AAAA, class = IN
    ANSWERS:
    ->  cloudflare.com
	has AAAA address 2611:1111::::
	ttl = 30
    ->  cloudflare.com
	has AAAA address 2611111:1111::::
	ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:	cloudflare.com
Address: 2611:1111::::
Name:	cloudflare.com
Address: 2611111:1111::::

$ kgn -o yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      alpha.kubernetes.io/provided-node-ip: 1.1.1.14,110:1111:1131:1101:119b:311:1191:1117
      etcd.k3s.cattle.io/local-snapshots-timestamp: "2024-09-16T22:20:09Z"
      etcd.k3s.cattle.io/node-address: 1.1.1.14
      etcd.k3s.cattle.io/node-name: ip-ip-8ead0505
      flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"ea:3b:e6:61:1a:94"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 172.31.11.144
      k3s.io/external-ip: 1.2.3.4
      k3s.io/hostname: ip-ip
      k3s.io/internal-ip: 1.1.1.14,110:1111:1131:1101:119b:311:1191:1117
      k3s.io/node-args: '["server","--node-external-ip","1.2.3.4","--node-external-dns","cloudflare.com","--node-external-dns","arc.k3s.com","--token","********","--write-kubeconfig-mode","644","--debug","true","--cluster-init","true","--embedded-registry","true"]'
      k3s.io/node-config-hash: 61111117Y565JXGI11111111111111111CWHBQ====
      k3s.io/node-env: '{}'
      node.alpha.kubernetes.io/ttl: "0"
      volumes.kubernetes.io/controller-managed-attach-detach: "true"

Different example of a known bad dns lookup that continues hunting out further

** server can't find arc.k3s.com.svc.cluster.local: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	arc.k3s.com.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
	origin = ns.dns.cluster.local
	mail addr = hostmaster.cluster.local
	serial = 1726525239
	refresh = 7200
	retry = 1800
	expire = 86400
	minimum = 5
	ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find arc.k3s.com.cluster.local: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	arc.k3s.com.us-east-2.compute.internal, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  us-east-2.compute.internal
	origin = ns0.us-east-2.compute.internal
	mail addr = hostmaster.amazon.com
	serial = 2012103100
	refresh = 3600
	retry = 3600
	expire = 3600
	minimum = 60
	ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find arc.k3s.com.us-east-2.compute.internal: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	arc.k3s.com, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  k3s.com
	origin = daisy.ns.cloudflare.com
	mail addr = dns.cloudflare.com
	serial = 2351630111
	refresh = 10000
	retry = 2400
	expire = 604800
	minimum = 1800
	ttl = 30
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
------------
    QUESTIONS:
	arc.k3s.com, type = AAAA, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  k3s.com
	origin = daisy.ns.cloudflare.com
	mail addr = dns.cloudflare.com
	serial = 2351630111
	refresh = 10000
	retry = 2400
	expire = 604800
	minimum = 1800
	ttl = 30
    ADDITIONAL RECORDS:
------------
*** Can't find arc.k3s.com: No answer

$ k exec -it dnsutils -- nslookup -debug google.com

Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	google.com.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
	origin = ns.dns.cluster.local
	mail addr = hostmaster.cluster.local
	serial = 1726525239
	refresh = 7200
	retry = 1800
	expire = 86400
	minimum = 5
	ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find google.com.default.svc.cluster.local: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	google.com.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
	origin = ns.dns.cluster.local
	mail addr = hostmaster.cluster.local
	serial = 1726525239
	refresh = 7200
	retry = 1800
	expire = 86400
	minimum = 5
	ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find google.com.svc.cluster.local: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	google.com.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
	origin = ns.dns.cluster.local
	mail addr = hostmaster.cluster.local
	serial = 1726525239
	refresh = 7200
	retry = 1800
	expire = 86400
	minimum = 5
	ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find google.com.cluster.local: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	google.com.us-east-2.compute.internal, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  us-east-2.compute.internal
	origin = ns0.us-east-2.compute.internal
	mail addr = hostmaster.amazon.com
	serial = 2012103100
	refresh = 3600
	retry = 3600
	expire = 3600
	minimum = 60
	ttl = 28
    ADDITIONAL RECORDS:
------------
** server can't find google.com.us-east-2.compute.internal: NXDOMAIN
Server:		10.43.0.10
Address:	10.43.0.10#53

------------
    QUESTIONS:
	google.com, type = A, class = IN
    ANSWERS:
    ->  google.com
	internet address = 172.217.1.110
	ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name:	google.com
Address: 172.217.1.110
------------
    QUESTIONS:
	google.com, type = AAAA, class = IN
    ANSWERS:
    ->  google.com
	has AAAA address 2607:f8b0:4009:81a::200e
	ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:	google.com
Address: 2607:f8b0:4009:81a::200e

command terminated with exit code 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@VestigeJ VestigeJ

Labels
None yet
Projects
K3s Development
Status: Next Up
Milestone
v1.31.2+k3s1
Development

No branches or pull requests
2 participants
@brandond @VestigeJ