Join node not working any more

[oleo65]

On v.1.20.2 joining a node is not working any more.

Syslog shows the following stack trace.

Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[567882]: error: error running snapctl: unknown service: "microk8s.daemon-apiservice"
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]: [2021-02-16 17:12:59,916] ERROR in app: Exception on /cluster/api/v1.0/join [POST]
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]: Traceback (most recent call last):
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     response = self.full_dispatch_request()
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     rv = self.handle_user_exception(e)
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     reraise(exc_type, exc_value, tb)
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     raise value
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     rv = self.dispatch_request()
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     return self.view_functions[rule.endpoint](**req.view_args)
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/scripts/cluster/agent.py", line 316, in join_node_etcd
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     service("restart", "apiservice")
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/scripts/cluster/common/utils.py", line 277, in service
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     "snapctl {} microk8s.daemon-{}".format(operation, service_name).split()
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:   File "/snap/microk8s/2048/usr/lib/python3.5/subprocess.py", line 581, in check_call
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]:     raise CalledProcessError(retcode, cmd)
Feb 16 17:12:59 pi-k8s-1 microk8s.daemon-cluster-agent[554105]: subprocess.CalledProcessError: Command '['snapctl', 'restart', 'microk8s.daemon-apiservice']' returned non-zero exit status 1

I believe it is a change introduced in this line. Probably should be apiserver instead of apiservice.

https://github.com/ubuntu/microk8s/blame/efde194e67312ab7f701d119a7bbbd183c096739/scripts/cluster/agent.py#L316

Setup is without ha-cluster. Joining node is a fresh snap.

[ktsakalozos]

Hi @oleo65 thank you for reporting this. We have reverted the offending change you should be able to add nodes now without problems.

[oleo65]

Thank you for your feedback. With the latest update the node joining is working again as expected. 👍

[balchua]

Thanks @oleo65 for confirming i will close this one then.

[sudhakso]

After all trial and errors, the only trick that has worked is to update /etc/hosts on each node with at least 2 entries.

hostname
hostname

Here, IP of the master node is what is shown post microk8s add-node command.

Also consider opening the firewall to allow the port : where microk8s master process is running (default on the master node).