k3d cluster create got stuck in "Injecting records for hostAliases..."

[zhitaoli-6]

I want to create a k3d cluster in my local vm with k3d cluster --verbose create --servers 1 --agents 1 main. However it got stuck without any response. Could anyone help me fix this issue?

k3d version:

[root@zhitao-pc zhitaoli]# k3d --version
k3d version v5.6.0
k3s version v1.27.4-k3s1 (default)

docker version:

[root@zhitao-pc zhitaoli]# docker --version
Docker version 24.0.5, build ced0996

ip addr:

[root@zhitao-pc zhitaoli]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 52:54:00:4d:08:46 brd ff:ff:ff:ff:ff:ff
    inet 192.168.23.18/20 brd 192.168.31.255 scope global dynamic noprefixroute ens4
       valid_lft 7037sec preferred_lft 7037sec
    inet6 fe80::b53:dc53:88ab:c31f/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: ens11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 52:54:00:1d:a4:24 brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:e7:d5:2a brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:e7:d5:2a brd ff:ff:ff:ff:ff:ff
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:42:e2:5b:0a brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
23: br-7ad921cfa466: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:e6:4f:c9:65 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.1/16 brd 172.19.255.255 scope global br-7ad921cfa466
       valid_lft forever preferred_lft forever
    inet6 fe80::42:e6ff:fe4f:c965/64 scope link
       valid_lft forever preferred_lft forever
69: br-20b8a974cb8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:aa:43:ae:1d brd ff:ff:ff:ff:ff:ff
    inet 172.24.0.1/16 brd 172.24.255.255 scope global br-20b8a974cb8c
       valid_lft forever preferred_lft forever
    inet6 fe80::42:aaff:fe43:ae1d/64 scope link
       valid_lft forever preferred_lft forever
73: veth02bef22@if72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-20b8a974cb8c state UP group default
    link/ether ca:81:b5:22:a2:5e brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::c881:b5ff:fe22:a25e/64 scope link
       valid_lft forever preferred_lft forever
75: vethe92995f@if74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-20b8a974cb8c state UP group default
    link/ether 52:90:7a:a7:1b:52 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::5090:7aff:fea7:1b52/64 scope link
       valid_lft forever preferred_lft forever
77: veth0d6ddf8@if76: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-20b8a974cb8c state UP group default
    link/ether e2:e4:de:87:ba:ff brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::e0e4:deff:fe87:baff/64 scope link
       valid_lft forever preferred_lft forever

The detail output with verbose option is as following:

[root@zhitao-pc sfs]# k3d cluster --verbose create --servers 1 --agents 1 main
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:24.0.5 OSType:linux OS:CentOS Linux 8 Arch:x86_64 CgroupVersion:1 CgroupDriver:cgroupfs Filesystem:xfs InfoName:zhitao-pc}
DEBU[0000] Additional CLI Configuration:
cli:
  api-port: ""
  env: []
  k3s-node-labels: []
  k3sargs: []
  ports: []
  registries:
    create: ""
  runtime-labels: []
  runtime-ulimits: []
  volumes: []
hostaliases: []
DEBU[0000] Configuration:
agents: 1
image: docker.io/rancher/k3s:v1.27.4-k3s1
network: ""
options:
  k3d:
    disableimagevolume: false
    disableloadbalancer: false
    disablerollback: false
    loadbalancer:
      configoverrides: []
    timeout: 0s
    wait: true
  kubeconfig:
    switchcurrentcontext: true
    updatedefaultkubeconfig: true
  runtime:
    agentsmemory: ""
    gpurequest: ""
    hostpidmode: false
    serversmemory: ""
registries:
  config: ""
  use: []
servers: 1
subnet: ""
token: ""
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha5} ObjectMeta:{Name:} Servers:1 Agents:1 ExposeAPI:{Host: HostIP: HostPort:} Image:docker.io/rancher/k3s:v1.27.4-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[] Ulimits:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
==========================
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha5} ObjectMeta:{Name:} Servers:1 Agents:1 ExposeAPI:{Host: HostIP: HostPort:35331} Image:docker.io/rancher/k3s:v1.27.4-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[] Ulimits:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
==========================
DEBU[0000] generated loadbalancer config:
ports:
  6443.tcp:
  - k3d-main-server-0
settings:
  workerConnections: 1024
DEBU[0000] ===== Merged Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:main Network:{Name:k3d-main ID: External:false IPAM:{IPPrefix:invalid Prefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0003eb040 0xc0003eb1e0 0xc0003eb380] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc000136f80 ServerLoadBalancer:0xc00025e2c0 ImageVolume: Volumes:[]} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== =====
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-main'
INFO[0000] Created image volume k3d-main-images
INFO[0000] Starting new tools node...
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] [Docker] DockerHost: '' ()
DEBU[0000] Created container k3d-main-tools (ID: fb0f0f841e6939b028b0dad93bdbe9551403a85d91eb8a52b38879884d994ca3)
DEBU[0000] Node k3d-main-tools Start Time: 2023-12-11 19:09:06.409726903 +0800 CST m=+0.469912302
INFO[0000] Starting Node 'k3d-main-tools'
DEBU[0001] Truncated 2023-12-11 11:09:06.8721228 +0000 UTC to 2023-12-11 11:09:06 +0000 UTC
INFO[0001] Creating node 'k3d-main-server-0'
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
DEBU[0001] Created container k3d-main-server-0 (ID: 0455509af9020e038966c219524c874b9bc3e3a5fa85e7d936e949be28d576f0)
DEBU[0001] Created node 'k3d-main-server-0'
INFO[0001] Creating node 'k3d-main-agent-0'
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
DEBU[0001] Created container k3d-main-agent-0 (ID: c4b36b053e416270f9d7bcf1be3bab49c43f60733c0918a627c01979737d7d1b)
DEBU[0001] Created node 'k3d-main-agent-0'
INFO[0001] Creating LoadBalancer 'k3d-main-serverlb'
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
DEBU[0001] Created container k3d-main-serverlb (ID: b96b6b5204e111343d86dec4b8e75f44812cc0b2f607a18aa1f5178b732a9223)
DEBU[0001] Created loadbalancer 'k3d-main-serverlb'
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
INFO[0001] Using the k3d-tools node to gather environment information
DEBU[0001] no netlabel present on container /k3d-main-tools
DEBU[0001] failed to get IP for container /k3d-main-tools as we couldn't find the cluster network
DEBU[0001] Deleting node k3d-main-tools ...
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
INFO[0001] HostIP: using network gateway 172.24.0.1 address
INFO[0001] Starting cluster 'main'
INFO[0001] Starting servers...
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
DEBU[0001] No fix enabled.
DEBU[0001] Node k3d-main-server-0 Start Time: 2023-12-11 19:09:07.793704307 +0800 CST m=+1.853889707
INFO[0001] Starting Node 'k3d-main-server-0'
DEBU[0002] Truncated 2023-12-11 11:09:08.244261763 +0000 UTC to 2023-12-11 11:09:08 +0000 UTC
DEBU[0002] Waiting for node k3d-main-server-0 to get ready (Log: 'k3s is up and running')
DEBU[0007] Finished waiting for log message 'k3s is up and running' from node 'k3d-main-server-0'
INFO[0007] Starting agents...
DEBU[0007] DOCKER_SOCK=/var/run/docker.sock
DEBU[0007] No fix enabled.
DEBU[0007] Node k3d-main-agent-0 Start Time: 2023-12-11 19:09:13.08331087 +0800 CST m=+7.143496273
INFO[0007] Starting Node 'k3d-main-agent-0'
DEBU[0007] Truncated 2023-12-11 11:09:13.577211798 +0000 UTC to 2023-12-11 11:09:13 +0000 UTC
DEBU[0007] Waiting for node k3d-main-agent-0 to get ready (Log: 'Successfully registered node')
DEBU[0010] Finished waiting for log message 'successfully registered node' from node 'k3d-main-agent-0'
INFO[0010] Starting helpers...
DEBU[0010] Node k3d-main-serverlb Start Time: 2023-12-11 19:09:16.933743854 +0800 CST m=+10.993929237
INFO[0011] Starting Node 'k3d-main-serverlb'
DEBU[0011] Truncated 2023-12-11 11:09:17.542233548 +0000 UTC to 2023-12-11 11:09:17 +0000 UTC
DEBU[0011] Waiting for node k3d-main-serverlb to get ready (Log: 'start worker processes')
DEBU[0017] Finished waiting for log message 'start worker processes' from node 'k3d-main-serverlb'
DEBU[0017] Found network {Name:k3d-main ID:20b8a974cb8ca59a45e15ce7f90301e2d298d997101f8bfa9632f4b76f38a583 Created:2023-12-11 19:09:05.978848819 +0800 CST Scope:local Driver:bridge EnableIPv6:false IPAM:{Driver:default Options:map[] Config:[{Subnet:172.24.0.0/16 IPRange: Gateway:172.24.0.1 AuxAddress:map[]}]} Internal:false Attachable:false Ingress:false ConfigFrom:{Network:} ConfigOnly:false Containers:map[0455509af9020e038966c219524c874b9bc3e3a5fa85e7d936e949be28d576f0:{Name:k3d-main-server-0 EndpointID:cd594ffd0ae33d912ee94f44ff24b5e5c2f8b3b03573d886842e8ce4740c7c87 MacAddress:02:42:ac:18:00:02 IPv4Address:172.24.0.2/16 IPv6Address:} b96b6b5204e111343d86dec4b8e75f44812cc0b2f607a18aa1f5178b732a9223:{Name:k3d-main-serverlb EndpointID:7e9d74845e0e6bf637665579dbbab0b5c909bc31b0e872acf58394bbc20096c4 MacAddress:02:42:ac:18:00:04 IPv4Address:172.24.0.4/16 IPv6Address:} c4b36b053e416270f9d7bcf1be3bab49c43f60733c0918a627c01979737d7d1b:{Name:k3d-main-agent-0 EndpointID:7f8b6d268f8a12c8088ff6df38997a2d7cfde665cbb9439ddbde440d87b39370 MacAddress:02:42:ac:18:00:03 IPv4Address:172.24.0.3/16 IPv6Address:}] Options:map[com.docker.network.bridge.enable_ip_masquerade:true] Labels:map[app:k3d] Peers:[] Services:map[]}
INFO[0017] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap...
DEBU[0018] Executing command '[sh -c cat /tmp/-etc-hosts-dUdfofYHjMqyJiXuVUgK > /etc/hosts]' in node 'k3d-main-server-0'
DEBU[0018] Executing command '[sh -c cat /tmp/-etc-hosts-pTuRZhaDEIfdfVYhCRQe > /etc/hosts]' in node 'k3d-main-agent-0'
DEBU[0019] Exec process in node 'k3d-main-server-0' exited with '0'
DEBU[0019] Exec process in node 'k3d-main-agent-0' exited with '0'