[KubeUP][Scaleout] 8 system pods failed to start when mizar with setting DISABLE_NETWORK_SERVICE_SUPPORT=true

[q131172019]

What happened:
Run kube-up.sh with mizar network and setting DISABLE_NETWORK_SERVICE_SUPPORT=true to start 1 x 1 scale-out cluster, 8 pods (coredns, coredns-default, event-exporter, kube-proxy, l7-default-backend, metrics-server) failed to start and are not in Running states, which are a little different from issue #1334.

$ kubectl get pods -AT |grep -v Running
TENANT   NAMESPACE     NAME                                                    HASHKEY               READY   STATUS              RESTARTS   AGE
system   kube-system   coredns-75c65c444f-w49dw                                6957727143446538512   0/1     ContainerCreating   0          19m
system   kube-system   event-exporter-v0.2.5-868dff6494-8qv8l                  1630321710437582987   0/1     ContainerCreating   0          19m
system   kube-system   fluentd-gcp-scaler-74b46b8776-jzsnk                     491406284915146892    0/1     ContainerCreating   0          19m
system   kube-system   heapster-v1.6.0-beta.1-57874ccf9d-wgjfh                 8829222027596640593   0/2     ContainerCreating   0          19m
system   kube-system   kube-dns-autoscaler-748b78969c-7wpmz                    7000028260306986928   0/1     ContainerCreating   0          19m
system   kube-system   kubernetes-dashboard-848965699-qxw8j                    5585600767979665499   0/1     ContainerCreating   0          19m
system   kube-system   l7-default-backend-6497bc5bf6-86gg8                     560727389528662025    0/1     ContainerCreating   0          19m
system   kube-system   metrics-server-v0.3.3-5f994fcb77-bxglj                  689528112758024915    0/2     ContainerCreating   0          19m

$ kubectl get pods -AT |more
TENANT   NAMESPACE     NAME                                                    HASHKEY               READY   STATUS              RESTARTS   AGE
system   default       mizar-daemon-carltenant-012722-rp-1-minion-group-2qr2   8536270369083156335   1/1     Running             0          12m
system   default       mizar-daemon-carltenant-012722-tp-1-master              8536270369083156335   1/1     Running             0          18m
system   default       mizar-operator-carltenant-012722-tp-1-master            4897509646032053362   1/1     Running             0          11m
system   kube-system   coredns-75c65c444f-w49dw                                6957727143446538512   0/1     ContainerCreating   0          19m
system   kube-system   etcd-empty-dir-cleanup-carltenant-012722-rp-1-master    6416262497714047119   1/1     Running             0          12m
system   kube-system   etcd-empty-dir-cleanup-carltenant-012722-tp-1-master    6416262497714047119   1/1     Running             0          18m
system   kube-system   etcd-server-carltenant-012722-rp-1-master               6268924934619826925   1/1     Running             0          10m
system   kube-system   etcd-server-carltenant-012722-tp-1-master               1228365644270276432   1/1     Running             0          18m
system   kube-system   etcd-server-events-carltenant-012722-rp-1-master        7474375747461145051   1/1     Running             0          11m
system   kube-system   etcd-server-events-carltenant-012722-tp-1-master        6619750444035813212   1/1     Running             0          17m
system   kube-system   event-exporter-v0.2.5-868dff6494-8qv8l                  1630321710437582987   0/1     ContainerCreating   0          19m
system   kube-system   fluentd-gcp-scaler-74b46b8776-jzsnk                     491406284915146892    0/1     ContainerCreating   0          18m
system   kube-system   fluentd-gcp-v3.2.0-5h5rz                                2288972835954854406   1/1     Running             0          11m
system   kube-system   fluentd-gcp-v3.2.0-8fgkc                                770887717618315565    1/1     Running             0          18m
system   kube-system   fluentd-gcp-v3.2.0-khv8w                                304005417228941619    1/1     Running             0          11m
system   kube-system   heapster-v1.6.0-beta.1-57874ccf9d-wgjfh                 8829222027596640593   0/2     ContainerCreating   0          18m
system   kube-system   kube-addon-manager-carltenant-012722-rp-1-master        5014754618061431440   1/1     Running             0          10m
system   kube-system   kube-addon-manager-carltenant-012722-tp-1-master        5014754618061431440   1/1     Running             0          18m
system   kube-system   kube-apiserver-carltenant-012722-rp-1-master            861923758851201239    1/1     Running             0          11m
system   kube-system   kube-apiserver-carltenant-012722-tp-1-master            5112260190289867119   1/1     Running             0          18m
system   kube-system   kube-controller-manager-carltenant-012722-rp-1-master   3850563164447459352   1/1     Running             0          11m
system   kube-system   kube-controller-manager-carltenant-012722-tp-1-master   1217068427044670755   1/1     Running             1          18m
system   kube-system   kube-dns-autoscaler-748b78969c-7wpmz                    7000028260306986928   0/1     ContainerCreating   0          18m
system   kube-system   kube-proxy-carltenant-012722-rp-1-master                8351578365153539748   1/1     Running             0          5m29s
system   kube-system   kube-proxy-carltenant-012722-rp-1-minion-group-2qr2     8351578365153539748   1/1     Running             0          12m
system   kube-system   kube-proxy-carltenant-012722-tp-1-master                1228465400893692972   1/1     Running             0          18m
system   kube-system   kube-scheduler-carltenant-012722-tp-1-master            701206762147788219    1/1     Running             6          18m
system   kube-system   kubernetes-dashboard-848965699-qxw8j                    5585600767979665499   0/1     ContainerCreating   0          18m
system   kube-system   l7-default-backend-6497bc5bf6-86gg8                     560727389528662025    0/1     ContainerCreating   0          19m
system   kube-system   l7-lb-controller-v1.2.3-carltenant-012722-rp-1-master   5013348416365870850   1/1     Running             0          11m
system   kube-system   l7-lb-controller-v1.2.3-carltenant-012722-tp-1-master   5013348416365870850   1/1     Running             0          17m
system   kube-system   metrics-server-v0.3.3-5f994fcb77-bxglj                  689528112758024915    0/2     ContainerCreating   0          19m

What you expected to happen:
The 8 pods should be in "Running" state.

How to reproduce it (as minimally and precisely as possible):
The codes are POC codes + mizar PR1320.

$ unset KUBE_GCE_MASTER_PROJECT KUBE_GCE_NODE_PROJECT KUBE_GCI_VERSION  KUBE_GCE_MASTER_IMAGE  KUBE_GCE_NODE_IMAGE KUBE_CONTAINER_RUNTIME NETWORK_PROVIDER DISABLE_NETWORK_SERVICE_SUPPORT

$ export KUBEMARK_NUM_NODES=100 NUM_NODES=1 SCALEOUT_CLUSTER=true SCALEOUT_TP_COUNT=1 SCALEOUT_RP_COUNT=1 RUN_PREFIX=carltenant-012722 NETWORK_PROVIDER=mizar DISABLE_NETWORK_SERVICE_SUPPORT=true

$ export MASTER_DISK_SIZE=500GB MASTER_ROOT_DISK_SIZE=500GB KUBE_GCE_ZONE=us-west2-b MASTER_SIZE=n1-highmem-32 NODE_SIZE=n1-highmem-16 NODE_DISK_SIZE=500GB GOPATH=$HOME/go KUBE_GCE_ENABLE_IP_ALIASES=true KUBE_GCE_PRIVATE_CLUSTER=true CREATE_CUSTOM_NETWORK=true KUBE_GCE_INSTANCE_PREFIX=${RUN_PREFIX} KUBE_GCE_NETWORK=${RUN_PREFIX} ENABLE_KCM_LEADER_ELECT=false ENABLE_SCHEDULER_LEADER_ELECT=false ETCD_QUOTA_BACKEND_BYTES=8589934592 SHARE_PARTITIONSERVER=false LOGROTATE_FILES_MAX_COUNT=200 LOGROTATE_MAX_SIZE=200M KUBE_ENABLE_APISERVER_INSECURE_PORT=true KUBE_ENABLE_PROMETHEUS_DEBUG=true KUBE_ENABLE_PPROF_DEBUG=true TEST_CLUSTER_LOG_LEVEL=--v=2 HOLLOW_KUBELET_TEST_LOG_LEVEL=--v=2 GCE_REGION=us-west2-b

$ ./cluster/kube-up.sh

Anything else we need to know?:

Environment:

• Arktos version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools: Kubeup scale-out 1 x 1
• Network plugin and version (if this is a network-related bug):
• Others: