[KubeUP][Scaleout] System tenants' nginx pods are in "ContainerCreating" states when mizar is with setting DISABLE_NETWORK_SERVICE_SUPPORT=true

[q131172019]

What happened:
Run kube-up.sh with mizar network and with setting DISABLE_NETWORK_SERVICE_SUPPOR=trueT to start 1 x 1 scale-out cluster, then create nginx pods for system tenant test tenants. These nginx pods are in "ContainerCreating" state.

ying-nginx-system-1                                     5417300996794749240   0/1     ContainerCreating   0          13m   <none>      carltenant-012722-rp-1-minion-group-2qr2   <none>           <none>
ying-nginx-system-2                                     1488824182988708395   0/1     ContainerCreating   0          13m   <none>      carltenant-012722-rp-1-minion-group-2qr2   <none>           <none>

What you expected to happen:
The system tenant's nginx pods should be in "Running" state.

How to reproduce it (as minimally and precisely as possible):
The codes are POC codes + mizar PR1320.

$ unset KUBE_GCE_MASTER_PROJECT KUBE_GCE_NODE_PROJECT KUBE_GCI_VERSION  KUBE_GCE_MASTER_IMAGE  KUBE_GCE_NODE_IMAGE KUBE_CONTAINER_RUNTIME NETWORK_PROVIDER DISABLE_NETWORK_SERVICE_SUPPORT

$ export KUBEMARK_NUM_NODES=100 NUM_NODES=1 SCALEOUT_CLUSTER=true SCALEOUT_TP_COUNT=1 SCALEOUT_RP_COUNT=1 RUN_PREFIX=carltenant-012722 NETWORK_PROVIDER=mizar DISABLE_NETWORK_SERVICE_SUPPORT=true

$ export MASTER_DISK_SIZE=500GB MASTER_ROOT_DISK_SIZE=500GB KUBE_GCE_ZONE=us-west2-b MASTER_SIZE=n1-highmem-32 NODE_SIZE=n1-highmem-16 NODE_DISK_SIZE=500GB GOPATH=$HOME/go KUBE_GCE_ENABLE_IP_ALIASES=true KUBE_GCE_PRIVATE_CLUSTER=true CREATE_CUSTOM_NETWORK=true KUBE_GCE_INSTANCE_PREFIX=${RUN_PREFIX} KUBE_GCE_NETWORK=${RUN_PREFIX} ENABLE_KCM_LEADER_ELECT=false ENABLE_SCHEDULER_LEADER_ELECT=false ETCD_QUOTA_BACKEND_BYTES=8589934592 SHARE_PARTITIONSERVER=false LOGROTATE_FILES_MAX_COUNT=200 LOGROTATE_MAX_SIZE=200M KUBE_ENABLE_APISERVER_INSECURE_PORT=true KUBE_ENABLE_PROMETHEUS_DEBUG=true KUBE_ENABLE_PPROF_DEBUG=true TEST_CLUSTER_LOG_LEVEL=--v=2 HOLLOW_KUBELET_TEST_LOG_LEVEL=--v=2 GCE_REGION=us-west2-b

$ ./cluster/kube-up.sh

$ cat ~/TMP/mizar/pod-vpc-1-without-annotation.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ying-nginx-system-1
  tenant: system
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
      - containerPort: 443
---
apiVersion: v1
kind: Pod
metadata:
  name: ying-nginx-system-2
  tenant: system
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
      - containerPort: 443


$ kubectl apply -f ~/TMP/mizar/pod-vpc-1-without-annotation.yaml
pod/ying-nginx-system-1 created
pod/ying-nginx-system-2 created

-- Wait for over 10 minutes
131172019@carltenant-012722-tp-1-master:~$ kubectl get pods --tenant system -o wide
NAME                                                    HASHKEY               READY   STATUS              RESTARTS   AGE   IP          NODE                                       NOMINATED NODE   READINESS GATES
mizar-daemon-carltenant-012722-rp-1-minion-group-2qr2   8536270369083156335   1/1     Running             0          30m   10.40.0.4   carltenant-012722-rp-1-minion-group-2qr2   <none>           <none>
mizar-daemon-carltenant-012722-tp-1-master              8536270369083156335   1/1     Running             0          37m   10.40.0.2   carltenant-012722-tp-1-master              <none>           <none>
mizar-operator-carltenant-012722-tp-1-master            4897509646032053362   1/1     Running             0          30m   10.40.0.2   carltenant-012722-tp-1-master              <none>           <none>
ying-nginx-system-1                                     5417300996794749240   0/1     ContainerCreating   0          13m   <none>      carltenant-012722-rp-1-minion-group-2qr2   <none>           <none>
ying-nginx-system-2                                     1488824182988708395   0/1     ContainerCreating   0          13m   <none>      carltenant-012722-rp-1-minion-group-2qr2   <none>           <none>

Anything else we need to know?:

Environment:

• Arktos version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools: Scale-out 1x1 KubeUp + PR1320
• Network plugin and version (if this is a network-related bug):
• Others: