why pod is always staying terminating status, and couldn't be operated unless forced delete.

[wang-xiaowu]

• resources:
  

• describe【no finalizers】

Name:                      xxx-master-58b4977765-wmlwb
Namespace:                 sopei-biz
Priority:                  0
Service Account:           default
Node:                      k3s-release-master1/xxx
Start Time:                Wed, 12 Jun 2024 02:21:09 +0000
Labels:                    app=xxx
                           pod-template-hash=58b4977765
                           security.istio.io/tlsMode=istio
                           service.istio.io/canonical-name=xxx
                           service.istio.io/canonical-revision=master
                           version=master
Annotations:               cattle.io/timestamp: 2024-01-31T11:09:41Z
                           istio.io/rev: default
                           kubectl.kubernetes.io/default-container: xxx
                           kubectl.kubernetes.io/default-logs-container: xxx
                           prometheus.io/path: /stats/prometheus
                           prometheus.io/port: 15020
                           prometheus.io/scrape: true
                           sidecar.istio.io/inject: true
                           sidecar.istio.io/status:
                             {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["workload-socket","credential-socket","workload-certs","istio-env...
Status:                    Terminating (lasts 13h)
Termination Grace Period:  30s
IP:                        10.42.0.242
IPs:
  IP:           10.42.0.242
Controlled By:  ReplicaSet/xxx-master-58b4977765
Init Containers:
  istio-init:
    Container ID:  containerd://46286da7a2759db875d900389c5a121cf4e85d459927f2ee61e0df7754a44e27
    Image:         rancher/mirrored-istio-proxyv2:1.20.3-distroless
    Image ID:      docker.io/rancher/mirrored-istio-proxyv2@sha256:74b658130ba04dc0b3b324865f659a6a3f8c14aedfb403ff3ecda229d6bba16f
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x
      
      -b
      *
      -d
      15090,15021,15020
      --log_output_level=default:info
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 12 Jun 2024 02:21:10 +0000
      Finished:     Wed, 12 Jun 2024 02:21:10 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:        100m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ss9rq (ro)
Containers:
  xxx:
    Container ID:   containerd://ee68832dbfc8dee7b8cbb037cc6aab86135a7874e86f3d87901c0a9705fdec88
    Image:          xxx:5000/master/xxx:v51704
    Image ID:       xxx:5000/master/xxx@sha256:dbe290148d62ae4866bab1fe0cc892675e37f1c5a78a4a6b37a5e8335a14cb6b
    Port:           34039/TCP
    Host Port:      0/TCP
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 12 Jun 2024 02:21:13 +0000
      Finished:     Tue, 18 Jun 2024 12:12:10 +0000
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:15020/app-health/xxx/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:15020/app-health/xxx/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      sopei-biz-configmap  ConfigMap  Optional: false
    Environment:
      POD_NAME:         xxx-master-58b4977765-wmlwb (v1:metadata.name)
      PROJECT_VERSION:  master
    Mounts:
      /var/log/sopei from sopei-log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ss9rq (ro)
  istio-proxy:
    Container ID:  containerd://9a533885efb752d715d3287efdc823b019b211d620b7409ff9b2fb15b75ebd7a
    Image:         rancher/mirrored-istio-proxyv2:1.20.3-distroless
    Image ID:      docker.io/rancher/mirrored-istio-proxyv2@sha256:74b658130ba04dc0b3b324865f659a6a3f8c14aedfb403ff3ecda229d6bba16f
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 12 Jun 2024 02:21:13 +0000
      Finished:     Tue, 18 Jun 2024 12:12:10 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=0s timeout=3s period=15s #success=1 #failure=4
    Startup:    http-get http://:15021/healthz/ready delay=0s timeout=3s period=1s #success=1 #failure=600
    Environment:
      JWT_POLICY:                    third-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      xxx-master-58b4977765-wmlwb (v1:metadata.name)
      POD_NAMESPACE:                 sopei-biz (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      ISTIO_CPU_LIMIT:               2 (limits.cpu)
      PROXY_CONFIG:                  {}
                                     
      ISTIO_META_POD_PORTS:          [
                                         {"name":"web","containerPort":34039,"protocol":"TCP"}
                                     ]
      ISTIO_META_APP_CONTAINERS:     xxx
      GOMEMLIMIT:                    1073741824 (limits.memory)
      GOMAXPROCS:                    2 (limits.cpu)
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_NODE_NAME:           (v1:spec.nodeName)
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_META_WORKLOAD_NAME:      xxx-master
      ISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/sopei-biz/deployments/xxx-master
      ISTIO_META_MESH_ID:            cluster.local
      TRUST_DOMAIN:                  cluster.local
      ISTIO_KUBE_APP_PROBERS:        {"/app-health/xxx/livez":{"tcpSocket":{"port":34039},"timeoutSeconds":1},"/app-health/xxx/readyz":{"tcpSocket":{"port":34039},"timeoutSeconds":1}}
    Mounts:
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/credential-uds from credential-socket (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ss9rq (ro)
      /var/run/secrets/tokens from istio-token (rw)
      /var/run/secrets/workload-spiffe-credentials from workload-certs (rw)
      /var/run/secrets/workload-spiffe-uds from workload-socket (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  workload-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  credential-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  workload-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  sopei-log:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sopei-log
    ReadOnly:   false
  kube-api-access-ss9rq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason         Age                  From     Message
  ----     ------         ----                 ----     -------
  Warning  FailedKillPod  54s (x323 over 11h)  kubelet  error killing pod: failed to "KillPodSandbox" for "2ce0ab01-eba9-4207-a2b7-9561f2cf4c91" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"