ingres-nginx-controller External IP Stuck on Pending and Load Balancer Not creating

[thoth2357]

This is such a beautiful tool but i have issues having my ingress-nginx-container show an external IP, nor the load balancer create.
I dont know where i went wrong, as i've been religiously following the wiki . On trying the

On trying the Step 11 of using helm to install ingress-nginx with the below command,

helm upgrade --install \
ingress-nginx ingress-nginx/ingress-nginx \
-f ~/.kube/ingress-nginx-annotations.yaml \
--namespace ingress-nginx \
--create-namespace

but it kept giving the nil pointer error. so i decided to use the old fashioned way of manually downloading the deploy.yaml and editing it , i.e Adding the annotations, ConfigMap, etc.
But after applying the service/ingress-nginx-controller, the External IP remains at and LB doesnt create. Please what else i'm i missing .

My configs

hetzner-k3s_cluster_config.yaml

---
hetzner_token: fO96B3.........
cluster_name: test
kubeconfig_path: "./kubeconfig"
k3s_version: v1.25.5+k3s1
public_ssh_key_path: "~/.ssh/OSI-LMS.pub"
private_ssh_key_path: "~/.ssh/OSI-LMS"
use_ssh_agent: false
ssh_allowed_networks:
- 0.0.0.0/0
api_allowed_networks:
- 0.0.0.0/0
schedule_workloads_on_masters: false
masters_pool:
  instance_type: cx21
  location: nbg1
  instance_count: 1
worker_node_pools:
- name: fsn-cpx21
  instance_type: cpx21
  location: fsn1
  instance_count: 2
- name: autoscaled-hel-cpx31
  instance_type: cpx31
  location: hel1
  autoscaling:
    enabled: true
    min_instances: 0
    max_instances: 5

The ingress-nginx service section in the deploy.yaml showing the annotations i added

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
  annotations: 
    load-balancer.hetzner.cloud/location: nbg1
    load-balancer.hetzner.cloud/use-private-ip: "true"
    load-balancer.hetzner.cloud/name: "lb1"
    load-balancer.hetzner.cloud/uses-proxyprotocol: 'true'
spec:
  externalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer

The ConfigMap section of the deploy.yaml showing the data i added

apiVersion: v1
data:
  allow-snippet-annotations: "false"
  use-proxy-protocol: "true"
  use-forwarded-headers: "true"
  compute-full-forwarded-for: "true"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-controller
  namespace: ingress-nginx

The stdout of my kubectl get all -A

NAMESPACE       NAME                                            READY   STATUS    RESTARTS   AGE
ingress-nginx   pod/ingress-nginx-admission-create-w99g8        0/1     Pending   0          104m
ingress-nginx   pod/ingress-nginx-admission-patch-rgtl2         0/1     Pending   0          104m
ingress-nginx   pod/ingress-nginx-controller-675c47d5f8-4lsx6   0/1     Pending   0          104m
kube-system     pod/coredns-597584b69b-x246g                    0/1     Pending   0          6h42m

NAMESPACE       NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default         service/kubernetes                           ClusterIP      10.43.0.1       <none>        443/TCP                      6h42m
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.43.87.12     <pending>     80:31683/TCP,443:30009/TCP   104m
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.43.160.206   <none>        443/TCP                      104m
kube-system     service/kube-dns                             ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP       6h42m

NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx   deployment.apps/ingress-nginx-controller   0/1     1            0           104m
kube-system     deployment.apps/coredns                    0/1     1            0           6h42m

NAMESPACE       NAME                                                  DESIRED   CURRENT   READY   AGE
ingress-nginx   replicaset.apps/ingress-nginx-controller-675c47d5f8   1         1         0       104m
kube-system     replicaset.apps/coredns-597584b69b                    1         1         0       6h42m

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE
ingress-nginx   job.batch/ingress-nginx-admission-create   0/1           104m       104m
ingress-nginx   job.batch/ingress-nginx-admission-patch    0/1           104m       104m

[vitobotta]

What do you see if you describe the load balancer service?

[thoth2357]

below is the output when i describe the load balance service.

Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.10.1
Annotations:              load-balancer.hetzner.cloud/location: nbg1
                          load-balancer.hetzner.cloud/name: lb1
                          load-balancer.hetzner.cloud/use-private-ip: true
                          load-balancer.hetzner.cloud/uses-proxyprotocol: true
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.87.12
IPs:                      10.43.87.12
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31683/TCP
Endpoints:                <none>
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30009/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     31336
Events:                   <none>

[vitobotta]

Also check the logs of the cloud controller manager. Perhaps there is something wrong with the API or something and the logs might help you see why the load balancer isn't being created as expected. Your configuration above looks ok I think.

[thoth2357]

for the cloud manager, i read it been installed automatically, so i checked with the kubectl get pods -n kube-system, not sure if i'm right but i get the below output..

NAME                       READY   STATUS    RESTARTS   AGE
coredns-597584b69b-x246g   0/1     Pending   0          8h

on further check to describe this pod and find out whats making it pending, i get the below output

Name:                 coredns-597584b69b-x246g
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      coredns
Node:                 <none>
Labels:               k8s-app=kube-dns
                      pod-template-hash=597584b69b
Annotations:          <none>
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-597584b69b
Containers:
  coredns:
    Image:       rancher/mirrored-coredns-coredns:1.9.4
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /etc/coredns/custom from custom-config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4d67n (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  custom-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns-custom
    Optional:  true
  kube-api-access-4d67n:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    Burstable
Node-Selectors:               beta.kubernetes.io/os=linux
Tolerations:                  CriticalAddonsOnly op=Exists
                              node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                              node-role.kubernetes.io/master:NoSchedule op=Exists
                              node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector k8s-app=kube-dns
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  32m (x99 over 8h)  default-scheduler  0/2 nodes are available: 2 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

[vitobotta]

That's coredns but I meant the cloud controller manager. Is it running?

[vitobotta]

Run kubectl -n kube-system get pod to see what's running in that namespace

[thoth2357]

below is the output and it seems the cloud controller manager isnt there?..

NAME                       READY   STATUS    RESTARTS   AGE
coredns-597584b69b-x246g   0/1     Pending   0          11h

[vitobotta]

That suggests that something went wrong during cluster creation and the cloud controller manager, as well as the CSI driver, the autoscaler and the Rancher system upgrade controller were not installed at all. Without the cloud controller manager load balancers cannot be provisioned.

Please try re-running the cluster create command and create a bin (on something like https://dpaste.org) with the output, then add a link to the bin here.

[oyewunmio]

I did try running the cluster create again, and everything works very fine as expected, thanks so much for lending a helping hand.
though, the load balancer status says Mixed, could there be a reason for that?

[vitobotta]

It may be normal, is nginx installed as daemonset or deployment? It's normal if it's a deployment.