tls / ingress

[chadleywilson]

Struggling with my onprem BareMetal k8s cluster
Kubectl 1.28.11
Calico v3.28.0
MetallB v0.14.5
I have setup nginx as follows
Version: helm.sh/chart: ingress-nginx-4.11.0

helm install ingress-nginx ingress-nginx/ingress-nginx --set controller.replicaCount=2 --set controller.service.type=LoadBalancer --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux

I first grabbed a copy of the values.yaml from git.

here is my ingress section:

  ingress:
    enabled: true
    hosts:
      # Keep 'localhost' host only if you want to access Dashboard using 'kubectl port-forward ...' on:
      # https://localhost:8443
      #- localhost
      - dashboard-ing.mshome.net
    ingressClassName: internal-nginx
    # Use only if your ingress controllers support default ingress classes.
    # If set to true ingressClassName will be ignored and not added to the Ingress resources.
    # It should fall back to using IngressClass marked as the default.
    useDefaultIngressClass: false
    # This will append our Ingress with annotations required by our default configuration.
    #    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    #    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    #    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    useDefaultAnnotations: true
    pathType: ImplementationSpecific
    # If path is not the default (/), rewrite-target annotation will be added to the Ingress.
    # It allows serving Kubernetes Dashboard on a sub-path. Make sure that the configured path
    # does not conflict with gateway route configuration.
    path: /
    issuer:
      name: selfsigned
      # Scope determines what kind of issuer annotation will be used on ingress resource
      # - default - adds 'cert-manager.io/issuer'
      # - cluster - adds 'cert-manager.io/cluster-issuer'
      # - disabled - disables cert-manager annotations
      scope: default
    tls:
      enabled: true
      # If provided it will override autogenerated secret name
      secretName: "kubernetes-dashboard-certs"
    labels: {}
    annotations: {}

In the test of the ingress I am still seeing the "Kubernetes Ingress Controller Fake Certificate" despite having secret declared

First question:

# If provided it will override autogenerated secret name
      secretName: "kubernetes-dashboard-certs"

Which namespace should this be in
I have tried default / kubernetes-dashboard / kube-system and now I am out of ideas?
its ignoring it and putting the fake cert there instead

I think sub question to that question is:
If I navigate to https://dashboard-ing.mshome.net using a web browser. I actrually get "not secure" message, if I choose to continue anyway or curl -k on the command line,
I hit a 404 not found - nginx

This tells me the ingress is working. But what needs fixing adding or tweaking is beyond my skills!

Second Question:
If I prefer to use a LoadBalancer IP instead of ingress to expose the dashboard, how would I define the TLS in the values file? Or do it post deployment?

[chadleywilson]

I have solved the ingress issue

It was an oversight in the values file. I should have checked what the helm chart called my ingressclass before editing the values file.

ubroot@master-01:~$ kubectl get ingressclass
NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       70m

So based on this, the wrong name was in the values.yaml file and all I needed to do was change the line here:

ingressClassName: internal-nginx

to

ingressClassName: nginx

The ingress is now working with my cert.

For people who are not sure what to do. Here is small guide. I hope it helps.
As this is my test environment my own wild card is not valid, but that's ok as long as I see my invalid cert instead of the fake cert, I know it works and I can fix my own cert to make it valid in production.

Note: This is purely a test environment that I using to work things out and now also demonstrating with.

this is my test cluster setup
Kubectl 1.28.11
Calico v3.28.0
MetallB v0.14.5

ubroot@master-01:~$ kubectl get nodes
NAME        STATUS   ROLES           AGE     VERSION
master-01   Ready    control-plane   4h33m   v1.28.11
worker-01   Ready    <none>          4h28m   v1.28.11
worker-02   Ready    <none>          4h28m   v1.28.11
worker-03   Ready    <none>          4h28m   v1.28.11

from this point you need to start with creating an ingressclass - note the helm chart here creates the ingressclass as nginx

helm install ingress-nginx ingress-nginx/ingress-nginx --set controller.replicaCount=2 --set controller.service.type=LoadBalancer --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux

create the secret from your own certs

kubectl -n kubernetes-dashboard create secret tls kubernetes-dashboard-certs --key ./certs/tls.key --cert ./certs/tls.crt

Next grab the values.yaml

change the following lines in the values file:

  clusterName: "Home Cluster"

To whatever you fancy.

Then in the ingress section around line 94 at the time of writing..

change enabled to
enabled: true

hash out #
- localhost

Add a line directly below local host with the host name in FQDN format:
- dashboard-ing.mshome.net

Next - on the secretName line, type your secret name between the quotes:

secretName: "kubernetes-dashboard-certs"

so the ingress section will look similar to this:

  ingress:
    enabled: true
    hosts:
      # Keep 'localhost' host only if you want to access Dashboard using 'kubectl port-forward ...' on:
      # https://localhost:8443
      #- localhost
      - dashboard-ing.mshome.net
    ingressClassName: internal-nginx
    # Use only if your ingress controllers support default ingress classes.
    # If set to true ingressClassName will be ignored and not added to the Ingress resources.
    # It should fall back to using IngressClass marked as the default.
    useDefaultIngressClass: false
    # This will append our Ingress with annotations required by our default configuration.
    #    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    #    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    #    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    useDefaultAnnotations: true
    pathType: ImplementationSpecific
    # If path is not the default (/), rewrite-target annotation will be added to the Ingress.
    # It allows serving Kubernetes Dashboard on a sub-path. Make sure that the configured path
    # does not conflict with gateway route configuration.
    path: /
    issuer:
      name: selfsigned
      # Scope determines what kind of issuer annotation will be used on ingress resource
      # - default - adds 'cert-manager.io/issuer'
      # - cluster - adds 'cert-manager.io/cluster-issuer'
      # - disabled - disables cert-manager annotations
      scope: default
    tls:
      enabled: true
      # If provided it will override autogenerated secret name
      secretName: "kubernetes-dashboard-certs"
    labels: {}
    annotations: {}

save your values file and simply re-run the helm chart

helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard --values values.yaml

You should see the IP address assigned via nginx ingress here:

ubroot@master-01:~$ kubectl -n kubernetes-dashboard get ingress
NAME                   CLASS   HOSTS                      ADDRESS          PORTS     AGE
kubernetes-dashboard   nginx   dashboard-ing.mshome.net   192.168.25.235   80, 443   133m

If you don't see this ingress in the kubernetes-dashboard namespace, make sure the name of your ingressclass correct in the yaml file.
Also if you did not setup your ingress class with the --set controller.service.type=LoadBalancer it won't work in this setup.

Now you can check and test that it works. (my cert is invalid so I still need the -k )

ubroot@master-01:~$ curl -k https://dashboard-ing.mshome.net
<!--
Copyright 2017 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

And you can see the certificate here: ( you can see the subject is *.mshome.net which is the self signed, albeit invalid cert I created to test that this works, if you see the fake cert it is not working)

ubroot@master-01:~$ curl --insecure -vvI https://dashboard-ing.mshome.net 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=JO; CN=*.mshome.net
*  start date: Jul  8 12:20:07 2024 GMT
*  expire date: Jul  6 12:20:07 2034 GMT
*  issuer: C=JO; CN=*.mshome.net
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha1WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://dashboard-ing.mshome.net/
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: dashboard-ing.mshome.net]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection #0 to host dashboard-ing.mshome.net left intact
*