Improvements to handling OpenShift Ingress

[sabre1041]

Functionality was added to enable the configuration of Ingress resources to support OpenShift to enable a streamlined way of integrating these charts to support their ecosystem.

These opinionated configurations work great for the simple use case. However, challenges have been experienced when customizing for more complex configurations and improvements can be made to support additional integrations.

Challenges

1. Annotations overwritten

Annotations that are set on Ingress resources are overwritten by opinionated configurations

2. Inability to fully define certain properties

Certain properties, such as path and pathPrefix cannot be defined

3. Opinionated TLS

OpenShift manages certain configurations via annotations along with applying certain defaults. For example, if route.openshift.io/termination annotation is set to edge, it is still possible to define the TLS certificate that will be applied to the ingress

4. Holistic alignment of Ingress Configurations

These charts include several Ingress resources exposing varying capabilities. The configurations and options are currently not uniform which causes challenges when looking to apply certain configurations

OpenShift Logic

OpenShift enables the "upconverting" of native Ingress resources to OpenShift Routes. The logic that is employed is located here

The following are several valid examples that can be defined within OpenShift to expose access to resources using Ingress resources

1. Passthrough Termination

No TLS termination occurs at the Ingress Controller as the backend pod is serving SSL certificates

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: spire
    meta.helm.sh/release-namespace: spire-mgmt
    route.openshift.io/termination: passthrough
  labels:
    app.kubernetes.io/instance: spire
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spiffe-oidc-discovery-provider
    app.kubernetes.io/version: 1.8.7
    helm.sh/chart: spiffe-oidc-discovery-provider-0.1.0
  name: spire-example-passthrough
  namespace: spire-server
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          service:
            name: spire-spiffe-oidc-discovery-provider
            port:
              number: 443
        pathType: ImplementationSpecific

2. Edge Termination using default Ingress Certificate

TLS termination occurs at OpenShift Ingress Router using the default certificate defined within the Ingress Controller. Traffic to the backend pod is unencrypted

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: spire
    meta.helm.sh/release-namespace: spire-mgmt
    route.openshift.io/termination: edge
  labels:
    app.kubernetes.io/instance: spire
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spiffe-oidc-discovery-provider
    app.kubernetes.io/version: 1.8.7
    helm.sh/chart: spiffe-oidc-discovery-provider-0.1.0
  name: spire-example-edge
  namespace: spire-server
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          service:
            name: spire-spiffe-oidc-discovery-provider
            port:
              number: 80
        path: /
        pathType: Prefix

3. Edge Termination using Specified Certificate

TLS termination occurs at OpenShift Ingress Router using the a provided certificate. Traffic to the backend pod is unencrypted

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: spire
    meta.helm.sh/release-namespace: spire-mgmt
    route.openshift.io/termination: edge
  labels:
    app.kubernetes.io/instance: spire
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spiffe-oidc-discovery-provider
    app.kubernetes.io/version: 1.8.7
    helm.sh/chart: spiffe-oidc-discovery-provider-0.1.0
  name: spire-example-edge
  namespace: spire-server
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          service:
            name: spire-spiffe-oidc-discovery-provider
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - <hostname>
    secretName: <frontend_cert_secret>

4. Reencrypt using default Ingress certificate and trusted destination certificate

TLS reencryption occurs using the default Ingress certificate and destination certificate that is trusted within the cluster. Most likely using a certificate generated via Service Serving Certificates

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: spire
    meta.helm.sh/release-namespace: spire-mgmt
    route.openshift.io/termination: reencrypt
  labels:
    app.kubernetes.io/instance: spire
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spiffe-oidc-discovery-provider
    app.kubernetes.io/version: 1.8.7
    helm.sh/chart: spiffe-oidc-discovery-provider-0.1.0
  name: spire-example-reencrypt
  namespace: spire-server
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          service:
            name: spire-spiffe-oidc-discovery-provider
            port:
              number: 443
        path: /
        pathType: Prefix

5. Reencrypt using provided ingress certificate and trusted destination certificate

TLS reencryption occurs using provided ingress certificate and destination certificate that is trusted within the cluster. Most likely using a certificate generated via Service Serving Certificates

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: spire
    meta.helm.sh/release-namespace: spire-mgmt
    route.openshift.io/termination: reencrypt
  labels:
    app.kubernetes.io/instance: spire
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spiffe-oidc-discovery-provider
    app.kubernetes.io/version: 1.8.7
    helm.sh/chart: spiffe-oidc-discovery-provider-0.1.0
  name: spire-example-reencrypt-frontend-dest
  namespace: spire-server
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          service:
            name: spire-spiffe-oidc-discovery-provider
            port:
              number: 443
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - <hostname>
    secretName: <frontend_cert_secret>

6. Reencrypt using provided Ingress certificate and provided destination certificate

TLS reencryption occurs using provided ingress certificate along with a provided certificate served by the pod

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: spire
    meta.helm.sh/release-namespace: spire-mgmt
    route.openshift.io/destination-ca-certificate-secret: <destination_cert_secret>
    route.openshift.io/termination: reencrypt
  labels:
    app.kubernetes.io/instance: spire
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spiffe-oidc-discovery-provider
    app.kubernetes.io/version: 1.8.7
    helm.sh/chart: spiffe-oidc-discovery-provider-0.1.0
  name: spire-example-reencrypt-frontend-dest
  namespace: spire-server
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          service:
            name: spire-spiffe-oidc-discovery-provider
            port:
              number: 443
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - <hostname>
    secretName: <frontend_cert_secret>

7. Reencrypt using default Ingress certificate and provided destination certificate

TLS reencryption occurs using provided ingress certificate along with a provided certificate served by the pod

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: spire
    meta.helm.sh/release-namespace: spire-mgmt
    route.openshift.io/destination-ca-certificate-secret: <destination_cert_secret>
    route.openshift.io/termination: reencrypt
  labels:
    app.kubernetes.io/instance: spire
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spiffe-oidc-discovery-provider
    app.kubernetes.io/version: 1.8.7
    helm.sh/chart: spiffe-oidc-discovery-provider-0.1.0
  name: spire-example-reencrypt-frontend-dest
  namespace: spire-server
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          service:
            name: spire-spiffe-oidc-discovery-provider
            port:
              number: 443
        path: /
        pathType: Prefix

[kfox1111]

Thanks for the good overview of the issues you've hit.

To help prioritize solutions, are there any of these that are more of a problem for you then others?

I think there are probably some cases where supporting every permutation of options is undesirable. For example, spire-server's port must be passed through. Configuring it for retermination will only produce an unusable install. We need to identify the gaps in configuration options along with the softwares constraints.

[faisal-memon]

@sabre1041 Any thoughts on the questions by @kfox1111 ?