ControlPlane cluster resource do not allow migration from <1.2 (owner ref link) to 1.3+ (label based owner ref)

[pmalek]

Problem statement

#259 introduced a new way to link ControlPlane's cluster wide resources to the managing ControlPlane via labels.

This approach works but it misses the migration approach:

• users install ControlPlanes using KGO 1.2 (or older)
• KGO creates owner reference bindings between ControlPlane's cluster wide resources to the managing ControlPlane
• users upgrade KGO to 1.3
• KGO looks up the cluster wide resources via the new, label based approach, and misses the old ones that are already created, and thus tries to create the already existing resources

Logs:

manager {"level":"debug","ts":"2024-06-24T09:47:57Z","logger":"controlplane","msg":"admission webhook enabled, enforcing admission webhook resources","controller":"controlplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"ControlPlane","ControlPlane":{"name":"kong-2r5z8","namespace":"default"},"namespace":"default","name":"kong-2r5z8","reconcileID":"44661c34-2d58-418f-afbf-cf2807687d6e","namespace":"default","name":"kong-2r5z8"}
manager {"level":"Level(-2)","ts":"2024-06-24T09:47:57Z","logger":"controlplane","msg":"ensuring admission webhook service","controller":"controlplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"ControlPlane","ControlPlane":{"name":"kong-2r5z8","namespace":"default"},"namespace":"default","name":"kong-2r5z8","reconcileID":"44661c34-2d58-418f-afbf-cf2807687d6e","namespace":"default","name":"kong-2r5z8"}
manager {"level":"Level(-2)","ts":"2024-06-24T09:47:57Z","logger":"controlplane","msg":"ensuring admission webhook certificate","controller":"controlplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"ControlPlane","ControlPlane":{"name":"kong-2r5z8","namespace":"default"},"namespace":"default","name":"kong-2r5z8","reconcileID":"44661c34-2d58-418f-afbf-cf2807687d6e","namespace":"default","name":"kong-2r5z8"}
manager {"level":"Level(-2)","ts":"2024-06-24T09:47:57Z","logger":"controlplane","msg":"ensuring admission webhook configuration","controller":"controlplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"ControlPlane","ControlPlane":{"name":"kong-2r5z8","namespace":"default"},"namespace":"default","name":"kong-2r5z8","reconcileID":"44661c34-2d58-418f-afbf-cf2807687d6e","namespace":"default","name":"kong-2r5z8"}
manager {"level":"error","ts":"2024-06-24T09:47:57Z","msg":"Reconciler error","controller":"controlplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"ControlPlane","ControlPlane":{"name":"kong-2r5z8","namespace":"default"},"namespace":"default","name":"kong-2r5z8","reconcileID":"44661c34-2d58-418f-afbf-cf2807687d6e","error":"failed to ensure webhook resources: validatingwebhookconfigurations.admissionregistration.k8s.io \"kong-2r5z8\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

This causes e2e upgrade tests to fail.

Steps to reproduce:

• Install KGO 1.2 using a helm chart
• Apply Gateway manifest e.g. https://github.com/Kong/gateway-operator/blob/fa93ae9b150388d18fd4d6e01bcceb77736aa57a/config/samples/gateway-with-gatewayconfiguration.yaml
• Update KGO to 1.3 (or nightly at this point)
• Observe the above mentioned logs and that Gateway does not get Programmed

Anything else ?

Old labels

	gateway-operator.konghq.com/managed-by: controlplane
	konghq.com/gateway-operator: controlplane

New labels

	gateway-operator.konghq.com/managed-by: controlplane
	gateway-operator.konghq.com/managed-by-name: kong-ncv7m
	gateway-operator.konghq.com/managed-by-namespace: default
	konghq.com/gateway-operator: controlplane

Since we're using the new label set for both the generation and existing resources lookup, existing objects (created with old labels set) are not matched and hence a create operation is performed which should be an update to migrate to the new set of labels.

Acceptance criteria

• When KGO is upgraded from 1.2 to 1.3 ControlPlane managed cluster wide resources have their labels properly migrated (updated) and are properly looked up
• E2E upgrade test assertion is added to https://github.com/Kong/gateway-operator/blob/0781df268b2b75a70895c46c6bf5eed5e08f8887/test/e2e/test_helm_install_upgrade.go and ensures that the above happens.

[pmalek]

#369 is an attempt to fix the issue but it fails to include the identifying criteria to list the managed cluster owned resources (e.g. in https://github.com/Kong/gateway-operator/pull/369/files#diff-279d78c12a72e62c17d2b732438c7bf2554ca9ebbae2b462d7c278d06b5e18a1R81) i.e. the list of returned cluster owned resources does not include 1 but possibly more entries.