Operator & CRD upgrade test harness

[pmalek]

Problem statement

Currently operator has 1 E2E tests which tests the upgrade path

gateway-operator/test/e2e/upgrade_test.go

Lines 30 to 40 in ec5a314

	// TestDeployAndUpgradeFromLatestTagToOverride tests an upgrade path from latest
	// released tag to an override version provided by environment variables.
	// The reason we need and override is to not test against latest main build because
	// that would not provide accurate test results because when one changes code in
	// a PR, the intention is to test this very code through CI, not a prior version
	// without this change.
	//
	// CI already has this override provided in
	// https://github.com/Kong/gateway-operator/blob/0f3b726c33/.github/workflows/tests.yaml#L180-L190
	// so anyone pushing changes can expect those changes to be tested in this test.
	func TestDeployAndUpgradeFromLatestTagToOverride(t *testing.T) {

.

This is not comprehensive enough since it only tests "latest tag" to current version upgrade path and it doesn't test CRD upgrades (is installing new version of CRDs safe and non-breaking?).

This issue tracks the effort of adding a new test suite which will test operator upgrades.
For example it could test upgrades from a known, predefined list of tags to a a known, predefined list of tags. Additionally this suite would also test installing new CRD versions so that those do not bring breakage.

Acceptance criteria

• New testing harness is added which tests upgrades from known operators tags to known operator tags, using a way of deployment closest to the one used by the end users.
• As a developer I can be sure that the changes I introduce will not break during upgrades from an older version to the version I'm developping.

[pmalek]

• ci: test samples on CI #256 introduces samples testing which applies and deletes sample manifests against a kind cluster on CI
• chore: update release template #257 adds mention of adding a test case if the released version if a new minor/major
• https://github.com/Kong/gateway-operator/blob/9f33d27ab875b91e50d7e750b45a293c1395da2d/test/e2e/test_upgrade.go already contains tests cases against the latest release tags, against latest and nightly and against latest tag vs tip of the tested branch

Merging of those PRs will conclude this issue. Further development of upgrade tests using helm can be tracked in separate issues and PRs.