Skip to content

Commit

Permalink
Merge pull request crossplane#4343 from phisco/e2e-alpha-all-2
Browse files Browse the repository at this point in the history
tests(e2e): introduce environment wrapper and test suites
  • Loading branch information
phisco authored Aug 11, 2023
2 parents e312b6e + c64c679 commit 45f7cbc
Show file tree
Hide file tree
Showing 14 changed files with 673 additions and 203 deletions.
5 changes: 3 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -239,8 +239,9 @@ jobs:
needs: detect-noop
if: needs.detect-noop.outputs.noop != 'true'
strategy:
fail-fast: false
matrix:
area: [lifecycle, pkg, apiextensions, xfn]
test-suite: [base, composition-webhook-schema-validation, composition-functions]

steps:
- name: Setup QEMU
Expand Down Expand Up @@ -297,7 +298,7 @@ jobs:
BUILD_ARGS: "--load"

- name: Run E2E Tests
run: make e2e E2E_TEST_FLAGS="-test.v -labels area=${{ matrix.area }}"
run: make e2e E2E_TEST_FLAGS="-test.v --test-suite ${{ matrix.test-suite }}"

publish-artifacts:
runs-on: ubuntu-22.04
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ require (
kernel.org/pub/linux/libs/security/libcap/cap v1.2.69
sigs.k8s.io/controller-runtime v0.15.0
sigs.k8s.io/controller-tools v0.12.1
sigs.k8s.io/e2e-framework v0.2.0
sigs.k8s.io/e2e-framework v0.2.1-0.20230716064705-49e8554b536f
sigs.k8s.io/kind v0.20.0
sigs.k8s.io/yaml v1.3.0
)
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -969,8 +969,8 @@ sigs.k8s.io/controller-runtime v0.15.0 h1:ML+5Adt3qZnMSYxZ7gAverBLNPSMQEibtzAgp0
sigs.k8s.io/controller-runtime v0.15.0/go.mod h1:7ngYvp1MLT+9GeZ+6lH3LOlcHkp/+tzA/fmHa4iq9kk=
sigs.k8s.io/controller-tools v0.12.1 h1:GyQqxzH5wksa4n3YDIJdJJOopztR5VDM+7qsyg5yE4U=
sigs.k8s.io/controller-tools v0.12.1/go.mod h1:rXlpTfFHZMpZA8aGq9ejArgZiieHd+fkk/fTatY8A2M=
sigs.k8s.io/e2e-framework v0.2.0 h1:gD6AWWAHFcHibI69E9TgkNFhh0mVwWtRCHy2RU057jQ=
sigs.k8s.io/e2e-framework v0.2.0/go.mod h1:E6JXj/V4PIlb95jsn2WrNKG+Shb45xaaI7C0+BH4PL8=
sigs.k8s.io/e2e-framework v0.2.1-0.20230716064705-49e8554b536f h1:BN6JOYAOMYCC8FPSfALNFvH9f6Sf4k+fM8OwuZfHL4g=
sigs.k8s.io/e2e-framework v0.2.1-0.20230716064705-49e8554b536f/go.mod h1:7k84BFZzTqYNO1qxF4gDmQRxEoSw62lSBDXSAf43e2A=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0=
sigs.k8s.io/kind v0.20.0 h1:f0sc3v9mQbGnjBUaqSFST1dwIuiikKVGgoTwpoP33a8=
Expand Down
68 changes: 59 additions & 9 deletions test/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,26 +45,34 @@ E2E_TEST_FLAGS="-test.v -test.failfast -destroy-kind-cluster=false"

# Use an existing Kubernetes cluster. Note that the E2E tests can't deploy your
# local build of Crossplane in this scenario, so you'll have to do it yourself.
E2E_TEST_FLAGS="-create-kind-cluster=false -destroy-kind-cluster=false -kubeconfig=$HOME/.kube/config"
E2E_TEST_FLAGS="-create-kind-cluster=false -destroy-kind-cluster=false -kubeconfig=$HOME/.kube/config" make e2e

# Run the CrossplaneUpgrade feature, against an existing kind cluster named
# "kind" (or creating it if it doesn't exist), # without installing Crossplane
# first, as the feature expects the cluster to be empty, but still loading the
# images to # it. Setting the tests to fail fast and not destroying the cluster
# images to it. Setting the tests to fail fast and not destroying the cluster
# afterward in order to allow debugging it.
E2E_TEST_FLAGS="-test.v -v 4 -test.failfast \
-destroy-kind-cluster=false \
-kind-cluster-name=kind \
-install-crossplane=false \
-feature=CrossplaneUpgrade" make e2e

# Run the all tests not installing or upgrading Crossplane against the currently
# Run all the tests not installing or upgrading Crossplane against the currently
# selected cluster where Crossplane has already been installed.
E2E_TEST_FLAGS="-test.v -v 4 -test.failfast \
-kubeconfig=$HOME/.kube/config \
-skip-labels modify-crossplane-installation=true \
-create-kind-cluster=false \
-install-crossplane=false" make go.build e2e-run-tests

# Run the composition-webhook-schema-validation suite of tests, which will
# result in all tests marked as "test-suite=base" or
# "test-suite=composition-webhook-schema-validation" being run against a kind
# cluster with Crossplane installed with composition-webhook-schema-validation
# enabled
E2E_TEST_FLAGS="-test.v -v 4 -test.failfast \
-test-suite=composition-webhook-schema-validation " make e2e
```

## Test Parallelism
Expand All @@ -76,9 +84,50 @@ and less error-prone to write tests when you don't have to worry about one test
potentially conflicting with another - for example by installing the same
provider another test would install.

In order to achieve some parallelism at the CI level all tests are labelled with
an area (e.g. `pkg`, `install`, `apiextensions`, etc). The [CI GitHub workflow]
uses a matrix strategy to invoke each area as its own job, running in parallel.
The [CI GitHub workflow] uses a matrix strategy to run multiple jobs in parallel,
each running a test suite, see the dedicated section for more details.

We are currently splitting the tests to be able to run all basic tests against
the default installation of Crossplane, and for each alpha feature covered we
run all basic tests plus the feature specific ones against a Crossplane
installation with the feature enabled.

In the future we could think of improving parallelism by, for example, adding
the area to the matrix or spinning multiple kind clusters for each job.

## Test Suite

In order to be able to run specific subsets of tests, we introduced the concept
of test suites. To run a specific test suite use the `-test-suite` flag.

A test suite is currently defined by:
- A key to be used as value of the `-test-suite` flag.
- A set of labels that will be used to filter the tests to run, which will be
added to the user-provided ones, preserving the latter ones in case of key
conflicts.
- A list of Helm install options defining the initial Crossplane installation
for the given test suite, which will be used to install Crossplane before
running the tests. E.g. adding flags to enable alpha features and feature
specific flags.
- A list of additional setup steps to be run before installing Crossplane and
running the tests. E.g. Loading additional images into the cluster.
- Whether the suite should include the default suite or not, meaning that
install options will be added to the default ones if not explicitly specified
not to do so.

Test suites enable use cases such as installing Crossplane with a specific
alpha feature enabled and running all the basic tests, plus the ones specific to
that feature, to make sure the feature is not breaking any default behavior.

In case a test needs a specific Crossplane configuration, it must still take
care of upgrading the installation to the desired configuration, but should then
use `environment.GetSelectedSuiteInstallOpts` to retrieve at runtime the baseline
installation options to be sure to restore the previous state. This allows tests
to run against any suite if needed.

Test suites can be combined with labels to run a subset of tests, e.g.
splitting by area, or with the usual Go `-run <regexp>` flag to run only
specific tests, in such case, suite labels will be ignored altogether.

## Adding a Test

Expand Down Expand Up @@ -154,12 +203,13 @@ func TestSomeFeature(t *testing.T) {
// ... other variables or constants ...

environment.Test(t,
features.New("ConfigurationWithDependency").
features.New(t.Name()).
WithLabel(LabelArea, ...).
WithLabel(LabelSize, ...).
// ...
WithLabel(config.LabelTestSuite, config.TestSuiteDefault).
// ...
WithSetup("ReadyPrerequisites", ... ).
// ... other setup steps ...
// ... other setup steps ...
Assess("DoSomething", ... ).
Assess("SomethingElseIsInSomeState", ... ).
// ... other assess steps ...
Expand Down
61 changes: 5 additions & 56 deletions test/e2e/apiextensions_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,11 @@ import (
"time"

"sigs.k8s.io/e2e-framework/pkg/features"
"sigs.k8s.io/e2e-framework/third_party/helm"

xpv1 "github.com/crossplane/crossplane-runtime/apis/common/v1"

apiextensionsv1 "github.com/crossplane/crossplane/apis/apiextensions/v1"
pkgv1 "github.com/crossplane/crossplane/apis/pkg/v1"
"github.com/crossplane/crossplane/test/e2e/config"
"github.com/crossplane/crossplane/test/e2e/funcs"
)

Expand All @@ -42,9 +41,10 @@ func TestCompositionMinimal(t *testing.T) {
manifests := "test/e2e/manifests/apiextensions/composition/minimal"

environment.Test(t,
features.New("CompositionMinimal").
features.New(t.Name()).
WithLabel(LabelArea, LabelAreaAPIExtensions).
WithLabel(LabelSize, LabelSizeSmall).
WithLabel(config.LabelTestSuite, config.TestSuiteDefault).
WithSetup("PrerequisitesAreCreated", funcs.AllOf(
funcs.ApplyResources(FieldManager, manifests, "setup/*.yaml"),
funcs.ResourcesCreatedWithin(30*time.Second, manifests, "setup/*.yaml"),
Expand Down Expand Up @@ -75,9 +75,10 @@ func TestCompositionPatchAndTransform(t *testing.T) {

manifests := "test/e2e/manifests/apiextensions/composition/patch-and-transform"
environment.Test(t,
features.New("CompositionPatchAndTransform").
features.New(t.Name()).
WithLabel(LabelArea, LabelAreaAPIExtensions).
WithLabel(LabelSize, LabelSizeSmall).
WithLabel(config.LabelTestSuite, config.TestSuiteDefault).
WithSetup("CreatePrerequisites", funcs.AllOf(
funcs.ApplyResources(FieldManager, manifests, "setup/*.yaml"),
funcs.ResourcesCreatedWithin(30*time.Second, manifests, "setup/*.yaml"),
Expand All @@ -104,55 +105,3 @@ func TestCompositionPatchAndTransform(t *testing.T) {
)

}

func TestCompositionValidation(t *testing.T) {
manifests := "test/e2e/manifests/apiextensions/composition/validation"

cases := features.Table{
{
// A valid Composition should be created when validated in strict mode.
Name: "ValidCompositionIsAccepted",
Assessment: funcs.AllOf(
funcs.ApplyResources(FieldManager, manifests, "composition-valid.yaml"),
funcs.ResourcesCreatedWithin(30*time.Second, manifests, "composition-valid.yaml"),
),
},
{
// An invalid Composition should be rejected when validated in strict mode.
Name: "InvalidCompositionIsRejected",
Assessment: funcs.ResourcesFailToApply(FieldManager, manifests, "composition-invalid.yaml"),
},
}
environment.Test(t,
cases.Build("CompositionValidation").
WithLabel(LabelStage, LabelStageAlpha).
WithLabel(LabelArea, LabelAreaAPIExtensions).
WithLabel(LabelSize, LabelSizeSmall).
WithLabel(LabelModifyCrossplaneInstallation, LabelModifyCrossplaneInstallationTrue).
// Enable our feature flag.
WithSetup("EnableAlphaCompositionValidation", funcs.AllOf(
funcs.AsFeaturesFunc(funcs.HelmUpgrade(HelmOptions(helm.WithArgs("--set args={--debug,--enable-composition-webhook-schema-validation}"))...)),
funcs.ReadyToTestWithin(1*time.Minute, namespace),
)).
WithSetup("CreatePrerequisites", funcs.AllOf(
funcs.ApplyResources(FieldManager, manifests, "setup/*.yaml"),
funcs.ResourcesCreatedWithin(30*time.Second, manifests, "setup/*.yaml"),
funcs.ResourcesHaveConditionWithin(1*time.Minute, manifests, "setup/definition.yaml", apiextensionsv1.WatchingComposite()),
funcs.ResourcesHaveConditionWithin(1*time.Minute, manifests, "setup/provider.yaml", pkgv1.Healthy(), pkgv1.Active()),
)).
WithTeardown("DeleteValidComposition", funcs.AllOf(
funcs.DeleteResources(manifests, "*-valid.yaml"),
funcs.ResourcesDeletedWithin(30*time.Second, manifests, "*-valid.yaml"),
)).
WithTeardown("DeletePrerequisites", funcs.AllOf(
funcs.DeleteResources(manifests, "setup/*.yaml"),
funcs.ResourcesDeletedWithin(3*time.Minute, manifests, "setup/*.yaml"),
)).
// Disable our feature flag.
WithTeardown("DisableAlphaCompositionValidation", funcs.AllOf(
funcs.AsFeaturesFunc(funcs.HelmUpgrade(HelmOptions()...)),
funcs.ReadyToTestWithin(1*time.Minute, namespace),
)).
Feature(),
)
}
86 changes: 86 additions & 0 deletions test/e2e/comp_schema_validation_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
package e2e

import (
"testing"
"time"

"sigs.k8s.io/e2e-framework/pkg/features"
"sigs.k8s.io/e2e-framework/third_party/helm"

apiextensionsv1 "github.com/crossplane/crossplane/apis/apiextensions/v1"
pkgv1 "github.com/crossplane/crossplane/apis/pkg/v1"
"github.com/crossplane/crossplane/test/e2e/config"
"github.com/crossplane/crossplane/test/e2e/funcs"
)

const (
// SuiteCompositionWebhookSchemaValidation is the value for the
// config.LabelTestSuite label to be assigned to tests that should be part
// of the Composition Webhook Schema Validation test suite.
SuiteCompositionWebhookSchemaValidation = "composition-webhook-schema-validation"
)

func init() {
environment.AddTestSuite(SuiteCompositionWebhookSchemaValidation,
config.WithHelmInstallOpts(
helm.WithArgs("--set args={--debug,--enable-composition-webhook-schema-validation}"),
),
config.WithLabelsToSelect(features.Labels{
config.LabelTestSuite: []string{SuiteCompositionWebhookSchemaValidation, config.TestSuiteDefault},
}),
)
}

func TestCompositionValidation(t *testing.T) {
manifests := "test/e2e/manifests/apiextensions/composition/validation"

cases := features.Table{
{
// A valid Composition should be created when validated in strict mode.
Name: "ValidCompositionIsAcceptedStrictMode",
Description: "A valid Composition should be created when validated in strict mode.",
Assessment: funcs.AllOf(
funcs.ApplyResources(FieldManager, manifests, "composition-valid.yaml"),
funcs.ResourcesCreatedWithin(30*time.Second, manifests, "composition-valid.yaml"),
),
},
{
// An invalid Composition should be rejected when validated in strict mode.
Name: "InvalidCompositionIsRejectedStrictMode",
Assessment: funcs.ResourcesFailToApply(FieldManager, manifests, "composition-invalid.yaml"),
},
}
environment.Test(t,
cases.Build(t.Name()).
WithLabel(LabelStage, LabelStageAlpha).
WithLabel(LabelArea, LabelAreaAPIExtensions).
WithLabel(LabelSize, LabelSizeSmall).
WithLabel(LabelModifyCrossplaneInstallation, LabelModifyCrossplaneInstallationTrue).
WithLabel(config.LabelTestSuite, SuiteCompositionWebhookSchemaValidation).
// Enable our feature flag.
WithSetup("EnableAlphaCompositionValidation", funcs.AllOf(
funcs.AsFeaturesFunc(environment.HelmUpgradeCrossplaneToSuite(SuiteCompositionWebhookSchemaValidation)),
funcs.ReadyToTestWithin(1*time.Minute, namespace),
)).
WithSetup("CreatePrerequisites", funcs.AllOf(
funcs.ApplyResources(FieldManager, manifests, "setup/*.yaml"),
funcs.ResourcesCreatedWithin(30*time.Second, manifests, "setup/*.yaml"),
funcs.ResourcesHaveConditionWithin(1*time.Minute, manifests, "setup/definition.yaml", apiextensionsv1.WatchingComposite()),
funcs.ResourcesHaveConditionWithin(1*time.Minute, manifests, "setup/provider.yaml", pkgv1.Healthy(), pkgv1.Active()),
)).
WithTeardown("DeleteValidComposition", funcs.AllOf(
funcs.DeleteResources(manifests, "*-valid.yaml"),
funcs.ResourcesDeletedWithin(30*time.Second, manifests, "*-valid.yaml"),
)).
WithTeardown("DeletePrerequisites", funcs.AllOf(
funcs.DeleteResources(manifests, "setup/*.yaml"),
funcs.ResourcesDeletedWithin(3*time.Minute, manifests, "setup/*.yaml"),
)).
// Disable our feature flag.
WithTeardown("DisableAlphaCompositionValidation", funcs.AllOf(
funcs.AsFeaturesFunc(environment.HelmUpgradeCrossplaneToBase()),
funcs.ReadyToTestWithin(1*time.Minute, namespace),
)).
Feature(),
)
}
Loading

0 comments on commit 45f7cbc

Please sign in to comment.