tests/e2e: Add auth registry libvirt tests #1932

stevenhorsman · 2024-07-18T12:36:10Z

Run the authenticated regsitry tests for libvirt cloud provider

Depends on #1931

stevenhorsman · 2024-07-25T17:32:33Z

There is some value in this PR in seeing if authenticated image pull is working at all, but I think there are issues with the invalid credentials test. In the course of trying to fix it up I think the case is that for this test, we pass through the correct pull secret into the peer pod (I think to make this incorrect we'd need to modify the kustomize/re-create the caa, so that might not even be worth testing?) and just give it an invalid imagePullSecret, which just checks that it errors on the host pull, not the guest and is the opposite way around to want we are trying to test, so I think there is still a bunch more to do along with a bunch of refactoring I've been working on. It might be that we just drop that test entirely and keep the valid creds works and no guest creds (but still correct image pull test) fails, if that even is sensible.

stevenhorsman · 2024-07-25T17:50:34Z

The tests are all failing for libvirt anyway:

=== RUN   TestLibvirtCreatePeerPodWithAuthenticatedImageWithValidCredentials
=== RUN   TestLibvirtCreatePeerPodWithAuthenticatedImageWithValidCredentials/ValidAuthImagePeerPod_test
    assessment_runner.go:261: timed out waiting for the condition
--- FAIL: TestLibvirtCreatePeerPodWithAuthenticatedImageWithValidCredentials (600.12s)
    --- FAIL: TestLibvirtCreatePeerPodWithAuthenticatedImageWithValidCredentials/ValidAuthImagePeerPod_test (600.12s)
=== RUN   TestLibvirtCreatePeerPodWithAuthenticatedImageWithInvalidCredentials
=== RUN   TestLibvirtCreatePeerPodWithAuthenticatedImageWithInvalidCredentials/InvalidAuthImagePeerPod_test
    assessment_runner.go:201: secrets "auth-json-secret-default" already exists
--- FAIL: TestLibvirtCreatePeerPodWithAuthenticatedImageWithInvalidCredentials (0.04s)
    --- FAIL: TestLibvirtCreatePeerPodWithAuthenticatedImageWithInvalidCredentials/InvalidAuthImagePeerPod_test (0.04s)
=== RUN   TestLibvirtCreatePeerPodWithAuthenticatedImageWithoutCredentials
=== RUN   TestLibvirtCreatePeerPodWithAuthenticatedImageWithoutCredentials/InvalidAuthImagePeerPod_test
    assessment_runner.go:201: secrets "auth-json-secret-default" already exists
--- FAIL: TestLibvirtCreatePeerPodWithAuthenticatedImageWithoutCredentials (0.02s)
    --- FAIL: TestLibvirtCreatePeerPodWithAuthenticatedImageWithoutCredentials/InvalidAuthImagePeerPod_test (0.02s)

stevenhorsman · 2024-08-12T17:18:26Z

The libvirt provider should be tested in this PR, we don't have a docker provider e2e run yet, so the manual test results is:

=== RUN   TestDockerCreatePeerPodWithAuthenticatedImageWithoutCredentials
=== RUN   TestDockerCreatePeerPodWithAuthenticatedImageWithoutCredentials/InvalidAuthImagePeerPod_test
time="2024-08-12T10:15:00-07:00" level=info msg="ServiceAccount default updated successfully."
    assessment_runner.go:245: Deleting pre-existing auth-json-secret-default...
    assessment_runner.go:249: Creating empty auth-json-secret-default...
=== RUN   TestDockerCreatePeerPodWithAuthenticatedImageWithoutCredentials/InvalidAuthImagePeerPod_test/Peer_pod_with_Authenticated_Image_without_Credentials_has_been_created
    assessment_runner.go:379: podEvent: &{EventType:Failed EventDescription:Failed to pull image "quay.io/kata-containers/confidential-containers-auth:test": failed to pull and unpack image "quay.io/kata-containers/confidential-containers-auth:test": failed to resolve reference "quay.io/kata-containers/confidential-containers-auth:test": unexpected status from HEAD request to https://quay.io/v2/kata-containers/confidential-containers-auth/manifests/test: 401 UNAUTHORIZED EventReason:}
    assessment_runner.go:381: Output Log from Pod: &{Failed Failed to pull image "quay.io/kata-containers/confidential-containers-auth:test": failed to pull and unpack image "quay.io/kata-containers/confidential-containers-auth:test": failed to resolve reference "quay.io/kata-containers/confidential-containers-auth:test": unexpected status from HEAD request to https://quay.io/v2/kata-containers/confidential-containers-auth/manifests/test: 401 UNAUTHORIZED }
=== NAME  TestDockerCreatePeerPodWithAuthenticatedImageWithoutCredentials/InvalidAuthImagePeerPod_test
    assessment_runner.go:617: Deleting pod authenticated-image-without-creds-4104531429-pod...
    assessment_runner.go:624: Pod authenticated-image-without-creds-4104531429-pod has been successfully deleted within 60s
--- PASS: TestDockerCreatePeerPodWithAuthenticatedImageWithoutCredentials (10.12s)
    --- PASS: TestDockerCreatePeerPodWithAuthenticatedImageWithoutCredentials/InvalidAuthImagePeerPod_test (10.12s)
        --- PASS: TestDockerCreatePeerPodWithAuthenticatedImageWithoutCredentials/InvalidAuthImagePeerPod_test/Peer_pod_with_Authenticated_Image_without_Credentials_has_been_created (0.01s)
=== RUN   TestDockerCreatePeerPodWithAuthenticatedImageWithValidCredentials
=== RUN   TestDockerCreatePeerPodWithAuthenticatedImageWithValidCredentials/ValidAuthImagePeerPod_test
    assessment_runner.go:264: Waiting for containers in pod: authenticated-image-with-creds-1450112107-pod are ready
=== RUN   TestDockerCreatePeerPodWithAuthenticatedImageWithValidCredentials/ValidAuthImagePeerPod_test/Peer_pod_with_Authenticated_Image_with_Valid_Credentials(Default_service_account)_has_been_created
=== NAME  TestDockerCreatePeerPodWithAuthenticatedImageWithValidCredentials/ValidAuthImagePeerPod_test
    assessment_runner.go:617: Deleting pod authenticated-image-with-creds-1450112107-pod...
    assessment_runner.go:624: Pod authenticated-image-with-creds-1450112107-pod has been successfully deleted within 60s
--- PASS: TestDockerCreatePeerPodWithAuthenticatedImageWithValidCredentials (20.12s)
    --- PASS: TestDockerCreatePeerPodWithAuthenticatedImageWithValidCredentials/ValidAuthImagePeerPod_test (20.12s)
        --- PASS: TestDockerCreatePeerPodWithAuthenticatedImageWithValidCredentials/ValidAuthImagePeerPod_test/Peer_pod_with_Authenticated_Image_with_Valid_Credentials(Default_service_account)_has_been_created (0.04s)
PASS

wainersm · 2024-09-16T14:47:31Z

src/cloud-api-adaptor/test/e2e/common_suite.go

-	imageName := os.Getenv("AUTHENTICATED_REGISTRY_IMAGE")
-	pod := NewPod(E2eNamespace, podName, podName, imageName, WithRestartPolicy(v1.RestartPolicyNever), WithImagePullSecrets(secretName))
-	NewTestCase(t, e, "InvalidAuthImagePeerPod", assert, "Peer pod with Authenticated Image with Invalid Credentials has been created").WithSecret(secret).WithPod(pod).WithAuthenticatedImage().WithAuthImageStatus(expectedAuthStatus).WithCustomPodState(v1.PodPending).Run()
+	NewTestCase(t, e, "ValidAuthImagePeerPod", assert, "Peer pod with Authenticated Image with Valid Credentials(Default service account) has been created").WithPod(pod).WithCustomPodState(v1.PodRunning).Run()


Hi @stevenhorsman !

I'm curious why originally it was checking for v1.PodPending but now v1.PodRunning (which seems the correct indeed).

I can't recall the logic as I think I re-wrote it 2 months ago, but I think I ended up surprised that some of these tests ever passed. We did only run them for the ibm cloud provider, which has not had well supported tests for a long long time, so it might have been that if you got lucky with pull timing then pending was enough, but I was trying to make them more robust.

ok, got it. thx!

wainersm · 2024-09-16T14:49:49Z

src/cloud-api-adaptor/test/e2e/common_suite.go

 	imageName := os.Getenv("AUTHENTICATED_REGISTRY_IMAGE")
 	pod := NewPod(E2eNamespace, podName, podName, imageName, WithRestartPolicy(v1.RestartPolicyNever))
-	NewTestCase(t, e, "InvalidAuthImagePeerPod", assert, "Peer pod with Authenticated Image without Credentials has been created").WithPod(pod).WithAuthenticatedImage().WithAuthImageStatus(expectedAuthStatus).WithCustomPodState(v1.PodPending).Run()
+	expectedErrorString := "401 UNAUTHORIZED"
+	NewTestCase(t, e, "InvalidAuthImagePeerPod", assert, "Peer pod with Authenticated Image without Credentials has been created").WithPod(pod).WithNoAuthJson().WithExpectedPodDescribe(expectedErrorString).WithCustomPodState(v1.PodPending).Run()


The generalization to allow running tests WithExpectedPodDescribe(expectedErrorString) is amazing! Thanks!

Yeah, the framework has lots of building blocks, which can be helpful, but many of them are quite specific, which just makes it a struggle to understand the flow and which path you are going down (for me at least), so I wanted to simplify and make some more basic ones that mimic more how I test things myself.

wainersm

Code looks good. Contain wanted refactor but CI doesn't like us recently

- Add new expectedPodDescribe check for general purpose pod describe message checking - Remove `GetAuthenticatedImageStatus` - Use the pod describe message as a better way to check for general errors, rather than bespoke auth image approach - Remove InvalidCredentials test as it's currently just using invalid credentials on the host side, not for the guest pull, so isn't useful and this feature is tested in kata-containers for bare-metal - Refactor auth.json file and auth-json-secret creation to be common and sharable by all cloud providers Signed-off-by: stevenhorsman <[email protected]>

Run the authenticated regsitry tests for libvirt cloud provider Signed-off-by: stevenhorsman <[email protected]>

Run the authenticated regsitry tests for docker cloud provider Signed-off-by: stevenhorsman <[email protected]>

wainersm · 2024-09-23T13:12:26Z

Hi @stevenhorsman !

My ack stands! I will find someone else to give the additional approval.

stevenhorsman · 2024-09-23T13:15:21Z

Hi @stevenhorsman !

My ack stands! I will find someone else to give the additional approval.

Thanks - it's nice to have some green tests!

stevenhorsman force-pushed the libvirt-auth-e2e-tests branch from e93d9cb to 7dc671a Compare July 18, 2024 12:47

stevenhorsman added the hold label Jul 18, 2024

stevenhorsman force-pushed the libvirt-auth-e2e-tests branch 6 times, most recently from 093e700 to 72da0f7 Compare July 25, 2024 15:55

stevenhorsman added test_e2e_libvirt Run Libvirt e2e tests and removed hold labels Jul 25, 2024

stevenhorsman force-pushed the libvirt-auth-e2e-tests branch 4 times, most recently from 49fe28e to 7638f97 Compare August 9, 2024 11:10

stevenhorsman force-pushed the libvirt-auth-e2e-tests branch 5 times, most recently from 8d48800 to 8707615 Compare August 12, 2024 17:10

stevenhorsman marked this pull request as ready for review August 13, 2024 08:31

stevenhorsman force-pushed the libvirt-auth-e2e-tests branch from 8707615 to e76838c Compare August 13, 2024 10:43

stevenhorsman requested review from mkulke, bpradipt and wainersm August 27, 2024 12:30

stevenhorsman force-pushed the libvirt-auth-e2e-tests branch from e76838c to e80b61f Compare September 3, 2024 13:51

wainersm reviewed Sep 16, 2024

View reviewed changes

wainersm added test_e2e_libvirt Run Libvirt e2e tests and removed test_e2e_libvirt Run Libvirt e2e tests labels Sep 16, 2024

stevenhorsman force-pushed the libvirt-auth-e2e-tests branch from e80b61f to b1b0b5f Compare September 17, 2024 14:12

wainersm approved these changes Sep 17, 2024

View reviewed changes

stevenhorsman added 3 commits September 23, 2024 12:06

tests/e2e: Add auth registry libvirt tests

519c1d3

Run the authenticated regsitry tests for libvirt cloud provider Signed-off-by: stevenhorsman <[email protected]>

tests/e2e: Add auth registry docker tests

58a9186

Run the authenticated regsitry tests for docker cloud provider Signed-off-by: stevenhorsman <[email protected]>

stevenhorsman force-pushed the libvirt-auth-e2e-tests branch from b1b0b5f to 58a9186 Compare September 23, 2024 11:07

stevenhorsman requested a review from a team as a code owner September 23, 2024 11:07

mkulke approved these changes Sep 23, 2024

View reviewed changes

stevenhorsman merged commit 0001c43 into confidential-containers:main Sep 23, 2024
28 checks passed

stevenhorsman deleted the libvirt-auth-e2e-tests branch September 23, 2024 13:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests/e2e: Add auth registry libvirt tests #1932

tests/e2e: Add auth registry libvirt tests #1932

stevenhorsman commented Jul 18, 2024

stevenhorsman commented Jul 25, 2024 •

edited

Loading

stevenhorsman commented Jul 25, 2024 •

edited

Loading

stevenhorsman commented Aug 12, 2024

wainersm Sep 16, 2024

stevenhorsman Sep 16, 2024

wainersm Sep 16, 2024

wainersm Sep 16, 2024

stevenhorsman Sep 16, 2024

wainersm left a comment

wainersm commented Sep 23, 2024

stevenhorsman commented Sep 23, 2024

tests/e2e: Add auth registry libvirt tests #1932

tests/e2e: Add auth registry libvirt tests #1932

Conversation

stevenhorsman commented Jul 18, 2024

stevenhorsman commented Jul 25, 2024 • edited Loading

stevenhorsman commented Jul 25, 2024 • edited Loading

stevenhorsman commented Aug 12, 2024

wainersm Sep 16, 2024

Choose a reason for hiding this comment

stevenhorsman Sep 16, 2024

Choose a reason for hiding this comment

wainersm Sep 16, 2024

Choose a reason for hiding this comment

wainersm Sep 16, 2024

Choose a reason for hiding this comment

stevenhorsman Sep 16, 2024

Choose a reason for hiding this comment

wainersm left a comment

Choose a reason for hiding this comment

wainersm commented Sep 23, 2024

stevenhorsman commented Sep 23, 2024

stevenhorsman commented Jul 25, 2024 •

edited

Loading

stevenhorsman commented Jul 25, 2024 •

edited

Loading