Use Infra E2E to test itself

[fgksgf]

Background

Recently, there are several features implemented in SkyWalking Infra E2E (credit to @chunriyeqiongsaigao, the OSPP 2022 student), including fail-fast/non-fail-fast mode and concurrent verification. However, these features make Infra E2E more complicated, and they are not easy to test. So I suggested the student test these new features of Infra E2E with httpbin, more details can be seen here.

This test method is very convenient when I want to verify Infra E2E’s updates. But we still need to do it manually (like modify e2e.yaml). So, how about automating it? Also, there is no E2E test for Infra E2E, maybe we can use Infra E2E itself to do this.

Overview

The basic flow of SkyWalking Infra E2E is like this:

1. Set up a environment for testing
2. Spin up the system under test (SUT), prepare necessary dependencies
3. Trigger or give inputs to SUT, get outputs from SUT
4. Compare the actual outputs and the expected outputs

If we use Infra E2E to test itself:

• the Infra E2E that controls the whole process can be the release version (the external Infra E2E)
• SUT will be the dev version of Infra E2E (the internal Infra E2E)
• the inputs will be e2e.yaml and expected files for internal Infra E2E
• the outputs can be a summary in YAML format about which case is passed or failed

Example

If we use Infra E2E to test itself, the test folder structure will be like this:

.
└── test/
    └── e2e/
        └── e2e.yaml # the config file used by the released version
        └── cases/
            ├── fail-fast/
            │   ├── verify.yaml   # the config file used by the dev version
            │   ├── expected.yaml
            │   └── internal/
            │       └── expected.yaml
            └── ...

For example, we want to test the fail-fast/non-fail-fast mode of the Infra E2E tool.

In the fail-fast mode, we expect that when any case fails, the tool will not continue to verify the rest of the cases.

So we can give three cases to the Infra E2E (the dev version), and the first case will fail (we can design these cases' results in advance), like:

# test/e2e/cases/fail-fast/verify.yaml (for the dev version)
verify:
  fail-fast: true
  cases:
    - name: case one # This case will fail because the arg is different from the expected.
      query: 'curl -s localhost:8080/get?case=fail -H "accept: application/json" | yq -P'
      expected: expected.yaml
    
    - name: case two # This case will success.
      query: 'curl -s localhost:8080/get?case=success -H "accept: application/json" | yq -P'
      expected: expected.yaml
    
    - name: case three # This case will fail.
      query: 'curl -s localhost:8080/get?case=fail -H "accept: application/json" | yq -P'
      expected: expected.yaml
---
# test/e2e/cases/fail-fast/internal/expected.yaml
args:
  case: success
headers:
  Accept: application/json
  Host: localhost:8080
  User-Agent: curl/7.71.1
origin: 172.17.0.1
url: http://localhost:8080/get?case=success

Then use the dev version to verify them in the fail-fast case, the expected result should be 0 passed, 1 failed (case 1), 2 skipped (case 2 and case 3). We can let the dev version output the result in YAML format (not support yet), then it can be verified by Infra E2E (the released version):

# test/e2e/e2e.yaml (for the release version)
setup:
  env: compose
  # Run a httpbin container, or other apps that can return YAML data
  file: docker-compose.yml
  timeout: 20m
  steps:
    - name: Build the infra e2e binary that to be tested
      command: #...

cleanup:
  on: always

verify:
  fail-fast: false
  concurrency: false
  cases:
    - name: "Test fail-fast mode"
      query: e2e-dev run -c test/e2e/cases/fail-fast/verify.yaml verify -o yaml
      expected: test/e2e/cases/fail-fast/internal/expected.yaml
---
# test/e2e/cases/fail-fast/internal/expected.yaml
passed: []
failed:
  - case one
skipped:
  - case two
  - case three

Summary

It's kind of like a programming language bootstrap. Welcome to discuss as it's still just a rough idea. We can transfer this to an issue when we think the idea is feasible.

[wu-sheng]

So, in this process, we are going to use a new built(PR version) e2e infra tool to drive some predictable cases, and verify whether this tool doesn't break anything existing.
Is this basically the idea?

[fgksgf]

More precisely, the idea is just like @kezhenxu94 said, there are two versions of Infra E2E involved, and I want to use the released version (more stable) to test the PR version.

As we all know, Infra E2E is used in some SkyWalking projects, but there is no E2E test for the tool currently. How can we ensure the tool work as expected, and how to know updates will not break something?

So I have this idea, but it's just one of the way to add E2E tests for the tool. We can also just use code to test Infra E2E itself.

[wu-sheng]

OK, could you share about how we could verify this tool? Could you provide more details.

[fgksgf]

Ok I will give a more detailed example later.

[kezhenxu94]

Thanks for raising this incredible idea, I've never thought about testing infra e2e with the tool itself!

From my understanding, there are two versions of e2e involved, one is a released version that will be used to drive the whole process, just like what we've been doing in all other projects, but the cases it verifies, are yaml files produced by the e2e built from the PR codes. This looks reasonable to me actually, but I'm wondering how to resolve the case:

If we find a verification bug, say, the actual data doesn't match the expected data file, and we want to verify this in a PR that trying to fix it, but the verification has a bug in all previous versions, including the released version, this means we have no idea whether the fix works or not, because the verification itself has a bug as it's using a version before the fix (the following part):

verify: # (1)
  fail-fast: false
  concurrency: false
  cases:
    - name: Test infra e2e's fail-fast mode
      query: e2e run -c case1/e2e.yaml verify -o yaml # the verification inside this `e2e` is OK because it uses PR codes
      expected: case1/expected.yaml

# but we can't guarantee the `verify` section above in (1) is not buggy 😅

Does it make sense if we run all the two e2e with the one built from the PR codes? @fgksgf

[wu-sheng]

My 2 cents of options

1. Repeat my Use Infra E2E to test itself #9558 (comment).
2. Same as Zhenxu's above comments.

I want to highlight that, from a testing methodology perspective, there is no perfect way. We need to set up prediction(s).
For <1>, it is based on

• e2e infra tool's features always have a second way to verify. Such as setting up env, we could use other CLI tools to verify whether it is well set by the latest env. Or some scripts go through logs to check some keywords.
• We should be able to design good test scenarios to cover wanted-to-test features, and these scenarios are very stable.

For <2>, it assumpts

• We could choose at least previous versions(even dev versions) to verify the latest version in the PR.

Also in <2>, we still face the preset conditions of <1>, how to verify the features.

---

In other repositories, such as the main repository, how to verify is based on our design rather than the e2e-infra tool itself.
The design is We could set up a cluster collecting telemetry data, and the aggregated/analyzed results could be verified through GraphQL/CLI..

[wu-sheng]

Before we dive into the two different methods to test, we should be certain about how to test(no matter how to use built versions or new versions), and more importantly, how to verify the features(a.k.a. what are the expectations in the assert statement). Like all other tests, we should not say, a runnable test means passed.

[fgksgf]

I added a more detailed example to demonstrate the idea. cc @wu-sheng @kezhenxu94

As for the question asked by @kezhenxu94, IMO, verification is one of the most basic features provided by the Infra E2E, we can absolutely make sure it's correct with unit tests, right?

[kezhenxu94]

I added a more detailed example to demonstrate the idea. cc @wu-sheng @kezhenxu94

As for the question asked by @kezhenxu94, IMO, verification is one of the most basic features provided by the Infra E2E, we can absolutely make sure it's correct with unit tests, right?

If we can have curated tests for the verify module, I agree 😇

[wu-sheng]

Once you already have a version(commit ID) to use as a verifier, and you are confident the features are good enough.

I think I am good.

From an engineering perspective of my understanding, this could easily revert by removing the GHA cases. So, don't be blocked by my previous questions. They are more likely some brainstorm to provide possible cases you would face one day or another.