Proposal for testing using GitHub runners

[martinkennelly]

Community CI - Github Runner

Problem

How do we enable testing that requires dedicated hardware

Proposal

1. We leverage the power GitHub workflows and GitHub runners. One runner for each NIC generation. Runner listens to all NPWG repos that need this HW.
   We have two template GitHub workflows per repo: virtual (Github VM) and hardware (self-hosted)
2. Tests are skipped if they do not detect the right requirements i.e. no switchdev tests if available NIC on test platform doesn't support it or no hugepages test if no hugepages available. See here for example of how we can skip tests.
3. K8 provided by KinD.
4. HW injected into correct namespace to test. HW under test can be looked up by workflow in self hosted runners environment variables and this is used by script to add correct device to test namespace:

TEST_HW_PCI: 0000:00:01.0

Note: Abdallah mentioned he had issue with GPU using KinD. I want to know more here.

Example GitHub workflows

Virtual

Example of virtual (run on GitHub VM) workflow for Network Resources Injector:

name: e2e
on: [pull_request, push]
jobs:
  e2e-test:
    name: E2E test
    runs-on: ubuntu-latest
    steps:
      - name: Set up Go version
        uses: actions/setup-go@v1
        with:
          go-version: 1.13

      - name: Checkout code into the Go module directory
        uses: actions/checkout@v2

      - name: Get tools
        run: ./scripts/e2e_get_tools.sh

      - name: Setup KinD cluster and setup test env
        run: ./scripts/e2e_setup_cluster.sh

      - name: Execute E2E tests
        run: go test -v ./test/e2e/...

Self hosted runners

Example of hardware (run on self hosted infra) workflow for Network Resources Injector:

name: e2e_hardware
on:  [pull_request, push]
jobs:
  e2e-test:
    if: contains(github.event.pull_request.labels.*.name, 'hardware')
    name: E2E test on hardware
    runs-on: [self-hosted, Linux]
    steps:
      - name: Set up Go version
        uses: actions/setup-go@v1
        with:
          go-version: 1.13

      - name: Checkout code into the Go module directory
        uses: actions/checkout@v2

      - name: Get tools
        run: ./scripts/e2e_get_tools.sh

      - name: Setup KinD cluster and setup test env
        run: ./scripts/e2e_setup_cluster.sh

      - name: Execute E2E tests
        run: go test -v ./test/e2e/...

      - name: Tear down KinD cluster
        run: ./scripts/e2e_teardown_cluster.sh

Now we have two workflows. One workflow (virtual), that executes whenever a PR or push and one workflow that executes when the maintainers apply a label.
Maintainers of the repo have the power to assign the hardware label to trigger the workflow only.

Adding GitHub runners

Maintainers can add their own runners directly to the repo or on a forked repository.
Outside the maintainer team, others can add their runners to NPWG or they can fork our repo and add their own runners with the simple steps outlined below:

Anyone who forks can just use our hardware GitHub workflow! All they need is the right hardware!

At NPWG org level, we need to get someone with the right privileges to add a runner for us. See here for more info.

A runner can listen to one or more repos and you can control this at NPWG level, by giving the runner visibility to whatever projects you want the runner to execute on. You can also control what a runner is supposed to run on by using runner labels.

Logs

Logs will be output to the 'Actions tab' or workflow output and there should be a GitHub action in the hardware GitHub workflow to print the logs to the 'action tab' when there is a failure.

What I want at Intel

Internally at Intel, I will have a host for each generation of NICs. One host with X700 series and one for E800 series. I will need a runner for each host/hardware. I will let my runner listen to multiple NPWG projects that need this hardware and this visibility can be configured when adding the runner to NPWG.

Also

This flow doesn't rule out using other runners like Jenkins. We shouldn't rule out a tried and tested test app like jenkins. I don't know for sure if using GitHub runners is the way to go especially when Abdallah said he had issues with GPUs but...

Ask

I want to prove this flow out and I think doing this in Network Resources Injector is the best place to start. Currently, tests we have created require hugepages support and this is an opportunity to test this proposal out, find issues and see if its worthwhile for us all to look into this more.

@zshi-redhat @adrianchiris @AbdYsn