Hub Testing strategy

[jortel]

The hub testing strategy should be based on contracts and layers. Further, tests the provide high ROI (Return on Investment) should be prioritized.

Hub Perspective

All (3) of the Hub's contracts are relatively simple, Test development/maintenance and runtime complexity and overhead should be fairly low but with a high ROI. All of the tests collectively will ensure against integration-related regressions. Most of the test can b executed by running the Hub locally (outside of a cluster).

The REST API

The REST API should be tested for regression. This suite of tests will guarantee that the hub conforms to the published API. This can be performed while running the hub locally.

The Addon binding.

The addon binding should be tested for regression. This suite of tests will guarantee that the binding conforms to it's published API and integrates with the hub RESET API. This can be performed while running the hub locally.

Task Pods.

Task pod configuration should be tested for regression. This suite of tests will ensure task pod are configured according the published contract. Further, it ensures integration with the addon binding. This test requires the hub run in a cluster and would employ a test addon image that runs the suite of addon binding tests. The image would be small the test would run very quickly.

Addon Perspective

Testing the contract offered by the Hub, ensure against regressions with addons (on the hub side). In addition to the contract between the hub and an addon, there is contract between the UI and each addon. It is the UI that creates the task and needs to conform to the addon's API used in the Task.Data. For an e2e test involving an addon to provide real value, it needs to include the UI. This testing falls outside the concern of Hub and individual addon testing.

Further, each addon should have it's own set of tests (in their repository) to ensure it conforms to it's published API (Task.Data) and conforms to the Hub protocols.

User Perspective

Konveyor should have e2e testing that is driven through the UI.

Summary

Hub test suites:

• Hub REST API
• Hub addon binding.
• Task pod (running binding test).

Each addon:

• Test integration with addon binding and task protocols (status updates etc).
• Test integration with internal technologies.

[aufi]

[SOLVED]
Thanks for this strategy, I'd like ask about "running the hub locally" in REST API and Addon binding, is it right understanding that it means running "make run" on cloned tackle2-hub repo that requires running Minikube or OpenShift anyway? - Yes unless disconnected Hub PR #242 gets merged.