feat: add e2e test for instructlab CI

[RobotSail]

This commit adds a modified version of the E2E NVIDIA T4 workflow from the instructlab/instructlab
repository to automatically test new PRs against an NVIDIA box to see how they perform end-to-end.

This PR follows up work from the following PRs:

• github: Add e2e test workflow #69
• Revert "remove incorrect end-to-end implementation" #77

Signed-off-by: Oleg S [email protected]

[danmcp]

LGTM

Note: Looks like the repo has a linting issue that needs to be fixed.

[RobotSail]

@danmcp Thanks, we already fixed it in another PR so will just need to rebase here.

[nathan-weinberg]

@RobotSail i remember you voiced concerns about the E2E script not living in this repo, is that still the case?

[RobotSail]

@nathan-weinberg

i remember you voiced concerns about the E2E script not living in this repo, is that still the case?

Well the current workflow is not ideal but it is less work to have them live there for now until someone is interested in maintaining a separate set of tests in this repo. In the future if we decide to separate repos between upstream & downstream equivalents, the downstream should continue to rely on the tests in the https://github.com/instructlab/instructlab repo. The upstream version, should one exist, would drop this test and would instead rely on its own e2e tests.

[nathan-weinberg]

@nathan-weinberg

i remember you voiced concerns about the E2E script not living in this repo, is that still the case?

Well the current workflow is not ideal but it is less work to have them live there for now until someone is interested in maintaining a separate set of tests in this repo. In the future if we decide to separate repos between upstream & downstream equivalents, the downstream should continue to rely on the tests in the https://github.com/instructlab/instructlab repo. The upstream version, should one exist, would drop this test and would instead rely on its own e2e tests.

SGTM - we could even have a dedicated repo for upstream tests that other repos could pull from - this was done by the OCP org at one point: https://github.com/openshift/openshift-tests

[RobotSail]

we could even have a dedicated repo for upstream tests that other repos could pull from - this was done by the OCP org at one point: https://github.com/openshift/openshift-tests

Just to be clear, by upstream here you mean upstream relative to the other OCP repos and not upstream as in the upstream Kubernetes, right?

[nathan-weinberg]

Yes! So in our case it would be something like:

• a public repo for InstructLab tests that could be used anywhere upstream for testing within library repos (e.g. instructlab-tests with different tests that each lib can pull from)
• for tests for downstream projects such as RHEL AI, these tests could be used but the development of them would focus on testing the InstructLab project first and foremost

[RobotSail]

Got it, that makes perfect sense.