Feasibility of Running Kubeflow Pipeline E2E Tests on ppc64le with Self-Hosted Runner

[valen-mascarenhas14]

Feasibility of Running Kubeflow Pipeline E2E Tests on ppc64le with Self-Hosted Runner

We are exploring the feasibility of running the Kubeflow Pipeline (KFP) E2E tests on ppc64le architecture using a self-hosted runner. Below are the key details, current challenges, and steps we’ve undertaken so far:

---

Current Setup

1. Self-Hosted Runner:

   • The runner is an existing ppc64le Ubuntu machine.
   • It is reused for all jobs (not created/destroyed per workflow).
   • To avoid stale cluster issues, we’ve added an action to delete the Kubernetes cluster at the end of each job.

2. Image Handling:

   • We are not building images during runtime due to ppc64le-specific challenges.
   • Pre-built images are stored in JFrog Artifactory and used during tests.

3. Dependency Management:

   • Certain dependencies already exist on the runner, and we've skipped installing them in the workflow.

---

Challenges & Current Progress

1. Test Failures:

   • Frontend-Integration Test: Currently failing. We have proposed skipping this test (Discussion Link).
   • Basic Sample Test: Debugging ongoing to identify and resolve issues.

2. Cluster Cleanup:

   • Added an action to ensure clusters are deleted after each job to prevent resource conflicts or leftover artifacts.

3. ppc64le Build Issues:

   • Building images for ppc64le requires architecture-specific changes that are being addressed.

4. Workflow Adjustments:

   • Some actions are skipped since dependencies are pre-installed on the runner.

---

References

• Existing discussion of issues: #11143
• Proposal for skipping Frontend-Integration Tests: #11195

---

Request for Feedback

We’ve created a PR to showcase the work done so far and seek feedback on:

1. The overall approach and feasibility.
2. Best practices for handling architecture-specific challenges for ppc64le.
3. Community opinion on skipping or modifying workflows/tests.

---

Let us know if you have any recommendations or insights that could help streamline this process. Looking forward to collaborating with the community to enable ppc64le support!