Enable kubeflow pipeline CI/CD for ppc64le

[mdafsanhossain]

We are trying to enable kubeflow pipelines prowjobs to run tests on ppc64le k8s clusters. To get started,

• How can we add our k8s clusters which are in IBM Cloud? Will it be via K8s ExternalSecrets? https://github.com/GoogleCloudPlatform/oss-test-infra/blob/master/prow/oss/cluster/kubernetes_external_secrets.yaml

• What changes are required in https://github.com/GoogleCloudPlatform/oss-test-infra/tree/master/prow/prowjobs/kubeflow/pipelines to enable prowjobs for ppc64le?

[mdafsanhossain]

I am thinking of a similar approach to that of knative for adding secrets related to ppc64le cluster

https://github.com/knative/infra/blob/a51593dc3677530150434ba6eb0c588bd520aefa/prow/cluster/build/secrets.yaml#L50

Any thoughts on this?

[lehrig]

@chensun @zijianjoy - can you help answering above questions?

(it's relevant for kubeflow/pipelines#8660 (comment))

[ghatwala]

For tekton CI upstream , power based k8s clusters are used for running nightly CI jobs .
initial proposal - https://github.com/tektoncd/community/blob/main/teps/0051-ppc64le-architecture-support.md
Nightly CI on power k8s cluster - https://dashboard.dogfooding.tekton.dev/#/namespaces/bastion-p/pipelineruns

[seth-priya]

@chensun @zijianjoy any thoughts on this?

[lehrig]

Any news on this @chensun @zijianjoy ?

[chensun]

How can we add our k8s clusters which are in IBM Cloud? Will it be via K8s ExternalSecrets? https://github.com/GoogleCloudPlatform/oss-test-infra/blob/master/prow/oss/cluster/kubernetes_external_secrets.yaml

What changes are required in https://github.com/GoogleCloudPlatform/oss-test-infra/tree/master/prow/prowjobs/kubeflow/pipelines to enable prowjobs for ppc64le?

I have no experience to the first question. You can give it a try following what you found from knative or tekton. We can test it in the field as long as the test--if not working--doesn't block existing workflow.

For the second question, you can follow this example to add a new test: b5340c0 (make sure to include optional: true).
You can probably fork this one and customize it to testing on ppc64le. You may need to expand the argument list of https://github.com/kubeflow/pipelines/blob/master/test/presubmit-tests-with-pipeline-deployment.sh or even fork it.

[mdafsanhossain]

@chensun I am assuming we will need to customize this workflow file https://github.com/kubeflow/pipelines/blob/master/test/e2e_test_gke_v2.yaml for ppc64le? Or should this workflow work without modifications?

[lehrig]

@chensun, can you comment on this? We'd like to get this sorted out...Thanks!

[chensun]

@chensun I am assuming we will need to customize this workflow file https://github.com/kubeflow/pipelines/blob/master/test/e2e_test_gke_v2.yaml for ppc64le? Or should this workflow work without modifications?

I don't know enough the difference between ppec64le and x86/amd64, it's probably up to your exploration to tell wether customization is needed or not.

[valen-mascarenhas14]

@chensun @lehrig
We successfully ran the prow job for kubeflow-pipelines-component-yaml which runs on a ppc64le cluster on our local prow setup . The tests have passed.

I've attached the link of the prow job here for your reference kubeflow-pipelines-component-yaml-ppc64le.

[lehrig]

@chensun, does that look good to you, so we can prepare a PR?

[chensun]

kubeflow-pipelines-component-yaml is only a Python SDK test, which I assume, does not have much dependency on the underlying system architecture. You might encounter other challenges when adding e2e tests.

That being said, seems like you've figured out how to run prow job on a ppc64le cluster. Feel free to send PRs to enable this route.

[valen-mascarenhas14]

Hey @chensun,
We are in the process of enabling the kubeflow-pipeline-e2e-test job.
As we are only running the tests on the ppcc64le cluster, it raises a question regarding the deployment platform configuration.

Given that the presubmit-tests-with-pipeline-deployment.sh script utilizes GCP as the deployment platform, we are uncertain whether we should leverage the existing GCP configuration or create a new one tailored for our specific use case.

Your guidance on this matter would be of great help. Thanks

[mkumatag]

I also see an option for the minikube as well, wondering if we can explore that path or something else to trigger the tests. Also see that they are using argocd, which may become one of the prerequisites for running these tests.

Given that the presubmit-tests-with-pipeline-deployment.sh script utilizes GCP as the deployment platform, we are uncertain whether we should leverage the existing GCP configuration or create a new one tailored for our specific use case.