Multi-Cluster KubeFlow Pipelines

[pdettori]

My organization has been encountering some use case where it is beneficial to run KFP pipelines in multiple clusters. These use cases include better utilization of existing clusters by distributing workloads to under-utilized clusters, heterogeneity of resources (some clusters have some type of accelerators, some not), and data constraints (running some pipelines on private data on-prem and bursting to cloud to run on public data). Today, at the KFP community meeting, I presented an initial PoC of enabling Multi-Cluster execution of KFP pipelines. There was some interest from the community to start a discussion on this topic using this discussion forum. These are the charts presented in the community call.