Tekton on Multi Cluster

[zhouhaibing089]

This was originally raised at here.

A kubernetes cluster usually sits within one AZ(availability zone). If that availability zone fails(for e.g, power outage, network issues), the cluster fails as well.

The current installation of Tekton works best for a single cluster which is a little worri-some. That means, if that cluster fails, we will lose the entire tekton service.

Within our own team, we have a pattern of running kube-apiserver and etcd in HA mode:

In this graph, we run etcd across three different datacenters. In a three-replicas setup, we can tolerate one region failure.

Then all the other controllers, we can point their kubeconfig to the GTM DNS, and can be deployed anywhere and in multiple clusters with leader-election enabled.

And coming back to Tekton, we initially tried to run the same mode, but we faced some issues:

1. Tekton pipeline controller only accepts one kubeconfig - Its API objects are stored at the same cluster where it spins up Pods.
2. We also tried to modify pipeline source code a bit, and let it take two kubeconfigs: One pointing to API cluster, and one pointing to pod cluster. But it is still odd since a bunch of other objects are related - Namespace/ServiceAccount/Secrets.

Then we came up with a different approach: We try to put Tekton APIs(CRD/Webhooks/RBAC) to the HA cluster as well as individual clusters. That means users can still use tekton in each clusters without HA guaranteed.

Now, if users want to have HA guaranteed, they can use the HA endpoint(the one described in the graph above), we will then add another controller to:

1. Find the available tekton clusters which: a) are healthy and b) have enough capacity.
2. Sync Tekton Resources into that Tekton cluster.
3. Gather status back.

The sync logic is different per resource type:

1. Task/Pipeline/Namespace/ConfigMap..: Since these are pure informational data, they should be synced into all the available clusters.
2. TaksRun/PipelineRun: These are execution instances, so we need sort of scheduling to make sure they are scheduled to one of the clusters.

Does this setup sound reasonable or maybe there is already a solution for this? Thanks in advance.

[ileixe]

Multi-cluster support is one of the major thing that I'm missing in Tekton. I voted +1 for more attention.