VCN Peering

[shyamradhakrishnan]

Problem Description
CAPOCI allows users to create Kubernetes Endpoint LoadBalancers on a private subnet. But the management clusters needs to communicate with the Endpoints as well as with the machines directly in some cases(for Machine Health Checks).
If the user creates the management cluster and workload cluster in same VCN this is fine. But the typical use case will be that each cluster will have its own VCN. If the management cluster and workload cluster are in separate VCN, then the 2 VCNs needs to be peered for the communication to happen. Right now, the onus is on the users to peer the VCNs manually. This discussion is to discuss a proposal for VCN peering facilitated by CAPOCI.

Solution
The solution is to use Dynamic Peering Gateways(DRG) - https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingDRGs.htm to peer VCNs. DRG is now the OCI recommended way to peer VCNs because of the richness on functionality it provides. The main functionality that CAPOCI gets from using DRGs is cross region VCN peering which some customers require. We will use DRG as the single solution for both in-region and cross-region VCN peering.

Case 1 - In Region
In this case, typically the management cluster and workload cluster will share the same DRG. The propsed
steps required by the user are

Step 1

User creates a DRG which is to be shared by the management and workload clusters. This is a one time activity. How the user creates this DRG depends on how the management cluster is setup. We will explain in next section if the management cluster is setup using CAPOCI, how users can do this automatically.

Step 2

User adds the route rules in the management cluster VCN to route requests to the various workload cluster VCNs via the dRG> Again this step can be manual or automated depending on how the management cluster is setup.

Step 3

User creates the workload cluster, and provides the DRG ID. CAPOCI will attach the newly created VCN to the DRG and add the required route rule specified in the spec. An example spec is shown below

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OCICluster
spec:
  networkSpec:
    vcnPeering:
      drg:
        // The DRG is will be provided by the user in the spec
        id: <drg-id>
      peerRouteRules:
       // User will specify any VCN CIDR ranges which has to be forwarded to the DRG> This typically is the management cluster VCN CIDR
        - vcnCIDRRange: "10.0.0.0/16"

Case 2 - In Region With New DRG Per Workload Cluster

In this case, a new DRG will be created per workload cluster. Here, CAPOCI will create a new DRG, but also create the required Remote Peering Connections(https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/scenario_e.htm#scenario_e) in both the local DRG as well as the PeerDRG and the connect both. This technique can be used if CAPOCI is being used to manage the management cluster itself.

CAPOCI will do the following

• Create a new DRG, called local DRG from now on, and attach to the new VCN
• Create RPC in local DRG
• Create RPC in peer DRG(management cluster DRG)
• Connect the 2 RPCs

Step 1

User creates a DRG for the management cluster. This is a one time activity. How the user creates this DRG depends on how the management cluster is setup. If management cluster is being managed by CAPOCI itself, then CAPOCI can be used to create the DRG as explained below.

Step 2

User adds the route rules in the management cluster VCN to route requests to the various workload cluster VCNs via the DRG. Again this step can be manual or automated depending on how the management cluster is setup.

Step 3

User creates the workload cluster, and provides the DRG ID. CAPOCI will attach the newly created VCN to the DRG and add the required route rule specified in the spec. An example spec is shown below

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OCICluster
spec:
  networkSpec:
    vcnPeering:
      drg:
       // the following flag will instruct CAPOCI to create the DRG
        manage: true
      peerRouteRules:
        - vcnCIDRRange: "10.0.0.0/16"
      remotePeeringConnections:
        - managePeerRPC: true.  -> This flag will instruct CAPOCI to create remote RPC as well
          // The DRG Id of the management cluster
          peerDRGId: <peer-drg-id>

If management cluster is being managed by CAPOCI itself, the users can use the above spec to create as well as manage the DRG and route rules via spec itself.

Case 3 - Cross Region With New DRG Per Workload Cluster

This is same as Case 2, but with the peer region name mentioned as below

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OCICluster
spec:
  networkSpec:
    vcnPeering:
      drg:
       // the following flag will instruct CAPOCI to create the DRG
        manage: true
      peerRouteRules:
        - vcnCIDRRange: "10.0.0.0/16"
      remotePeeringConnections:
        - managePeerRPC: true.  -> This flag will instruct CAPOCI to create remote RPC as well
          // The DRG Id of the management cluster
          peerDRGId: <peer-drg-id>
          peerRegionName: "region-name"

[Jenksho]

I am good with this. It makes sense. It implies that we will only have 1 Workload Cluster per VCN but I think that is a sensible stance.

[joekr]

Is there a benefit for two or more clusters to share a VCN?

[Jenksho]

I don't think there is any particular advantage it's just that I can't really think of a reason why not. Not an issue just me brain dumping without really thinking enough I suppose.

[shyamradhakrishnan]

For any customisation, customers can bring their own network. We can of-course add capabilities later if customers ask for it. Hope that makes sense.