Dual-Homed EKS Nodes with IPs from distinct VPC CIDRs

[vchintal]

I am working on a unique use case with the following requirement:

1. Two ENIs with IPs from distinct VPC CIDRs for nodes of a Managed Node Group (MNG)
2. MNG is deployed to a single AZ only
3. Each node spun up by this MNG hosts only single pod which uses both the ENIs using hostNetwork = true where the pod will bind to both the ENIs.

Since this is not possible out of the box with Launch Templates (LT) for the MNG, as the MNG doesn't allow any LTs with more than one network interface, I have resorted to using user data scripts to add the additional secondary ENI.

This is how I have automated the solution using Terraform:

1. Extract the data of the LT EKS cluster creates by default
2. Extract the user data of the LT
3. The role attached to the Instance Profile given adequate permissions to add/attach new ENIs
4. Insert the create-network-interface and attach-network-interface commands in the extracted user data at the end, after bootstrap script.
5. Replace the managed node group name to a new name in the parameters for the boostrap.sh script
6. Create a new launch template based on the updated user data script
7. Create a new managed node group based on the new LT created

The above steps works fine as the new ENI displaces the secondary ENI attached to the node. Also, the new ENI come up as managed by the AWS VPC CNI with an IP address assigned from a distinct secondary VPC CIDR/subnet. I can also see secondary IP addresses, half belonging to the first ENI and half belonging to the second.

My questions are:

1. Is the second WARM ENI displacement with the user data created ENI, an expected behavior?
2. Are there any drawbacks to this approach of having two VPC CNI managed ENIs on an EKS node?
3. Should the secondary ENI be tagged with node.k8s.amazonaws.com/no_manage: true? If yes, why?

[jdn5126]

@vchintal For the first part, do you mean that you want the IPs on the primary ENI to come from CIDR X and the IPs on the secondary ENI to come from CIDR Y?

I am not sure I understand the third part. If the pods are in host networking mode, they do not get a pod IP from the VPC CNI, they just use the primary IP on the primary ENI. What does "use both ENIs" mean?

[vchintal]

Your understanding of the first requirement is correct. For the third, I do not have the example deployment YAML from the customer I am working with but the application deployed in the pod is using both interfaces/ENIs of the EC2 instance in hostNetwork mode.

[vchintal]

Here is the same code I built with TF to achieve a solution for the customer: https://github.com/vchintal/aws-dual-homed-instances#option-1---vpc-cni-managed-secondary-eni