v1.5.3 Release

This release includes a rewrite of the the v1.5.2 bash code to go and adds a fix for a startup issue with stuck ENIs.

v1.5.3

• Bug fix - Copy the binary and config after ipamd is ready (#576, @mogren)
• Improvement - Update Calico version to v3.8.1 (#554, @lmm)
• Improvement - Add env var to override introspection bind address (#501, @jacksontj)
• Improvement - Remove unused env variable (#578, @mogren)
• Improvement - Exit early if MAC address doesn't match (#582, @mogren)

All changes since v1.5.1: v1.5.1...v1.5.3

To use this version in a cluster:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.5/config/v1.5/aws-k8s-cni.yaml

v1.5.2 Release

The major change in this release is that worker nodes will stay in NotReady status until ipamd is ready to assign IPs to pods. (#553)

Extra thanks to @uthark and @nithu0115 for debugging and contributing some important fixes.

v1.5.2

• Bug fix - Fix formatting flag (#521, @uthark)
• Bug fix - Fix formatting issue (#524, @uthark)
• Bug fix - Detach ENI before deleting (#538, @uthark)
• Improvement - Adding healthz endpoint to IPamD (#548, @nithu0115)
• Improvement - Adding new m5 and r5 instances (#518, @mogren)
• Improvement - t3a.small only have 2 ENIs (#543, @mogren)
• Improvement - Updating AWS Go SDK version (#549, Nordlund, Eric)
• Improvement - Reduce the wait time when checking for pods without IPs (#552, @mogren)
• Improvement - Update start script to wait for ipamd health (#553, @mogren)
• Improvement - Hide health check output (#569, @mogren)
• Improvement - Support c5.12xlarge and c5.24xlarge (#510, @mogren)

Full changelog: 2ceb59c...e5f2b85

v1.5.2 Release Candidate 1

First release candidate for v1.5.2.

Note! This release is not meant for production use, it is still being tested.

The major change in this release is that the node will not become ready until ipamd is ready to assign IPs to pods by using the newly added health endpoint. This release candidate includes the eniconfig fix from v1.5.1.

Other notable changes:

• 628f2c2 - Updating AWS Go SDK version.
• 081539e - Detach ENI before deleting
• 31821f9 - Reduce the wait time when checking for pods without IPs
• 6ecba52 - Ignore namespace for custom eniconfig watch (v1.5.1)

Full changelog of v1.5.2-rc1

If you want to test this release candidate in your own cluster, patch the aws-node daemonset:

kubectl patch daemonset aws-node \
-n kube-system \
-p '{"spec": {"template": {"spec": {"containers": [{"image": "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.2-rc1","name":"aws-node"}]}}}}'

Verify the update:

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2

amazon-k8s-cni:v1.5.2-rc1

v1.5.1 Release

This release is a minor patch release for clusters with a custom network config.

v1.5.1

• Bug fix - Ignore namespace for custom eniconfig watch (#561, @mogren)

v1.5.1 Release Candidate 1

First release candidate for v1.5.1.

Note! This release is not meant for production use, it is still being tested.

The major change in this release is that the node will not become ready until ipamd is ready to assign IPs to pods by using the newly added health endpoint.

Other notable changes:

• 628f2c2 - Updating AWS Go SDK version.
• 081539e - Detach ENI before deleting
• 31821f9 - Reduce the wait time when checking for pods without IPs

Full changelog of v1.5.1-rc1

If you want to test this release candidate in your own cluster, patch the aws-node daemonset:

kubectl patch daemonset aws-node \
-n kube-system \
-p '{"spec": {"template": {"spec": {"containers": [{"image": "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.1-rc1","name":"aws-node"}]}}}}'

Verify the update:

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2

amazon-k8s-cni:v1.5.1-rc1

v1.5.0 Release

This release improves the CNI behavior when the number of IPs available to the cluster is constrained. Setting a WARM_IP_TARGET will now return IPs that are not assigned to pods back to the subnet.

Note! Setting a low WARM_IP_TARGET will increase the number of calls being made to EC2, and if the cluster is large, or the pod churn high, there is a risk of getting throttled. If that happens, no ENIs or IPs will be available untill the throttling stops.

The EKS team wishes to express appreciation to the FICO engineering team for their collaboration on this release.

Also thanks to our external contributors @forsberg, @venkatesh-eb, @Zyqsempai and @bboreham that have contributed to this release.

v1.5.0

• Bug fix - Fix spelling on annotation (#482, @forsberg)
• Bug fix - Avoid using force detach of ENIs (#458, @mogren)
• Bug fix - Flush logs before exiting (#451, @venkatesh-eb)
• Improvement - Add IPs to existing ENIs first (#487, @mogren)
• Improvement - Added error handling for GetENIipLimit (#484, @Zyqsempai)
• Improvement - Moved all GetEnv's calls to init step (#445, @Zyqsempai)
• Improvement - On start up, wait for pods with no IP (#480, @mogren)
• Improvement - Don't modify maxENI (#472, @nckturner)
• Improvement - Improve WARM_IP_TARGET handling (#461, @nckturner)
• Improvement - Update logging format to align messages (#473, @mogren)
• Improvement - Added -W (wait for xlock's) flag to iptables commands (#439, @Zyqsempai)
• Improvement - Remove error message from Prometheus labels (#467, @bboreham)
• Improvement - Update instance types (#459, @mogren)

v1.5.0 Release Candidate 1

First release candidate for the v1.5.0 branch.

Note! This release is not meant for production use, it is still being tested.

The major change in this release is the change in behavior of WARM_IP_TARGET(#461). Set this variable to the number of IPs you want to have available and ready to be assigned to new pods. This number applies to each node. This setting requires one additional IAM permission for the node role that has not yet been added to the managed policy. If you plan to set WARM_IP_TARGET, please add this IAM permission:

"ec2:UnassignPrivateIpAddresses"

For example, if WARM_IP_TARGET is set to 3 with a m5.4xlarge instance, the CNI will first allocate all 30 IPs on the first ENI, one being used by the ENI itself and the other 29 available to pods. If no pods are scheduled on the node, 26 IPs will be released back to the subnet and 3 unassigned IPs will be available for pods to use.

Note! If WARM_IP_TARGET is set to a low value, scheduled pods might get stuck for a long time before they can actually get an IP. If you have a lot of pod churn, either don't set this variable, or give it a bigger value. Not having WARM_IP_TARGET set means that all IPs will be allocated on each ENI, and no IPs will be freed until there are no pods using any IP on the whole ENI.

For testing this release candidate in your own cluster, patch the aws-node daemonset:

kubectl patch daemonset aws-node \
-n kube-system \
-p '{"spec": {"template": {"spec": {"containers": [{"image": "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.0-rc1","name":"aws-node"}]}}}}'

Other notable changes:

• Use RouteReplace instead of RouteAdd (#379, @mogren)
• system-node-critical priority class (#404, @nckturner)
• Prometheus metric improperly named (#403, @shraykay)
• Add i3en, r5ad and t3a instance types (#459, @mogren)
• Avoid using force detach of ENIs (#458, @mogren)
• Added -W (wait for xlock's) flag to iptables commands (#439, @Zyqsempai)

v1.4.1 Release

This release adds the environment variables DISABLE_INTROSPECTION and DISABLE_METRICS
to make it possible to turn off metrics and debugging.

The metrics agent has been updated and can be added by first adding an IAM policy with the following permissions the node group's IAM role:

ec2:DescribeTags
cloudwatch:PutMetricData

And then applying the config:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.4/config/v1.4/cni-metrics-helper.yaml

v1.4.1

• Feature - Add flag to disable metrics and introspection (#436, @mogren)
• Bug fix - Adding additional CRD for Calico that was missing (#410, @wmorgan6796)
• Improvement - Update CNI metrics (#413, @mogren)

v1.3.4 Release

This release adds the environment variables DISABLE_INTROSPECTION and DISABLE_METRICS
to make it possible to turn off metrics and debugging.

• Feature - Add flag to disable metrics and introspection

v1.4.0 Release

We are happy to note that this release includes community contributions from @mikkeloscar, @rickardrosen, @nak3, @pdbogen, @ikatson, @sftim, @etopeter, @ankon, @tustvold, @ewbankkit, @errordeveloper, @rudoi, @max-rocket-internet, @2ffs2nns, @peterbroadhurst and @StevenACoffman.

Notable changes in release v1.4.0

• Feature - Add an environment variable to limit the number of ENIs (#251, @pdbogen)
  • Makes it possible to limit how many ENIs that are allocated per node.
• Feature - Randomize outgoing port for connections in the SNAT iptables rule (#246, @taylorb-syd)
  • To avoid a race condition when using SNAT, select ports randomly instead of sequentially.
• Feature - ENIConfig set by custom annotation or label names (#280, @etopeter)
  • Enables users to set a custom annotation or label key to define ENIConfig name.
• Improvement - Update Calico to 3.3.6 (#368, @2ffs2nns)
• Improvement - Add new instance types (#366, @mogren)
  • Adds m5ad and r5ad families.
• Improvement - Actually enable prometheus metrics (#361, @mogren)
• Improvement - Retry LinkByMac when link not found (#360, @peterbroadhurst)
  • Sometimes it takes a few seconds for a new ENI to be available, so we retry 5 times.
• Improvement - Run yum clean all to reduce image size (#351, @mogren)
• Improvement - Renaming Prometheus metrics with "awscni_" prefix (#348, @max-rocket-internet)
• Improvement - Allow configuring docker image when running make (#178, @mikkeloscar)
• Improvement - Add support for stdout logging (#342, @rudoi)
  • Adds environment variable AWS_VPC_K8S_CNI_LOG_FILE that can be set to stdout or a file path.
• Improvement - Some cleanups related to #234 (#244, @mogren)
• Improvement - Use apps/v1 for DaemonSet (#341, @errordeveloper)
• Improvement - Clean up aws-cni-support.sh and update the documentation (#320, @mogren)
• Improvement - Fix tiny typo in log message (#323, #324, @ankon)
• Improvement - Collect rp_filter from all network interface in aws-cni-support.sh (#338, @nak3)
• Improvement - Use device number 0 for primary device in unit test (#247, @nak3)
• Improvement - Collect iptables -nvL -t mangle in support script (#304, @nak3)
• Improvement - Return the err from f.Close() (#249, @mogren)
• Improvement - Explicitly set the IP on secondary ENIs (#271, @ewbankkit)
  • Fixes IP bug on older kernels.
• Improvement - Update instance ENI and IP mapping table (#275, @hmizuma)
  • Adds a1 and c5n instances. (Already included in v1.3.2)
• Improvement - Add ENI entries for p3dn.24xlarge instance (#274, @hmizuma)
  • p3dn.24xlarge was already included in v1.3.2
• Improvement - Use InClusterConfig when CreateKubeClient() was called without args (#293, @nak3)
• Improvement - Expose configuration variables via ipamD to make it debug friendly (#287, @nak3)
• Improvement - Allow cross compile on different platform (#292, @nak3)
• Improvement - Add changes to support multiple platform build (#286, @mbartsch)
  • arm64 build support
• Improvement - Improve setup advice in README around ENI / IP (#276 @sftim)
• Improvement - Use unix.RT_TABLE_MAIN for main routing table number (#269, @nak3)
• Improvement - Detect if mockgen and goimports are in the path (#278, @nak3)
• Improvement - Increment IP address safely (#258, @nak3)
  • Calculate the gateway IP in a safe way.
• Improvement - Remove unused options from rpc.proto (#252, @nak3)
• Improvement - Add missing unit tests execution to Makefile (#253, @nak3)
• Improvement - Bump TravisCI to use 1.11 (#243, @mogren)
• Bug fix - Fix typos in json types for ENIConfig (#393, @tiffanyfay)
• Bug fix - Avoid unbound variable error in aws-cni-support.sh (#382, @StevenACoffman)
• Bug fix - Output CIDR in correct format (#267, @nak3)
• Bug fix - Use replace when adding host route (#367, @mogren)
• Bug fix - Update k8sapi to use operator-framework inClusterConfig (#364, @tiffanyfay)
  • If the environment variables are missing, fall back to DNS lookup.
• Bug fix - Set mainENIRule mask (#340, @tustvold)
  • In order to match the connmark correctly, we need to mask it out when checking.
• Bug fix - Use primary interface to add iptables for connmark entry (#305, @nak3)
• Bug fix - Stop wrapping and returning nil (#245, @nak3)
• Bug fix - Fix return path of NodePort traffic when using Calico network policy (#263, @ikatson)
• Bug fix - Remove scope: Cluster from spec.names (#199, @rickardrosen)
• Bug fix - Remove unneeded spec entry in v1.3 manifest (#262, @hmizuma)
• Bug fix - Add formatter to errors.Wrapf in driver (#241, @nak3)

For running this in your own cluster:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.4/config/v1.4/aws-k8s-cni.yaml

Note! If you are still running a Kubernetes 1.10 or older, you need to use the old CRD configuration:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.4/config/v1.4/aws-k8s-cni-1.10.yaml

Check that the CNI version got updated:

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni:v1.4.0