Changing authenticationMode of EKS cluster should trigger a re-create

[flostadler]

Courtesy of @jkodroff.

EKS cluster authentication modes can only be changed in the following order CONFIG_MAP -> API_AND_CONFIG_MAP -> API.
Other transitions are not allowed by AWS and require tearing the cluster down before recreating it.

We should detect such transitions and mark the diff to require a replacement. Regular transitions should not lead to replacements as this would break the upgrade path.

The change most likely needs to be made upstream.

Example

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as pcloud from "@pulumi/pulumiservice";

const vpc = new awsx.ec2.Vpc("k8s-better-together", {
  // Using a single NAT Gateway (as opposed to the default of one per AZ) helps
  // reduce cost and may improve provisioning time. In production scenarios, you
  // should typically use one per AZ.
  natGateways: {
    strategy: "Single"
  }
});

const eksCluster = new eks.Cluster("k8s-better-together", {
  authenticationMode: "API", // <-- Omit this line on the first run of the problem.
  vpcId: vpc.vpcId,
  publicSubnetIds: vpc.publicSubnetIds,
  privateSubnetIds: vpc.privateSubnetIds,
  // The CoreDNS add-on takes a while to install (about 10 minutes). Installing
  // the add-on keeps it automatically up to date and is helpful for day 2
  // operations. Since this is just a temporary cluster, we'll leave it
  // disabled.
  corednsAddonOptions: {
    enabled: false,
  },
  createOidcProvider: true
});

[jkodroff]

@flostadler Do we definitively know that you can't do API -> API_AND_CONFIGMAP?

[flostadler]

Yeah, once you set the auth mode to API you cannot go back. Tried this with an actual cluster a while ago.