Add config option to enable the encryption of AWS EKS secrets

[joneszc]

Reference Issues or PRs

Fixes #2681
Fixes #2746
Modifies PR#2723 (Failing Tests / Pytest)
Modifies PR#2752 (Failing Tests / Pytest)

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

How to test this PR?

Any other comments?

Allows user to set EKS encryption of secrets by specifying a KMS key ARN in nebari-config.yaml

amazon_web_services:
  eks_kms_arn: 'arn:aws:kms:us-east-1:010101010:key/3xxxxxxx-xxxxx-xxxxx-xxxxx'

The KMS key must meet the following conditions:

• Symmetric
• Can encrypt and decrypt data
• Created in the same AWS Region as the cluster
• If the KMS key was created in a different account, the IAM principal must have access to the KMS key.

[viniciusdc]

@joneszc, there are two PRs which seem to add the same thing, this one and #2752 -- I assume the first one was the original; can you close this one? (or move any relevant changes back to the other PR?)

[dcmcand]

@joneszc can we close #2752 and #2723 since we have this one?

[joneszc]

@joneszc can we close #2752 and #2723 since we have this one?

@dcmcand @viniciusdc
Yes, those two PRs were built on forks of the old develop branch that is now main
Thanks for help determining that the branch was not the issue causing Pytest failures. #2752 and #2723 can be closed.

[joneszc]

@viniciusdc
I've opened PR#537 to update the docs per your request

Also in follow-up to your ask, it appears that re-deploying to set KMS encryption on an existing Nebari EKS Cluster, without previous encryption set, does succeed. However, attempting thereafter to re-deploy to remove the previously set EKS secrets encryption will fail as terraform attempts to delete and rebuild the EKS cluster but cannot due to existing node groups.

[dcmcand]

Do we actually need a paginator here? It doesn't seem like we are using it. It seems like lines 137-141 could be replaced by

kms_keys = [i["KeyId"] for i in  client.list_keys().get("Keys")]

Which seems easier to read to me since it avoids nested loops.

[dcmcand]

or if you want it more explicit

key_id_list = client.list_keys().get("Keys")
kms_keys = [i.get("KeyId") for i in key_id_list]

[joneszc]

updated

[dcmcand]

I don't feel like this line is super readable. I would suggest extracting it into a separate function.

class Kms_key:
    Arn: str
    KeyUsage: str
    KeySpec: str

def check_kms_keys(key_ids: list[str], client: boto3.Client) -> dict[str, Kms_key]:
    keys = []
    for id in key_ids:
        key = client.describe_key(KeyId=id).get("KeyMetadata")
        if key.get("Enabled"):
            keys.append(kms_key(
                Arn=key.get("Arn"),
                KeyUsage=key.get("KeyUsage"),
                KeySpec=key.get("KeySpec"),
                )
            )
    return keys

Would be way easier to follow and accomplish the same thing

[viniciusdc]

I do prefer this as well!

[joneszc]

I've made the function more readable but I did not separate into two functions.
Note: the filter on the dictionary serves no other purpose other than to avoid passing more data than necessary for validating keys for EKS encryption. It's possible this function could be utilized in the future for collecting metadata on kms keys employed for services other than EKS encryption.

This function could just as easily be written:

def kms_key_arns(region: str) -> Dict[str, dict]:
    """Return dict of available/enabled KMS key IDs and associated KeyMetadata for the AWS region."""
    session = aws_session(region=region)
    client = session.client("kms")
    kms_keys = {}
    for key in client.list_keys().get("Keys"):
        key_id = key["KeyId"]
        key_data = client.describe_key(KeyId=key_id).get("KeyMetadata")
        if key_data.get("Enabled"):
            kms_keys[key_id] = key_data
    return kms_keys

[dcmcand]

You could avoid a lot of nesting here by flipping the logic on these if statements. Something like:

        available_kms_keys = amazon_web_services.kms_key_arns(data["region"])
        # don't check if eks_kms_arn is not set
        if "eks_kms_arn" not in data or data["eks_kms_arn"] is None:
            return data
        key_id = [
            id for id in available_kms_keys.keys() if id in data["eks_kms_arn"]
        ]
        # Raise error if key_id is not found in available_kms_keys
        if (
            len(key_id) != 1
            or available_kms_keys[key_id[0]]["Arn"] != data["eks_kms_arn"]
        ):
            raise ValueError(
                f"Amazon Web Services KMS Key with ARN {data['eks_kms_arn']} not one of available/enabled keys={[v['Arn'] for v in available_kms_keys.values()]}"
            )

        key_id = key_id[0]
        # EKS cluster encryption requires a Symmetric key that is set to encrypt and decrypt data
        if available_kms_keys[key_id]["KeySpec"] != "SYMMETRIC_DEFAULT":
            raise ValueError(
                f"Amazon Web Services KMS Key with ID {key_id} is not of type Symmetric"
            )
        if available_kms_keys[key_id]["KeyUsage"] != "ENCRYPT_DECRYPT":
            raise ValueError(
                f"Amazon Web Services KMS Key with ID {key_id} KeyUsage not set to 'Encrypt and decrypt' data"
            )        available_kms_keys = amazon_web_services.kms_key_arns(data["region"])
        # don't check if eks_kms_arn is not set
        if "eks_kms_arn" not in data or data["eks_kms_arn"] is None:
            return data
        key_id = [
            id for id in available_kms_keys.keys() if id in data["eks_kms_arn"]
        ]
        # Raise error if key_id is not found in available_kms_keys
        if (
            len(key_id) != 1
            or available_kms_keys[key_id[0]]["Arn"] != data["eks_kms_arn"]
        ):
            raise ValueError(
                f"Amazon Web Services KMS Key with ARN {data['eks_kms_arn']} not one of available/enabled keys={[v['Arn'] for v in available_kms_keys.values()]}"
            )

        key_id = key_id[0]
        # EKS cluster encryption requires a Symmetric key that is set to encrypt and decrypt data
        if available_kms_keys[key_id]["KeySpec"] != "SYMMETRIC_DEFAULT":
            raise ValueError(
                f"Amazon Web Services KMS Key with ID {key_id} is not of type Symmetric"
            )
        if available_kms_keys[key_id]["KeyUsage"] != "ENCRYPT_DECRYPT":
            raise ValueError(
                f"Amazon Web Services KMS Key with ID {key_id} KeyUsage not set to 'Encrypt and decrypt' data"
            )

Has less deeply nested logic and is therefore easier to read and test.

[joneszc]

@dcmcand
my hesitation to invoke return data in numerous places in this section is that could cause issues in the future if additional validators are appended after the KMS-specific validator lines under _check_input and validators are potentially skipped prematurely for aws services.

[viniciusdc]

However, attempting thereafter to re-deploy to remove the previously set EKS secrets encryption will fail as terraform attempts to delete and rebuild the EKS cluster but cannot due to existing node groups.

Hi @joneszc, thanks for checking that out! I was already expecting it to fail, but I had another thing in mind: they might be connected. Can you post a sanitized output of the terraform error and any error messages you might encounter in the CloudTrail history? I suspect you will find something related to the KMS key in there.

the main reason for this request is to validate if it will be beneficial to have this as an immutable field or, depending on the error, we can add manual steps to the user in our docs to disable it.