Skip to content

Using SpecMesh with AWS MSK

Jump to bottom
Neil Avery edited this page Aug 10, 2023 · 2 revisions

A walkthrough showing how SpecMesh hides topic provisioning and messy Kafka ACL configuration to create an effortless experience

From: https://specmesh.io/blog/specmeshandmsk/

Technical TLDR;

Run a cluster; associate user/secrets/principles in the Secrets Manager with the cluster; use these user/secret pairs in the client connections. SpecMesh will create the ACLs to ensure Access control works and you have self-service governance!

MSK has a few limitations

  1. AWS want you to use IAM - however, this only works with Java clients using their jars
  2. Non-Java clients (RUST, Go) will need to use SASL/SCRAM with MSK - IAM won't work without building your own IAM integration.
  3. IAM isn't that easy to setup and/or automate
  4. Broker property: super.users parameter is not supported on MSK (this property bypasses all ACL Checks - see https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/security/authorizer/AclAuthorizer.scala). A workaround is to use multi-authentication (below)

MSK Multi-Authentication - SASL/SCRAM + IAM

When multi-authentication is enabled on MSK cluster authorization depends on which of those access control methods a client is using to access MSK cluster.

Let's consider the following example: both IAM and SASL/SCRAM are enabled, and 'client A' is accessing the MSK cluster via IAM authentication, while 'client B' is accessing the cluster via SASL/SCRAM. In this scenario, you can still invoke Apache Kafka ACL APIs and add ACLs for an MSK cluster that uses IAM access control. However, ACLs stored in Apache ZooKeeper will have no effect on authorization for IAM roles. Therefore, access and authorization for 'client A,' which uses IAM auth, will be controlled by the IAM policy alone, as the added ACLs do not affect 'client A' in any way, even though they are added.

On the other hand, when a client is using non-IAM authentication, the added ACLs (including the "allow.everyone.if.no.acl.found" setting) will have an effect. In this case, authorization will be controlled by ACLs. So, when 'client B,' which uses SASL/SCRAM, attempts to perform any operations, it will be validated against the ACLs that were added.

In short, to fill in the gaps in the above table:

Authn & Authz mech | Kafka client authn | Kafka client authz | Kafka ACL behaviour | Property allow.everyone.if.no.acl.found --- | --- | --- | --- | --- | --- | SASL/IAM clients | SASL/IAM | IAM | No effect | No effect | SASL/SCRAM clients | SASL/SCRAM | ACLs | Applies/Does have an effect | Applies/Does have an effect |

Requirements

  1. Admin access to your AWS console (ability to create MSK cluster, start an instance, configure IAM, roles, start instances, configure secrets etc)
  2. Basic understanding of Kafka broker (broker properties), and client eco-system (producer, consumer, client.properties)

SASL/SCRAM on MSK has the following limitations:

  1. super.users parameter is not supported
  2. MSK only supports SCRAM-SHA-512 authentication
  3. An MSK cluster can have up to 1000 users
  4. You must use an AWS KMS key with your Secret
  5. You cannot use a Secret that uses the default Secrets Manager encryption key with Amazon MSK
  6. You can't use an asymmetric KMS key with Secrets Manager
  7. You can associate up to 10 secrets with a cluster at a time using the BatchAssociateScramSecret operation
  8. The name of secrets associated with an Amazon MSK cluster must have the prefix AmazonMSK_
  9. Secrets associated with an Amazon MSK cluster must be in the same Amazon Web Services account and AWS region as the cluster

Source: https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html#msk-password-limitations

Steps

1. Start your MSK Kafka cluster with SASL/SCRAM AND IAM

  • Sign in to the AWS Management Console and open the Amazon MSK console
  • Choose Create cluster
  • Enter a cluster name, and leave all other settings unchanged
  • From the table under All cluster settings, copy the values of the following settings and save them because you need them later in this tutorial: VPC, Subnets, Security groups associated with VPC
  • Choose Create cluster

Note: Creation will take about 15 minutes.

Later we will change the default env configuration for the SimpleAclAuthorizer: KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "TRUE"

Source: https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration.html

2. Make the cluster public and enable SASL + IAM

a. Navigate to the AWS MSK console
b. Choose the MSK cluster you just created in Step 1
c. Click on the Properties tab
d. In the Security settings section, choose Edit
e. Check the checkbox next to SASL/SCRAM authentication and IAM
f. Click Save changes

You can find more details about updating a cluster’s security configurations here.

Create a Symmetric Key

a. Now go to the AWS Key Management Service (AWS KMS) console
b. Click Create Key
c. Choose Symmetric and click Next
d. Give the key and Alias and click Next
e. Under Administrative permissions, check the checkbox next to the AWSServiceRoleForKafka and click Next
f. Under Key usage permissions, again check the checkbox next to the AWSServiceRoleForKafka and click Next
g. Click on Create secret
h. Review the details and click Finish

You can find more details about creating a symmetric key here.

Store a new Secret

a. Go to the AWS Secrets Manager console
b. Click Store a new secret
c. Choose Other type of secret (e.g. API key) for the secret type
d. Under Key/value pairs click on Plaintext
e. Paste the following in the space below it and replace and with the username and password you want to set for the cluster

{
"username": "<your-username>",
"password": "<your-password>"
}

f. On the next page, give a Secret name that starts with AmazonMSK_
g. Under Encryption Key, select the symmetric key you just created in the previous sub-section from the dropdown
h. Go forward to the next steps and finish creating the secret. Once created, record the ARN (Amazon Resource Name) value for your secret You can find more details about creating a secret using AWS Secrets Manager here.

Associate secrets with MSK cluster

a. Navigate back to the AWS MSK console and click on the cluster you created in Step 1
b. Click on the Properties tab
c. In the Security settings section, under SASL/SCRAM authentication, click on Associate secrets
d. Paste the ARN you recorded in the previous subsection and click Associate secrets

Create the cluster’s configuration

a. Go to the AWS CloudShell console
b. Create a file (eg. msk-config.txt) with the following line allow.everyone.if.no.acl.found = false

c. Run the following AWS CLI command, replacing with the path to the file where you saved your configuration in the previous step

aws kafka create-configuration --name "MakePublic" \
--description "Set allow.everyone.if.no.acl.found = false" \
--kafka-versions "2.6.2" \
--server-properties fileb://<config-file-path>/msk-config.txt

You can find more information about making your cluster public here.

3. Create a client machine (using IAM)

If you already have a client machine set up that can interact with your cluster, then you can skip this step. If not, you can create an EC2 client machine and then add the security group of the client to the inbound rules of the cluster’s security group from the VPC console. You can find more details about how to do that here.

4. Install Apache Kafka on Client machine

We install Apache Kafka to test and check ACLs, Topic creation etc. You can find more information about how to do that here.

5. Create the domain/user secret (principle) for use with SASL/SCRAM

SpecMesh specs have an ‘id’ - this ‘id’ is used as the principle (user). This way SpecMesh will automatically generate and apply ACLs that govern which ‘id’s can access various topics.

Create the SASL/SCRAM secrets for each id and associate them with the cluster.

https://catalog.workshops.aws/msk-labs/en-US/securityencryption/saslscram/authorization

With SASL/SCRAM the username/secret is stored within the secrets-manger. The username is is also called the principle, it and the secret are provided to clients (via properties files). Each secret is associated with your cluster. When the client connects to MSK, the broker retrieves the credentials from SecretsManager and applies the associated ACLs

6. Create client properties files with security credentials

Configure Kafka (Java) clients with the appropriate SASL/SCRAM credentials as shown on the following pages

https://catalog.workshops.aws/msk-labs/en-US/securityencryption/saslscram/authorization#client-setup

Associate secrets: https://catalog.workshops.aws/msk-labs/en-US/securityencryption/saslscram/authorization

7. Provision Specs

Run the SpecMesh CLI provision command against each Spec. This will create Topics, Publish schemas and configure ACLs according to the Spec. The ‘id’ of the spec matches the principle.

  1. Log onto the client machine**
  2. Checkout/Pull the PR
  3. Execute SpecMesh CLI provision via docker - as shown here
 % docker run --rm -v "$(pwd)/resources:/app" ghcr.io/specmesh/specmesh-build-cli provision -bs kafka:9092 -sr http://schema-registry:8081 -spec /app/simple_schema_demo-api.yaml -schemaPath /app

Note: the docker runtime can access the broker and schema registry url

Use the SpecMesh CLI provision - learn more here: https://github.com/specmesh/specmesh-build/blob/main/cli/README.md

Learn more here

8. Verify Topics and ACLs exist

From the client machine (it has super-user permissions via IAM, MSK service role):

  1. list topics: $ bin/kafka-topics.sh --list --zookeeper localhost:2181
  2. list ACLs: kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --list --topic SommeTopic

See:

Clone this wiki locally