Use CDK To Deploy a Complete Solution With MSK Serverless, App Runner, EKS, and DynamoDB

A previous post covered how to deploy a Go Lambda function and trigger it in response to events sent to a topic in an MSK Serverless cluster.

This blog will take it a notch further.

• The solution consists of an MSK Serverless cluster, a producer application on AWS App Runner, and a consumer application in Kubernetes (EKS) persisting data to DynamoDB.
• The core components (MSK cluster, EKS, and DynamoDB) and the producer application will be provisioned using Infrastructure-as-code with AWS CDK.
• Since the consumer application on EKS will interact with both MSK and DynamoDB, you will also need to configure appropriate IAM roles.

All the components in this solution have been written in Go.

• The MSK producer and consumer app use the franz-go library (it also supports MSK IAM authentication).
• The CDK stacks have been written using CDK Go library.

Prerequisites

You will need the following:

• An AWS account
• Install AWS CDK, AWS CLI, Docker, eksctl and curl.

Use CDK to Provision MSK, EKS, and DynamoDB

AWS CDK is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language.

All the code and config are present in this GitHub repo. Clone the GitHub repo and change it to the right directory:

git clone https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb
cd msk-cdk-apprunner-eks-dynamodb/cdk

Deploy the first CDK stack:

cdk deploy MSKDynamoDBEKSInfraStack

Wait for all the components to get provisioned, including the MSK Serverless cluster, EKS cluster and DynamoDB.

You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code here.

Deploy MSK Producer Application to App Runner Using CDK

Deploy the second CDK stack.

Note that in addition to deploying the producer application to App Runner, it also builds and uploads the consumer application Docker image to an ECR repository.

Make sure to enter the MSK Serverless broker endpoint URL.

export MSK_BROKER=<enter endpoint>
export MSK_TOPIC=test-topic

cdk deploy AppRunnerServiceStack

Wait for the producer application to get deployed to App Runner. You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code and the producer application.

Once complete, make a note of the App Runner application public endpoint as well as the ECR repository for the consumer application. You should see these in the stack output as such:

Outputs:
AppRunnerServiceStack.AppURL = <app URL>
AppRunnerServiceStack.ConsumerAppDockerImage = <ecr docker image>
....

Now, you can verify if the application is functioning properly. Get the publicly accessible URL for the App Runner application and invoke it using curl. This will create the MSK topic and send data specified in the HTTP POST body.

export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"user1@foo.com","name":"user1"}' $APP_RUNNER_URL

Now you can deploy the consumer application to the EKS cluster. Before that, execute the steps to configure appropriate permissions for the application to interact with MSK and DynamoDB.

Configure IRSA for Consumer Application

Applications in a pod's containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. Applications must sign their AWS API requests with AWS credentials. IAM roles for service accounts provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance's role, you associate an IAM role with a Kubernetes service account and configure your pods to use the service account.

Exit the cdk directory and change to the root of the project:

cd ..

Create an IAM OIDC Identity Provider for Your Cluster With eksctl

export EKS_CLUSTER_NAME=<EKS cluster name>

oidc_id=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

aws iam list-open-id-connect-providers | grep $oidc_id

eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve

Define IAM Roles for the Application

Configure IAM Roles for Service Accounts (also known as IRSA). Refer to the documentation for details.

Start by creating a Kubernetes Service Account:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-app-sa
EOF

To verify:

kubectl get serviceaccount/eks-app-sa -o yaml

Set your AWS Account ID and OIDC Identity provider as environment variables:

ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

export AWS_REGION=<enter region e.g. us-east-1>

OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

Create a JSON file with Trusted Entities for the role:

read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:eks-app-sa"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json

To verify:

cat trust.json

Now, create the IAM role:

export ROLE_NAME=msk-consumer-app-irsa

aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for MSK consumer app on EKS"

You will need to create and attach the policy to the role, since we only want the consumer application to consume data from the MSK cluster and put data to DynamoDB table. This needs to be fine-grained.

In the policy.json file, replace values for MSK cluster and DynamoDB. Create and attach the policy to the role you just created:

aws iam create-policy --policy-name msk-consumer-app-policy --policy-document file://policy.json

aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::$ACCOUNT_ID:policy/msk-consumer-app-policy

Finally, associate the IAM role with the Kubernetes Service Account that you created earlier:

kubectl annotate serviceaccount -n default eks-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME

#confirm
kubectl get serviceaccount/eks-app-sa -o yaml

Deploy MSK Consumer Application to EKS

You can refer to the consumer application code here.

Make sure to update the consumer application manifest (app-iam.yaml) with the MSK cluster endpoint and ECR image (obtained from the stack output).

kubectl apply -f msk-consumer/app-iam.yaml

# verify Pods
kubectl get pods -l=app=msk-iam-consumer-app

Verify End-To-End Solution

Continue to send records using the App Runner producer application:

export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"user2@foo.com","name":"user2"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"user3@foo.com","name":"user3"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"user4@foo.com","name":"user4"}' $APP_RUNNER_URL

Check consumer app logs on EKS to verify:

kubectl logs -f $(kubectl get pods -l=app=msk-iam-consumer-app -o jsonpath='{.items[0].metadata.name}')

Scale-Out Consumer App

The MSK topic created by the producer application has three topic partitions, so we can have a maximum of three consumer instances. 

Scale-out to three replicas:

kubectl scale deployment/msk-iam-consumer-app --replicas=3

Verify the number of Pods and check logs for each of them. Notice how the data consumption is balanced across the three instances.

kubectl get pods -l=app=msk-iam-consumer-app

Conclusion

You were able to deploy the end-to-end application using CDK. This comprised of a producer on App Runner sending data to MSK Serverless cluster and a consumer on EKS persisting data to DynamoDB. All the components were written using the Go programming language!