Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New serverless pattern - API GW to SQS to Lambda to DDB #1480

Merged
merged 5 commits into from
Jul 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions apigw-sqs-lambda-ddb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Amazon API Gateway to Amazon SQS to AWS Lambda to Amazon DynamoDB

This pattern explains how to deploy a SAM application with Amazon API Gateway, Amazon SQS, AWS Lambda, AWS Secrets Manager and Amazon DynamoDB.

This pattern is useful to accept and respond to requests quickly but offloading the processing as asynchronous process e.g., webhook endpoints. Once the request is placed in SQS, API gateway responds back to the caller immediately without waiting for those messages to be processed.

When an HTTP POST request is made to the Amazon API Gateway endpoint, Gateway authorizes the request by checking Basic auth credentials against values in AWS Secrets manager. When credentials are valid, request payload is sent to Amazon Simple Queue Service. AWS Lambda function consumes event from the Queue and inserts the event/payload into the Amazon DynamoDB table. Amazon Simple Queue Service is also configured with a Dead Letter Queue where events are sent when retries to process those messages are repeatedly failed.

Key Benefits:
- Operations: AWS services used in this pattern can scale easily and quickly based on incoming traffic.
- Security: The APIs created with Amazon API Gateway expose HTTPS endpoints only. All user data stored in Amazon DynamoDB and Amazon SQS is fully encrypted at rest. Secrets are securely stored in AWS Secrets Manager.
- Reliability: A dead-letter queue ensures no messages are lost due to issues. Helps debugging failed messages and once underlying issue is resolved, messages can be redrived back to source queues for processing.
- Performance: AWS Lambda supports batching with Amazon SQS. Lambda reads messages in batches and invokes the function once for each batch.
- Cost: Pay only for what you use. No minimum fee.


Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a great pattern and very useful, so i think it would be good to mention in the intro of this pattern why is useful.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expanded the intro section along with some key benefits


## Requirements

* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed

## Deployment Instructions

1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
```
git clone https://github.com/aws-samples/serverless-patterns
```
1. Change directory to the pattern directory:
```
cd apigw-sqs-lambda-ddb
```
1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:
```
sam deploy --guided
```
1. During the prompts:
* Enter a stack name
* Enter the desired AWS Region
* Allow SAM CLI to create IAM roles with the required permissions.

Once you have run `sam deploy -guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults.

1. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing.

## How it works

<img src="docs/apigw-sqs-lambda-ddb.drawio.png" alt="architecture diagram"/>

- This pattern deploys an Amazon API Gateway HTTP API with route /submit configured with basic authentication.
- On receiving a request, API Gateway will invoke a Lambda authorizer which validates the request and returns a policy informing API Gateway to accept or deny the request.
- When request is accepted, API Gateway sends the message payload to a queue in SQS. SQS uses another queue as Dead Letter Queue to send the messages in case of continued failures to process the messages from downstream lambda.
- Messages from SQS is posted to a lambda function to process them.
- Lambda function receives the messages from SQS and saves them into a DynamoDB table.

## Testing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as you add the DLQ it would be great to have an example on how to test that that is working.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a section to test DLQ


Once the application is deployed, follow the below steps to test this pattern:

### Retrieve Information
- Retrieve the HttpApiEndpoint value from CloudFormation Outputs
- Retrieve the username and password from Secrets Manager in AWS Console. Base64 encoding of username and password joined by a single colon : will be used as {Credential} in Authorization header.

### Construct Request
- Replace the placeholders in the Curl request given below with retrieved values.
- Region: AWS Region used for deployment e.g., us-east-1

Example POST Request (Curl):
```
curl --location 'https://{HttpApiEndpoint}.execute-api.{Region}.amazonaws.com/submit' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic {Credentials}' \
--data '{
"eventId": "f5337ee6-e13e-4bcb-8082-872898b5d1a8",
"message": "Sample message for testing"
}'
```

### Verify
- Execute the fully formed Curl request and verify that you received a successful response with an XML message. Response is basically an XML representation of SQS SendMessage response.
Example:
```
<?xml version="1.0"?>
<SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/">
<SendMessageResult>
<MessageId>9c22e4eb-09f6-4374-95b8-f6030f33da33</MessageId>
<MD5OfMessageBody>93355f3c10001c53dcae387ea9d5b71e</MD5OfMessageBody>
</SendMessageResult>
<ResponseMetadata>
<RequestId>84728dba-affe-50d7-8e62-d08a5bb32a38</RequestId>
</ResponseMetadata>
</SendMessageResponse>
```

- View the contents of DynamoDB Table "EventTable" from AWS Console and verify that JSON data used in your payload is saved as a record.


### Steps To Verify Dead-Letter Queue

- Messages should be moved to Dead-Letter Queue when there's repeated failures in processing the message by Lambda function.
- Lambda which consumes the message from source queue (BufferingQueue) expects the message to be in JSON format. When the message is not in JSON format, it will fail to process it. To verify that messages are moved to Dead-Letter Queue, use a request which doesn't use a JSON payload. Example:
```
curl --location 'https://{HttpApiEndpoint}.execute-api.{Region}.amazonaws.com/submit' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic {Credentials}' \
--data 'Test Message'
```
- After few minutes, login to AWS console and verify that the Dead-Letter queue (DeadLetterQueue) has available messages greater than 0. You can also try "Poll for messages" in this queue to verify the message body.

## Cleanup

1. Delete the stack
```bash
sam delete
```

----
Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.

SPDX-License-Identifier: MIT-0
30 changes: 30 additions & 0 deletions apigw-sqs-lambda-ddb/api.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
openapi: "3.0.1"
info:
title: "Http Api"
version: "2023-06-25 17:32:29UTC"
paths:
/submit:
post:
responses:
default:
description: "Response for POST /"
x-amazon-apigateway-integration:
type: "aws_proxy"
integrationSubtype: "SQS-SendMessage"
credentials:
Fn::GetAtt: [ApiGatewayRole, Arn]
connectionType: "INTERNET"
payloadFormatVersion: "1.0"
requestParameters:
MessageBody: "$request.body"
QueueUrl:
Ref: BufferingQueue

x-amazon-apigateway-cors:
allowMethods:
- "*"
maxAge: 0
allowCredentials: false
allowOrigins:
- "*"
x-amazon-apigateway-importexport-version: "1.0"
67 changes: 67 additions & 0 deletions apigw-sqs-lambda-ddb/docs/apigw-sqs-lambda-ddb.drawio
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
<mxfile host="Electron" modified="2023-07-05T13:00:07.956Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.2.8 Chrome/112.0.5615.165 Electron/24.2.0 Safari/537.36" etag="DzfVa3Ub_cXn1OShkBAz" version="21.2.8" type="device">
<diagram id="_h-H2UqnfAHlt3tjEZla" name="Pattern">
<mxGraphModel dx="1114" dy="822" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
<mxCell id="CeDZHfQ7SwTlkByDnDzr-10" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" parent="1" source="CeDZHfQ7SwTlkByDnDzr-2" target="CeDZHfQ7SwTlkByDnDzr-3" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-2" value="Client" style="verticalLabelPosition=bottom;html=1;verticalAlign=top;strokeWidth=1;align=center;outlineConnect=0;dashed=0;outlineConnect=0;shape=mxgraph.aws3d.client;aspect=fixed;strokeColor=none;fillColor=#777777;" parent="1" vertex="1">
<mxGeometry x="50" y="70" width="60" height="104" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-11" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" parent="1" source="CeDZHfQ7SwTlkByDnDzr-3" target="CeDZHfQ7SwTlkByDnDzr-4" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="Pq9tvVw2lUp87TCCF5MC-6" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="CeDZHfQ7SwTlkByDnDzr-3" target="Pq9tvVw2lUp87TCCF5MC-4" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-3" value="API Gateway" style="sketch=0;points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;gradientColor=#FF4F8B;gradientDirection=north;fillColor=#BC1356;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.api_gateway;" parent="1" vertex="1">
<mxGeometry x="220" y="83" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-12" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" source="CeDZHfQ7SwTlkByDnDzr-4" target="CeDZHfQ7SwTlkByDnDzr-5" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-15" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.5;entryY=0;entryDx=0;entryDy=0;entryPerimeter=0;dashed=1;dashPattern=12 12;" parent="1" source="CeDZHfQ7SwTlkByDnDzr-4" target="CeDZHfQ7SwTlkByDnDzr-14" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-4" value="SQS" style="sketch=0;points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;gradientColor=#FF4F8B;gradientDirection=north;fillColor=#BC1356;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.sqs;" parent="1" vertex="1">
<mxGeometry x="420" y="83" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-13" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" source="CeDZHfQ7SwTlkByDnDzr-5" target="CeDZHfQ7SwTlkByDnDzr-6" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-5" value="Lambda" style="sketch=0;points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;gradientColor=#F78E04;gradientDirection=north;fillColor=#D05C17;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.lambda;" parent="1" vertex="1">
<mxGeometry x="620" y="83" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-6" value="DynamoDB&lt;br&gt;(Pay Per Request)" style="sketch=0;points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;gradientColor=#4D72F3;gradientDirection=north;fillColor=#3334B9;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.dynamodb;" parent="1" vertex="1">
<mxGeometry x="830" y="83" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="CeDZHfQ7SwTlkByDnDzr-14" value="DLQ" style="sketch=0;points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;gradientColor=#FF4F8B;gradientDirection=north;fillColor=#BC1356;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.sqs;labelPosition=center;" parent="1" vertex="1">
<mxGeometry x="420" y="240" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="Pq9tvVw2lUp87TCCF5MC-4" value="" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;dashPattern=12 12;" parent="1" vertex="1">
<mxGeometry x="194" y="240" width="130" height="70" as="geometry" />
</mxCell>
<mxCell id="Pq9tvVw2lUp87TCCF5MC-5" value="Authorizer" style="sketch=0;points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;gradientColor=#F78E04;gradientDirection=north;fillColor=#D05C17;strokeColor=#ffffff;dashed=0;verticalLabelPosition=middle;verticalAlign=middle;align=left;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.lambda;labelPosition=right;" parent="1" vertex="1">
<mxGeometry x="207" y="250" width="50" height="50" as="geometry" />
</mxCell>
<mxCell id="o9DxeRwTcoc4fpytT8ro-1" value="1" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;fontColor=#FFFFFF;labelBackgroundColor=none;fillColor=#000000;" parent="1" vertex="1">
<mxGeometry x="150" y="100" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="ZOjlcuqnHgMgoiZsBKUH-1" value="2" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;fontColor=#FFFFFF;labelBackgroundColor=none;fillColor=#000000;" parent="1" vertex="1">
<mxGeometry x="262" y="190" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="ZOjlcuqnHgMgoiZsBKUH-2" value="3" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;fontColor=#FFFFFF;labelBackgroundColor=none;fillColor=#000000;" parent="1" vertex="1">
<mxGeometry x="350" y="100" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="ZOjlcuqnHgMgoiZsBKUH-3" value="4" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;fontColor=#FFFFFF;labelBackgroundColor=none;fillColor=#000000;" parent="1" vertex="1">
<mxGeometry x="550" y="100" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="ZOjlcuqnHgMgoiZsBKUH-4" value="5" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;fontColor=#FFFFFF;labelBackgroundColor=none;fillColor=#000000;" parent="1" vertex="1">
<mxGeometry x="750" y="100" width="20" height="20" as="geometry" />
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading