Skip to content

Releases: aws-powertools/powertools-lambda-typescript

v1.18.0

25 Jan 15:48
@github-actions github-actions
v1.18.0
7715df8
Compare
Choose a tag to compare
v1.18.0

Summary

This minor release introduces improvements around how the Idempotency utility handles conditional writes when used with DynamoDB as persistence layer. Additionally the release fixes an issue with expired AppConfig session tokens that affected long lived execution environments.

Idempotency: DynamoDB storage optimization

The Idempotency utility uses conditional writes to persist the idempotency state of a request. A failed conditional write signals to the utility that the request payload being processed has already been tried or is currently being processed.

Previously, condition check errors in single write operations did not return a copy of the item in the event of a condition check error. A separate read request was necessary to get the item and investigate the cause of the error.

Now that AWS introduced the ability to return a copy of the item as it was during the write attempt the Idempotency utility simplifies and lowers the cost of handling retries by removing the need to perform a separate read operation to retrieve the idempotency record of the request already processed.

Note that this feature requires you to use version v3.363.0 of the @aws-sdk/client-dynamodb or newer together with @aws-lambda-powertools/idempotency.

Parameters: AppConfig Session Token handling

When retrieving a configuration from AppConfig the Parameters utility retrieves a session token that can be used to retrieve the next value within 24 hours. Prior to this release the utility mistakenly assumed that the execution environment would be recycled before that time due to the Lambda timeout of 15 minutes.

For those customers who use provisioned concurrency or use the Parameters utility outside of Lambda however this was an issue as it caused the utility to fail retrieving new configurations from AppConfig due to an expired token. Starting from this release the utility keeps track of the token expiration timestamp and retrieves a new one before attempting to call AppConfig if the token has already expired.

Acknowledgements

Congrats to @daschaa, @tolutheo, and @yamatatsu for getting their first PR merged 🎉

Changes

  • chore(maintenance): add overwrite to artifact arn in layer pipeline (#1970) by @dreamorosi
  • chore(maintenance): group CDK cli with other CDK dependencies (#1968) by @dreamorosi
  • chore(maintenance): add environment scope to npm token (#1957) by @dreamorosi
  • chore(maintenance): fine tune dependabot config (#1935) by @dreamorosi
  • chore(ci): add aws-sdk group and ignore middy upgrades in examples (#1893) by @am29d
  • chore(ci): allow deps-dev for semantic PR (#1861) by @am29d
  • chore(ci): allow dependabot PRs (#1860) by @heitorlessa
  • chore(ci): [StepSecurity] Apply security best practices (#1839) by @step-security-bot
  • chore(internal): broken link in boring-cyborg app (#1807) by @daschaa
  • chore(ci): Update log retention for layers (#1809) by @sthulb
  • chore(ci): Update permissions in workflows (#1810) by @sthulb
  • chore(ci): sets base permissions on all workflows (#1801) by @sthulb

🌟New features and non-breaking changes

  • feat(idempotency): leverage new dynamodB Failed conditional writes behavior (#1779) by @tolutheo

📜 Documentation updates

🐛 Bug and hot fixes

  • fix(parameters): refresh AppConfig session token after 24 hrs (#1916) by @dreamorosi

🔧 Maintenance

  • chore(deps-dev): bump husky from 9.0.3 to 9.0.5 (#1971) by @dependabot
  • chore(deps): bump the aws-cdk group with 2 updates (#1972) by @dependabot
  • chore(deps-dev): bump axios from 1.6.5 to 1.6.6 (#1964) by @dependabot
  • chore(deps-dev): bump @types/node from 20.11.5 to 20.11.6 (#1963) by @dependabot
  • chore(deps-dev): bump husky from 8.0.3 to 9.0.3 (#1962) by @dependabot
  • chore(docs): add link to performance tuning demo (#1960) by @dreamorosi
  • chore(deps): bump the aws-sdk group with 9 updates (#1959) by @dependabot
  • chore(deps-dev): bump aws-sdk from 2.1540.0 to 2.1543.0 (#1958) by @dependabot
  • chore(deps-dev): bump @typescript-eslint/eslint-plugin from 6.19.0 to 6.19.1 (#1948) by @dependabot
  • chore(deps-dev): bump @typescript-eslint/parser from 6.19.0 to 6.19.1 (#1946) by @dependabot
  • chore(deps): bump esbuild from 0.19.11 to 0.19.12 (#1952) by @dependabot
  • chore(deps-dev): bump ts-jest from 29.1.1 to 29.1.2 (#1943) by @dependabot
  • chore(deps-dev): bump aws-sdk from 2.1538.0 to 2.1540.0 (#1944) by @dependabot
  • chore(deps): bump the aws-sdk group with 9 updates (#1940) by @dependabot
  • chore(deps): bump the aws-cdk group with 1 update (#1936) by @dependabot
  • chore(deps): bump the aws-sdk group with 4 updates (#1937) by @dependabot
  • chore(deps): bump aws-cdk from 2.121.1 to 2.122.0 (#1938) by @dependabot
  • chore(deps-dev): bump aws-sdk from 2.1537.0 to 2.1538.0 (#1933) by @dependabot
  • chore(deps-dev): bump aws-sdk from 2.1536.0 to 2.1537.0 (#1926) by @dependabot
  • chore(deps-dev): bump prettier from 3.2.2 to 3.2.4 (#1927) by @dependabot
  • chore(deps-dev): bump @types/node from 20.11.4 to 20.11.5 (#1928) by @dependabot
  • chore(deps-dev): bump lerna from 7.4.2 to 8.0.2 (#1923) by @dependabot
  • chore(maintenance): fix install command for .devcontainer setup (#1924) by @am29d
  • chore(deps-dev): bump @types/node from 20.11.2 to 20.11.4 (#1920) by @dependabot
  • chore(deps): fix dependencies and dependabot config (#1917) by @dreamorosi
  • chore(deps-dev): bump typedoc-plugin-missing-exports from 2.1.0 to 2.2.0 (#1911) by @dependabot
  • chore(deps-dev): bump @types/lodash.merge from 4.6.7 to 4.6.9 (#1902) by @dependabot
  • chore(deps-dev): bump @types/node from 20.11.0 to 20.11.2 (#1912) by @dependabot
  • chore(deps-dev): bump @types/aws-lambda from 8.10.129 to 8.10.131 (#1901) by @dependabot
  • chore(deps): bump vscode/devcontainers/javascript-node from b01b5db to ff7fcaf in /.devcontainer (#1892) by @dependabot
  • chore(deps-dev): bump aws-sdk-client-mock-jest from 3.0.0 to 3.0.1 (#1897) by @dependabot
  • chore(deps-dev): bump eslint-config-prettier from 9.0.0 to 9.1.0 (#1872) by @dependabot
  • chore(deps-dev): bump @types/node from 20.10.7 to 20.11.0 in /examples/cdk (#1885) by @dependabot
  • chore(deps-dev): bump @types/node from 20.10.7 to 20.11.0 in /examples/sam (#1890) by @dependabot
  • chore(deps-dev): bump @typescript-eslint/parser from 6.12.0 to 6.18.1 (#1896) by @dependabot
  • chore(deps-dev): bump eslint-import-resolver-typescript from 3.6.0 to 3.6.1 (#1874) by @dependabot
  • chore(deps-dev): bump follow-redirects from 1.15.2 to 1.15.4 (#1842) by @dependabot
  • chore(deps): bump esbuild from 0.19.3 to 0.19.11 (#1868) by @dependabot
  • chore(deps-dev): bump typedoc from 0.25.1 to 0.25.7 (#1869) by @dependabot
  • chore(deps): bump vscode/devcontainers/javascript-node from a20adf8 to b01b5db in /.devcontainer (#1840) by @dependabot
  • chore(deps-dev): bump @aws-sdk/client-cloudwatch from 3.438.0 to 3.485.0 (#1857) by @dependabot
  • chore(deps-dev): bump @types/promise-retry from 1.1.3 to 1.1.6 (#1866) by @dependabot
  • chore(ci): Dependabot fine tuning (#1862) by @am29d
  • chore(deps): bump constructs from 10.2.70 to 10.3.0 (#1846) by @dependabot
  • chore(deps-dev): bump @types/node from 20.6.1 to 20.10.7 in /examples/cdk (#1851) by @dependabot
  • chore(deps-dev): bump @types/node from 20.6.1 to 20.10.7 in /examples/sam (#1853) by @dependabot
  • chore(deps): bump aws-cdk from 2.107.0 to 2.118.0 (#1854) by @dependabot
  • chore(deps): bump aws-cdk-lib from 2.107.0 to 2.118.0 (#1856) by @dependabot
  • docs: fix typos (#1834) by @am29d
  • docs: fix some typos (#1824) by @yamatatsu
  • chore(docs): add AppYourself reference customer (#1826) by @dreamorosi

This release was made possible by the following contributors:

@am29d, @daschaa, @dreamorosi, @heitorlessa, @sthulb, @tolutheo, and @yamatatsu

Contributors

sthulb, am29d, and 7 other contributors
Assets 2
Loading

v1.17.0

24 Nov 19:15
@github-actions github-actions
v1.17.0
33de405
Compare
Choose a tag to compare
v1.17.0

Summary

Starting from this release Powertools for AWS Lambda (TypeScript) will no longer officially support the Node.js 14 runtime for AWS Lambda.

On November 27, AWS will begin the deprecation process for the Node.js 14 runtime, which means it will not be possible to deploy new functions using this runtime after that date. Given this change by Lambda, we can no longer run integration tests or provide full support for Node.js 14.

The previous version, v1.16.0, will remain the last release that is fully compatible with Node.js 14. Upgrading to v1.17.0 will not change any behavior or API - this release represents only a change in official support for the deprecated Node.js 14 runtime.

We announced plans for ending Node.js 14 support three months ago. We recommend upgrading your Lambda functions to Node.js 18 or later to avoid any disruption.

As always, we welcome your feedback and questions.

Changes

📜 Documentation updates

🔧 Maintenance

This release was made possible by the following contributors:

@dreamorosi, @sthulb

Contributors

sthulb and dreamorosi
Assets 2
Loading

v1.16.0

16 Nov 15:51
@github-actions github-actions
v1.16.0
1e51228
Compare
Choose a tag to compare
v1.16.0

Summary

In this minor release we are adding support for two new environments variables to configure the log level in Logger.

You can now configure the log level of for Logger using two new environment variables: AWS_LAMBDA_LOG_LEVEL and POWERTOOLS_LOG_LEVEL. The new environment variables will work along the existing LOG_LEVEL variable that is now considered legacy and will be removed in the future.

Setting the log level now follows this order:

  1. AWS_LAMBDA_LOG_LEVEL environment variable
  2. Setting the log level in code using the logLevel constructor option, or by calling the logger.setLogLevel() method
  3. POWERTOOLS_LOG_LEVEL environment variable

We have also added a new section to the docs to highlight the new behavior.

Changes

🌟New features and non-breaking changes

  • feat(logger): add support for AWS_LAMBDA_LOG_LEVEL and POWERTOOLS_LOG_LEVEL (#1795) by @dreamorosi

📜 Documentation updates

  • feat(logger): add support for AWS_LAMBDA_LOG_LEVEL and POWERTOOLS_LOG_LEVEL (#1795) by @dreamorosi

This release was made possible by the following contributors:

@dreamorosi

Contributors

dreamorosi
Assets 2
Loading

v1.15.0

14 Nov 11:52
@github-actions github-actions
v1.15.0
d508078
Compare
Choose a tag to compare
v1.15.0

Summary

This release brings support for the new Node.js 20 AWS Lambda managed runtime as well as tweaking how the Metrics utility emits logs under the hood.

Node.js 20 support

With this release we are excited to announce that Powertools for AWS Lambda (TypeScript) is compatible with the nodejs20.x AWS Lambda managed runtime 🎉.

The toolkit and our public Lambda Layers are both compatible with the new runtime and no code change should be required on your part.

Metrics

The Metrics utility emits logs using the Embedded Metric Format (EMF). Prior to this release, the logs were emitted using the global console object. This makes it so that in addition to the payload of the log, AWS Lambda adds the request id and timestamp of the log.

For most customers, and especially those who consume the metrics in Amazon CloudWatch, this is fine as CloudWatch is able to parse the EMF content and create custom metrics. For customers who instead want to send the metrics to third party observability providers the presence of these strings means having an extra parsing step.

To support these use cases, and to align with the behavior of the Logger utility, the Metrics utility now uses a dedicated instance of the Console object, which allows it to emit only the content of EMF metric. Just like for the Logger, this behavior can be reverted for development environments by setting the POWERTOOLS_DEV environment variable to a truthy value (i.e. true, yes, 1, on, etc.).

When POWERTOOLS_DEV is enabled, the Metrics utility reverts to using the global console object. This allows customers to place mock and spies and optionally override the implementation for testing purposes.

Changes

🌟New features and non-breaking changes

🔧 Maintenance

This release was made possible by the following contributors:

@dreamorosi

Contributors

dreamorosi
Assets 2
Loading

v1.14.2

03 Nov 11:16
@github-actions github-actions
v1.14.2
1944156
Compare
Choose a tag to compare
v1.14.2

Summary

In this patch release we are fixing a bug that affected the Metrics utility.

When using the utility you can set default dimensions that will be added to every metric emitted by your application.

carbon

Before this release, when setting a dimension using an existing key, the emitted EMF blob would contain duplicate keys. This release fixes the bug and makes sure that keys are deduplicated correctly.

Additionally we have also improved our Gitpod configuration which should make it easier for contributors to get up and running.

Changes

📜 Documentation updates

  • docs(maintenance): add clarification about async decorators (#1778) by @dreamorosi

🐛 Bug and hot fixes

  • fix(metrics): deduplicate dimensions when serialising (#1780) by @am29d

🔧 Maintenance

This release was made possible by the following contributors:

@am29d, @dreamorosi

Contributors

am29d and dreamorosi
Assets 2
Loading

v1.14.1

01 Nov 01:19
@github-actions github-actions
v1.14.1
1805a34
Compare
Choose a tag to compare
v1.14.1

Summary

In this patch release we have improved the logic around creating AWS SDK clients in the Parameters & Idempotency utility, as well as improving our documentation to include sections dedicated to how to contribute and how we manage the project.

Idempotency & Parameters

Both the Idempotency utility and the Parameters one allow you to bring your own AWS SDK to interact with AWS APIs. This is useful when there's a need to customize the SDK client or share an existing one already used in other parts of the function. Prior to this release, both utilities were instantiating a new AWS SDK client by default, only to then replace it with the customer provided one. In these cases, we were needlessly instantiating a client leading to wasted cycles.

Starting from this version both utilities include a refactored logic that instantiate a new SDK client only when a valid one is not provided by the customer. This way customers bringing their own client don't have to pay the performance hit of instantiating multiple clients.

Documentation

As part of this release we have also added a new section to our documentation dedicated to explain our processes. The section includes our roadmap, the maintainers' handbook, and a few sections dedicated to contributing. These sections are designed to be a companion to the contributing guidelines, which we also refreshed to make them more focused, and provide a deeper look around specific areas like setting your development environment, finding you first contribution, project's conventions, and testing.

Changes

  • chore(idempotency): refactor aws sdk init logic (#1768) by @dreamorosi
  • chore(commons): update Powertools UA middleware detection (#1762) by @dreamorosi
  • chore(parameters): refactor provider constructor to init client as needed (#1757) by @dreamorosi
  • chore(ci): add workflow to publish v2 docs on merge (#1756) by @dreamorosi
  • chore(docs): add invisible unicode char to decorator docstrings (#1755) by @dreamorosi
  • chore(maintenance): set removeComments to false in tsconfig.json (#1754) by @dreamorosi
  • chore(tracer): update warning to better format segment name (#1750) by @dreamorosi
  • chore(ci): update v2 release workflow (#1745) by @dreamorosi

📜 Documentation updates

🔧 Maintenance

This release was made possible by the following contributors:

@dreamorosi

Contributors

dreamorosi and dependabot
Assets 2
Loading

v1.14.0

29 Sep 07:27
@github-actions github-actions
v1.14.0
1dc1805
Compare
Choose a tag to compare
v1.14.0

Summary

This release brings all the generally available utilities to the Lambda Layers, improves the Idempotency utility with the addition of a new @idempotent class method decorator, and makes Tracer more reliable.

Lambda Layers

Starting from version 21, which corresponds to this release, of our Lambda Layer includes the Idempotency, Parameters, and Batch Processing utilities. The layer comes with its own reduced copy of AWS SDK v3 for JavaScript clients so you can easily attach it to Lambda functions running on Node.js 16 without having to bundle the SDK.

The layers are available in most commercial AWS Regions, go here to learn more about how to use them and find the ARN for your region.

Idempotency

If you use decorators you can now make your class methods idempotent thanks to the new @idempotent decorator.

idempotent

You can use the decorator on your Lambda handler, like shown in the image above, or on any method that returns a response. This is useful when you want to make a specific part of your code idempotent, for example when your Lambda handler performs multiple side effects and you only want to make part of it safe to retry.

Tracer

When segments generated by your code are about to be sent to the AWS X-Ray daemon, the AWS X-Ray SDK for Node.js serializes them into a JSON string. If the segment contains exotic objects like BigInt, Set, or Map in the metadata the serialization can throw an error because the native JSON.stringify() function doesn't know how to handle these objects.

To guard against this type of runtime errors we have wrapped within try/catch logic the branches of code in Tracer where this issue could happen. Now, when an error gets thrown during the serialization of a segment within Tracer, we will catch it and log a warning instead.

We are also working with the X-Ray team to add a replacer function to the serialization logic directly in the X-Ray SDK so that the issue can be better mitigated.

Acknowledgements

Congratulations to @HaaLeo and @KhurramJalil for having your first contribution to the project, thank you for helping make Powertools better for everyone 🎉

Note
We have officially started working on the next major version of Powertools for AWS (TypeScript) 🔥 We have published a Request For Comment (RFC) that details most of the changes that we have planned and in the coming weeks we'll work on an upgrade guide. We would love to hear what you think about our plan and hear any concern you might have.

Changes

  • chore(ci): create v2 alpha release workflow (#1719) by @dreamorosi
  • chore(layers): add Idempotency, Batch, and Parameters to layer (#1712) by @am29d

🌟New features and non-breaking changes

  • feat(idempotency): add idempotency decorator (#1723) by @am29d
  • feat(tracer): add try/catch logic to decorator and middleware close (#1716) by @dreamorosi
  • feat(layers): add arm64 to integration test matrix (#1720) by @dreamorosi

🌟 Minor Changes

📜 Documentation updates

  • docs(idempotency): address comments (#1724) by @am29d
  • feat(idempotency): add idempotency decorator (#1723) by @am29d
  • docs: typo for serverless framework (#1701) by @HaaLeo

🔧 Maintenance

This release was made possible by the following contributors:

@HaaLeo, @KhurramJalil, @am29d, @dreamorosi

Contributors

am29d, dreamorosi, and 2 other contributors
Assets 2
Loading

v1.13.1

21 Sep 11:18
@github-actions github-actions
v1.13.1
1c7a169
Compare
Choose a tag to compare
v1.13.1

Summary

In this minor release we have removed the upper bound for middy/core 4.x in our peerDependency. We only support middy 3.x and in the last release we added peerDependency section to make it explicit. But some of Powertools user had already middy 4.x in their dependencies and the recent change broke their builds. We have now removed the upper bound to give you the freedom to use middy/core 4.x though we do not support it yet. With more recent requests we will address this issue soon to bring middy 4.x support earlier than we anticipated.

Changes

  • chore: bump version in commons for user agent (#1698) by @am29d
  • chore(tracer): mark createTracer helper function as deprecated (#1692) by @dreamorosi

📜 Documentation updates

  • docs(tracer): update annotation & metadata docs to include full code (#1704) by @dreamorosi
  • docs: remove beta warning from batch and idempotency readme (#1696) by @am29d
  • docs: upgrade mkdocs to fix dark mode custom highlight style (#1695) by @am29d

🐛 Bug and hot fixes

🔧 Maintenance

  • fix(maintenance): remove upper peer dependency Middy (#1705) by @dreamorosi
  • docs: remove beta warning from batch and idempotency readme (#1696) by @am29d

This release was made possible by the following contributors:

@am29d, @dreamorosi, Alexander Melnyk, Alexander Schueren and Release bot[bot]

Contributors

am29d and dreamorosi
Assets 2
Loading

v1.13.0

18 Sep 16:27
@github-actions github-actions
v1.13.0
ed969f3
Compare
Choose a tag to compare
v1.13.0

Summary

In this release we are excited to announce the General Availability of two utilities: Idempotency and Batch.

Batch

Warning
Breaking Change

We have introduced a breaking change in the batch utility. We have initially followed the python implementation for this utility and after multiple reviews we have realised that the choice of async/sync processors is different in NodeJS ecosystem compared to Python. Async functions are often the preferred choice of synchronous functions and thus we renamed AsyncBatchProcessor to BatchProcessor, making it also a default choice. When you need to process a batch synchronously (i.e. SQS Fifo) use the explicit BatchProcessorSync processor. We have added a dedicated section in the documentation to clarify the implications and when to pick the right processor.

Idempotency

We have improved the docs on the customer feedback we have received over the month. You have now more details how to implement your own persistence store. In addition, Batch and Idempotency are often used together, so we added a section how to integrate Batch into Idempotency, here is an example:

import {
  BatchProcessor,
  EventType,
  processPartialResponse,
} from '@aws-lambda-powertools/batch';
import type {
  Context,
  SQSBatchResponse,
  SQSEvent,
  SQSRecord,
} from 'aws-lambda';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';

const processor = new BatchProcessor(EventType.SQS);

const dynamoDBPersistence = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTable',
});
const idempotencyConfig = new IdempotencyConfig({
  eventKeyJmesPath: 'messageId',
});

const processIdempotently = makeIdempotent(
  async (_record: SQSRecord) => {
    // process your event
  },
  {
    persistenceStore: dynamoDBPersistence,
    config: idempotencyConfig,
  }
);

export const handler = async (
  event: SQSEvent,
  context: Context
): Promise<SQSBatchResponse> => {
  idempotencyConfig.registerLambdaContext(context);

  return processPartialResponse(event, processIdempotently, processor, {
    context,
  });
};

Layers

Welcome TelAviv! We have added il-central-1 to our regions and the layer is now available in this region. Of course, we bumped the version to the same number, so you only have to change the region in your ARN:

arn:aws:lambda:il-central-1:094274105915:layer:AWSLambdaPowertoolsTypeScript:19

Acknowledgements

A big thanks again to @erikayao93 for the work on the Batch Processing utility that goes GA in this release, we appreciate your work 🎉

Changes

🌟New features and breaking changes

  • feat(batch): rename AsyncBatchProcessor to default BatchProcessor (#1683) by @am29d

🌟 Minor Changes

📜 Documentation updates

  • feat(batch): rename AsyncBatchProcessor to default BatchProcessor (#1683) by @am29d
  • chore(maintenance): bump dependencies + apply peerDependencies (#1685) by @dreamorosi
  • docs(idempotency): bring your own persistent store (#1681) by @dreamorosi
  • improv(idempotency): expose record status & expiry config + make DynamoDB Client optional (#1679) by @dreamorosi
  • docs(idempotency): add batch integration to idempotency docs (#1676) by @am29d
  • docs(batch): add section on how to trace batch processing (#1673) by @dreamorosi
  • chore(maintenance): adopt TypeScript 5.x (#1672) by @dreamorosi
  • chore(internal): revisit tsconfig files (#1667) by @dreamorosi
  • chore(docs): improve idempotency documentation (#1655) by @am29d
  • chore(ci): add il-central-1 to layer deployment pipeline (#1656) by @am29d
  • chore(docs): added tecRacer GmbH & Co. KG to project supporters (#1653) by @dreamorosi
  • docs(batch): added flow charts (#1640) by @dreamorosi
  • chore(docs): update roadmap, labels, and issue templates (#1632) by @dreamorosi

🐛 Bug and hot fixes

  • fix(parameters): return type when options without transform is used (#1671) by @dreamorosi
  • fix(batch): Update processor to pass only context to handler (#1637) by @erikayao93

🔧 Maintenance

This release was made possible by the following contributors:

@am29d, @dreamorosi, @erikayao93, @sthulb and Release bot[bot]

Contributors

sthulb, am29d, and 2 other contributors
Assets 2
Loading

v1.12.1

25 Jul 13:20
@github-actions github-actions
v1.12.1
2aa1d88
Compare
Choose a tag to compare
v1.12.1

Summary

This release brings another new utility to Powertools for AWS Lambda (TypeScript): introducing the Batch Processing utility ✨ The release also improves the Logger utility, which can now include the cause field in error logs.

Batch Processing Beta

Warning
This utility is currently released as beta developer preview and is intended strictly for feedback and testing purposes and not for production workloads. The version and all future versions tagged with the -beta suffix should be treated as not stable. Up until before the General Availability release we might introduce significant breaking changes and improvements in response to customers feedback.

The batch processing utility handles partial failures when processing batches from Amazon SQS, Amazon Kinesis Data Streams, and Amazon DynamoDB Streams.

Key Features

  • Reports batch item failures to reduce number of retries for a record upon errors
  • Simple interface to process each batch record
  • Build your own batch processor by extending primitives

Problem Statement

When using SQS, Kinesis Data Streams, or DynamoDB Streams as a Lambda event source, your Lambda functions are triggered with a batch of messages.

If your function fails to process any message from the batch, the entire batch returns to your queue or stream. This same batch is then retried until either condition happens first: a) your Lambda function returns a successful response, b) record reaches maximum retry attempts, or c) when records expire.

With this utility, batch records are processed individually – only messages that failed to be processed return to the queue or stream for a further retry.

Getting Started

To get started, install the utility by running:

npm install @aws-lambda-powertools/batch

Then, define a record handler function:

record handler

This function will be called by the Batch Processing utility for each record in the batch. If the function throws an error, the record will be marked as failed and reported once the main handler returns.

Record handlers can be both synchronous and asynchronous, in the latter case the utility will process all the records of your batch in concurrently. To learn more about when it's safe to use async handlers, check the dedicated section in our docs.

SQS Processor

When using SQS as a Lambda event source, you can specify the EventType.SQS to process the records. The response will be a SQSBatchResponse which contains a list of items that failed to be processed.

sqs processing

To learn more about this mode, as well as how to process SQS FIFO queues, check the docs.

Kinesis Processor

When using Kinesis Data Streams as a Lambda event source, you can specify the EventType.KinesisDataStreams to process the records. The response will be a KinesisStreamBatchResponse which contains a list of items that failed to be processed.

kinesis processing

Learn more on the docs.

DynamoDB Stream Processor

When using DynamoDB Streams as a Lambda event source, you can use the BatchProcessor with the EventType.DynamoDBStreams to process the records. The response will be a DynamoDBBatchResponse which contains a list of items that failed to be processed.

dynamodb stream processing

Check the docs to learn more about this processor.

Logger

Starting from this release, when logging an error with the logger.error() method, the Logger utility will include the cause field as part of the JSON-formatted log entry:

log error cause

The cause field is available in the Error class starting from Node.js v16.9.0 and allows to specify the error that caused the one being thrown. This is useful when you are catching an error and throwing your own, but still want to preserve the original cause of the error.

Acknowledgements

Congratulations and a big thank you to @erikayao93 for the work on the new Batch Processing utility 🎉

Changes

🌟New features and non-breaking changes

📜 Documentation updates

  • docs(parameters): add parameters examples cdk and sam (#1622) by @am29d
  • feat(batch): add batch processing utility (#1625) by @dreamorosi
  • feat(batch): Implementation of base batch processing classes (#1588) by @erikayao93

🔧 Maintenance

  • docs(parameters): add parameters examples cdk and sam (#1622) by @am29d
  • chore(maintenance): bump word-wrap from 1.2.3 to 1.2.4 (#1618) by @dependabot
  • chore(ci): restore dependencies & fix e2e tests (#1615) by @dreamorosi
  • chore(maintenance): remove vm2 from package-lock.json (#1613) by @am29d
  • chore(maintenance): remove proxy-agent from dependencies (#1611) by @am29d
  • feat(batch): Implementation of base batch processing classes (#1588) by @erikayao93

This release was made possible by the following contributors:

@am29d, @dreamorosi, @erikayao93

Contributors

am29d, dreamorosi, and 2 other contributors
Assets 2
Loading