RFC: Add Idempotency Package

[hossambarakat]

Is this related to an existing feature request or issue?

No

Which AWS Lambda Powertools utility does this relate to?

Other

Summary

The idempotency package that provides a simple solution to convert your Idempotent functions into idempotent operations which are safe to retry.

The developer will annotate the Lambda function with Idempotent attribute and the package will take care of making the function idempotent by:

• Extracting unique identifier to represent the event
• Search the persistence layer for the extracted identifier
  • if response exists in the persistence layer; return the response.
  • if response doesn't exist in the persistence layer; execute the function, save result in persistence layer and return the response.

Use case

Idempotency is a very useful design characteristic of any system. It enables safely retrying requests without accidentally performing the same operation twice.

Networks are unreliable which leads to situations where the Lambda Function receives duplicate events. Idempotency enables overcoming this unreliable environment.

It is also mentioned on Stream processing section of the AWS Well Architected Framework - Serverless Lens that "Duplicated records may occur, and you must use both retries and idempotency within your application
for both consumers and producers"

Please note that Idempotency module is already implemented for Lambda Powertools for Python and Java.

Proposal

Getting Started

Developer can quickly start by configuring Idempotency and using it with the Idempotent attribute on your Lambda function.

public class CoolFunction
{
    public CoolFunction()
    {
        Idempotency.Configure(builder => builder.UseDynamoDb("idempotency_table"));
    }
    
    [Idempotent]
    public Task<string> FunctionHandler(string input, ILambdaContext context)
    {
        return Task.FromResult(input.ToUpper());
    }
}

The package comes with a built-in DynamoDB persistence store. The table has to be created before using Idempotency

Configuration	Value	Notes
Partition key	id
TTL attribute name	expiration	This can only be configured after your table is created if you're using AWS Console

Advanced Configurations

Idempotency behavior can be further configured with IdempotencyOptions using a builder:

Idempotency.Configure(builder =>
            builder
                .WithOptions(optionsBuilder =>
                    optionsBuilder.WithEventKeyJmesPath("id")
                        .WithPayloadValidationJmesPath("paymentId")
                        .WithThrowOnNoIdempotencyKey(true)
                        .WithExpiration(TimeSpan.FromMinutes(1))
                        .WithUseLocalCache(true)
                        .WithHashFunction("MD5"))
                .UseDynamoDb("idempotency_table"));

These are the available options for further configuration:

Parameter	Default	Description
EventKeyJMESPath	""	JMESPath expression to extract the idempotency key from the event record.
PayloadValidationJMESPath	""	JMESPath expression to validate whether certain parameters have changed in the event
ThrowOnNoIdempotencyKey	False	Throw exception if no idempotency key was found in the request
ExpirationInSeconds	3600	The number of seconds to wait before a record is expired
UseLocalCache	false	Whether to locally cache idempotency results (LRU cache)
HashFunction	MD5	Algorithm to use for calculating hashes, as supported by System.Security.Cryptography.HashAlgorithm (eg. SHA1, SHA-256, ...)

When using DynamoDB as a persistence layer, you can alter the attribute names by passing these parameters when initializing the persistence layer:

Use the builder to customize the table structure:

Idempotency.Configure(builder =>
            builder
                .UseDynamoDb(storeBuilder => storeBuilder
                    .WithTableName("TABLE_NAME")
                    .WithKeyAttr("idempotency_key")
                    .WithExpiryAttr("expires_at")
                    .WithStatusAttr("current_status")
                    .WithDataAttr("result_data")
                    .WithValidationAttr("validation_key")));

Here is the description of each attribute

Parameter	Required	Default	Description
TableName	Y		Table name to store state
KeyAttr		id	Partition key of the table. Hashed representation of the payload (unless SortKeyAttr is specified)
ExpiryAttr		expiration	Unix timestamp of when record expires
StatusAttr		status	Stores status of the Lambda execution during and after invocation
DataAttr		data	Stores results of successfully idempotent methods
ValidationAttr		validation	Hashed representation of the parts of the event used for validation
SortKeyAttr			Sort key of the table (if table is configured with a sort key).
StaticPkValue		idempotency#{LAMBDA_FUNCTION_NAME}	Static value to use as the partition key. Only used when SortKeyAttr is set.

Disabling the idempotency utility
When testing your code, you may wish to disable the idempotency logic altogether and focus on testing your business logic. To do this, you can set the environment variable POWERTOOLS_IDEMPOTENCY_DISABLED to true.

Out of scope

Implementing other persistence layers such as Redis, MySQL,...

Potential challenges

• This solution will have increase the execution time of Lambda functions. Every execution would require accessing the persistence layer.

• When using the built-in DynamoDB persistence store, the function's responses must be smaller than 400KB.

• The solution supports functions that has return type Task<T> because the AmazonDynamoDBClient supports only async methods for .NET.

Dependencies and Integrations

The implementation will leverage the AspectInjector package to build the Idempotent attribute.

Alternative solutions

No other designs considered at the moment.

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Python, Java, TypeScript