Description
Expected Behavior
When parsing an event coming from Kafka, either Self managed or from MSK, the schemas for the Parser utility should return a list of values. These values should be coming from the items within each record and should be base64-decoded into strings.
Current Behavior
Currently the result of the parsing is this: [['{"key":"value"}']] aka an array of arrays with the values inside.
This is because during our initial implementation we followed the event structure and assumed that there was a relationship many:many between the record keys and the record items. I.e. considering the following sample event:
{
"eventSource": "aws:kafka",
"eventSourceArn": "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4",
"bootstrapServers": "b-2.demo-cluster-1.a1bcde.c1.kafka.us-east-1.amazonaws.com:9092,b-1.demo-cluster-1.a1bcde.c1.kafka.us-east-1.amazonaws.com:9092",
"records": {
"mytopic-0": [
{
"topic": "mytopic",
"partition": 0,
"offset": 15,
"timestamp": 1545084650987,
"timestampType": "CREATE_TIME",
"key": "cmVjb3JkS2V5",
"value": "eyJrZXkiOiJ2YWx1ZSJ9",
"headers": [
{
"headerKey": [104, 101, 97, 100, 101, 114, 86, 97, 108, 117, 101]
}
]
}
]
}
}We wrote the envelope result to accommodate the potential of multiple groups of values, one for each record, with a record being mytopic-0 and it having a list of items.
In reality this can never happen in Lambda because the integration enforces a relationship between 1 topic and one consumer, so in practice the result can be flattened to a simple list of values, i.e. ['{"key":"value"}'].
Code snippet
N/A
Steps to Reproduce
N/A
Possible Solution
No response
Powertools for AWS Lambda (TypeScript) version
latest
AWS Lambda function runtime
22.x
Packaging format used
npm
Execution logs
Metadata
Metadata
Assignees
Labels
Type
Projects
Status