Skip to content

Kinesis Firehose lambda api doesn't support routing to multiple table for iceberg #575

@amalakar

Description

@amalakar

Is your feature request related to a problem? Please describe.
As per: https://docs.aws.amazon.com/firehose/latest/dev/apache-iceberg-format-input-record-different.html
In order to set the target table while processing via lambda needs to set metadata as:

    "metadata":{
        "otfMetadata":{
            "destinationTableName":"gameevents",
            "destinationDatabaseName":"PlayerSpawn",
            "operation":"insert"
        }
    }

But the response metadata struct only supports partition keys

PartitionKeys map[string]string `json:"partitionKeys"`

type KinesisFirehoseResponseRecord struct {
.....
	Metadata KinesisFirehoseResponseRecordMetadata `json:"metadata"`
}

type KinesisFirehoseResponseRecordMetadata struct {
	PartitionKeys map[string]string `json:"partitionKeys"`
}

Describe the solution you'd like
Please support otfMetadata as part of metadata for lambda response object.

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/eventsissue or feature request related to the events package

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions