Skip to content

ProvisionedThroughputExceededException while checkpointing causes ShardConsumer illegal state #190

@acm19

Description

@acm19

When KinesisMessageDrivenChannelAdapter finds an exhausted shard it tries to checkpoint its endingSequenceNumber to ensure it is marked as exhausted. If during this process the checkpoint operation throws ProvisionedThroughputExceededException then the ShardConsumer is not marked as closed but its shardIterator is null making it loop forever throwing the following exception over and over until it's killed for other reason.

com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Invalid ShardIterator: Missing ShardIterator (Service: AmazonDynamoDBStreams; Status Code: 400; Error Code: ValidationException; Request ID: <REQUEST_ID>; Proxy: null)

This is particularly important because when the app starts it goes through all these shards causing more contention on the checkpoint table therefore increasing the probability of this situation.

I'll submit a PR to swallow the exception in those cases to avoid illegal states in the ShardConsumer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions