KinesisMessageDrivenChannelAdapter doesn't read from all shards

[0x006EA1E5]

In what version(s) of Spring Integration are you seeing this issue?

spring-integration-aws v2.3.3.RELEASE

Describe the bug

KinesisMessageDrivenChannelAdapter doesn't read from all shards if the number of shards is greater than the default MaxResults of 100.

The response from amazonKinesis.listShards(...) is paginated.

If the number of shards available for a stream is greater than MaxResults, then the response will include a NextToken value. This token should then be used in the next request to get the next page.

This should be repeated until all shards have been returned, indicated by the absence of the NextToken value in the last response.

To Reproduce

Create a kinesis stream with more than 100 shards.
Start KinesisMessageDrivenChannelAdapter for the above stream.
See that only 100 shards are read from.

Or mock the response in a unit test

Or go look at the code here:

spring-integration-aws/src/main/java/org/springframework/integration/aws/inbound/kinesis/KinesisMessageDrivenChannelAdapter.java

Line 544 in ee79a1c

ListShardsResult listShardsResult = this.amazonKinesis.listShards(listShardsRequest);

And see here https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ListShards.html

Expected behavior

We should read from all shards, not just the ones listed in the first page of results

[0x006EA1E5]

Here is a PR which should fix this: #176

[0x006EA1E5]

The context of this issue is that we have implemented an "auto-sharding" lambda for a heavily used stream, to raise and lower the shard count in response to data volume in order to decrease cost. This can result in many shards (most of which are closed, but still available), and so we are sometimes hitting the page limit