SimpleConsumer and KafkaConsumer may fetch and return committed messages if compression is enabled

[ecanzonieri]

Looking at the Fetch API on the Kafka documentation it seems that when compression is enabled fetch request may return messages with offsets smaller than the FetchOffset in the request.
It is up to the client library to filter those messages out.
It looks like both SimpleConsumer and KafkaConsumer do not check if the offset in the message is actually higher than the one in the request. So they may return duplicate messages.

[dpkp]

good catch.

[dpkp]

This is fixed in master and will go out in the next release.