Setting the Kafka Streams property default.dsl.store has no effect #3176
Description
Between 2.9.x and 3.2.x (main)
Describe the bug
In Kafka Streams, all DSL operators use RocksDB stores by default. With KIP-591 (introduced with Kafka 3.2), a new Kafka Streams property called default.dsl.store was proposed, which allows to set the default built-in store type for the whole application. When using Spring for Apache Kafka and setting default.dsl.store to in_memory, this has no effect and RocksDB is still used for all the stores.
I have already fixed this bug and I will create a pull request soon.
To Reproduce
In order to reproduce the bug, one needs to set the property default.dsl.store to in_memory. Despite this setting, RocksDB files will still be created under the directory used as state.dir.
A minimal reproducible sample is available under the chapter "Sample" of this bug report.
Expected behavior
Spring for Apache Kafka must comply with the setting and use in-memory stores when asked to do so.
As a side remark, KIP-591 has been deprecated with Kafka 3.7 in favor KIP-954. However, despite the setting default.dsl.store now being deprecated, it is still available and should therefore be supported until it is completely removed, especially in spring-kafka 2.9.x -> 3.1.x, where the mechanism introduced by KIP-954 is not yet available.
It is also important to mention that KIP-954 is already properly supported by Spring for Apache Kafka and is in no way affected by the bug described here.
Sample
A minimal reproducible sample can be found under https://github.com/cedric-schaller/default-dsl-store-bug. When building it, the test DefaultDslStoreBugApplicationIT will fail, indicating that a RocksDB state store is being used despite the instruction to use in_memory state stores by default.
This sample uses the latest Spring Boot version (currently 3.2.4) which relies on spring-kafka 3.1.3. The PR fixing the bug will however be made on the main branch (spring-kafka 3.2.x). It can then be back-ported (e.g. by cherry-picking) to all the versions affected:
- 2.9.x
- 3.0.x
- 3.1.x