feat: upgrade to Kafka 4.0.0 #562

gaeljw · 2025-03-21T17:49:07Z

Resolves #561, supersedes #560

Tasks

Update code
Unit tests OK
Update docs
Remove any reference to Zookeeper
Integration tests
- Private company projects (basic usage)
- zio-kafka, works partially but most errors seem unrelated to embedded-kafka (Kafka 4.x (Integration tests for embedded-kafka 4.x - DO NOT MERGE) zio/zio-kafka#1516)
Changelog / upgrade notes

erikvanoosten · 2025-03-21T19:18:02Z

Since kafka is now compiled for scala 3 (right?) do we still need "org.apache.kafka" %% "kafka" % Versions.Kafka cross CrossVersion.for3Use2_13 in the sbt files?

gaeljw · 2025-03-21T19:42:47Z

Since kafka is now compiled for scala 3 (right?) do we still need "org.apache.kafka" %% "kafka" % Versions.Kafka cross CrossVersion.for3Use2_13 in the sbt files?

Unless I missed something, it's not published for Scala 3. Only 2.13.

https://mvnrepository.com/artifact/org.apache.kafka/kafka

erikvanoosten · 2025-03-22T07:22:57Z

Okay, sorry, my bad.

gaeljw · 2025-03-23T16:26:01Z

For some reason, unit tests relying on "arbitrary available ports" (port = 0) are failing on my laptop, waiting to see if also fails in CI. If so, I probably have changed something that make them fail but not obvious for now.

EDIT: also fails on CI. Will fix tomorrow.

EDIT 2: error is likely due to the fact that the controller is started on a random port, but the broker is configured to reach the controller on port 0 instead of the actual port. This is a bit more tricky than expected because the random port is chosen by the controller when it starts. I may have to change the config of the broker after the controller has started.

build.sbt

gaeljw · 2025-03-23T16:53:19Z

.github/workflows/coverage-report.yml

      with:
        distribution: 'temurin'
-        java-version: 8
+        java-version: 17


Kafka Connect & Server are not available for Java < 17 anymore: https://kafka.apache.org/40/documentation/compatibility.html

gaeljw · 2025-03-23T16:56:28Z

project/Dependencies.scala

    lazy val prodDeps: Seq[ModuleID] = Seq(
      "org.apache.kafka" %% "kafka" % Versions.Kafka cross CrossVersion.for3Use2_13
    )
+    lazy val testDeps: Seq[ModuleID] = Seq(


ℹ️ The library is not pulled transitively anymore but we do use it in our tests.

embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/kafkaOps.scala

gaeljw · 2025-03-24T16:10:45Z

I got unit tests to pass. Next step: run some integration tests on projects of my company + on OSS projects like zio-kafka.

erikvanoosten · 2025-03-25T09:02:42Z

on OSS projects like zio-kafka

I can run the test suite of zio-kafka. Is there a snapshot version somewhere?

gaeljw · 2025-03-25T12:38:41Z

on OSS projects like zio-kafka

I can run the test suite of zio-kafka. Is there a snapshot version somewhere?

Not yet, I was planning to build one locally and run the tests locally on my machine.

gaeljw · 2025-03-26T17:32:15Z

@erikvanoosten just to keep you informed that I started trying on the zio-kafka codebase. There are a couple of errors. Some are due to changes in 4.x that need zio-kafka to adapt as well, some may be due to issues with embedded-kafka. I haven't had time to look at it closely for now.

[error] Failed tests:
[error] 	zio.kafka.admin.AdminSaslSpec
[error] 	zio.kafka.consumer.ConsumerSpec
[error] 	zio.kafka.admin.AdminSpec
[error] 	zio.kafka.ProducerSpec
[error] 	zio.kafka.utils.SslHelperSpec

Feel free to checkout this PR and build it locally if you want to try on your end for zio-kafka.

I haven't tried yet on other projects.

erikvanoosten · 2025-03-26T17:56:18Z

@gaeljw Did you already try it on our kafka 4 branch?

gaeljw · 2025-03-26T18:30:30Z

@gaeljw Did you already try it on our kafka 4 branch?

I had not see there was already a branch. Oops! I'll start from it then :)

gaeljw · 2025-03-30T12:15:16Z

embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/AdminOps.scala

+    * @return
+    *   the list of topic names
+    */
+  def listTopics()(implicit config: C): Try[Set[String]] = {


I've added listTopics() and describeTopics() methods for our own tests. Previously we were asserting topics creation/properties by using a ZK client and reading from ZK but now we have to do differently. I chose to reuse the Kafka AdminClient.

embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/kafkaOps.scala

gaeljw · 2025-03-30T14:11:09Z

I've tested on a few of my companies projects (simple usages), no issue found. Even though the upgrade PR in zio-kafka is failing, I believe these errors are not directly related to embedded-kafka and I would suggest to release a 4.0.0 of embedded-kafka. This will allow the community to use it and if they're any issue, we'll obviously fix them by releasing new 4.0.0.x versions.

Copilot

Pull Request Overview

This pull request upgrades the project to Kafka 4.0.0, updating documentation and build configurations to reflect the new requirements and changes in functionality. Key changes include:

Updated README.md with a version compatibility matrix, upgrade notes, and replacement of Zookeeper settings with controller settings.
Adjusted GitHub workflow configurations to use Java 17 instead of Java 8.

Reviewed Changes

Copilot reviewed 6 out of 22 changed files in this pull request and generated 1 comment.

File	Description
README.md	Updated documentation for upgraded Kafka version and configuration changes.
.github/workflows/test.yml	Changed Java version from 8 to 17 for tests.
.github/workflows/release.yml	Changed Java version from 8 to 17 for release workflows.
.github/workflows/coverage-report.yml	Changed Java version from 8 to 17 for coverage reporting.

Files not reviewed (16)

embedded-kafka/src/main/scala/io/github/embeddedkafka/EmbeddedKafka.scala: Language not supported
embedded-kafka/src/main/scala/io/github/embeddedkafka/EmbeddedKafkaConfig.scala: Language not supported
embedded-kafka/src/main/scala/io/github/embeddedkafka/EmbeddedServer.scala: Language not supported
embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/AdminOps.scala: Language not supported
embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/ConsumerOps.scala: Language not supported
embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/embeddedKafkaOps.scala: Language not supported
embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/kafkaOps.scala: Language not supported
embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/zooKeeperOps.scala: Language not supported
embedded-kafka/src/test/scala/io/github/embeddedkafka/EmbeddedKafkaMethodsSpec.scala: Language not supported
embedded-kafka/src/test/scala/io/github/embeddedkafka/EmbeddedKafkaObjectSpec.scala: Language not supported
embedded-kafka/src/test/scala/io/github/embeddedkafka/EmbeddedKafkaWithRunningKafkaOnFoundPortSpec.scala: Language not supported
embedded-kafka/src/test/scala/io/github/embeddedkafka/EmbeddedKafkaWithRunningKafkaSpec.scala: Language not supported
kafka-connect/src/main/scala/io/github/embeddedkafka/connect/EmbeddedKafkaConnect.scala: Language not supported
kafka-connect/src/test/scala/io/github/embeddedkafka/connect/ExampleKafkaConnectSpec.scala: Language not supported
kafka-streams/src/main/scala/io/github/embeddedkafka/streams/EmbeddedKafkaStreams.scala: Language not supported
kafka-streams/src/test/scala/io/github/embeddedkafka/streams/ExampleKafkaStreamsSpec.scala: Language not supported

README.md

Co-authored-by: Copilot <[email protected]>

gaeljw · 2025-04-06T15:29:19Z

@francescopellegrini , @guizmaii or @NeQuissimus: would one of you have time to give a look to the PR?

erikvanoosten

LGTM

gaeljw · 2025-04-13T10:09:21Z

Thanks @erikvanoosten (and welcome on board!) . I'll ship the release in the afternoon :)

gaeljw assigned gaeljw and unassigned gaeljw Mar 21, 2025

gaeljw force-pushed the chore/kafka-400 branch from dbbda89 to 90c3865 Compare March 23, 2025 16:24

gaeljw commented Mar 23, 2025

View reviewed changes

build.sbt Show resolved Hide resolved

gaeljw force-pushed the chore/kafka-400 branch 3 times, most recently from 1e713a4 to 23b41b5 Compare March 23, 2025 16:52

gaeljw commented Mar 23, 2025

View reviewed changes

embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/kafkaOps.scala Outdated Show resolved Hide resolved

gaeljw linked an issue Mar 23, 2025 that may be closed by this pull request

Support for kraft mode #415

Closed

gaeljw force-pushed the chore/kafka-400 branch 3 times, most recently from 0b751c9 to ee9ab47 Compare March 24, 2025 13:18

gaeljw mentioned this pull request Mar 26, 2025

Update kafka-clients and embedded-kafka to 4.0.0 zio/zio-kafka#1504

Merged

gaeljw commented Mar 30, 2025

View reviewed changes

embedded-kafka/src/main/scala/io/github/embeddedkafka/ops/kafkaOps.scala Show resolved Hide resolved

gaeljw mentioned this pull request Mar 30, 2025

Kafka 4.x (Integration tests for embedded-kafka 4.x - DO NOT MERGE) zio/zio-kafka#1516

Closed

feat: upgrade for Kafka 4.0

aefce5f

gaeljw force-pushed the chore/kafka-400 branch from 6a93855 to aefce5f Compare March 30, 2025 14:07

gaeljw requested review from NeQuissimus, Copilot, francescopellegrini and guizmaii March 30, 2025 14:11

Copilot AI reviewed Mar 30, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

gaeljw marked this pull request as ready for review March 30, 2025 14:13

Update README.md

2f2d29d

Co-authored-by: Copilot <[email protected]>

erikvanoosten approved these changes Apr 13, 2025

View reviewed changes

gaeljw merged commit c8bca4c into master Apr 13, 2025
3 checks passed

gaeljw deleted the chore/kafka-400 branch April 13, 2025 11:47

feat: upgrade to Kafka 4.0.0 #562

feat: upgrade to Kafka 4.0.0 #562

Uh oh!

Conversation

gaeljw commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

erikvanoosten commented Mar 21, 2025

Uh oh!

gaeljw commented Mar 21, 2025

Uh oh!

erikvanoosten commented Mar 22, 2025

Uh oh!

gaeljw commented Mar 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

gaeljw Mar 23, 2025

Choose a reason for hiding this comment

Uh oh!

gaeljw Mar 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gaeljw commented Mar 24, 2025

Uh oh!

erikvanoosten commented Mar 25, 2025

Uh oh!

gaeljw commented Mar 25, 2025

Uh oh!

gaeljw commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erikvanoosten commented Mar 26, 2025

Uh oh!

gaeljw commented Mar 26, 2025

Uh oh!

gaeljw Mar 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gaeljw commented Mar 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

gaeljw commented Apr 6, 2025

Uh oh!

erikvanoosten left a comment

Choose a reason for hiding this comment

Uh oh!

gaeljw commented Apr 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gaeljw commented Mar 21, 2025 •

edited

Loading

gaeljw commented Mar 23, 2025 •

edited

Loading

gaeljw commented Mar 26, 2025 •

edited

Loading

gaeljw commented Apr 13, 2025 •

edited

Loading