From 0be4415d6469f55db351f0e31269659e40412383 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Wed, 26 Jul 2017 09:18:42 +0200 Subject: [PATCH 1/5] Reuse the build in tags, as Docker Hub's no-cache policy ... typically causes duplicates, leading to confusion whey you identify images using SHAs (which you should for other people's builds). --- tags/0.11.0.0/Dockerfile | 1 + 1 file changed, 1 insertion(+) create mode 100644 tags/0.11.0.0/Dockerfile diff --git a/tags/0.11.0.0/Dockerfile b/tags/0.11.0.0/Dockerfile new file mode 100644 index 0000000..4b94014 --- /dev/null +++ b/tags/0.11.0.0/Dockerfile @@ -0,0 +1 @@ +FROM solsson/kafka@sha256:75ecbf6c7fb9f814a0ab0e44c5d48d3cb4dd688a244a377f6986eefcb1498614 From 4a7b8daef0ae665baef5f09758f41003755ba0be Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Wed, 26 Jul 2017 09:56:59 +0200 Subject: [PATCH 2/5] Explains why we're here --- README.md | 53 ++++++++++++++++++++++++++++------------------------- 1 file changed, 28 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 50f8bf5..4c0ccc4 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,19 @@ # Kafka docker builds -Automated [Kafka](http://kafka.apache.org/) builds for [solsson/kafka](https://hub.docker.com/r/solsson/kafka/) -and related `kafka-` images under https://hub.docker.com/u/solsson/. +The no-surprises Kafka docker image. Public build at [solsson/kafka](https://hub.docker.com/r/solsson/kafka/), with 100k+ pulls to date. ---- +Design goals: + * Transparent build: self-contained in Dockerfile && directly from source. + * Recommend use of image SHAs for security and stability ([#11](https://github.com/solsson/dockerfiles/pull/11)). + * Same basic platform choices as the more [thoroughly validated](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) [Confluent Platform distribution](https://hub.docker.com/r/confluentinc/cp-kafka/) ([#5](https://github.com/solsson/dockerfiles/pull/5), [#9](https://github.com/solsson/dockerfiles/pull/9)). + * Supports the other tools bundled with Kafka distributions - Zookeeper, topic admin, Connect & Streams ([#7](https://github.com/solsson/dockerfiles/pull/7)). + * Help Kafka beginners through conventions to expose config changes. + * Avoid any [recommendations to use `--net=host`](http://docs.confluent.io/current/cp-docker-images/docs/quickstart.html) because it is impractical in orchestrated multi-node environments. + * Support Kubernetes; transparent and tweakable cluster setups like [Yolean/kubernetes-kafka](https://github.com/Yolean/kubernetes-kafka). -This repo used to contain misc dockerfiles, but they've moved to separate repositories for dockerization projects. -We've kept the repository name to avoid breaking the automated build of solsson/kafka in Docker Hub. - -For legacy Dockerfiles from this repo (if you navigated to here from a Docker Hub [solsson](https://hub.docker.com/u/solsson/) image), -see https://github.com/solsson/dockerfiles/tree/misc-dockerfiles. - ---- +## How to use -Our kafka images are tested in production with https://github.com/Yolean/kubernetes-kafka/. +The default entrypoint `docker run solsson/kafka` will list "bin" scripts and sample config files. Make a guess like `docker run --entrypoint ./bin/kafka-server-start.sh solsson/kafka` or `docker run --entrypoint ./bin/kafka-topics.sh solsson/kafka` to see tool-specific help. You most likely need to mount your own config files, or for `./bin/kafka-server-start.sh` use overrides like: ``` @@ -24,25 +24,20 @@ You most likely need to mount your own config files, or for `./bin/kafka-server- --override advertised.listener=PLAINTEXT://kafka-0:9092 ``` -## One image to rule them all - -Official [Kafka distributions](http://kafka.apache.org/downloads) contain startup scripts and config for various services and clients. Thus `./kafka` produces a multi-purpose image for direct use and specialized docker builds. - -We could build specialized images like `kafka-server` but we have two reasons not to: - * Won't be as transparent in Docker Hub because you can't use Automated Build without scripting. - * In reality you'll need to control your own config anyway. +Beware of `log4j.properties`' location if you mount config. Kafka's bin scripts will guess path unless you set a `KAFKA_LOG4J_OPTS` env. -### Example of downstream image: Kafka Connect +## Upgrade from pre 0.11 images -See ./connect-jmx +Earlier images used `./bin/kafka-server-start.sh` as entrypoint +and had the `zookeeper.connect=zookeeper:2181` (instead of localhost:2181) built in. At upgrade use the command recommended above to restore that functionality. -### Example downstream image: Kafka Streams +## Build and test locally -TODO +To build your own kafka image simply run `docker build ./kafka`. -## Building - -Rudimentary compliance with kubernetes-kafka is tested using a [build-contract](https://github.com/Yolean/build-contract/). +When we develop locally --- stream processing images, monitoring, +compliance with kubernetes-kafka etc --- +we use a [build-contract](https://github.com/Yolean/build-contract/). Build and test using: `docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/:/source solsson/build-contract test`. However... while timing issues remain you need some manual intervention: @@ -60,3 +55,11 @@ docker-compose -f build-contracts/docker-compose.files-aggregation.yml up # demo the JMX->kafka image docker-compose -f build-contracts/docker-compose.monitoring.yml up ``` + +## Why is the repo named `dockerfiles`? + +This repo used to contain misc dockerfiles, but they've moved to separate repositories for dockerization projects. +We've kept the repository name to avoid breaking the automated build of solsson/kafka in Docker Hub. + +For legacy Dockerfiles from this repo (if you navigated to here from a Docker Hub [solsson](https://hub.docker.com/u/solsson/) image), +see https://github.com/solsson/dockerfiles/tree/misc-dockerfiles. From c5a5938bf5e5bf030e58dc0932f9647d6d3ebe68 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Wed, 26 Jul 2017 10:06:10 +0200 Subject: [PATCH 3/5] The experimental from-source 0.11 build https://github.com/solsson/dockerfiles/pull/6 --- tags/0.11.0.0-alpine/Dockerfile | 1 + 1 file changed, 1 insertion(+) create mode 100644 tags/0.11.0.0-alpine/Dockerfile diff --git a/tags/0.11.0.0-alpine/Dockerfile b/tags/0.11.0.0-alpine/Dockerfile new file mode 100644 index 0000000..7fe84db --- /dev/null +++ b/tags/0.11.0.0-alpine/Dockerfile @@ -0,0 +1 @@ +FROM solsson/kafka:0.11.0.0@sha256:4c194db2ec15698aca6f1aa8a2fd5e5c566caed82b4bf43446c388f315397756 From 815f2e44d10d23e3761dbb31b7288216826a379b Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Wed, 26 Jul 2017 10:07:37 +0200 Subject: [PATCH 4/5] The previous production build, on Alpine https://github.com/solsson/dockerfiles/pull/3 --- tags/0.10.2.0-alpine/Dockerfile | 1 + 1 file changed, 1 insertion(+) create mode 100644 tags/0.10.2.0-alpine/Dockerfile diff --git a/tags/0.10.2.0-alpine/Dockerfile b/tags/0.10.2.0-alpine/Dockerfile new file mode 100644 index 0000000..7fe84db --- /dev/null +++ b/tags/0.10.2.0-alpine/Dockerfile @@ -0,0 +1 @@ +FROM solsson/kafka:0.11.0.0@sha256:4c194db2ec15698aca6f1aa8a2fd5e5c566caed82b4bf43446c388f315397756 From 26d5465d962ba93ba8e9edc00e8b518f71b85c0a Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Wed, 26 Jul 2017 10:13:11 +0200 Subject: [PATCH 5/5] Preserves the build used in the community helm chart https://github.com/kubernetes/charts/pull/144 Beware: This image is 841 MB, compared to 301 for the current :latest. See README.md for upgrade instructions. --- tags/0.10.0.1/Dockerfile | 1 + 1 file changed, 1 insertion(+) create mode 100644 tags/0.10.0.1/Dockerfile diff --git a/tags/0.10.0.1/Dockerfile b/tags/0.10.0.1/Dockerfile new file mode 100644 index 0000000..41dc513 --- /dev/null +++ b/tags/0.10.0.1/Dockerfile @@ -0,0 +1 @@ +FROM solsson/kafka:0.10.0.1@sha256:301b1ca59d62e5cb9c030513ac30ebe6c4c020546611e265dd282380b42be6ee