[SPARK-37354][K8S][TESTS] Make the Java version installed on the container image used by the K8s integration tests with SBT configurable #34628

sarutak · 2021-11-17T08:18:47Z

What changes were proposed in this pull request?

This PR fixes an issue that the Java version installed on the container image used by K8s integration tests are not configurable in case the tests are launched by SBT.

bin/docker-image-tool.sh supports -b java_image_tag=<java image tag> option to the Java version installed on the container image configurable.
But project/SparkBuild.scala doesn't provide the way to pass the option so we can't change the Java version if the tests are launched by SBT.

To allow to change the version, this PR changes project/SparkBuild.scala to recognized the environment variable JAVA_IMAGE_TAG, which is the same manner that resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh does.

Why are the changes needed?

The K8s integration tests should be launched by any supported Java versions.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Confirmed that the Java version installed on the container image.

In case JAVA_IMAGE_TAG is not set:

# Run the integration tests to create the container image
$ build/sbt -Phive -Pkubernetes  -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"  

# Create and login the container.
docker run -it <image>:<tag> /bin/bash

# Confirm the version in the container.
185@4023f49aa0b0:/opt/spark/work-dir$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

In case JAVA_IMAGE_TAG is set:

# Run the integration tests to create the container image
$ JAVA_IMAGE_TAG=11-jre-slim build/sbt -Phive -Pkubernetes  -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"  

# Create and login the container.
docker run -it <image>:<tag> /bin/bash

# Confirm the version in the container.
185@fdf2d9d5ee26:/opt/spark/work-dir$ java -version
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)

… java_image_tag.

sarutak · 2021-11-17T08:20:04Z

project/SparkBuild.scala

          "-t", imageTag.value,
          "-p", s"$bindingsDir/python/Dockerfile",
          "-R", s"$bindingsDir/R/Dockerfile",
+          "-b", s"java_image_tag=${sys.env.getOrElse("JAVA_IMAGE_TAG", "8-jre-slim")}",


The default Java version is 8 if the K8s integration tests are launched by Maven so this PR complies with that.

SparkQA · 2021-11-17T09:25:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49796/

SparkQA · 2021-11-17T10:09:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49796/

SparkQA · 2021-11-17T11:22:31Z

Test build #145324 has finished for PR 34628 at commit 2a63f02.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2021-11-17T20:50:37Z

LGTM, the K8s failures seem unrelated I've got a follow up PR for the decom failure - #34636

sarutak · 2021-11-21T17:25:10Z

I'll merge this as I have one LGTM from @holdenk . But I'll rebase before merge, just in case.

…e-java-version

SparkQA · 2021-11-21T18:31:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49957/

SparkQA · 2021-11-21T19:33:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49957/

SparkQA · 2021-11-21T20:28:13Z

Test build #145485 has finished for PR 34628 at commit 9b03228.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class DayTimeIntervalType(AtomicType):
class DayTimeIntervalTypeConverter(object):
case class OptimizeSkewedJoin(ensureRequirements: EnsureRequirements)
// When this is enabled, this class does additional lookup on write operations (put/delete) to

sarutak · 2021-11-22T02:47:21Z

Merging to master.
I think the failed Kubernetes integration tests are not related to this change and they are flaky ones. I saw those tests failed in another PR.
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49651/console

Thank you @holdenk for reviewing !

### What changes were proposed in this pull request? This PR aims to support K8s integration tests for Java 17 using Maven and SBT. The new system property `spark.kubernetes.test.dockerFile` is introduced to specify a Dockerfile. By setting `Dockerfile.java17` to the property, the integration tests run with Java 17. This PR also revised the change brought by SPARK-37354 (#34628) by changing `SparkBuild.scala` so that it can recognize the system property `spark.kubernetes.test.javaImageTag`, like the integration tests with Maven do. If both `spark.kubernetes.test.dockerFile` and `spark.kubernetes.test.javaImageTag` are set, `spark.kubernetes.test.dockerFile` is preferred. ### Why are the changes needed? To ensure Spark works on K8s with Java 17. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed that the intended version of Java is used for each pattern. * Neither `spark.kubernetes.test.javaImageTag` nor `spark.kubernetes.test.dockerFile` is set (SBT) ``` # Run the integration tests to create the container image $ build/sbt -Pkubernetes -Pkubernetes-integration-tests "kubernetes-integration-tests/test" # Create and login the container. docker run -it <hash> /bin/bash 185b40ef8aaa56c:/opt/spark/work-dir$ java -version openjdk version "1.8.0_312" OpenJDK Runtime Environment (build 1.8.0_312-b07) OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode) ``` * Neither `spark.kubernetes.test.javaImageTag` nor `spark.kubernetes.test.dockerFile` is set (Maven) ``` # Run the integration tests to create the container image $ build/mvn -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test # Create and login the container. docker run -it <hash> /bin/bash 18536ae5c5c21f4:/opt/spark/work-dir$ java -version openjdk version "1.8.0_312" OpenJDK Runtime Environment (build 1.8.0_312-b07) OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode) ``` * `spark.kubernetes.test.javaImageTag` is set (SBT): ``` # Run the integration tests to create the container image $ build/sbt -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests "kubernetes-integration-tests/test" # Create and login the container. docker run -it <hash> /bin/bash 1855b896d0e5dd8:/opt/spark/work-dir$ java -version openjdk version "11.0.13" 2021-10-19 OpenJDK Runtime Environment 18.9 (build 11.0.13+8) OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing) ``` * `spark.kubernetes.test.javaImageTag` is set (Maven): ``` # Run the integration tests to create the container image $ build/mvn -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test # Create and login the container. docker run -it <hash> /bin/bash 1855d3c228e7521:/opt/spark/work-dir$ java -version openjdk version "11.0.13" 2021-10-19 OpenJDK Runtime Environment 18.9 (build 11.0.13+8) OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing) ``` * `spark.kubernetes.test.dockerFile` is set (SBT) ``` $ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes -Pkubernetes-integration-tests package "kubernetes-integration-tests/test" # Create and login the container. docker run -it <hash> /bin/bash 185550515c76582:/opt/spark/work-dir$ java -version openjdk version "17.0.1" 2021-10-19 OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2) OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing) ``` * `spark.kubernetes.test.dockerFile` is set (Maven) ``` build/mvn -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test # Create and login the container. docker run -it <hash> /bin/bash 185be35bde0ebfa:/opt/spark/work-dir$ java -version openjdk version "17.0.1" 2021-10-19 OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2) OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing) ``` * Both `spark.kubernetes.test.javaImageTag` and `spark.kubernetes.test.dockerFile` are set (SBT) ``` $ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests package "kubernetes-integration-tests/test" # Create and login the container. docker run -it <hash> /bin/bash 1857ed1891b2921:/opt/spark/work-dir$ java -version openjdk version "17.0.1" 2021-10-19 OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2) OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing) ``` * Both `spark.kubernetes.test.javaImageTag` and `spark.kubernetes.test.dockerFile` are set (Maven) ``` $ build/mvn -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test # Create and login the container. docker run -it <hash> /bin/bash 18584f3a8b7b4a6:/opt/spark/work-dir$ java -version openjdk version "17.0.1" 2021-10-19 OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2) OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing) ``` Closes #34790 from sarutak/kube-integration-test-java17. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

Change SparkBuild.scala for Kubernetes integration tests to recognize…

2a63f02

… java_image_tag.

github-actions bot added the BUILD label Nov 17, 2021

sarutak commented Nov 17, 2021

View reviewed changes

Merge branch 'master' of https://github.com/apache/spark into sbt-kub…

9b03228

…e-java-version

sarutak closed this in a85c51f Nov 22, 2021

sarutak mentioned this pull request Dec 3, 2021

[SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17 #34790

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-37354][K8S][TESTS] Make the Java version installed on the container image used by the K8s integration tests with SBT configurable #34628

[SPARK-37354][K8S][TESTS] Make the Java version installed on the container image used by the K8s integration tests with SBT configurable #34628

Uh oh!

sarutak commented Nov 17, 2021

Uh oh!

sarutak Nov 17, 2021

Uh oh!

SparkQA commented Nov 17, 2021

Uh oh!

SparkQA commented Nov 17, 2021

Uh oh!

SparkQA commented Nov 17, 2021

Uh oh!

holdenk commented Nov 17, 2021

Uh oh!

sarutak commented Nov 21, 2021

Uh oh!

SparkQA commented Nov 21, 2021

Uh oh!

SparkQA commented Nov 21, 2021

Uh oh!

SparkQA commented Nov 21, 2021

Uh oh!

sarutak commented Nov 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-37354][K8S][TESTS] Make the Java version installed on the container image used by the K8s integration tests with SBT configurable #34628

[SPARK-37354][K8S][TESTS] Make the Java version installed on the container image used by the K8s integration tests with SBT configurable #34628

Uh oh!

Conversation

sarutak commented Nov 17, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sarutak Nov 17, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 17, 2021

Uh oh!

SparkQA commented Nov 17, 2021

Uh oh!

SparkQA commented Nov 17, 2021

Uh oh!

holdenk commented Nov 17, 2021

Uh oh!

sarutak commented Nov 21, 2021

Uh oh!

SparkQA commented Nov 21, 2021

Uh oh!

SparkQA commented Nov 21, 2021

Uh oh!

SparkQA commented Nov 21, 2021

Uh oh!

sarutak commented Nov 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants