[SPARK-25828][K8S] Bumping Kubernetes-Client version to 4.1.0 #22820

ifilonenko · 2018-10-25T00:25:54Z

What changes were proposed in this pull request?

Changed the kubernetes-client version and refactored code that broke as a result

How was this patch tested?

Unit and Integration tests

ifilonenko · 2018-10-25T00:27:21Z

@erikerlandson @skonto @liyinan926 for review
Do you see additional places where we would want to leverage the newer version for optimality?

SparkQA · 2018-10-25T00:31:29Z

Test build #97992 has finished for PR 22820 at commit de62f07.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2018-10-25T00:36:35Z

resource-managers/kubernetes/core/pom.xml

  <properties>
    <sbt.project.name>kubernetes</sbt.project.name>
-    <kubernetes.client.version>3.0.0</kubernetes.client.version>
+    <kubernetes.client.version>4.0.0</kubernetes.client.version>


4.1.0? https://mvnrepository.com/artifact/io.fabric8/kubernetes-client/4.1.0 what are the differences compared to 4.0.0?

hmm, agreed. Maybe 4.1.0 is a more stable version

Yeah just wondering I don't know. What matters is what each version supports. Still no version supports 1.11 fabric8io/kubernetes-client#1235

SparkQA · 2018-10-25T00:51:56Z

Test build #97994 has finished for PR 22820 at commit 3d7a0b9.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

...netes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala

felixcheung

should there be some doc changes on supported client version

SparkQA · 2018-10-25T18:06:57Z

Test build #98037 has finished for PR 22820 at commit a28ebcf.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

liyinan926

LGTM. But looks like the build failed on dependency tests. Please take a look.

SparkQA · 2018-10-25T18:14:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4494/

ifilonenko · 2018-10-25T18:16:36Z

But looks like the build failed on dependency tests. Please take a look.

@liyinan926 that is In reference to manifests that need to be updated in dev/deps/spark-deps-hadoop-2.7 and dev/deps/spark-deps-hadoop-3.1. This most recent commit should fix

SparkQA · 2018-10-25T18:25:50Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4494/

SparkQA · 2018-10-25T18:39:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4496/

SparkQA · 2018-10-25T18:50:56Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4496/

erikerlandson · 2018-10-25T20:22:53Z

docs/running-on-kubernetes.md


-The driver and executor pod scheduling is handled by Kubernetes. It is possible to schedule the
+The driver and executor pod scheduling is handled by Kubernetes. Communication to the Kubernetes API is done via fabric8, and we are 
+currently running <code>kubernetes-client</code> version <code>4.1.0</code>. Make sure that any infrastructure additions are weary of said version. It is possible to schedule the


nit: "wary" ? or "aware of " ?

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

SparkQA · 2018-10-25T22:38:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4509/

SparkQA · 2018-10-25T22:47:41Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4509/

SparkQA · 2018-10-26T00:58:52Z

Test build #98041 has finished for PR 22820 at commit c6b6723.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-26T01:46:15Z

Test build #98054 has finished for PR 22820 at commit 728d70a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ifilonenko · 2018-10-26T03:13:55Z

retest this please

SparkQA · 2018-10-26T03:28:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4519/

SparkQA · 2018-10-26T03:39:47Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4519/

SparkQA · 2018-10-26T07:05:02Z

Test build #98064 has finished for PR 22820 at commit 728d70a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

rvesse

+1 Would like to get this in ASAP so I can rebase PR #22805 and avoid mixing unrelated changes into the integration testing enhancement PR per @liyinan926's request

rvesse · 2018-10-26T09:16:49Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

-    if (time != null) time.getTime else "N/A"
+  def formatTime(time: String): String = {
+    // TODO: Investigate form `time` comes in as there was a DataType
+    // change from Time to String


The string returned is ISO8601 format so no need to do anything special here

e.g. a sample from the logs from my PR where I had made a similar change:

creation time: 2018-10-24T17:17:03Z

SparkQA · 2018-10-26T17:38:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4545/

SparkQA · 2018-10-26T17:49:46Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4545/

SparkQA · 2018-10-26T21:34:43Z

Test build #98102 has finished for PR 22820 at commit b0ba951.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ifilonenko · 2018-10-26T21:42:09Z

@felixcheung and @erikerlandson and @liyinan926 for merge

erikerlandson · 2018-10-26T23:03:04Z

merging

## What changes were proposed in this pull request? Changed the `kubernetes-client` version and refactored code that broke as a result ## How was this patch tested? Unit and Integration tests Closes apache#22820 from ifilonenko/SPARK-25828. Authored-by: Ilan Filonenko <[email protected]> Signed-off-by: Erik Erlandson <[email protected]>

bump to 4.0.0 for kubernetes-client version

de62f07

skonto reviewed Oct 25, 2018

View reviewed changes

ifilonenko changed the title ~~[SPARK-25828][K8S] Bumping Kubernetes-Client version to 4.0.0~~ [SPARK-25828][K8S] Bumping Kubernetes-Client version to 4.1.0 Oct 25, 2018

bump to 4.1.0

3d7a0b9

ifilonenko changed the title ~~[SPARK-25828][K8S] Bumping Kubernetes-Client version to 4.1.0~~ [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client version to 4.1.0 Oct 25, 2018

liyinan926 reviewed Oct 25, 2018

View reviewed changes

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala Show resolved Hide resolved

...netes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala Show resolved Hide resolved

felixcheung reviewed Oct 25, 2018

View reviewed changes

comments

a28ebcf

liyinan926 mentioned this pull request Oct 25, 2018

[SPARK-25809][K8S][TEST] New K8S integration testing backends #22805

Closed

4 tasks

liyinan926 reviewed Oct 25, 2018

View reviewed changes

replaced manifests

c6b6723

erikerlandson reviewed Oct 25, 2018

View reviewed changes

Resolve comments

728d70a

ifilonenko changed the title ~~[SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client version to 4.1.0~~ [SPARK-25828][K8S] Bumping Kubernetes-Client version to 4.1.0 Oct 26, 2018

rvesse approved these changes Oct 26, 2018

View reviewed changes

took out comment

b0ba951

asfgit closed this in e9b71c8 Oct 26, 2018

[SPARK-25828][K8S] Bumping Kubernetes-Client version to 4.1.0 #22820

[SPARK-25828][K8S] Bumping Kubernetes-Client version to 4.1.0 #22820

Uh oh!

Conversation

ifilonenko commented Oct 25, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

ifilonenko commented Oct 25, 2018

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

skonto Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ifilonenko Oct 25, 2018

Choose a reason for hiding this comment

Uh oh!

skonto Oct 25, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

Uh oh!

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

liyinan926 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

ifilonenko commented Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

erikerlandson Oct 25, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

SparkQA commented Oct 25, 2018

Uh oh!

SparkQA commented Oct 26, 2018

Uh oh!

SparkQA commented Oct 26, 2018

Uh oh!

ifilonenko commented Oct 26, 2018

Uh oh!

SparkQA commented Oct 26, 2018

Uh oh!

SparkQA commented Oct 26, 2018

Uh oh!

SparkQA commented Oct 26, 2018

Uh oh!

rvesse left a comment

Choose a reason for hiding this comment

Uh oh!

rvesse Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

rvesse Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 26, 2018

Uh oh!

SparkQA commented Oct 26, 2018

Uh oh!

SparkQA commented Oct 26, 2018

skonto Oct 25, 2018 •

edited

Loading

liyinan926 left a comment •

edited

Loading

ifilonenko commented Oct 25, 2018 •

edited

Loading

ifilonenko commented Oct 26, 2018 •

edited

Loading