[SPARK-28937][SPARK-28936][KUBERNETES] Reduce test flakyness #25765

holdenk · 2019-09-11T22:39:37Z

What changes were proposed in this pull request?

Switch from using a Thread sleep for waiting for commands to finish to just waiting for the command to finish with a watcher & improve the error messages in the SecretsTestsSuite.

Why are the changes needed?

Currently some of the Spark Kubernetes tests have race conditions with command execution, and the frequent use of eventually makes debugging test failures difficult.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests pass after removal of thread.sleep

… of Spark K8s integration tests

…do a more complete env check

SparkQA · 2019-09-11T22:42:41Z

Test build #110490 has finished for PR 25765 at commit 16ba894.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-11T23:04:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15465/

SparkQA · 2019-09-11T23:19:53Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15465/

SparkQA · 2019-09-12T00:10:38Z

Test build #110492 has finished for PR 25765 at commit 2ab68b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-12T00:14:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15467/

SparkQA · 2019-09-12T00:19:30Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15467/

holdenk · 2019-09-12T18:00:49Z

jenkins retest this please

SparkQA · 2019-09-12T18:17:01Z

Test build #110527 has finished for PR 25765 at commit 2ab68b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-12T18:22:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15502/

SparkQA · 2019-09-12T18:27:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15502/

shaneknapp · 2019-09-12T20:12:34Z

test this please

shaneknapp · 2019-09-12T20:15:49Z

(i triggered this because i renamed the k8s prb job, and just want to be sure nothing breaks)

SparkQA · 2019-09-12T20:28:11Z

Test build #110540 has finished for PR 25765 at commit 2ab68b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-12T20:35:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15516/

SparkQA · 2019-09-12T20:40:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15516/

SparkQA · 2019-09-13T00:40:07Z

Test build #110549 has finished for PR 25765 at commit de35d67.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-13T00:52:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15525/

SparkQA · 2019-09-13T01:10:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15525/

holdenk · 2019-09-13T04:32:20Z

Jenkins retest this please

SparkQA · 2019-09-13T04:46:45Z

Test build #110557 has finished for PR 25765 at commit de35d67.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-13T04:50:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15532/

SparkQA · 2019-09-13T05:07:41Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15532/

shaneknapp · 2019-09-13T19:04:53Z

test this please

shaneknapp · 2019-09-13T19:05:19Z

(sorry for barging in again but i'm testing a small infra change on the ubuntu workers)

;)

SparkQA · 2019-09-13T19:26:35Z

Test build #110571 has finished for PR 25765 at commit de35d67.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2019-09-13T19:33:16Z

It's 100% ok :)

SparkQA · 2019-09-13T19:34:36Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15546/

shaneknapp · 2019-09-13T19:41:21Z

It's 100% ok :)

awesome. i'm getting rid of those crappy, unrelated-to-the-build lsof errors in the logs.

SparkQA · 2019-09-13T19:50:16Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15546/

shaneknapp · 2019-09-13T19:52:17Z

It's 100% ok :)

awesome. i'm getting rid of those crappy, unrelated-to-the-build lsof errors in the logs.

aaaaaand those error messages are gone!

+ /home/jenkins/bin/kill_zinc_nailgun.py --zinc-port 3714

holdenk · 2019-09-13T22:18:33Z

Awesome @shaneknapp :D

srowen · 2019-09-14T15:20:57Z

...ion-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/SecretsTestsSuite.scala

  val SECRET_MOUNT_PATH = "/etc/secret"
  val ENV_SECRET_KEY_1 = "username"
  val ENV_SECRET_KEY_2 = "password"
+  val ENV_SECRET_KEY_1_CAP = ENV_SECRET_KEY_1.toUpperCase(Locale.ROOT)


Is this part of the change related, necessary?

No, although I figured cleaning up hard coded strings while I was there would be good.

SparkQA · 2019-09-16T18:23:38Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15617/

SparkQA · 2019-09-16T18:38:53Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15617/

skonto · 2019-09-19T11:04:30Z

...tes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala

      .writingOutput(out)
      .writingError(System.err)
      .withTTY()
+      .usingListener(listener)


Cool I was looking for something like that in the past :)

holdenk · 2019-09-19T17:08:20Z

If no one objects I plan to merge this tomorrow.

holdenk · 2019-09-20T17:09:08Z

Merged to master.

holdenk added 6 commits August 30, 2019 22:02

Work on debugging the secrets test

e9bf47f

Test pass (yay)

805fe3a

Add a listener in place of Thread.sleep for executing commands inside…

86f9e75

… of Spark K8s integration tests

Remove printlns from my debugging

3ccddbf

Switch some constant uses of username/password to the object ref and …

916372c

…do a more complete env check

cammel fix

16ba894

Style fixes

2ab68b5

dongjoon-hyun added the KUBERNETES label Sep 12, 2019

Add missing Locale import

de35d67

srowen reviewed Sep 14, 2019

View reviewed changes

srowen approved these changes Sep 18, 2019

View reviewed changes

skonto reviewed Sep 19, 2019

View reviewed changes

asfgit closed this in 4080c4b Sep 20, 2019

[SPARK-28937][SPARK-28936][KUBERNETES] Reduce test flakyness #25765

[SPARK-28937][SPARK-28936][KUBERNETES] Reduce test flakyness #25765

Uh oh!

Conversation

holdenk commented Sep 11, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Sep 11, 2019

Uh oh!

SparkQA commented Sep 11, 2019

Uh oh!

SparkQA commented Sep 11, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

holdenk commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

shaneknapp commented Sep 12, 2019

Uh oh!

shaneknapp commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

holdenk commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

shaneknapp commented Sep 13, 2019

Uh oh!

shaneknapp commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

holdenk commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

shaneknapp commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

shaneknapp commented Sep 13, 2019

Uh oh!

holdenk commented Sep 13, 2019

Uh oh!

srowen Sep 14, 2019

Choose a reason for hiding this comment

Uh oh!

holdenk Sep 18, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 16, 2019

Uh oh!