[SPARK-27460][FOLLOW-UP][TESTS] Fix flaky tests #24434

gatorsmile · 2019-04-22T01:34:44Z

What changes were proposed in this pull request?

This patch makes several test flakiness fixes.

How was this patch tested?

N/A

gatorsmile · 2019-04-22T01:35:36Z

cc @gengliangwang

SparkQA · 2019-04-22T04:08:44Z

Test build #104794 has finished for PR 24434 at commit 6cbf7e7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2019-04-22T06:17:03Z

LGTM, I did see some of these flaky tests in #24373 . Increasing the timeout makes the parallel tests more robust.
I will trigger a round of concurrent Jenkins jobs to see if any flaky tests still existed.

SparkQA · 2019-04-22T07:05:01Z

Test build #4760 has finished for PR 24434 at commit 6cbf7e7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T07:05:02Z

Test build #4757 has finished for PR 24434 at commit 6cbf7e7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T07:05:02Z

Test build #4759 has finished for PR 24434 at commit 6cbf7e7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T07:05:02Z

Test build #4758 has finished for PR 24434 at commit 6cbf7e7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T07:05:02Z

Test build #4761 has finished for PR 24434 at commit 6cbf7e7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-04-22T07:26:45Z

retest this please

SparkQA · 2019-04-22T08:50:23Z

Test build #104798 has finished for PR 24434 at commit 6cbf7e7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T11:34:03Z

Test build #4762 has finished for PR 24434 at commit 6cbf7e7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-04-22T11:51:25Z

resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala


    try {
-      eventually(timeout(30.seconds), interval(100.milliseconds)) {
+      eventually(timeout(180.seconds), interval(100.milliseconds)) {


You can write "3.minutes" in cases like this and others below, but it doesn't matter.

SparkQA · 2019-04-22T12:16:07Z

Test build #4763 has finished for PR 24434 at commit 6cbf7e7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T12:38:32Z

Test build #4764 has finished for PR 24434 at commit 6cbf7e7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T12:41:55Z

Test build #4765 has finished for PR 24434 at commit 6cbf7e7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T16:17:25Z

Test build #104810 has finished for PR 24434 at commit 3224966.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T19:18:44Z

Test build #4767 has finished for PR 24434 at commit 3224966.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-04-22T20:38:33Z

That's a weird error... it's from javadoc:

[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/core/target/java/org/apache/spark/serializer/SerializationDebugger.java:159: error: cannot find symbol
[error]   static private  org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassReflection reflect ()  { throw new RuntimeException(); }
[error]                                                                    ^
[error]   symbol:   class ObjectStreamClassReflection
[error]   location: class SerializationDebugger
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/core/target/java/org/apache/spark/serializer/SerializationDebugger.java:22: error: class SerializationDebugger is already defined in package org.apache.spark.serializer
[error]   static private  class SerializationDebugger {
[error]                   ^
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/external/kafka-0-10-token-provider/target/java/org/apache/spark/kafka010/KafkaDelegationTokenTest.java:6: error: illegal combination of modifiers: public and private
[error]   private  class KafkaJaasConfiguration extends javax.security.auth.login.Configuration {
[error]            ^
...

It's from javadoc'ing the Java code auto-generated from Scala, and this kind of thing is kind of a known problem; I thought we didn't run javadoc on this part. I have no idea so far why it came up in running these tests!

cloud-fan · 2019-04-23T03:16:53Z

ok to test

HyukjinKwon · 2019-04-23T03:22:10Z

If it still fails, let me fix it against this branch since I am kind of used to it.

SparkQA · 2019-04-23T03:28:59Z

Test build #104819 has finished for PR 24434 at commit 3224966.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-04-23T03:31:34Z

Will take a look late today (KST)

SparkQA · 2019-04-23T03:35:57Z

Test build #104820 has finished for PR 24434 at commit 3224966.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-04-23T13:38:37Z

Note that these errors also appear in 'normal' builds; they're just warnings. I still have no idea why they are rendered as errors due to this change. It doesn't touch the build. I could try separately updating sbt-unidoc, but don't think that's the issue. Still looking...

HyukjinKwon · 2019-04-23T13:56:49Z

Yes .. I think we talked about this few times long long ago @srowen. I made some analysis in gatorsmile#5

HyukjinKwon · 2019-04-24T04:45:05Z

retest this please

SparkQA · 2019-04-24T05:02:44Z

Test build #104853 has finished for PR 24434 at commit 3224966.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-24T07:05:01Z

Test build #104857 has finished for PR 24434 at commit e5a5991.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-04-24T07:06:33Z

retest this please

SparkQA · 2019-04-24T09:15:27Z

Test build #104860 has finished for PR 24434 at commit e5a5991.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-04-24T09:36:44Z

thanks, merging to master!

…rked JVMs for higher parallelism ## What changes were proposed in this pull request? This is a backport of #24373 , #24404 and #24434 This patch modifies SparkBuild so that the largest / slowest test suites (or collections of suites) can run in their own forked JVMs, allowing them to be run in parallel with each other. This opt-in / whitelisting approach allows us to increase parallelism without having to fix a long-tail of flakiness / brittleness issues in tests which aren't performance bottlenecks. See comments in SparkBuild.scala for information on the details, including a summary of why we sometimes opt to run entire groups of tests in a single forked JVM . The time of full new pull request test in Jenkins is reduced by around 53%: before changes: 4hr 40min after changes: 2hr 13min ## How was this patch tested? Unit test Closes #25861 from dongjoon-hyun/SPARK-27460. Lead-authored-by: Gengliang Wang <[email protected]> Co-authored-by: gatorsmile <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

fix the flaky tests.

6cbf7e7

gatorsmile changed the title ~~[SPARK-27460][SPARK-27460] Fix flaky tests~~ [SPARK-27460][FOLLOW-UP][TESTS] Fix flaky tests Apr 22, 2019

srowen approved these changes Apr 22, 2019

View reviewed changes

felixcheung approved these changes Apr 22, 2019

View reviewed changes

gatorsmile added 2 commits April 22, 2019 08:53

more

857657d

address comment.

3224966

cloud-fan closed this Apr 23, 2019

cloud-fan reopened this Apr 23, 2019

HyukjinKwon mentioned this pull request Apr 23, 2019

Fix Javadoc (Unidoc) generation issue in PR 24434 gatorsmile/spark#5

Merged

Fix Javadoc generation issue in PR 24434 (#5)

e5a5991

cloud-fan closed this in cd4a284 Apr 24, 2019

dongjoon-hyun mentioned this pull request Sep 20, 2019

[SPARK-27460][TESTS][2.4] Running slowest test suites in their own forked JVMs for higher parallelism #25861

Closed

[SPARK-27460][FOLLOW-UP][TESTS] Fix flaky tests #24434

[SPARK-27460][FOLLOW-UP][TESTS] Fix flaky tests #24434

Uh oh!

Conversation

gatorsmile commented Apr 22, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

gatorsmile commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

gengliangwang commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

HyukjinKwon commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

srowen Apr 22, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

srowen commented Apr 22, 2019

Uh oh!

cloud-fan commented Apr 23, 2019

Uh oh!

HyukjinKwon commented Apr 23, 2019

Uh oh!

SparkQA commented Apr 23, 2019

Uh oh!

HyukjinKwon commented Apr 23, 2019

Uh oh!

SparkQA commented Apr 23, 2019

Uh oh!

srowen commented Apr 23, 2019

Uh oh!

HyukjinKwon commented Apr 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Apr 24, 2019

Uh oh!

SparkQA commented Apr 24, 2019

Uh oh!

SparkQA commented Apr 24, 2019

Uh oh!

cloud-fan commented Apr 24, 2019

Uh oh!

SparkQA commented Apr 24, 2019

Uh oh!

cloud-fan commented Apr 24, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

HyukjinKwon commented Apr 23, 2019 •

edited

Loading