[SPARK-27460][TESTS] Running slowest test suites in their own forked JVMs for higher parallelism #24373

gengliangwang · 2019-04-15T11:43:35Z

What changes were proposed in this pull request?

This patch modifies SparkBuild so that the largest / slowest test suites (or collections of suites) can run in their own forked JVMs, allowing them to be run in parallel with each other. This opt-in / whitelisting approach allows us to increase parallelism without having to fix a long-tail of flakiness / brittleness issues in tests which aren't performance bottlenecks.

See comments in SparkBuild.scala for information on the details, including a summary of why we sometimes opt to run entire groups of tests in a single forked JVM .

The time of full new pull request test in Jenkins is reduced by around 53%:
before changes: 4hr 40min
after changes: 2hr 13min

How was this patch tested?

Unit test

SparkQA · 2019-04-15T13:27:57Z

Test build #104584 has finished for PR 24373 at commit 2600785.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-04-15T13:53:26Z

It's worth trying, but it won't help the Maven build. What about rewriting the test suites to just run the test cases in parallel? http://doc.scalatest.org/3.0.1-2.12/org/scalatest/ParallelTestExecution.html does this.

gengliangwang · 2019-04-15T14:23:45Z

project/SparkBuild.scala

+  //       addition to JVM startup time and JIT warmup, it appears that initialization of Derby
+  //       metastores can be very slow so creating a fresh warehouse per suite is inefficient.
+  //
+  //    2. When parallelizing within a project we need to give each forked JVM a different tmpdir


@srowen It is hard to successfully run test cases in parallel. See the comment here. E.g, it possible that multiple test cases use the same table location spark-warehouse/t for a temporary table.

I get it, but this won't help the Maven build and is kind of brittle. Is it really hard to just set temp dirs differently for different suites?

Can a suite run suites in scalatest? and parallelize suites that way?

I get it, but this won't help the Maven build and is kind of brittle.

From the Jenkins log, I can see that we are using SBT for test PRs. And I can see that the test time of this PR is about 106 minutes from the email notification, which is much better now (before changes it takes around 3.5 hours from https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104580/testReport/history/)

Is it really hard to just set temp dirs differently for different suites?

I think fixing that problem would be a huge amount of work for limited payoff in most cases because most test suites are short-running.

Can a suite run suites in scalatest? and parallelize suites that way?

Do you mean run all test suite in parallel? We can enable parallelExecution in Test (http://www.scalatest.org/user_guide/using_scalatest_with_sbt). But we still face the problem of colliding warehouse paths.

gengliangwang · 2019-04-15T15:46:02Z

@gatorsmile @cloud-fan

SparkQA · 2019-04-15T16:12:56Z

Test build #104591 has finished for PR 24373 at commit 2e2c8f1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-15T18:24:52Z

Test build #104594 has finished for PR 24373 at commit fded60e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-04-15T20:13:00Z

@srowen SBT tests are being used for PR tests. Let us improve the SBT tests first and at least it can speed up the development speeds.

srowen · 2019-04-15T20:17:05Z

Yeah I think that's a good argument. it's a bit of a hack but not bad.

gatorsmile · 2019-04-15T20:21:01Z

I just triggered 6 tests. Let us see whether the tests become more flaky, or it could introduce new flaky tests.

SparkQA · 2019-04-15T21:27:58Z

Test build #4712 has finished for PR 24373 at commit fded60e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-15T21:48:39Z

Test build #4710 has finished for PR 24373 at commit fded60e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-15T21:50:53Z

Test build #4708 has finished for PR 24373 at commit fded60e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-15T21:54:38Z

Test build #4709 has finished for PR 24373 at commit fded60e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-15T22:16:34Z

Test build #4711 has finished for PR 24373 at commit fded60e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-15T22:38:30Z

Test build #4707 has finished for PR 24373 at commit fded60e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-16T04:29:16Z

Test build #104609 has finished for PR 24373 at commit 475c8ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-16T04:32:25Z

Test build #4713 has finished for PR 24373 at commit 475c8ef.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-16T04:48:25Z

Test build #4714 has finished for PR 24373 at commit 475c8ef.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-16T05:29:33Z

Test build #4715 has finished for PR 24373 at commit 475c8ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2019-04-16T06:50:07Z

retest this please.

gengliangwang · 2019-04-16T06:50:12Z

retest this please.

gengliangwang · 2019-04-16T06:50:16Z

retest this please.

SparkQA · 2019-04-16T07:05:01Z

Test build #104615 has finished for PR 24373 at commit d2ac3af.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2019-04-16T07:07:06Z

retest this please

SparkQA · 2019-04-16T08:40:33Z

Test build #104616 has finished for PR 24373 at commit d2ac3af.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-17T16:29:00Z

Test build #4731 has finished for PR 24373 at commit 5047e5a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-17T18:33:57Z

Test build #4735 has finished for PR 24373 at commit 57e18c2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-17T18:46:46Z

Test build #104670 has finished for PR 24373 at commit 57e18c2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-17T18:47:48Z

Test build #4734 has finished for PR 24373 at commit 57e18c2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-17T18:53:35Z

Test build #4736 has finished for PR 24373 at commit 57e18c2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-17T19:48:16Z

Test build #4737 has finished for PR 24373 at commit 57e18c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-04-18T04:34:44Z

+1 Looks worth trying.

SparkQA · 2019-04-18T05:22:53Z

Test build #4740 has finished for PR 24373 at commit 70874b7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-18T06:06:49Z

Test build #104690 has finished for PR 24373 at commit 70874b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-18T06:17:59Z

Test build #4739 has finished for PR 24373 at commit 70874b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-18T08:37:50Z

Test build #4742 has finished for PR 24373 at commit 70874b7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-18T09:15:15Z

Test build #4741 has finished for PR 24373 at commit 70874b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-04-18T12:50:35Z

thanks, merging to master! Let's see how the PR build works.

cloud-fan · 2019-04-18T12:52:50Z

Oops, I triggered the merge script at 70874b7 , and then got stuck by network until now. @gengliangwang can you send a new PR for the commits after 70874b7?

SparkQA · 2019-04-18T13:26:22Z

Test build #104699 has finished for PR 24373 at commit 4b61d0a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2019-04-18T13:29:51Z

Oops, I triggered the merge script at 70874b7 , and then got stuck by network until now. @gengliangwang can you send a new PR for the commits after 70874b7?

Sure, I have created #24404 .

SparkQA · 2019-04-18T14:13:38Z

Test build #4744 has finished for PR 24373 at commit 4b61d0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-18T14:22:14Z

Test build #4743 has finished for PR 24373 at commit 4b61d0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-18T14:33:16Z

Test build #4746 has finished for PR 24373 at commit 4b61d0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-18T14:33:20Z

Test build #4747 has finished for PR 24373 at commit 4b61d0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-18T14:41:18Z

Test build #4745 has finished for PR 24373 at commit 4b61d0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-09-20T00:11:29Z

Hi, All.
We are hitting the timeout in branch-2.4 SBT build, too.
Instead of increasing the timeout, it would be great if we can have this in branch-2.4.
I'll make a backporting PR since branch-2.4 is our LTS branch.

…rked JVMs for higher parallelism ## What changes were proposed in this pull request? This is a backport of #24373 , #24404 and #24434 This patch modifies SparkBuild so that the largest / slowest test suites (or collections of suites) can run in their own forked JVMs, allowing them to be run in parallel with each other. This opt-in / whitelisting approach allows us to increase parallelism without having to fix a long-tail of flakiness / brittleness issues in tests which aren't performance bottlenecks. See comments in SparkBuild.scala for information on the details, including a summary of why we sometimes opt to run entire groups of tests in a single forked JVM . The time of full new pull request test in Jenkins is reduced by around 53%: before changes: 4hr 40min after changes: 2hr 13min ## How was this patch tested? Unit test Closes #25861 from dongjoon-hyun/SPARK-27460. Lead-authored-by: Gengliang Wang <[email protected]> Co-authored-by: gatorsmile <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

parallelize tests

2600785

gengliangwang commented Apr 15, 2019

View reviewed changes

try increasing the number of concurrent test groups

2e2c8f1

use 8 concurrentRestrictions again; parallize more test suites

fded60e

try reducing ForkedTestGroup

475c8ef

remove some test suites

d2ac3af

fix

57e18c2

add org.apache.spark.sql.hive.StatisticsSuite

70874b7

add HiveClientVersions

4b61d0a

cloud-fan closed this in 9c238b8 Apr 18, 2019

gengliangwang mentioned this pull request Apr 18, 2019

[SPARK-27460][TESTS][FollowUp] Add HiveClientVersions to parallel test suite list #24404

Closed

gengliangwang mentioned this pull request Apr 22, 2019

[SPARK-27460][FOLLOW-UP][TESTS] Fix flaky tests #24434

Closed

dongjoon-hyun mentioned this pull request Sep 20, 2019

[SPARK-27460][TESTS][2.4] Running slowest test suites in their own forked JVMs for higher parallelism #25861

Closed

[SPARK-27460][TESTS] Running slowest test suites in their own forked JVMs for higher parallelism #24373

[SPARK-27460][TESTS] Running slowest test suites in their own forked JVMs for higher parallelism #24373

Uh oh!

Conversation

gengliangwang commented Apr 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

srowen commented Apr 15, 2019

Uh oh!

gengliangwang Apr 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srowen Apr 15, 2019

Choose a reason for hiding this comment

Uh oh!

gengliangwang Apr 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

gatorsmile commented Apr 15, 2019

Uh oh!

srowen commented Apr 15, 2019

Uh oh!

gatorsmile commented Apr 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 16, 2019

Uh oh!

SparkQA commented Apr 16, 2019

Uh oh!

SparkQA commented Apr 16, 2019

Uh oh!

SparkQA commented Apr 16, 2019

Uh oh!

gengliangwang commented Apr 16, 2019

Uh oh!

gengliangwang commented Apr 16, 2019

Uh oh!

gengliangwang commented Apr 16, 2019

Uh oh!

SparkQA commented Apr 16, 2019

Uh oh!

dilipbiswal commented Apr 16, 2019

Uh oh!

SparkQA commented Apr 16, 2019

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

SparkQA commented Apr 17, 2019

gengliangwang commented Apr 15, 2019 •

edited

Loading

gengliangwang Apr 15, 2019 •

edited

Loading

gengliangwang Apr 15, 2019 •

edited

Loading

gatorsmile commented Apr 15, 2019 •

edited

Loading