Skip to content

[SPARK-27460][TESTS][FollowUp] Add HiveClientVersions to parallel test suite list #24404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-27460][TESTS][FollowUp] Add HiveClientVersions to parallel test suite list #24404

wants to merge 2 commits into from

Conversation

@gengliangwang
Copy link
Member

@gengliangwang gengliangwang commented Apr 18, 2019 •
edited
Loading

What changes were proposed in this pull request?

The test time of HiveClientVersions is around 3.5 minutes.
This PR is to add it into the parallel test suite list. To make sure there is no colliding warehouse location, we can change the warehouse path to a temporary directory.

How was this patch tested?

Unit test

@gengliangwang
Copy link
Member Author

gengliangwang commented Apr 18, 2019

@cloud-fan

@gengliangwang gengliangwang mentioned this pull request Apr 18, 2019
Closed
@gengliangwang
Copy link
Member Author

gengliangwang commented Apr 18, 2019

From #24373 (comment) we can see that the tests become stable after the changes in this PR.
I will trigger more rounds of concurrent Jenkins tests in this PR.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #104703 has finished for PR 24404 at commit a031c47.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #4751 has finished for PR 24404 at commit dd57470.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #4752 has finished for PR 24404 at commit dd57470.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #4750 has finished for PR 24404 at commit dd57470.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #4749 has finished for PR 24404 at commit dd57470.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #104704 has finished for PR 24404 at commit dd57470.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #4748 has finished for PR 24404 at commit dd57470.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #4755 has finished for PR 24404 at commit dd57470.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #4754 has finished for PR 24404 at commit dd57470.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 18, 2019

Test build #4753 has finished for PR 24404 at commit dd57470.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun approved these changes Apr 18, 2019
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Merged to master.
According to the multiple result, I can see HiveClientVersions suite results are stable. The failures are irrelevant to this.

@dongjoon-hyun dongjoon-hyun mentioned this pull request Sep 20, 2019
Closed
dongjoon-hyun pushed a commit that referenced this pull request Sep 20, 2019
@gengliangwang @gatorsmile @dongjoon-hyun
[SPARK-27460][TESTS][2.4] Running slowest test suites in their own fo…
0d26cfc 
…rked JVMs for higher parallelism

## What changes were proposed in this pull request?

This is a backport of #24373 , #24404 and #24434

This patch modifies SparkBuild so that the largest / slowest test suites (or collections of suites) can run in their own forked JVMs, allowing them to be run in parallel with each other. This opt-in / whitelisting approach allows us to increase parallelism without having to fix a long-tail of flakiness / brittleness issues in tests which aren't performance bottlenecks.

See comments in SparkBuild.scala for information on the details, including a summary of why we sometimes opt to run entire groups of tests in a single forked JVM .

The time of full new pull request test in Jenkins is reduced by around 53%:
before changes: 4hr 40min
after changes: 2hr 13min

## How was this patch tested?

Unit test

Closes #25861 from dongjoon-hyun/SPARK-27460.

Lead-authored-by: Gengliang Wang <[email protected]>
Co-authored-by: gatorsmile <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowen srowen srowen approved these changes

@felixcheung felixcheung felixcheung approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@gengliangwang @SparkQA @srowen @felixcheung @dongjoon-hyun