[WIP] [SQL] Experiment with speculatively running without reset() in HiveCompatibilitySuite #6663

JoshRosen · 2015-06-05T02:07:59Z

This is an experiment to see if we can easily speed up HiveCompatibilitySuite by speculatively running without calling TestHive.reset(), then retrying if that fails.

Note: This PR is a bit of a mess since it's now serving as a quick playground for me to rapidly prototype perf. patches and test them with Jenkins.

This reverts commit 3c9c944.

…th it.

SparkQA · 2015-06-05T02:44:32Z

Test build #34243 has finished for PR 6663 at commit 417f50e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-06-05T02:57:40Z

Cool, HiveCompatibilitySuite now runs in 12 minutes.

JoshRosen · 2015-06-05T06:23:56Z

core/src/main/scala/org/apache/spark/SerializableWritable.scala

I think that this, and SerilaizableWritable more generally, may be a huge source of perf. bottlenecks for short tasks. A common use of SerializableWritable is in serializing Hadoop Configurations, but it seems kind of crazy to create and discard a new Configuration in order to be able to deserialize the driver-provided conf. Maybe we can make a substitute for SerializableWritable which only deals with Configuration subclasses and just calls writeFields() and readFields() directly. This would sidestep a lot of the performance penalties involved in creating Configuration objects and having them spend tons of time loading defaults.

Apparently other folks have noticed Configuration's expensive instantiation costs, too: https://issues.apache.org/jira/browse/MAPREDUCE-5399

SparkQA · 2015-06-05T06:40:08Z

Test build #34257 has finished for PR 6663 at commit f90dc94.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-06-05T06:41:34Z

Wow, this test suite seems to run an insane number of queries against Derby. From YourKit's performance charts view:

That's nearly 2.5 million queries for ~700 test cases or roughly 3500 queries / statements per test. Can this be right or is YourKit measuring something different than I think it is?

SparkQA · 2015-06-05T08:42:35Z

Test build #34258 has finished for PR 6663 at commit 480d20a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-05T09:19:49Z

Test build #34261 has finished for PR 6663 at commit 137a0c7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-05T09:49:26Z

Test build #34262 has finished for PR 6663 at commit 55041d2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-05T10:02:33Z

Test build #34263 has finished for PR 6663 at commit 57c2cb4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-05T10:21:07Z

Test build #34265 has finished for PR 6663 at commit 9db0abc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit 480d20a.

…ations

SparkQA · 2015-06-05T22:17:50Z

Test build #34312 has finished for PR 6663 at commit 3fe16e8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit 55041d2.

This reverts commit 57c2cb4.

…table" This reverts commit f90dc94.

SparkQA · 2015-06-05T23:19:35Z

Test build #34327 has finished for PR 6663 at commit b1e3f82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit 9e116d1.

…iating Configuration

SparkQA · 2015-06-06T00:06:40Z

Test build #34336 has finished for PR 6663 at commit 5113370.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-06T00:18:46Z

Test build #34339 has finished for PR 6663 at commit fac08d5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-06-06T00:24:22Z

Jenkins, retest this please.

SparkQA · 2015-06-06T01:04:46Z

Test build #34341 has finished for PR 6663 at commit fac08d5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…ations

SparkQA · 2015-06-08T18:41:32Z

Test build #34455 has finished for PR 6663 at commit 8c1e1ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-06-09T19:03:37Z

Jenkins, retest this please.

SparkQA · 2015-06-09T19:49:41Z

Test build #34523 has finished for PR 6663 at commit 8c1e1ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-12T20:27:23Z

Test build #903 has finished for PR 6663 at commit 8c1e1ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-12T20:30:53Z

Test build #34789 has finished for PR 6663 at commit 2b500b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-12T20:36:13Z

Test build #904 has finished for PR 6663 at commit 2b500b9.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-12T20:43:28Z

Test build #905 has finished for PR 6663 at commit 2b500b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-12T20:46:28Z

Test build #34793 has finished for PR 6663 at commit fbd3d03.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…t-tests.log?)

SparkQA · 2015-06-12T20:56:46Z

Test build #906 has finished for PR 6663 at commit fbd3d03.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-12T21:02:26Z

Test build #34795 has finished for PR 6663 at commit 46dd005.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-12T21:05:58Z

Test build #907 has finished for PR 6663 at commit fbd3d03.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…g up TestHive.reset() When profiling HiveCompatibilitySuite, I noticed that most of the time seems to be spent in expensive `TestHive.reset()` calls. This patch speeds up suites based on HiveComparisionTest, such as HiveCompatibilitySuite, with the following changes: - Avoid `TestHive.reset()` whenever possible: - Use a simple set of heuristics to guess whether we need to call `reset()` in between tests. - As a safety-net, automatically re-run failed tests by calling `reset()` before the re-attempt. - Speed up the expensive parts of `TestHive.reset()`: loading the `src` and `srcpart` tables took roughly 600ms per test, so we now avoid this by using a simple heuristic which only loads those tables by tests that reference them. This is based on simple string matching over the test queries which errs on the side of loading in more situations than might be strictly necessary. After these changes, HiveCompatibilitySuite seems to run in about 10 minutes. This PR is a revival of apache#6663, an earlier experimental PR from June, where I played around with several possible speedups for this suite. Author: Josh Rosen <[email protected]> Closes apache#10055 from JoshRosen/speculative-testhive-reset.

…g up TestHive.reset() When profiling HiveCompatibilitySuite, I noticed that most of the time seems to be spent in expensive `TestHive.reset()` calls. This patch speeds up suites based on HiveComparisionTest, such as HiveCompatibilitySuite, with the following changes: - Avoid `TestHive.reset()` whenever possible: - Use a simple set of heuristics to guess whether we need to call `reset()` in between tests. - As a safety-net, automatically re-run failed tests by calling `reset()` before the re-attempt. - Speed up the expensive parts of `TestHive.reset()`: loading the `src` and `srcpart` tables took roughly 600ms per test, so we now avoid this by using a simple heuristic which only loads those tables by tests that reference them. This is based on simple string matching over the test queries which errs on the side of loading in more situations than might be strictly necessary. After these changes, HiveCompatibilitySuite seems to run in about 10 minutes. This PR is a revival of #6663, an earlier experimental PR from June, where I played around with several possible speedups for this suite. Author: Josh Rosen <[email protected]> Closes #10055 from JoshRosen/speculative-testhive-reset. (cherry picked from commit ef6790f) Signed-off-by: Reynold Xin <[email protected]>

JoshRosen added 6 commits May 17, 2015 17:08

Avoid IO operations on empty files in BlockObjectWriter.

00bcf8a

Do not create empty files at all.

8fd89b4

Reduce scope of FileOutputStream in ExternalSorter

0db87c3

Revert "Reduce scope of FileOutputStream in ExternalSorter"

7e2340d

This reverts commit 3c9c944.

Hacky WIP towards speculatively running w/o reset(), then retrying wi…

8113eac

…th it.

Hackily comment out most of dev/run-tests to speed up Jenkins iteration.

417f50e

Don't pass configuration to ObjectWritable in SerializableWritable

f90dc94

JoshRosen reviewed Jun 5, 2015
View reviewed changes

Broadcast configuration in hiveWriterContainers (WIP hack)

480d20a

Use local[*] instead of local[2]

55041d2

JoshRosen force-pushed the hive-compat-suite-speedup branch from 137a0c7 to 55041d2 Compare June 5, 2015 09:09

JoshRosen added 2 commits June 5, 2015 02:22

try in-memory Derby

57c2cb4

Avoid writing empty files in BypassMergeSortShuffleWriter

9db0abc

JoshRosen added 4 commits June 5, 2015 14:35

Rework SPARK-7041 for BypassMergeSort split

9e116d1

Revert "Broadcast configuration in hiveWriterContainers (WIP hack)"

3fe16e8

This reverts commit 480d20a.

Merge remote-tracking branch 'origin/master' into file-handle-optimiz…

54cd5ce

…ations

Rework SPARK-7041 for BypassMergeSort split

5c777cf

JoshRosen added 3 commits June 5, 2015 15:38

Revert "Use local[*] instead of local[2]"

5ac11d1

This reverts commit 55041d2.

Revert "try in-memory Derby"

895de59

This reverts commit 57c2cb4.

Revert "Don't pass configuration to ObjectWritable in SerializableWri…

bf30fee

…table" This reverts commit f90dc94.

JoshRosen and others added 2 commits June 5, 2015 16:25

Revert "Rework SPARK-7041 for BypassMergeSort split"

5113370

This reverts commit 9e116d1.

SPARK-8135. In SerializableWritable, don't load defaults when instant…

fac08d5

…iating Configuration

JoshRosen added 2 commits June 5, 2015 22:21

Merge remote-tracking branch 'origin/master' into file-handle-optimiz…

c7caa5c

…ations

Merge branch 'file-handle-optimizations' into hive-compat-suite-speedup

8c1e1ff

Only run unsafe tests (testing a jenkins job)

2b500b9

Add log4j test properties to unsafe project.

fbd3d03

Try only testing bagel instead (since I don't think Maven logs to uni…

46dd005

…t-tests.log?)

JoshRosen closed this Jun 13, 2015

JoshRosen deleted the hive-compat-suite-speedup branch November 24, 2015 00:35

JoshRosen restored the hive-compat-suite-speedup branch November 24, 2015 00:35

JoshRosen mentioned this pull request Dec 1, 2015

[SPARK-12075][SQL] Speed up HiveComparisionTest by avoiding / speeding up TestHive.reset() #10055

Closed

JoshRosen deleted the hive-compat-suite-speedup branch August 29, 2016 19:21

[WIP] [SQL] Experiment with speculatively running without reset() in HiveCompatibilitySuite #6663

[WIP] [SQL] Experiment with speculatively running without reset() in HiveCompatibilitySuite #6663

Uh oh!

Conversation

JoshRosen commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

JoshRosen commented Jun 5, 2015

Uh oh!

JoshRosen Jun 5, 2015

Choose a reason for hiding this comment

Uh oh!

JoshRosen Jun 5, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

JoshRosen commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 6, 2015

Uh oh!

SparkQA commented Jun 6, 2015

Uh oh!

JoshRosen commented Jun 6, 2015

Uh oh!

SparkQA commented Jun 6, 2015

Uh oh!

SparkQA commented Jun 8, 2015

Uh oh!

JoshRosen commented Jun 9, 2015

Uh oh!

SparkQA commented Jun 9, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants