[SPARK-10708] Consolidate sort shuffle implementations

[JoshRosen]

There's a lot of duplication between SortShuffleManager and UnsafeShuffleManager. Given that these now provide the same set of functionality, now that UnsafeShuffleManager supports large records, I think that we should replace SortShuffleManager's serialized shuffle implementation with UnsafeShuffleManager's and should merge the two managers together.

[SparkQA]

Test build #42693 has finished for PR 8829 at commit 803f62f.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

TODO: update these comments.

[SparkQA]

Test build #42782 has finished for PR 8829 at commit 26ecf5c.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42784 has finished for PR 8829 at commit af2794c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42866 has finished for PR 8829 at commit 68c3a25.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42868 has finished for PR 8829 at commit 426e016.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42931 has finished for PR 8829 at commit 8276eb0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42978 has finished for PR 8829 at commit 3ffa137.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42990 has finished for PR 8829 at commit 8fe9094.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

This patch should now be ready for a preliminary round of review. There are still some comment updates to make, plus an update to preserve certain aspects of the old bypass-merge-sort shuffle fallback path, but the basics are good to go.

/cc @rxin and @sryza for feedback. The key idea here is to consolidate on a single implementation of serialized buffering in sort-based shuffle in order to easy maintainability.

[JoshRosen]

Jenkins, retest this please.

[rxin]

Can you add some comment explaining why this only works when aggregator & mapsidecombine is off?

[JoshRosen]

Fixed via a larger refactoring / cleanup of this logic.

[rxin]

I took a pass. Looks pretty good.

[SparkQA]

Test build #44011 has finished for PR 8829 at commit f7c620c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #44027 has started for PR 8829 at commit f7c620c.

[rxin]

LGTM

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #44090 has finished for PR 8829 at commit f7c620c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #1934 has finished for PR 8829 at commit f7c620c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44101 has finished for PR 8829 at commit f7c620c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #1937 has finished for PR 8829 at commit db0cd28.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44135 has finished for PR 8829 at commit db0cd28.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Woohoo, this passed tests and looks good in benchmarking, so I'm going to merge it to master (1.6).