[SPARK-10403] Allow UnsafeRowSerializer to work with tungsten-sort ShuffleManager

[JoshRosen]

This patch attempts to fix an issue where Spark SQL's UnsafeRowSerializer was incompatible with the tungsten-sort ShuffleManager.

[JoshRosen]

Not 100% sure if this is sufficient for a proper fix.

[SparkQA]

Test build #42865 has finished for PR 8873 at commit e6fdd46.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class WriterThread(

[JoshRosen]

@davies, I was puzzled about why UnsafeRowSerializer didn't fail with similar exceptions when re-ordering serialized rows using the PartitionedSerializedPairBuffer. It turns out that the ExternalSorter's serialized shuffle sort path ends up using a DiskBlockObjectWriter in such a way that the serializer instance's close() method is called when closing the writer. The UnsafeShuffleWriter, on the other hand, uses a different write path which does not do this, leading to the error that you saw.

[davies]

LGTM

[SparkQA]

Test build #42867 has finished for PR 8873 at commit 5419ca4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Merging to master and 1.5