[SPARK-12539][follow-up] always sort in partitioning writer

[cloud-fan]

address comments in #10498 , especially #10498 (comment)

[cloud-fan]

cc @davies @rxin

[SparkQA]

Test build #48926 has finished for PR 10638 at commit e523245.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

Could we generate a partition key (excluding the sorting key)? we just need to compare the key

[SparkQA]

Test build #49001 has finished for PR 10638 at commit cceb7a8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49051 has finished for PR 10638 at commit 247ddad.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

LGTM.

[rxin]

I manually resolved the conflict during merging and merged it. Thanks.

[davies]

@cloud-fan Should we create a patch for 1.6 branch?

[cloud-fan]

only backport it without bucket-write?

[davies]

Yes, cc @marmbrus

[rxin]

We discussed this and decided not to do it, because this is a simple config change users can apply. Changing the behavior automatically for users in a point release is also bad.