[SPARK-7215] made coalesce and repartition a part of the query plan

[brkyvz]

Coalesce and repartition now show up as part of the query plan, rather than resulting in a new DataFrame.

cc @rxin

[rxin]

import order

[rxin]

LGTM otherwise

[SparkQA]

Test build #31191 has finished for PR 5762 at commit f2e6af1.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Coalesce(numPartitions: Int, shuffle: Boolean, child: LogicalPlan) extends UnaryNode
  • case class Coalesce(numPartitions: Int, shuffle: Boolean, child: SparkPlan) extends UnaryNode
• This patch does not change any dependencies.

[SparkQA]

Test build #31192 has finished for PR 5762 at commit 2c349b5.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Coalesce(numPartitions: Int, shuffle: Boolean, child: LogicalPlan) extends UnaryNode
  • case class Coalesce(numPartitions: Int, shuffle: Boolean, child: SparkPlan) extends UnaryNode
• This patch does not change any dependencies.

[brkyvz]

@rxin Should I rename this to Repartition? There are a lot of conflicts coming from catalyst and sql. In fact, the Coalesce function in catalyst fits it's usage, which is to combine (elements) in a mass or whole.. Here, we are basically repartitioning the dataset. Coalesce with a higher number of partitions sounds weird. Also it might be weird to have two different types of Coalesce. What do you think?

[rxin]

Yea repartition sounds better.

[marmbrus]

Oh, or repartition sounds even better than CoalescePartitions

[SparkQA]

Test build #31215 has finished for PR 5762 at commit fa4509f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class CoalescePartitions(numPartitions: Int, shuffle: Boolean, child: LogicalPlan)
  • case class CoalescePartitions(numPartitions: Int, shuffle: Boolean, child: SparkPlan)
• This patch does not change any dependencies.

[SparkQA]

Test build #31234 has finished for PR 5762 at commit b1e76dd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Repartition(numPartitions: Int, shuffle: Boolean, child: LogicalPlan)
  • case class RepartitionByExpression(partitionExpressions: Seq[Expression], child: LogicalPlan)
  • case class Repartition(numPartitions: Int, shuffle: Boolean, child: SparkPlan)
• This patch does not change any dependencies.

[rxin]

Thanks. I've merged this in master.