[SPARK-17271] [SQL] Planner adds un-necessary Sort even if child ordering is semantically same as required ordering

[tejasapatil]

What changes were proposed in this pull request?

Jira : https://issues.apache.org/jira/browse/SPARK-17271

Planner is adding un-needed SORT operation due to bug in the way comparison for SortOrder is done at https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala#L253
SortOrder needs to be compared semantically because Expression within two SortOrder can be "semantically equal" but not literally equal objects.

eg. In case of sql("SELECT * FROM table1 a JOIN table2 b ON a.col1=b.col1")

Expression in required SortOrder:

      AttributeReference(
        name = "col1",
        dataType = LongType,
        nullable = false
      ) (exprId = exprId,
        qualifier = Some("a")
      )

Expression in child SortOrder:

      AttributeReference(
        name = "col1",
        dataType = LongType,
        nullable = false
      ) (exprId = exprId)

Notice that the output column has a qualifier but the child attribute does not but the inherent expression is the same and hence in this case we can say that the child satisfies the required sort order.

This PR includes following changes:

• Added a semanticEquals method to SortOrder so that it can compare underlying child expressions semantically (and not using default Object.equals)
• Fixed EnsureRequirements to use semantic comparison of SortOrder

How was this patch tested?

• Added a test case to PlannerSuite. Ran rest tests in PlannerSuite

[tejasapatil]

ok to test

[SparkQA]

Test build #64512 has finished for PR 14841 at commit 7192418.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tejasapatil]

cc'ing @rxin and @hvanhovell for review

[hvanhovell]

LGTM - merging to master. Thanks!

[cloud-fan]

Expression has a default implementation of semanticEquals, doesn't it work?

[tejasapatil]

@cloud-fan : If you look at the old version of EnsureRequirements below at L253, it compared raw SortOrder objects which will use equals() generated for it. In scala, equals() for case classes is merely doing equals() over all its fields so that lead to Expression's equals() being used instead of its semanticEquals().

My fix here was to introduce a semanticEquals in SortOrder which compares the underlying Expression semantically.

[cloud-fan]

yea I understand in EnsureRequirements we should use semanticEquals instead of == to compare SortOrder, but why we need to implement samanticEquals again in SortOrder? What's wrong with the default implementation?

I mean, there is no need to "introduce" a semanticEquals in SortOrder, it already has, because SortOrder is also Expression

[tejasapatil]

@cloud-fan : I see what you were trying to say before. I tried that and it worked. I have created a PR to clean it up : #14910 Thanks for pointing this out !!

[rxin]

Should this go into 2.0?

[cloud-fan]

yea, I think so, it's pretty safe. @tejasapatil do you mind sending a new PR against branch 2.0 to backport this PR and the follow-up one? thanks!