Skip to content

Conversation

@prakharjain09
Copy link
Contributor

What changes were proposed in this pull request?

This is a followup of #30302 . As part of this PR, sameOrderExpressions set is made part of children of SortOrder node - so that they don't need any special handling as done in #30302 .

Why are the changes needed?

sameOrderExpressions should get same treatment as child. So making them part of children helps in transforming them easily.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing UTs

@github-actions github-actions bot added the SQL label Nov 19, 2020
@prakharjain09
Copy link
Contributor Author

@cloud-fan @maropu I have made the changes as suggested in #30302 (comment) .

@HyukjinKwon
Copy link
Member

ok to test

@maropu
Copy link
Member

maropu commented Nov 20, 2020

@prakharjain09 Thanks for working on this. btw, could you assign a new jira ID to this?

child: Expression,
direction: SortDirection,
nullOrdering: NullOrdering,
sameOrderExpressions: Set[Expression])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to use Set for this variable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The set property of this variable is used at couple of places in SortMergeJoin (here, here).

We can use ExpressionSet maybe at those 2 places and make it a Seq in constructor here. any thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ExpressionSet maybe at those 2 places and make it a Seq in constructor here. any thoughts?

Yea, I think it is okay to just deduplicate it before storing it in the class as you said, and then make it Seq.

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35990/

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35990/

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Test build #131386 has finished for PR 30430 at commit f3a8ded.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@prakharjain09 prakharjain09 changed the title [SPARK-33400][SQL][FOLLOWUP] Make sameOrderExpressions part of SortOrder childrens [SPARK-33503][SQL] Refactor SortOrder class to allow multiple childrens Nov 20, 2020
@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36032/

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36032/

@SparkQA
Copy link

SparkQA commented Nov 20, 2020

Test build #131426 has finished for PR 30430 at commit 9510b5f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@prakharjain09 prakharjain09 requested a review from maropu November 23, 2020 07:33
@SparkQA
Copy link

SparkQA commented Nov 25, 2020

Test build #131785 has finished for PR 30430 at commit 1c38e97.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 27, 2020

Test build #131885 has finished for PR 30430 at commit b82540e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 27, 2020

Test build #131886 has finished for PR 30430 at commit 815396e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnixTimestamp(
  • case class SerdeInfo(
  • case class FormatClasses(input: String, output: String)
  • case class CoalesceShufflePartitions(session: SparkSession) extends CustomShuffleReaderRule
  • trait CustomShuffleReaderRule extends Rule[SparkPlan]

@maropu maropu closed this in cf4ad21 Dec 1, 2020
@maropu
Copy link
Member

maropu commented Dec 1, 2020

The GA tests passed, so I merged to master. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants