Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Aug 18, 2017

What changes were proposed in this pull request?

This PR support for pushing down filters for DateType in ORC

How was this patch tested?

Pass the Jenkins with newly add and updated test cases.

@dongjoon-hyun
Copy link
Member Author

Hi, @cloud-fan, @gatorsmile, @rxin , @sameeragarwal .
I tested this on #18953, this only works with Apache ORC 1.4.0.
This issue is reported saparately. And, to keep #18953 minimally, I didn't include this into that.

@SparkQA
Copy link

SparkQA commented Aug 18, 2017

Test build #80854 has finished for PR 18995 at commit 7f63202.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 18, 2017

Test build #80857 has finished for PR 18995 at commit b31acc5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-21787 branch November 2, 2017 18:11
@dongjoon-hyun dongjoon-hyun restored the SPARK-21787 branch November 10, 2017 19:28
@dongjoon-hyun dongjoon-hyun reopened this Nov 10, 2017
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC [WIP][SPARK-21787][SQL] Support for pushing down filters for DateType in ORC Nov 10, 2017
@dongjoon-hyun
Copy link
Member Author

This will be supported in a new ORCFileFormat after merging #18953.

@SparkQA
Copy link

SparkQA commented Nov 10, 2017

Test build #83700 has finished for PR 18995 at commit b31acc5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun changed the title [WIP][SPARK-21787][SQL] Support for pushing down filters for DateType in ORC [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC Dec 3, 2017
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please note that this suite use Apache ORC SearchArgument and PredicateLeaf classes.
We cannot share the test code for this suite.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By using full canonical class names, we will explicitly test new OrcFileFormat in sql/core and old OrcFileFormat in sql/hive without considering SQL ORC switching conf.

@dongjoon-hyun
Copy link
Member Author

Hi, @cloud-fan .
This is the first followup after #19651 .
Could you review this PR, too?

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC [SPARK-21787][SPARK-22672][SQL] Support for pushing down filters for DateType in ORC Dec 3, 2017
@SparkQA
Copy link

SparkQA commented Dec 4, 2017

Test build #84405 has finished for PR 18995 at commit 9f6d75b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

maybe have a PR to move the tests first?

@dongjoon-hyun
Copy link
Member Author

@cloud-fan .

For the other part, I'm restructuring to remove redundancy in the same manner of InMemoryCatalogedDDLSuite, HiveCatalogedDDLSuite, and DDLSuite.

This one cannot be the part of that restructuring, because the test function signatures are different due to PredicateLeaf and SearchArgument.

Could you review this first?

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 4, 2017

BTW, @cloud-fan . Do you mean literally move? I'm wondering if I'm thinking in a different way.

Otherwise, I had a set of ported duplicated test suites in the previous PRs. Is it okay?

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-21787][SPARK-22672][SQL] Support for pushing down filters for DateType in ORC [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC Dec 7, 2017
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC [SPARK-21787][SQL] Support for pushing down filters for DateType in native OrcFileFormat Dec 7, 2017
@SparkQA
Copy link

SparkQA commented Dec 7, 2017

Test build #84609 has finished for PR 18995 at commit 18accdd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

Hi, @cloud-fan .
Could you review this PR again?

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@asfgit asfgit closed this in 0ba8f4b Dec 8, 2017
@dongjoon-hyun
Copy link
Member Author

Thank you so much, @cloud-fan !

@dongjoon-hyun dongjoon-hyun deleted the SPARK-21787 branch December 8, 2017 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants