Skip to content

Conversation

@viirya
Copy link
Member

@viirya viirya commented Aug 24, 2020

What changes were proposed in this pull request?

This PR proposes to fix ORC predicate pushdown under case-insensitive analysis case. The field names in pushed down predicates don't need to match in exact letter case with physical field names in ORC files, if we enable case-insensitive analysis.

This is re-submitted for #29457. Because #29457 has a hive-1.2 error and there were some tests failed with hive-1.2 profile at the same time, #29457 was reverted to unblock others.

Why are the changes needed?

Currently ORC predicate pushdown doesn't work with case-insensitive analysis. A predicate "a < 0" cannot pushdown to ORC file with field name "A" under case-insensitive analysis.

But Parquet predicate pushdown works with this case. We should make ORC predicate pushdown work with case-insensitive analysis too.

Does this PR introduce any user-facing change?

Yes, after this PR, under case-insensitive analysis, ORC predicate pushdown will work.

How was this patch tested?

Unit tests.

@viirya viirya changed the title [SPARK-32646][SQL] ORC predicate pushdown should work with case-insensitive analysis [SPARK-32646][SQL][test-hadoop2.7][test-hive1.2] ORC predicate pushdown should work with case-insensitive analysis Aug 24, 2020
@viirya
Copy link
Member Author

viirya commented Aug 24, 2020

retest this please

@SparkQA
Copy link

SparkQA commented Aug 24, 2020

Test build #127848 has finished for PR 29530 at commit 5807075.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Member Author

viirya commented Aug 24, 2020

retest this please

@SparkQA
Copy link

SparkQA commented Aug 24, 2020

Test build #127847 has finished for PR 29530 at commit 5807075.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 25, 2020

Test build #127855 has finished for PR 29530 at commit 5807075.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Member Author

viirya commented Aug 25, 2020

OK. Passed tests of both hive-2.3 and hive-1.2 profiles. This is the same diff as previous #29457, but fixed a hive-1.2 compilation error. cc @cloud-fan @HyukjinKwon @dongjoon-hyun

Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine as it was already reviewed, and the compilation issue looks minor.

@HyukjinKwon
Copy link
Member

I am going to merge this. The last commit is just about accessor and explicit return type which should be caught during the build.

@HyukjinKwon
Copy link
Member

Merged to master.

@viirya
Copy link
Member Author

viirya commented Aug 25, 2020

Thanks @HyukjinKwon

@SparkQA
Copy link

SparkQA commented Aug 25, 2020

Test build #127867 has finished for PR 29530 at commit 50f0037.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya viirya deleted the fix-orc-pushdown branch December 27, 2023 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants