-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-10829][SQL]Filter combine partition key and attribute doesn't work in DataSource scan #8916
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ttributes doesn't work
|
cc @liancheng @yhuai |
|
Test build #43012 has finished for PR 8916 at commit
|
|
cc @liancheng @yhuai Just in case you missed this PR. :) |
|
What is our current behavior? Do we pushdown |
|
Currently, it simply ignore the filter condition which contains both partition key and the other column. For the unit test case, it simply ignore the part |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of using groupBy, can we just use filter 3 times to split these 3 kinds of filters? I think performance doesn't matter here.
|
Test build #43611 has finished for PR 8916 at commit
|
|
Thank you @cloud-fan for the reviewing, updated, and passed the unit test. |
|
LGTM, cc @liancheng to take another look. |
|
LGTM, merging to master. Thanks! |
…d columns (1.5 backport) [SPARK-10829](#8916) Filter combine partition key and attribute doesn't work in DataSource scan [SPARK-11301](#9271) fix case sensitivity for filter on partitioned columns Author: Wenchen Fan <[email protected]> This patch had conflicts when merged, resolved by Committer: Yin Huai <[email protected]> Closes #9371 from cloud-fan/branch-1.5.
We expect the result to be:
But got