[SPARK-13871][SQL] Support for inferring filters from data constraints #11665
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
Test build #52963 has finished for PR 11665 at commit
|
|
Test build #53051 has finished for PR 11665 at commit
|
sameeragarwal
changed the title
|
Test build #53101 has finished for PR 11665 at commit
|
Member
Author
|
test this please |
|
Test build #53104 has finished for PR 11665 at commit
|
Member
Author
|
test this please |
|
Test build #53113 has finished for PR 11665 at commit
|
Contributor
There was a problem hiding this comment.
This rule doesn't seem related to the comment above it (same for nullfiltering)
Contributor
There was a problem hiding this comment.
This comment seems like a clearer version of the first sentence in the object comment. I'd remove this here and replace the first sentence of the above comment.
sameeragarwal
force-pushed
the
infer-filters
branch
from
e3d84ce to
336a18e
Compare
Member
Author
|
thanks, all comments addressed! |
sameeragarwal
force-pushed
the
infer-filters
branch
from
336a18e to
92d935f
Compare
|
Test build #53330 has finished for PR 11665 at commit
|
|
Test build #53331 has finished for PR 11665 at commit
|
Contributor
|
LGTM |
roygao94
pushed a commit
to roygao94/spark
that referenced
this pull request
Mar 22, 2016
## What changes were proposed in this pull request? This PR generalizes the `NullFiltering` optimizer rule in catalyst to `InferFiltersFromConstraints` that can automatically infer all relevant filters based on an operator's constraints while making sure of 2 things: (a) no redundant filters are generated, and (b) filters that do not contribute to any further optimizations are not generated. ## How was this patch tested? Extended all tests in `InferFiltersFromConstraintsSuite` (that were initially based on `NullFilteringSuite` to test filter inference in `Filter` and `Join` operators. In particular the 2 tests ( `single inner join with pre-existing filters: filter out values on either side` and `multiple inner joins: filter out values on all sides on equi-join keys` attempts to highlight/test the real potential of this rule for join optimization. Author: Sameer Agarwal <[email protected]> Closes apache#11665 from sameeragarwal/infer-filters.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR generalizes the
NullFilteringoptimizer rule in catalyst toInferFiltersFromConstraintsthat can automatically infer all relevant filters based on an operator's constraints while making sure of 2 things:(a) no redundant filters are generated, and
(b) filters that do not contribute to any further optimizations are not generated.
How was this patch tested?
Extended all tests in
InferFiltersFromConstraintsSuite(that were initially based onNullFilteringSuiteto test filter inference inFilterandJoinoperators.In particular the 2 tests (
single inner join with pre-existing filters: filter out values on either sideandmultiple inner joins: filter out values on all sides on equi-join keysattempts to highlight/test the real potential of this rule for join optimization.