-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-14137][SPARK-14150][SQL] Infer IsNotNull constraints from non-nullable attributes #11953
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we break chains like this down with intermediate variables and comments? they are becoming fairly difficult to understand.
Also the foldLeft/union ... seems like a pretty inefficient way to run this and also difficult to read.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, for sure
|
Test build #54157 has finished for PR 11953 at commit
|
|
cc @davies: I verified that this patch fixes |
|
Test build #54159 has finished for PR 11953 at commit
|
|
#11828 Has been merged, could you enable |
6421ba7 to
2f43d41
Compare
|
thanks, done |
|
LGTM |
|
Test build #54191 has finished for PR 11953 at commit
|
|
Merging this into master |
What changes were proposed in this pull request?
This PR adds support for automatically inferring
IsNotNullconstraints from any non-nullable attributes that are part of an operator's output. This also fixes the issue that causes the optimizer to hit the maximum number of iterations for certain queries in #11828.How was this patch tested?
Unit test in
ConstraintPropagationSuite