-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-27880][SQL] Add bool_and for every and bool_or for any as function aliases #26126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #112103 has finished for PR 26126 at commit
|
| expression[EveryAgg]("every"), | ||
| expression[AnyAgg]("any"), | ||
| expression[SomeAgg]("some"), | ||
| expression[BoolAnd]("bool_and"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is bool_and just an alias of every? If so we can do expression[EveryAgg]("bool_and")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it is. https://github.com/apache/spark/pull/26126/files#diff-fdccd9945d709da6c561b67a2e46a0d8R296, is there a way to keep the bool_and string in the schema?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need to keep the function alias in the schema. There are many examples in FunctionRegistry, that a function has more than one name and we always use one name when displaying.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Then any, some and bool_or are mathematically equivalent, I guess we can only left AnyAgg and remove other implemtations
|
Can we make the title more explicit that this just adds function alias? |
|
Test build #112142 has finished for PR 26126 at commit
|
|
retest this please |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should enable these tests: aggregates_part2.sql#L116-L160 and udf-aggregates_part2.sql#L118-L162. Are we doing this in this PR or in the follow-up PR?
|
Test build #112145 has finished for PR 26126 at commit
|
|
@wangyum thanks for noticing. But why these same tests in two different test files? |
|
Test build #112160 has finished for PR 26126 at commit
|
|
Test build #112162 has finished for PR 26126 at commit
|
|
thanks, merging to master! |
What changes were proposed in this pull request?
bool_or(x) <=> any/some(x) <=> max(x)
bool_and(x) <=> every(x) <=> min(x)
Args:
x: boolean
Why are the changes needed?
PostgreSQL, Presto and Vertica, etc also support this feature:
Does this PR introduce any user-facing change?
add new functions support
How was this patch tested?
add ut