[SPARK-29883][SQL] Implement a helper method for aliasing bool_and() and bool_or() #26712

amanomer · 2019-11-29T09:35:47Z

What changes were proposed in this pull request?

This PR introduces a method expressionWithAlias in class FunctionRegistry which is used to register function's constructor. Currently, expressionWithAlias is used to register BoolAnd & BoolOr.

Why are the changes needed?

Error message is wrong when alias name is used for BoolAnd & BoolOr.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Tested manually.

For query,
select every('true');

Output before this PR,

Error in query: cannot resolve 'bool_and('true')' due to data type mismatch: Input to function 'bool_and' should have been boolean, but it's [string].; line 1 pos 7;

After this PR,

Error in query: cannot resolve 'every('true')' due to data type mismatch: Input to function 'every' should have been boolean, but it's [string].; line 1 pos 7;

amanomer · 2019-11-29T09:37:38Z

@gatorsmile I have now handled for BoolAnd and BoolOr. Kindly review the changes.

amanomer · 2019-11-29T09:37:55Z

cc @srowen @cloud-fan

srowen · 2019-11-29T14:06:07Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

Hm, do functions not already have names somewhere to use, that can already be set differently per alias? it looks like that's what nodeName is for, and it's already overridden in the aliases, so I'm missing why this is different.

do functions not already have names somewhere to use, that can already be set differently per alias?

I think, No? Currently, when we use alias of bool_and (i.e, every), it will be resolved as a constructor of BoolAnd using FunctionRegistry#expressions, which sets `bool_and' as a nodeName.

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/UnevaluableAggs.scala

Lines 55 to 57 in 9351e3e

case class BoolAnd(arg: Expression) extends UnevaluableBooleanAggBase(arg) {

override def nodeName: String = "bool_and"

}

This may not make sense, I don't know this code well, but: is it not that "Exists" needs to customize its nodeName, for example? or does it never exist as such a node.

every here does not have any node. It will be resolved as a BoolAnd.

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

Lines 316 to 320 in 32af700

expression[BoolAnd]("every"),

expression[BoolAnd]("bool_and"),

expression[BoolOr]("any"),

expression[BoolOr]("some"),

expression[BoolOr]("bool_or"),

I see. I don't know enough to evaluate the effect of changing nodeName for all implementations, which seems like a broader change than required, but it does make some sense. Maybe @liancheng or @yhuai has an opinion.

amanomer · 2019-12-02T03:11:20Z

@gatorsmile Kindly review this PR.

cloud-fan · 2019-12-02T03:48:25Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

It's a bad practice to make the Expression mutable for trivial things like this. How about using RuntimeReplaceable?

Argument's data types are matched in Analysis phase of planning and its optimizer's task to replace RuntimeReplaceable, correct me if I'm wrong.
So, optimization rules (ReplaceExpressions) will never be applied on queries like select every('true').

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

Line 130 in 85cb388

ReplaceExpressions,

cloud-fan · 2019-12-02T07:41:26Z

...yst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/UnevaluableAggs.scala

how about

case class BoolAnd(functionName: String, arg: Expression) ... with MultiNamesFunction { def nodeName = functionName }

We can update FunctionRegistry.expression to detect MultiNamesFunction and pass name to the constructor.

Updated PR. cc @cloud-fan

cloud-fan · 2019-12-03T08:23:17Z

ok to test

cloud-fan · 2019-12-03T08:32:04Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

        val params = Seq.fill(expressions.size)(classOf[Expression])
-        val f = constructors.find(_.getParameterTypes.toSeq == params).getOrElse {
+        val f = constructors.find(e => e.getParameterTypes.toSeq == params
+            || e.getParameterTypes.head == classOf[String]).getOrElse {


Seems like it's less hacky to create a new expressionWithAlias method, with only the necessary logic

def expressionWithAlias ... = { val constructors = tag.runtimeClass.getConstructors .filter(c => e.getParameterTypes.head == classOf[String]) assert(constructors.length == 1) try { constructors.head.newInstance(name, expressions : _*).asInstanceOf[Expression] } ... }

then we don't even need the MultiNamedExpression trait. We just need to register bool_and, bool_or with expressionWithAlias

@cloud-fan updated as per your suggestions.

SparkQA · 2019-12-03T09:29:30Z

Test build #114761 has finished for PR 26712 at commit 3387eef.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-12-03T10:54:13Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

+      .filter(_.getParameterTypes.head == classOf[String])
+    assert(constructors.length == 1)
+    val builder = (expressions: Seq[Expression]) => {
+      Try(constructors.head.newInstance(name.toString, expressions.head).asInstanceOf[Expression])


seems better to use the normal try catch?

try { constructors.head.newInstance(name.toString, expressions.head).asInstanceOf[Expression] } catch { // the original comment ... case e => throw new AnalysisException(e.getCause.getMessage) }

We can update def expression as well.

Updated in latest commit.

SparkQA · 2019-12-03T11:49:30Z

Test build #114764 has finished for PR 26712 at commit 4476e7e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class BoolAnd(funcName: String, arg: Expression) extends UnevaluableBooleanAggBase(arg)
case class BoolOr(funcName: String, arg: Expression) extends UnevaluableBooleanAggBase(arg)

SparkQA · 2019-12-03T16:09:44Z

Test build #114786 has finished for PR 26712 at commit 77ad53f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-03T16:25:59Z

Test build #114787 has finished for PR 26712 at commit 404d829.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-03T19:30:10Z

Test build #114797 has finished for PR 26712 at commit 27f1a7f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

Looks good to me too if tests pass

SparkQA · 2019-12-04T08:05:02Z

Test build #114828 has finished for PR 26712 at commit 2952898.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-12-04T08:08:48Z

retest this please

SparkQA · 2019-12-04T11:41:32Z

Test build #114842 has finished for PR 26712 at commit 2952898.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-04T22:20:10Z

Test build #114875 has finished for PR 26712 at commit bb665e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

amanomer · 2019-12-05T05:26:37Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

+    assert(constructors.length == 1)
+    val builder = (expressions: Seq[Expression]) => {
+      try {
+        constructors.head.newInstance(name.toString, expressions.head).asInstanceOf[Expression]


Since, we are not validating arguments, queries like
SELECT EVERY(true, false); will result true.

We can validate arguments with assert or as used in expression?
cc @cloud-fan @HyukjinKwon

how is it done in def expression?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

Lines 602 to 620 in ebd83a5

val params = Seq.fill(expressions.size)(classOf[Expression])

val f = constructors.find(_.getParameterTypes.toSeq == params).getOrElse {

val validParametersCount = constructors

.filter(_.getParameterTypes.forall(_ == classOf[Expression]))

.map(_.getParameterCount).distinct.sorted

val invalidArgumentsMsg = if (validParametersCount.length == 0) {

s"Invalid arguments for function $name"

} else {

val expectedNumberOfParameters = if (validParametersCount.length == 1) {

validParametersCount.head.toString

} else {

validParametersCount.init.mkString("one of ", ", ", " and ") +

validParametersCount.last

}

s"Invalid number of arguments for function $name. " +

s"Expected: $expectedNumberOfParameters; Found: ${params.length}"

}

throw new AnalysisException(invalidArgumentsMsg)

}

Since, in expressionWithAlias we are always passing expressions.head to function's constructor. We can use assert statement

... val builder = (expressions: Seq[Expression]) => { assert(expressions.size == 1, s"Invalid number of arguments for function $name. " + s"Expected: 1; Found: ${expressions.size}") assert(expressions.head == classOf[Expression], s"Invalid arguments for function $name") try { constructors.head.newInstance(name.toString, expressions.head).asInstanceOf[Expression] } ...

BTW is it possible to do newInstance(name.toString, expressions: _*)? Then it can work for other expressions that take more than 1 parameter.

is it possible to do newInstance(name.toString, expressions: _*)?

No, compilation error.

how about newInstance((name.toString +: expressions): _*)?

It works. Updated in latest commit. cc @cloud-fan

amanomer · 2019-12-05T05:34:52Z

There are other function with alias name example VarianceSamp, StddevSamp. Should we change them in this PR?

cloud-fan · 2019-12-05T05:58:18Z

Should we change them in this PR?

I'm fine either way.

amanomer · 2019-12-05T06:18:18Z

Ok, I will handle them in different PR.

SparkQA · 2019-12-05T10:43:12Z

Test build #114899 has finished for PR 26712 at commit 3c87222.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-05T16:15:47Z

Test build #114903 has finished for PR 26712 at commit 17c91f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

amanomer · 2019-12-06T04:30:29Z

cc @cloud-fan

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

amanomer · 2019-12-06T11:51:01Z

cc @cloud-fan

SparkQA · 2019-12-06T15:51:47Z

Test build #114940 has finished for PR 26712 at commit d07d261.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-12-08T17:43:09Z

I'm OK with it if @cloud-fan is.

dongjoon-hyun · 2019-12-08T19:05:56Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

-          case Failure(e) =>
-            // the exception is an invocation exception. To get a meaningful message, we need the
-            // cause.
-            throw new AnalysisException(e.getCause.getMessage)


Hi, @amanomer . I'm wondering if this change is required in this PR.

This reformatting of try-catch block can be raised in different PR.

FWIW I think this is fine and cleaner, so think it's OK to change here.

I think there are similar try-catch block format on other files too, which can be reformatted like this.

dongjoon-hyun · 2019-12-08T19:06:07Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

-          case Failure(e) =>
-            // the exception is an invocation exception. To get a meaningful message, we need the
-            // cause.
-            throw new AnalysisException(e.getCause.getMessage)


maropu

Can you make the title clearer like Implement a helper method for aliasing registered functions?

cloud-fan · 2019-12-09T05:23:47Z

thanks, merging to master!

amanomer · 2019-12-09T05:36:21Z

Thanks all for reviewing and merging

### What changes were proposed in this pull request? This PR is to use `expressionWithAlias` for remaining functions for which alias name can be used. Remaining functions are: `Average, First, Last, ApproximatePercentile, StddevSamp, VarianceSamp` PR #26712 introduced `expressionWithAlias` ### Why are the changes needed? Error message is wrong when alias name is used for above mentioned functions. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Manually Closes #26808 from amanomer/fncAlias. Lead-authored-by: Aman Omer <[email protected]> Co-authored-by: Aman Omer <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

srowen reviewed Nov 29, 2019

View reviewed changes

amanomer mentioned this pull request Dec 2, 2019

[SPARK-29628][SQL] Forcibly create a temporary view in CREATE VIEW if referencing a temporary view #26317

Closed

cloud-fan reviewed Dec 2, 2019

View reviewed changes

Added MultiNamedExpression

dbf6b4f

amanomer force-pushed the 29883 branch from bc02d23 to dbf6b4f Compare December 2, 2019 16:15

Removed unused import

3387eef

amanomer requested review from cloud-fan and srowen December 2, 2019 16:20

cloud-fan reviewed Dec 3, 2019

View reviewed changes

Added expressionWithAlias

4476e7e

cloud-fan reviewed Dec 3, 2019

View reviewed changes

cloud-fan approved these changes Dec 3, 2019

View reviewed changes

amanomer added 2 commits December 3, 2019 19:52

Updated try-catch

77ad53f

Updated TC ExplainSuite.scala

404d829

TC update

27f1a7f

HyukjinKwon approved these changes Dec 4, 2019

View reviewed changes

Updated TC in group-by.sql.out

2952898

amanomer commented Dec 5, 2019

View reviewed changes

validate expression

3c87222

Handled for expressions that take more than 1 parameter

17c91f4

dongjoon-hyun added the SQL label Dec 5, 2019

cloud-fan reviewed Dec 6, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala Show resolved Hide resolved

Add meaningfull error message

d07d261

cloud-fan approved these changes Dec 6, 2019

View reviewed changes

amanomer changed the title ~~[SPARK-29883][SQL] Improve error messages when function name is an alias~~ [SPARK-29883][SQL] Improve error messages when bool_and() and bool_or() is called using alias Dec 7, 2019

dongjoon-hyun reviewed Dec 8, 2019

View reviewed changes

maropu approved these changes Dec 9, 2019

View reviewed changes

amanomer changed the title ~~[SPARK-29883][SQL] Improve error messages when bool_and() and bool_or() is called using alias~~ [SPARK-29883][SQL] IImplement a helper method for aliasing bool_and() and bool_or() Dec 9, 2019

amanomer changed the title ~~[SPARK-29883][SQL] IImplement a helper method for aliasing bool_and() and bool_or()~~ [SPARK-29883][SQL] Implement a helper method for aliasing bool_and() and bool_or() Dec 9, 2019

cloud-fan closed this in dcea7a4 Dec 9, 2019

amanomer mentioned this pull request Dec 9, 2019

[SPARK-30184][SQL] Implement a helper method for aliasing functions #26808

Closed

	case class BoolAnd(arg: Expression) extends UnevaluableBooleanAggBase(arg) {
	override def nodeName: String = "bool_and"
	}

	expression[BoolAnd]("every"),
	expression[BoolAnd]("bool_and"),
	expression[BoolOr]("any"),
	expression[BoolOr]("some"),
	expression[BoolOr]("bool_or"),

	val params = Seq.fill(expressions.size)(classOf[Expression])
	val f = constructors.find(_.getParameterTypes.toSeq == params).getOrElse {
	val validParametersCount = constructors
	.filter(_.getParameterTypes.forall(_ == classOf[Expression]))
	.map(_.getParameterCount).distinct.sorted
	val invalidArgumentsMsg = if (validParametersCount.length == 0) {
	s"Invalid arguments for function $name"
	} else {
	val expectedNumberOfParameters = if (validParametersCount.length == 1) {
	validParametersCount.head.toString
	} else {
	validParametersCount.init.mkString("one of ", ", ", " and ") +
	validParametersCount.last
	}
	s"Invalid number of arguments for function $name. " +
	s"Expected: $expectedNumberOfParameters; Found: ${params.length}"
	}
	throw new AnalysisException(invalidArgumentsMsg)
	}

[SPARK-29883][SQL] Implement a helper method for aliasing bool_and() and bool_or() #26712

[SPARK-29883][SQL] Implement a helper method for aliasing bool_and() and bool_or() #26712

Uh oh!

Conversation

amanomer commented Nov 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

amanomer commented Nov 29, 2019

Uh oh!

amanomer commented Nov 29, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amanomer commented Dec 2, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Dec 3, 2019

Uh oh!

cloud-fan Dec 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 3, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 3, 2019

Uh oh!

SparkQA commented Dec 3, 2019

Uh oh!

SparkQA commented Dec 3, 2019

Uh oh!

SparkQA commented Dec 3, 2019

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 4, 2019

Uh oh!

cloud-fan commented Dec 4, 2019

Uh oh!

SparkQA commented Dec 4, 2019

Uh oh!

SparkQA commented Dec 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amanomer commented Nov 29, 2019 •

edited

Loading

cloud-fan Dec 3, 2019 •

edited

Loading

amanomer Dec 5, 2019 •

edited

Loading

amanomer commented Dec 5, 2019 •

edited

Loading

amanomer commented Dec 5, 2019 •

edited

Loading