[SPARK-34086][SQL] RaiseError generates too much code and may fails codegen in length check for char varchar #31150
Conversation
…odegen in length check for char varchar
|
cc @cloud-fan @maropu @HyukjinKwon @gatorsmile @dongjoon-hyun, thanks for reviewing |
| override def foldable: Boolean = false | ||
| override def nullable: Boolean = true | ||
| override def dataType: DataType = NullType | ||
| override def dataType: DataType = returnType |
There was a problem hiding this comment.
nit: the constructor parameter can be dataType directly, then we don't need this override.
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #133968 has finished for PR 31150 at commit
|
| since = "3.1.0", | ||
| group = "misc_funcs") | ||
| case class RaiseError(child: Expression) extends UnaryExpression with ImplicitCastInputTypes { | ||
| case class RaiseError private[spark] (child: Expression, dataType: DataType) |
There was a problem hiding this comment.
catalyst is already a private module. We can just remove private[spark].
|
Test build #133970 has finished for PR 31150 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #133990 has finished for PR 31150 at commit
|
|
thanks, merging to master! |
|
it conflicts with 3.1, @yaooqinn can you send a new PR? |
|
w/ pleasure~ |
…ils codegen in length check for char varchar A backport for #31150 to branch 3.1 ### What changes were proposed in this pull request? https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133928/testReport/org.apache.spark.sql.execution/LogicalPlanTagInSparkPlanSuite/q41/ We can reduce more than 8000 bytes by removing the unnecessary CONCAT expression. W/ this fix, for q41 in TPCDS with [Using TPCDS original definitions for char/varchar columns](#31012) applied, we can reduce the stage code-gen size from 22523 to 14369 ``` 14369 - 22523 = - 8154 ``` ### Why are the changes needed? fix the perf regression(we need other improvements for q41 works), there will be a huge performance regression if codegen fails ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? modified uts Closes #31168 from yaooqinn/SPARK-34086-31. Authored-by: Kent Yao <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
4 participants
What changes were proposed in this pull request?
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133928/testReport/org.apache.spark.sql.execution/LogicalPlanTagInSparkPlanSuite/q41/
We can reduce more than 8000 bytes by removing the unnecessary CONCAT expression.
W/ this fix, for q41 in TPCDS with Using TPCDS original definitions for char/varchar columns applied, we can reduce the stage code-gen size from 22523 to 14369
Why are the changes needed?
fix the perf regression(we need other improvements for q41 works), there will be a huge performance regression if codegen fails
Does this PR introduce any user-facing change?
no
How was this patch tested?
modified uts