Skip to content

Commit b99a64b

Browse files
JoshRosendongjoon-hyun
authored andcommitted
[SPARK-48081] Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type
### What changes were proposed in this pull request? While migrating the `NTile` expression's type check failures to the new error class framework, PR #38457 removed a pair of not-unnecessary `return` statements and thus caused certain branches' values to be discarded rather than returned. As a result, invalid usages like ``` select ntile(99.9) OVER (order by id) from range(10) ``` trigger internal errors like errors like ``` java.lang.ClassCastException: class org.apache.spark.sql.types.Decimal cannot be cast to class java.lang.Integer (org.apache.spark.sql.types.Decimal is in unnamed module of loader 'app'; java.lang.Integer is in module java.base of loader 'bootstrap') at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:99) at org.apache.spark.sql.catalyst.expressions.NTile.checkInputDataTypes(windowExpressions.scala:877) ``` instead of clear error framework errors like ``` org.apache.spark.sql.catalyst.ExtendedAnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "ntile(99.9)" due to data type mismatch: The first parameter requires the "INT" type, however "99.9" has the type "DECIMAL(3,1)". SQLSTATE: 42K09; line 1 pos 7; 'Project [unresolvedalias(ntile(99.9) windowspecdefinition(id#0L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())))] +- Range (0, 10, step=1, splits=None) at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.dataTypeMismatch(package.scala:73) at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$7(CheckAnalysis.scala:315) ``` ### Why are the changes needed? Improve error messages. ### Does this PR introduce _any_ user-facing change? Yes, it improves an error message. ### How was this patch tested? Added a new test case to AnalysisErrorSuite. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46333 from JoshRosen/SPARK-48081. Authored-by: Josh Rosen <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 5bbbc6c commit b99a64b

File tree

2 files changed

+36
-2
lines changed

2 files changed

+36
-2
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -853,7 +853,7 @@ case class NTile(buckets: Expression) extends RowNumberLike with SizeBasedWindow
853853
// for each partition.
854854
override def checkInputDataTypes(): TypeCheckResult = {
855855
if (!buckets.foldable) {
856-
DataTypeMismatch(
856+
return DataTypeMismatch(
857857
errorSubClass = "NON_FOLDABLE_INPUT",
858858
messageParameters = Map(
859859
"inputName" -> toSQLId("buckets"),
@@ -864,7 +864,7 @@ case class NTile(buckets: Expression) extends RowNumberLike with SizeBasedWindow
864864
}
865865

866866
if (buckets.dataType != IntegerType) {
867-
DataTypeMismatch(
867+
return DataTypeMismatch(
868868
errorSubClass = "UNEXPECTED_INPUT_TYPE",
869869
messageParameters = Map(
870870
"paramIndex" -> ordinalNumber(0),

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -360,6 +360,40 @@ class AnalysisErrorSuite extends AnalysisTest with DataTypeErrorsBase {
360360
"inputType" -> "\"BOOLEAN\"",
361361
"requiredType" -> "\"INT\""))
362362

363+
errorClassTest(
364+
"the buckets of ntile window function is not foldable",
365+
testRelation2.select(
366+
WindowExpression(
367+
NTile(Literal(99.9f)),
368+
WindowSpecDefinition(
369+
UnresolvedAttribute("a") :: Nil,
370+
SortOrder(UnresolvedAttribute("b"), Ascending) :: Nil,
371+
UnspecifiedFrame)).as("window")),
372+
errorClass = "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
373+
messageParameters = Map(
374+
"sqlExpr" -> "\"ntile(99.9)\"",
375+
"paramIndex" -> "first",
376+
"inputSql" -> "\"99.9\"",
377+
"inputType" -> "\"FLOAT\"",
378+
"requiredType" -> "\"INT\""))
379+
380+
381+
errorClassTest(
382+
"the buckets of ntile window function is not int literal",
383+
testRelation2.select(
384+
WindowExpression(
385+
NTile(AttributeReference("b", IntegerType)()),
386+
WindowSpecDefinition(
387+
UnresolvedAttribute("a") :: Nil,
388+
SortOrder(UnresolvedAttribute("b"), Ascending) :: Nil,
389+
UnspecifiedFrame)).as("window")),
390+
errorClass = "DATATYPE_MISMATCH.NON_FOLDABLE_INPUT",
391+
messageParameters = Map(
392+
"sqlExpr" -> "\"ntile(b)\"",
393+
"inputName" -> "`buckets`",
394+
"inputExpr" -> "\"b\"",
395+
"inputType" -> "\"INT\""))
396+
363397
errorClassTest(
364398
"unresolved attributes",
365399
testRelation.select($"abcd"),

0 commit comments

Comments
 (0)