Skip to content

Commit 8a28e91

Browse files
committed
[SPARK-40382][SQL][TESTS][FOLLOW-UP] Make the tests pass with ANSI enabled
### What changes were proposed in this pull request? This PR is a followup of #37825, that change the types in the test relation to make the tests pass with ANSI enalbed. ### Why are the changes needed? To recover the test coverage. Currently it fails with ANSI mode on: https://github.com/apache/spark/actions/runs/3246829492/jobs/5326051798#step:9:20487 ``` [info] - SPARK-40382: eliminate multiple distinct groups due to superficial differences *** FAILED *** (5 milliseconds) [info] org.apache.spark.sql.AnalysisException: [DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE] Cannot resolve "(b + c)" due to data type mismatch: the binary operator requires the input type ("NUMERIC" or "INTERVAL DAY TO SECOND" or "INTERVAL YEAR TO MONTH" or "INTERVAL"), not "STRING". [info] at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.dataTypeMismatch(package.scala:68) [info] at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$5(CheckAnalysis.scala:223) [info] at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$5$adapted(CheckAnalysis.scala:210) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:295) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:294) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:294) [info] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) [info] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) [info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:294) [info] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:2 ``` ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? Manually ran the tests locally. Closes #38250 from HyukjinKwon/SPARK-40382-followup. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 9d28909 commit 8a28e91

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregatesSuite.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ class RewriteDistinctAggregatesSuite extends PlanTest {
2828
val nullInt = Literal(null, IntegerType)
2929
val nullString = Literal(null, StringType)
3030
val testRelation = LocalRelation($"a".string, $"b".string, $"c".string, $"d".string, $"e".int)
31+
val testRelation2 = LocalRelation($"a".double, $"b".int, $"c".int, $"d".int, $"e".int)
3132

3233
private def checkRewrite(rewrite: LogicalPlan): Unit = rewrite match {
3334
case Aggregate(_, _, Aggregate(_, _, _: Expand)) =>
@@ -77,7 +78,7 @@ class RewriteDistinctAggregatesSuite extends PlanTest {
7778
}
7879

7980
test("SPARK-40382: eliminate multiple distinct groups due to superficial differences") {
80-
val input = testRelation
81+
val input = testRelation2
8182
.groupBy($"a")(
8283
countDistinct($"b" + $"c").as("agg1"),
8384
countDistinct($"c" + $"b").as("agg2"),
@@ -92,7 +93,7 @@ class RewriteDistinctAggregatesSuite extends PlanTest {
9293
}
9394

9495
test("SPARK-40382: reduce multiple distinct groups due to superficial differences") {
95-
val input = testRelation
96+
val input = testRelation2
9697
.groupBy($"a")(
9798
countDistinct($"b" + $"c" + $"d").as("agg1"),
9899
countDistinct($"d" + $"c" + $"b").as("agg2"),

0 commit comments

Comments
 (0)