Skip to content

Commit e284b2c

Browse files
committed
[SPARK-50525][SQL][TESTS][FOLLOWUP] Fix DataFrameSuite.repartition by MapType test assumption
### What changes were proposed in this pull request? This is a follow-up to recover the NON-ANSI mode CI failure by adding a test assumption clearly. - #49144 ### Why are the changes needed? **BEFORE** ``` $ SPARK_ANSI_SQL_MODE=false build/sbt "sql/testOnly *.DataFrameSuite -- -z MapType" [info] *** 1 TEST FAILED *** [error] Failed tests: [error] org.apache.spark.sql.DataFrameSuite ``` **AFTER** ``` $ SPARK_ANSI_SQL_MODE=false build/sbt "sql/testOnly *.DataFrameSuite -- -z MapType" [info] All tests passed. ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually test with `SPARK_ANSI_SQL_MODE=false`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49457 from dongjoon-hyun/SPARK-50525. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 898f7af commit e284b2c

File tree

1 file changed

+17
-15
lines changed

1 file changed

+17
-15
lines changed

sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -430,21 +430,23 @@ class DataFrameSuite extends QueryTest
430430

431431
test("repartition by MapType") {
432432
Seq("int", "long", "float", "double", "decimal(10, 2)", "string", "varchar(6)").foreach { dt =>
433-
val df = spark.range(20)
434-
.withColumn("c1",
435-
when(col("id") % 3 === 1, typedLit(Map(1 -> 1)))
436-
.when(col("id") % 3 === 2, typedLit(Map(1 -> 1, 2 -> 2)))
437-
.otherwise(typedLit(Map(2 -> 2, 1 -> 1))).cast(s"map<$dt, $dt>"))
438-
.withColumn("c2", typedLit(Map(1 -> null)).cast(s"map<$dt, $dt>"))
439-
.withColumn("c3", lit(null).cast(s"map<$dt, $dt>"))
440-
441-
assertPartitionNumber(df.repartition(4, col("c1")), 2)
442-
assertPartitionNumber(df.repartition(4, col("c2")), 1)
443-
assertPartitionNumber(df.repartition(4, col("c3")), 1)
444-
assertPartitionNumber(df.repartition(4, col("c1"), col("c2")), 2)
445-
assertPartitionNumber(df.repartition(4, col("c1"), col("c3")), 2)
446-
assertPartitionNumber(df.repartition(4, col("c1"), col("c2"), col("c3")), 2)
447-
assertPartitionNumber(df.repartition(4, col("c2"), col("c3")), 2)
433+
withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
434+
val df = spark.range(20)
435+
.withColumn("c1",
436+
when(col("id") % 3 === 1, typedLit(Map(1 -> 1)))
437+
.when(col("id") % 3 === 2, typedLit(Map(1 -> 1, 2 -> 2)))
438+
.otherwise(typedLit(Map(2 -> 2, 1 -> 1))).cast(s"map<$dt, $dt>"))
439+
.withColumn("c2", typedLit(Map(1 -> null)).cast(s"map<$dt, $dt>"))
440+
.withColumn("c3", lit(null).cast(s"map<$dt, $dt>"))
441+
442+
assertPartitionNumber(df.repartition(4, col("c1")), 2)
443+
assertPartitionNumber(df.repartition(4, col("c2")), 1)
444+
assertPartitionNumber(df.repartition(4, col("c3")), 1)
445+
assertPartitionNumber(df.repartition(4, col("c1"), col("c2")), 2)
446+
assertPartitionNumber(df.repartition(4, col("c1"), col("c3")), 2)
447+
assertPartitionNumber(df.repartition(4, col("c1"), col("c2"), col("c3")), 2)
448+
assertPartitionNumber(df.repartition(4, col("c2"), col("c3")), 2)
449+
}
448450
}
449451
}
450452

0 commit comments

Comments
 (0)