Skip to content

Commit 940ac0c

Browse files
LuciferYangMaxGekk
authored andcommitted
[SPARK-38198][SQL][3.2] Fix QueryExecution.debug#toFile use the passed in maxFields when explainMode is CodegenMode
### What changes were proposed in this pull request? `QueryExecution.debug#toFile` method supports passing in `maxFields` and this parameter will be passed down when `explainMode` is `SimpleMode`, `ExtendedMode`, or `CostMode`. But the passed down `maxFields` was ignored when `explainMode` is `CostMode` because `QueryExecution#stringWithStats` overrides it with `SQLConf.get.maxToStringFields` at present, so this pr removes the override behavior to let passed in `maxFields` take effect. ### Why are the changes needed? Bug fix ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA and add a new test case Closes #35515 from LuciferYang/SPARK-38198-32. Authored-by: yangjie01 <[email protected]> Signed-off-by: Max Gekk <[email protected]>
1 parent 75c7726 commit 940ac0c

File tree

2 files changed

+18
-2
lines changed

2 files changed

+18
-2
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -304,8 +304,6 @@ class QueryExecution(
304304
}
305305

306306
private def stringWithStats(maxFields: Int, append: String => Unit): Unit = {
307-
val maxFields = SQLConf.get.maxToStringFields
308-
309307
// trigger to compute stats for logical plans
310308
try {
311309
// This will trigger to compute stats for all the nodes in the plan, including subqueries,

sql/core/src/test/scala/org/apache/spark/sql/execution/QueryExecutionSuite.scala

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -264,4 +264,22 @@ class QueryExecutionSuite extends SharedSparkSession {
264264
assert(cmdResultExec.commandPhysicalPlan.asInstanceOf[ExecutedCommandExec]
265265
.cmd.isInstanceOf[ShowTablesCommand])
266266
}
267+
268+
test("SPARK-38198: check specify maxFields when call toFile method") {
269+
withTempDir { dir =>
270+
val path = dir.getCanonicalPath + "/plans.txt"
271+
// Define a dataset with 6 columns
272+
val ds = spark.createDataset(Seq((0, 1, 2, 3, 4, 5), (6, 7, 8, 9, 10, 11)))
273+
// `CodegenMode` and `FormattedMode` doesn't use the maxFields, so not tested in this case
274+
Seq(SimpleMode.name, ExtendedMode.name, CostMode.name).foreach { modeName =>
275+
val maxFields = 3
276+
ds.queryExecution.debug.toFile(path, explainMode = Some(modeName), maxFields = maxFields)
277+
Utils.tryWithResource(Source.fromFile(path)) { source =>
278+
val tableScan = source.getLines().filter(_.contains("LocalTableScan"))
279+
assert(tableScan.exists(_.contains("more fields")),
280+
s"Specify maxFields = $maxFields doesn't take effect when explainMode is $modeName")
281+
}
282+
}
283+
}
284+
}
267285
}

0 commit comments

Comments
 (0)