Skip to content

Commit 567e65e

Browse files
belieferHyukjinKwon
authored andcommitted
[SPARK-31372][SQL][TEST][FOLLOW-UP] Improve ExpressionsSchemaSuite so that easy to track the diff
### What changes were proposed in this pull request? This PR follows up #28194. As discussed at https://github.com/apache/spark/pull/28194/files#r418418796. This PR will improve `ExpressionsSchemaSuite` so that easy to track the diff. Although `ExpressionsSchemaSuite` at line https://github.com/apache/spark/blob/b7cde42b04b21c9bfee6535199cf385855c15853/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala#L165 just want to compare the total size between expected output size and the newest output size, the scalatest framework will output the extra information contains all the content of expected output and newest output. This PR will try to avoid this issue. After this PR, the exception looks like below: ``` [info] - Check schemas for expression examples *** FAILED *** (7 seconds, 336 milliseconds) [info] 340 did not equal 341 Expected 332 blocks in result file but got 333. Try regenerate the result files. (ExpressionsSchemaSuite.scala:167) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529) [info] at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560) [info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:503) [info] at org.apache.spark.sql.ExpressionsSchemaSuite.$anonfun$new$1(ExpressionsSchemaSuite.scala:167) ``` ### Why are the changes needed? Make the exception more concise and clear. ### Does this PR introduce _any_ user-facing change? 'No'. ### How was this patch tested? Jenkins test. Closes #28430 from beliefer/improve-expressions-schema-suite. Authored-by: beliefer <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit b949420) Signed-off-by: HyukjinKwon <[email protected]>
1 parent c4b292e commit 567e65e

File tree

1 file changed

+24
-22
lines changed

1 file changed

+24
-22
lines changed

sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -141,38 +141,39 @@ class ExpressionsSchemaSuite extends QueryTest with SharedSparkSession {
141141
}
142142
}
143143

144+
val header = Seq(
145+
s"<!-- Automatically generated by${getClass.getSimpleName} -->",
146+
"## Summary",
147+
s" - Number of queries: ${outputs.size}",
148+
s" - Number of expressions that missing example: ${missingExamples.size}",
149+
s" - Expressions missing examples: ${missingExamples.mkString(",")}",
150+
"## Schema of Built-in Functions",
151+
"| Class name | Function name or alias | Query example | Output schema |",
152+
"| ---------- | ---------------------- | ------------- | ------------- |"
153+
)
154+
144155
if (regenerateGoldenFiles) {
145-
val missingExampleStr = missingExamples.mkString(",")
146-
val goldenOutput = {
147-
s"<!-- Automatically generated by${getClass.getSimpleName} -->\n" +
148-
"## Summary\n" +
149-
s" - Number of queries: ${outputs.size}\n" +
150-
s" - Number of expressions that missing example: ${missingExamples.size}\n" +
151-
s" - Expressions missing examples: $missingExampleStr\n" +
152-
"## Schema of Built-in Functions\n" +
153-
"| Class name | Function name or alias | Query example | Output schema |\n" +
154-
"| ---------- | ---------------------- | ------------- | ------------- |\n" +
155-
outputBuffer.mkString("\n")
156-
}
156+
val goldenOutput = (header ++ outputBuffer).mkString("\n")
157157
val parent = resultFile.getParentFile
158158
if (!parent.exists()) {
159159
assert(parent.mkdirs(), "Could not create directory: " + parent)
160160
}
161161
stringToFile(resultFile, goldenOutput)
162162
}
163163

164+
val outputSize = outputs.size
165+
val headerSize = header.size
164166
val expectedOutputs: Seq[QueryOutput] = {
165-
val goldenOutput = fileToString(resultFile)
166-
val lines = goldenOutput.split("\n")
167+
val expectedGoldenOutput = fileToString(resultFile)
168+
val lines = expectedGoldenOutput.split("\n")
169+
val expectedSize = lines.size
167170

168-
// The header of golden file has one line, plus four lines of the summary and three
169-
// lines of the header of schema table.
170-
assert(lines.size == outputs.size + 8,
171-
s"Expected ${outputs.size + 8} blocks in result file but got ${lines.size}. " +
172-
s"Try regenerate the result files.")
171+
assert(expectedSize == outputSize + headerSize,
172+
s"Expected $expectedSize blocks in result file but got " +
173+
s"${outputSize + headerSize}. Try regenerate the result files.")
173174

174-
Seq.tabulate(outputs.size) { i =>
175-
val segments = lines(i + 8).split('|')
175+
Seq.tabulate(outputSize) { i =>
176+
val segments = lines(i + headerSize).split('|')
176177
QueryOutput(
177178
className = segments(1).trim,
178179
funcName = segments(2).trim,
@@ -182,7 +183,8 @@ class ExpressionsSchemaSuite extends QueryTest with SharedSparkSession {
182183
}
183184

184185
// Compare results.
185-
assert(expectedOutputs.size == outputs.size, s"Number of queries not equals")
186+
assert(expectedOutputs.size == outputSize,
187+
"The number of queries not equals the number of expected queries.")
186188

187189
outputs.zip(expectedOutputs).foreach { case (output, expected) =>
188190
assert(expected.sql == output.sql, "SQL query did not match")

0 commit comments

Comments
 (0)