[SPARK-15498][TESTS] fix slow tests #13273
Conversation
| val trimmed = line.trim | ||
| formatter.addLine(code.comment.getOrElse(trimmed, trimmed)) | ||
| } | ||
| formatter.result() |
There was a problem hiding this comment.
cc @sarutak , here I assume the placeholder will always take an entire line, is it corrected?
There was a problem hiding this comment.
How slow is it if we use an regexp to match the placeholder here?
|
Test build #59178 has finished for PR 13273 at commit
|
|
retest this please |
|
Test build #59179 has finished for PR 13273 at commit
|
|
test this please |
|
Test build #59205 has finished for PR 13273 at commit
|
|
Thank you for fixing it! |
|
LGTM |
| val line = l.trim() | ||
| val skip = lastLine.startsWith("/*") && lastLine.endsWith("*/") && | ||
| line.startsWith("/*") && line.endsWith("*/") && | ||
| map(lastLine).substring(3).contains(map(line).substring(3)) |
There was a problem hiding this comment.
The previous one doesn't handle the case that the line is not in the comments map.
|
Test build #59210 has finished for PR 13273 at commit
|
|
Test build #59219 has finished for PR 13273 at commit
|
|
retest this please |
|
Test build #59220 has finished for PR 13273 at commit
|
| m => code.comment.getOrElse(m.group(1), m.group(0))) | ||
| formatter.addLine(comment) | ||
| } | ||
| formatter.result() |
|
Test build #59230 has finished for PR 13273 at commit
|
|
retest this please |
|
Test build #59231 has finished for PR 13273 at commit
|
|
Test build #59242 has finished for PR 13273 at commit
|
|
Merging to master and branch-2.0. |
## What changes were proposed in this pull request? This PR fixes 3 slow tests: 1. `ParquetQuerySuite.read/write wide table`: This is not a good unit test as it runs more than 5 minutes. This PR removes it and add a new regression test in `CodeGenerationSuite`, which is more "unit". 2. `ParquetQuerySuite.returning batch for wide table`: reduce the threshold and use smaller data size. 3. `DatasetSuite.SPARK-14554: Dataset.map may generate wrong java code for wide table`: Improve `CodeFormatter.format`(introduced at #12979) can dramatically speed this it up. ## How was this patch tested? N/A Author: Wenchen Fan <[email protected]> Closes #13273 from cloud-fan/test. (cherry picked from commit 50b660d) Signed-off-by: Cheng Lian <[email protected]>
7 participants
What changes were proposed in this pull request?
This PR fixes 3 slow tests:
ParquetQuerySuite.read/write wide table: This is not a good unit test as it runs more than 5 minutes. This PR removes it and add a new regression test inCodeGenerationSuite, which is more "unit".ParquetQuerySuite.returning batch for wide table: reduce the threshold and use smaller data size.DatasetSuite.SPARK-14554: Dataset.map may generate wrong java code for wide table: ImproveCodeFormatter.format(introduced at [SPARK-15165][SPARK-15205][SQL] Introduce place holder for comments in generated code #12979) can dramatically speed this it up.How was this patch tested?
N/A