Commit 593b423
committed
[SPARK-31958][SQL] normalize special floating numbers in subquery
### What changes were proposed in this pull request?
This is a followup of #23388 .
#23388 has an issue: it doesn't handle subquery expressions and assumes they will be turned into joins. However, this is not true for non-correlated subquery expressions.
This PR fixes this issue. It now doesn't skip `Subquery`, and subquery expressions will be handled by `OptimizeSubqueries`, which runs the optimizer with the subquery.
Note that, correlated subquery expressions will be handled twice: once in `OptimizeSubqueries`, once later when it becomes join. This is OK as `NormalizeFloatingNumbers` is idempotent now.
### Why are the changes needed?
fix a bug
### Does this PR introduce _any_ user-facing change?
yes, see the newly added test.
### How was this patch tested?
new test
Closes #28785 from cloud-fan/normalize.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 6fb9c80)
Signed-off-by: Wenchen Fan <[email protected]>1 parent d1a3fad commit 593b423
File tree
2 files changed
+18
-4
lines changed- sql
- catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer
- core/src/test/scala/org/apache/spark/sql
2 files changed
+18
-4
lines changedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NormalizeFloatingNumbers.scala
Lines changed: 0 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | 59 | | |
64 | 60 | | |
65 | 61 | | |
| |||
Lines changed: 18 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3449 | 3449 | | |
3450 | 3450 | | |
3451 | 3451 | | |
| 3452 | + | |
| 3453 | + | |
| 3454 | + | |
| 3455 | + | |
| 3456 | + | |
| 3457 | + | |
| 3458 | + | |
| 3459 | + | |
| 3460 | + | |
| 3461 | + | |
| 3462 | + | |
| 3463 | + | |
| 3464 | + | |
| 3465 | + | |
| 3466 | + | |
| 3467 | + | |
| 3468 | + | |
| 3469 | + | |
3452 | 3470 | | |
3453 | 3471 | | |
3454 | 3472 | | |
0 commit comments