Commit 6fb9c80
committed
[SPARK-31958][SQL] normalize special floating numbers in subquery
### What changes were proposed in this pull request?
This is a followup of #23388 .
#23388 has an issue: it doesn't handle subquery expressions and assumes they will be turned into joins. However, this is not true for non-correlated subquery expressions.
This PR fixes this issue. It now doesn't skip `Subquery`, and subquery expressions will be handled by `OptimizeSubqueries`, which runs the optimizer with the subquery.
Note that, correlated subquery expressions will be handled twice: once in `OptimizeSubqueries`, once later when it becomes join. This is OK as `NormalizeFloatingNumbers` is idempotent now.
### Why are the changes needed?
fix a bug
### Does this PR introduce _any_ user-facing change?
yes, see the newly added test.
### How was this patch tested?
new test
Closes #28785 from cloud-fan/normalize.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>1 parent 56d4f27 commit 6fb9c80
File tree
2 files changed
+18
-4
lines changed- sql
- catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer
- core/src/test/scala/org/apache/spark/sql
2 files changed
+18
-4
lines changedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NormalizeFloatingNumbers.scala
Lines changed: 0 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | 59 | | |
64 | 60 | | |
65 | 61 | | |
| |||
Lines changed: 18 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3503 | 3503 | | |
3504 | 3504 | | |
3505 | 3505 | | |
| 3506 | + | |
| 3507 | + | |
| 3508 | + | |
| 3509 | + | |
| 3510 | + | |
| 3511 | + | |
| 3512 | + | |
| 3513 | + | |
| 3514 | + | |
| 3515 | + | |
| 3516 | + | |
| 3517 | + | |
| 3518 | + | |
| 3519 | + | |
| 3520 | + | |
| 3521 | + | |
| 3522 | + | |
| 3523 | + | |
3506 | 3524 | | |
3507 | 3525 | | |
3508 | 3526 | | |
0 commit comments