Commit c8c6be2
[SPARK-48922][SQL][3.5] Avoid redundant array transform of identical expression for map type
### What changes were proposed in this pull request?
Backports #50245 to 3.5
Similar to #47843, this patch avoids ArrayTransform in `resolveMapType` function if the resolution expression is the same as input param.
### Why are the changes needed?
My previous pr #47381 was not merged, but I still think it is an optimization, so I reopened it.
During the upgrade from Spark 3.1.1 to 3.5.0, I found a performance regression in map type inserts.
There are some extra conversion expressions in project before insert, which doesn't seem to be always necessary.
```
map_from_arrays(transform(map_keys(map#516), lambdafunction(lambda key#652, lambda key#652, false)), transform(map_values(map#516), lambdafunction(lambda value#654, lambda value#654, false))) AS map#656
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
added unit test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #50245 from wForget/SPARK-48922.
Authored-by: wforget <643348094qq.com>
Signed-off-by: beliefer <beliefer163.com>
(cherry picked from commit 1be108e)
Closes #50265 from wForget/SPARK-48922-3.5.
Authored-by: wforget <[email protected]>
Signed-off-by: beliefer <[email protected]>1 parent 4231d58 commit c8c6be2
File tree
2 files changed
+53
-5
lines changed- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/analysis
- test/scala/org/apache/spark/sql/catalyst/analysis
2 files changed
+53
-5
lines changedLines changed: 22 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
418 | 418 | | |
419 | 419 | | |
420 | 420 | | |
421 | | - | |
422 | | - | |
423 | | - | |
424 | | - | |
425 | | - | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
426 | 443 | | |
427 | 444 | | |
428 | 445 | | |
| |||
Lines changed: 31 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
334 | 334 | | |
335 | 335 | | |
336 | 336 | | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
337 | 368 | | |
338 | 369 | | |
339 | 370 | | |
| |||
0 commit comments