Skip to content

Conversation

@TongWei1105
Copy link
Contributor

What changes were proposed in this pull request?

This PR enhances ConstantPropagation to support more cases.

Propagated through other binary comparisons.
Propagated across equality comparisons. This can be further optimized to false.

Why are the changes needed?

Improve query performance. Denodo also has a similar optimization. For example:

CREATE TABLE t1(a int, b int) using parquet;
CREATE TABLE t2(x int, y int) using parquet;

CREATE TEMP VIEW v1 AS                                        
SELECT * FROM t1 JOIN t2 WHERE a = x AND a = 0                
UNION ALL                                                     
SELECT * FROM t1 JOIN t2 WHERE a = x AND (a IS NULL OR a <> 0);

SELECT * FROM v1 WHERE x > 1;

Before this PR:

== Optimized Logical Plan ==
Union false, false
:- Project [a#0 AS a#12, b#1 AS b#13, x#2 AS x#14, y#3 AS y#15]
:  +- Join Inner
:     :- Filter (isnotnull(a#0) AND (a#0 = 0))
:     :  +- Relation spark_catalog.default.t1[a#0,b#1] parquet
:     +- Filter (isnotnull(x#2) AND ((0 = x#2) AND (x#2 > 1)))
:        +- Relation spark_catalog.default.t2[x#2,y#3] parquet
+- Join Inner, (a#16 = x#18)
   :- Filter ((isnull(a#16) OR NOT (a#16 = 0)) AND ((a#16 > 1) AND isnotnull(a#16)))
   :  +- Relation spark_catalog.default.t1[a#16,b#17] parquet
   +- Filter ((isnotnull(x#18) AND (x#18 > 1)) AND (isnull(x#18) OR NOT (x#18 = 0)))
      +- Relation spark_catalog.default.t2[x#18,y#19] parquet

After this PR:

== Optimized Logical Plan ==
Join Inner, (a#16 = x#18)
:- Filter ((isnull(a#16) OR NOT (a#16 = 0)) AND ((a#16 > 1) AND isnotnull(a#16)))
:  +- Relation spark_catalog.default.t1[a#16,b#17] parquet
+- Filter ((isnotnull(x#18) AND (x#18 > 1)) AND (isnull(x#18) OR NOT (x#18 = 0)))
   +- Relation spark_catalog.default.t2[x#18,y#19] parquet

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

@github-actions github-actions bot added the SQL label Jul 17, 2023
@TongWei1105 TongWei1105 force-pushed the SPARK-42500 branch 3 times, most recently from 5712bb5 to ce86f25 Compare July 17, 2023 08:59
@TongWei1105 TongWei1105 changed the title SPARK-42500: ConstantPropagation support more case [SPARK-42500][SQL] ConstantPropagation support more case Jul 17, 2023
@wangyum
Copy link
Member

wangyum commented Jul 17, 2023

cc @cloud-fan @peter-toth

@peter-toth
Copy link
Contributor

I think this PR is very similar to my #40268 so I'm fine with this this change.
But my PR uses a mutable map to avoid the might be costly val equalityPredicates = equalityPredicatesLeft ++ equalityPredicatesRight map addition so please consider that PR too.

@TongWei1105 TongWei1105 force-pushed the SPARK-42500 branch 3 times, most recently from 20d286f to 4d1e3a9 Compare July 28, 2023 05:46
@wangyum
Copy link
Member

wangyum commented Aug 6, 2023

@peter-toth This map should usually be small. This change is small and easy to maintain.

@wangyum wangyum closed this in 74ae1e3 Aug 6, 2023
@wangyum
Copy link
Member

wangyum commented Aug 6, 2023

Merged to master.

turboFei pushed a commit to turboFei/spark that referenced this pull request Nov 6, 2025
…apache#256)

### What changes were proposed in this pull request?

This PR enhances ConstantPropagation to support more cases.

Propagated through other binary comparisons.
Propagated across equality comparisons. This can be further optimized to false.

### Why are the changes needed?

Improve query performance. [Denodo](https://community.denodo.com/docs/html/browse/latest/en/vdp/administration/optimizing_queries/automatic_simplification_of_queries/removing_redundant_branches_of_queries_partitioned_unions) also has a similar optimization. For example:
```
CREATE TABLE t1(a int, b int) using parquet;
CREATE TABLE t2(x int, y int) using parquet;

CREATE TEMP VIEW v1 AS
SELECT * FROM t1 JOIN t2 WHERE a = x AND a = 0
UNION ALL
SELECT * FROM t1 JOIN t2 WHERE a = x AND (a IS NULL OR a <> 0);

SELECT * FROM v1 WHERE x > 1;
```
Before this PR:
```
== Optimized Logical Plan ==
Union false, false
:- Project [a#0 AS a#12, b#1 AS b#13, x#2 AS x#14, y#3 AS y#15]
:  +- Join Inner
:     :- Filter (isnotnull(a#0) AND (a#0 = 0))
:     :  +- Relation spark_catalog.default.t1[a#0,b#1] parquet
:     +- Filter (isnotnull(x#2) AND ((0 = x#2) AND (x#2 > 1)))
:        +- Relation spark_catalog.default.t2[x#2,y#3] parquet
+- Join Inner, (a#16 = x#18)
   :- Filter ((isnull(a#16) OR NOT (a#16 = 0)) AND ((a#16 > 1) AND isnotnull(a#16)))
   :  +- Relation spark_catalog.default.t1[a#16,b#17] parquet
   +- Filter ((isnotnull(x#18) AND (x#18 > 1)) AND (isnull(x#18) OR NOT (x#18 = 0)))
      +- Relation spark_catalog.default.t2[x#18,y#19] parquet
```
After this PR:
```
== Optimized Logical Plan ==
Join Inner, (a#16 = x#18)
:- Filter ((isnull(a#16) OR NOT (a#16 = 0)) AND ((a#16 > 1) AND isnotnull(a#16)))
:  +- Relation spark_catalog.default.t1[a#16,b#17] parquet
+- Filter ((isnotnull(x#18) AND (x#18 > 1)) AND (isnull(x#18) OR NOT (x#18 = 0)))
   +- Relation spark_catalog.default.t2[x#18,y#19] parquet
```
### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Unit test.

Closes apache#42038 from TongWei1105/SPARK-42500.

Authored-by: TongWei1105 <[email protected]>
Signed-off-by: Yuming Wang <[email protected]>
(cherry picked from commit 74ae1e3)

Co-authored-by: TongWei1105 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants