[SPARK-38918][SQL][3.2] Nested column pruning should filter out attributes that do not belong to the current relation #36386

allisonwang-db · 2022-04-27T22:48:18Z

What changes were proposed in this pull request?

Backport #36216 to branch-3.2

Why are the changes needed?

To fix a bug in SchemaPruning.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test

viirya · 2022-05-01T19:34:56Z

@allisonwang-db Can you fix the test failure?

allisonwang-db · 2022-05-03T18:12:43Z

@viirya I tried a few times but can't reproduce this TPCDSV1_4_PlanStabilitySuite test failure locally...

viirya · 2022-05-03T18:15:26Z

@allisonwang-db Maybe you can just re-trigger the CI?

viirya · 2022-05-07T05:48:13Z

Hmm, from the logs, seems there are unmatched plans:

2022-05-04T01:30:18.1915772Z ESC[0m[ESC[0mESC[0minfoESC[0m] ESC[0mESC[0mESC[31m- check simplified (tpcds-v1.4/q5) *** FAILED *** (361 milliseconds)ESC[0mESC[0m
2022-05-04T01:30:18.1916346Z ESC[0m[ESC[0mESC[0minfoESC[0m] ESC[0mESC[0mESC[31m  Plans did not match:ESC[0mESC[0m
2022-05-04T01:30:18.1917069Z ESC[0m[ESC[0mESC[0minfoESC[0m] ESC[0mESC[0mESC[31m  last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/simplified.txtESC[0mESC[0m
2022-05-04T01:30:18.1918103Z ESC[0m[ESC[0mESC[0minfoESC[0m] ESC[0mESC[0mESC[31m  last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/explain.txtESC[0mESC[0m

viirya · 2022-05-07T06:24:58Z

@allisonwang-db I can reproduce it locally by running build/sbt "sql/testOnly *PlanStabilitySuite -- -z (tpcds-v1.4/q5)".

viirya · 2022-05-09T17:59:41Z

sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q4/explain.txt

Seems cosmetic change only in the explain string?

Yes and there is no plan change

viirya

lgtm if CI can pass.

viirya · 2022-05-09T20:53:52Z

CI still doesn't pass:

check simplified (tpcds-v1.4/q4) *** FAILED ***

allisonwang-db · 2022-05-09T22:06:32Z

Hmm let me try again

allisonwang-db · 2022-05-10T16:36:01Z

This test failure is strange. When I ran it individually (for q4 and q5), it failed but when I ran the entire suite together build/sbt "sql/testOnly *PlanStabilitySuite" the individual test case can pass.

dongjoon-hyun

Hi, @allisonwang-db . Could you rebase this PR once more, please?

allisonwang-db · 2022-05-24T02:31:44Z

@dongjoon-hyun Looks like it still has the same issue:

# Test passed
build/sbt "sql/testOnly *PlanStabilitySuite"

# Test failed for q4 and q5
build/sbt "sql/testOnly *PlanStabilitySuite -- -z (tpcds-v1.4/q5)"

dongjoon-hyun · 2022-05-24T05:31:46Z

Thank you for rechecking.

olaky · 2022-06-07T12:36:10Z

I am facing the same issues here: #36753

olaky · 2022-06-07T12:43:08Z

So the only change in the plan I can see that makes the test fail is that the last plan node has a source filename in it now, for example

Scan parquet default.web_site [web_site_sk,web_site_id] (PlanStabilitySuite.scala:156)

viirya · 2022-06-10T07:49:36Z

Can #36828 help on stablizing PlanStabilitySuite?

dongjoon-hyun · 2022-06-10T08:32:33Z

+1 for @viirya 's comment. @cloud-fan found the root cause and is fixing now on that PR.

dongjoon-hyun · 2022-06-10T08:32:53Z

We can revisit this PR after merging that PR.

viirya · 2022-06-10T16:39:42Z

@allisonwang-db #36828 is merged, can you rebase to trigger CI? Thanks.

… that do not belong to the current relation This PR updates `ProjectionOverSchema` to use the outputs of the data source relation to filter the attributes in the nested schema pruning. This is needed because the attributes in the schema do not necessarily belong to the current data source relation. For example, if a filter contains a correlated subquery, then the subquery's children can contain attributes from both the inner query and the outer query. Since the `RewriteSubquery` batch happens after early scan pushdown rules, nested schema pruning can wrongly use the inner query's attributes to prune the outer query data schema, thus causing wrong results and unexpected exceptions. To fix a bug in `SchemaPruning`. No Unit test Closes apache#36216 from allisonwang-db/spark-38918-nested-column-pruning. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]> (cherry picked from commit 150434b) Signed-off-by: Liang-Chi Hsieh <[email protected]> (cherry picked from commit 793ba60) Signed-off-by: allisonwang-db <[email protected]>

dongjoon-hyun

+1, LGTM.

viirya

Pending CI

…butes that do not belong to the current relation ### What changes were proposed in this pull request? Backport #36216 to branch-3.2 ### Why are the changes needed? To fix a bug in `SchemaPruning`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Closes #36386 from allisonwang-db/spark-38918-branch-3.2. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2022-06-10T22:50:06Z

Merged to branch-3.2. Thank you so much, @allisonwang-db , @viirya .

…butes that do not belong to the current relation ### What changes were proposed in this pull request? Backport apache#36216 to branch-3.2 ### Why are the changes needed? To fix a bug in `SchemaPruning`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Closes apache#36386 from allisonwang-db/spark-38918-branch-3.2. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

github-actions bot added the SQL label Apr 27, 2022

viirya reviewed May 9, 2022

View reviewed changes

viirya approved these changes May 9, 2022

View reviewed changes

dongjoon-hyun reviewed May 22, 2022

View reviewed changes

allisonwang-db force-pushed the spark-38918-branch-3.2 branch from 15c9ccc to 9652212 Compare May 23, 2022 18:04

olaky mentioned this pull request Jun 7, 2022

[SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries #36753

Closed

allisonwang-db added 3 commits June 10, 2022 13:27

update explain.txt

ff83285

update golden files

83dba72

allisonwang-db force-pushed the spark-38918-branch-3.2 branch from 9652212 to 83dba72 Compare June 10, 2022 20:53

allisonwang-db mentioned this pull request Jun 10, 2022

[3.2][SPARK-39437][SQL][TEST] Normalize plan id separately in PlanStabilitySuite #36828

Closed

dongjoon-hyun approved these changes Jun 10, 2022

View reviewed changes

viirya approved these changes Jun 10, 2022

View reviewed changes

dongjoon-hyun closed this Jun 10, 2022

[SPARK-38918][SQL][3.2] Nested column pruning should filter out attributes that do not belong to the current relation #36386

[SPARK-38918][SQL][3.2] Nested column pruning should filter out attributes that do not belong to the current relation #36386

Uh oh!

Conversation

allisonwang-db commented Apr 27, 2022 • edited by viirya Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

viirya commented May 1, 2022

Uh oh!

allisonwang-db commented May 3, 2022

Uh oh!

viirya commented May 3, 2022

Uh oh!

viirya commented May 7, 2022

Uh oh!

viirya commented May 7, 2022

Uh oh!

viirya May 9, 2022

Choose a reason for hiding this comment

Uh oh!

allisonwang-db May 9, 2022

Choose a reason for hiding this comment

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

viirya commented May 9, 2022

Uh oh!

allisonwang-db commented May 9, 2022

Uh oh!

allisonwang-db commented May 10, 2022

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

allisonwang-db commented May 24, 2022

Uh oh!

dongjoon-hyun commented May 24, 2022

Uh oh!

olaky commented Jun 7, 2022

Uh oh!

olaky commented Jun 7, 2022

Uh oh!

viirya commented Jun 10, 2022

Uh oh!

dongjoon-hyun commented Jun 10, 2022

Uh oh!

dongjoon-hyun commented Jun 10, 2022

Uh oh!

viirya commented Jun 10, 2022

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

allisonwang-db commented Apr 27, 2022 •

edited by viirya

Loading

dongjoon-hyun commented Jun 10, 2022 •

edited

Loading