[SPARK-53155][SQL] Global lower agggregation should not be replaced with a project #51884
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
There was a problem hiding this comment.
According to the JIRA issue, is this bug starting at Spark 3.4? Or, is it a long-standing bug which we have before?
Member
Author
There was a problem hiding this comment.
Yea, the issue seems there since RemoveRedundantAggregates was introduced.
Member
|
Thank you, @viirya . cc @peter-toth , too. |
viirya
force-pushed
the
fix_remove_redundant_agg
branch
from
9c626df to
808a35c
Compare
peter-toth
reviewed
Aug 7, 2025
| .groupBy()(Literal(1).as("col1"), Literal(2).as("col2"), Literal(3).as("col3")) | ||
| .groupBy($"col1")(max($"col1")) | ||
| .analyze | ||
| val expected = relation |
Member
Author
There was a problem hiding this comment.
ah, as seems it doesn't look like necessary so I didn't change this to save another CI round. Thank you.
peter-toth
approved these changes
Aug 7, 2025
viirya
added a commit
that referenced
this pull request
Aug 7, 2025
…ith a project This patch fixes the optimization rule `RemoveRedundantAggregates`. The optimizer rule `RemoveRedundantAggregates` removes redundant lower aggregation from a query plan and replace it with a project of referred non-aggregate expressions. However, if the removed aggregation is a global one, that is not correct because a project is different with a global aggregation in semantics. For example, if the input relation is empty, a project might be optimized to an empty relation, while a global aggregation will return a single row. Yes, this fixes a user-facing bug. Previously, a global aggregation under another aggregation might be treated as redundant and replaced as a project with non-aggregation expressions. If the input relation is empty, the replacement is incorrect and might produce incorrect result. This patch adds a new unit test to show the difference. Unit test, manual test. No Closes #51884 from viirya/fix_remove_redundant_agg. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]> (cherry picked from commit 3aa8c9d) Signed-off-by: Liang-Chi Hsieh <[email protected]>
viirya
added a commit
that referenced
this pull request
Aug 7, 2025
…ith a project This patch fixes the optimization rule `RemoveRedundantAggregates`. The optimizer rule `RemoveRedundantAggregates` removes redundant lower aggregation from a query plan and replace it with a project of referred non-aggregate expressions. However, if the removed aggregation is a global one, that is not correct because a project is different with a global aggregation in semantics. For example, if the input relation is empty, a project might be optimized to an empty relation, while a global aggregation will return a single row. Yes, this fixes a user-facing bug. Previously, a global aggregation under another aggregation might be treated as redundant and replaced as a project with non-aggregation expressions. If the input relation is empty, the replacement is incorrect and might produce incorrect result. This patch adds a new unit test to show the difference. Unit test, manual test. No Closes #51884 from viirya/fix_remove_redundant_agg. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]> (cherry picked from commit 3aa8c9d) Signed-off-by: Liang-Chi Hsieh <[email protected]>
Member
Author
|
Thanks @dongjoon-hyun @peter-toth Merged to master/4.0/3.5. |
Member
Author
|
The CI of branch-4.0 failed on https://github.com/apache/spark/actions/runs/16808660184/job/47607970234 This patch doesn't touch any AQE or shuffle stuffs. I also verified it locally and the test passed without issue. Seems just a flaky test. For branch-3.5, all tests were passed but just documentation build failed: https://github.com/apache/spark/actions/runs/16808722611/job/47608228583 I think it is not related too. |
zifeif2
pushed a commit
to zifeif2/spark
that referenced
this pull request
Nov 14, 2025
…ith a project This patch fixes the optimization rule `RemoveRedundantAggregates`. The optimizer rule `RemoveRedundantAggregates` removes redundant lower aggregation from a query plan and replace it with a project of referred non-aggregate expressions. However, if the removed aggregation is a global one, that is not correct because a project is different with a global aggregation in semantics. For example, if the input relation is empty, a project might be optimized to an empty relation, while a global aggregation will return a single row. Yes, this fixes a user-facing bug. Previously, a global aggregation under another aggregation might be treated as redundant and replaced as a project with non-aggregation expressions. If the input relation is empty, the replacement is incorrect and might produce incorrect result. This patch adds a new unit test to show the difference. Unit test, manual test. No Closes apache#51884 from viirya/fix_remove_redundant_agg. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]> (cherry picked from commit 776b68f) Signed-off-by: Liang-Chi Hsieh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This patch fixes the optimization rule
RemoveRedundantAggregates.Why are the changes needed?
The optimizer rule
RemoveRedundantAggregatesremoves redundant lower aggregation from a query plan and replace it with a project of referred non-aggregate expressions. However, if the removed aggregation is a global one, that is not correct because a project is different with a global aggregation in semantics.For example, if the input relation is empty, a project might be optimized to an empty relation, while a global aggregation will return a single row.
Does this PR introduce any user-facing change?
Yes, this fixes a user-facing bug. Previously, a global aggregation under another aggregation might be treated as redundant and replaced as a project with non-aggregation expressions. If the input relation is empty, the replacement is incorrect and might produce incorrect result. This patch adds a new unit test to show the difference.
How was this patch tested?
Unit test, manual test.
Was this patch authored or co-authored using generative AI tooling?
No