[SPARK-33122][SQL][FOLLOWUP] Extend RemoveRedundantAggregates optimizer rule to apply to more cases

[tanelk]

What changes were proposed in this pull request?

Addressed the @dongjoon-hyun comments on the previous PR #30018.
Extended the RemoveRedundantAggregates rule to remove redundant aggregations in even more queries. For example in

dataset
   .dropDuplicates()
   .groupBy('a)
   .agg(max('b))

the dropDuplicates is not needed, because the result on max does not depend on duplicate values.

Why are the changes needed?

Improve performance.

Does this PR introduce any user-facing change?

No

How was this patch tested?

UT

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40890/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40890/

[SparkQA]

Test build #136308 has finished for PR 31914 at commit 19748dc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @tanelk .
Since the commit title is precious resource, please don't repeat the original JIRA title. The JIRA ID is enough for that purpose. It would be great if you give a more meaningful and specific PR title.

[tanelk]

@maropu , this is a followup to a PR you reviewed a while back, but it has gone unnoticed.

[tanelk]

This is only behaviour change

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43395/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43395/

[SparkQA]

Test build #138873 has finished for PR 31914 at commit 3fe6985.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

I've checked the GA tests passed, so I will merge this. Thank you, @tanelk

[maropu]

Merged to master.