Fix two window function bugs

[scwf]

@yhuai @guowei2 This PR fix two issues
1 HiveQl line number 796, it should be withDistinct, otherwise leads to test failure

2 Fix the case of sql in my added unit tests:

select area, sum(product), sum(sum(product)) over (partition by area) from windowData group by month, area

This sql want to aggregate product for (month, area) and (area)

select area, sum(product), sum(product) / sum(sum(product)) over (partition by area) from windowData group by month, area

this sql want to aggregate product for (month, area) and want to know the ratio over the product of all area.

[yhuai]

I think instead of having regularExpressionWithAlias, a better way is to extract all expressions from an WindowExpression and push them to the underlying Projects and Aggregates. Then, a WindowExpression will only have AttributeReference inside.

[scwf]

correct me if i misunderstanding,
1

a better way is to extract all expressions from an WindowExpression and push them to the underlying Projects and Aggregates

Do you mean push down all expressions of WindowExpression including self?
val allexpressions = windowExpressions.flatMap(_.collect {
case e: Expression => e
})

then allexpressions here including some expression we do not want to push down,such as
HiveWindowFunction
WindowSpecDefinition
CAST ...

2 i think we can not push down all expressions, see this example:
select sum(key) as s1, sum(sum(key)) over (...) from src group by value
we can not push down key to underlying aggregate because key is not in group by expressions, this will lead to analysis error. Here we just want to replace sum(sum(key)) with sum(s1)

3 IMO, we should only consider Alias in regularExpressions, user may also use them in windowexpression such as my examples in description above, so i substitute them.
And here this case actually is not a pushing down expression case, the underlying Aggregates already has the output attribute, and in windowexpression we want to use the attribute(otherwise analysis error), so do a substitution here.

[guowei2]

@scwf I think it is not a basic solution。
It is still error when select sum(key) -1 , sum(sum(key)) over (...) from src group by value

[scwf]

this is ok in hive?

[scwf]

yeah this is a case i did not considered, i am trying to fix this.

[scwf]

And @guowei2 can you merge this firstly?, let's see what jenkins say

[guowei2]

expression 'key' is neither present in the group by, nor is it an aggregate function
The key issue is that sum(key) -1 is a unit compute expression in aggregate. so it doesn't work just to substitute sum(key) to attribute

[yhuai]

I have merged it.

[scwf]

@yhuai , i have not fixed all the issues, this PR has test failure

[yhuai]

What test failures? Tests you added and we already have or tests you have not added? There is no failure in the current pr. If you have any case that has not been added in your mind, please comment on the pr. Thanks!

[yhuai]

btw, after I merged your pr, I made some changes on the analysis part.

[scwf]

YES, i have noticed that you have fixed it. it is in analyzer changes leads to window.q failure