[SPARK-8608][SPARK-8609][SPARK-9083][SQL] reset mutable states of nondeterministic expression before evaluation and fix PullOutNondeterministic

[cloud-fan]

We will do local projection for LocalRelation, and thus reuse the same Expression object among multiply evaluations. We should reset the mutable states of Expression before evaluate it.

Fix PullOutNondeterministic rule to make it work for Sort.

Also got a chance to cleanup the dataframe test suite.

[cloud-fan]

This case should be fixed by #7593. However, PullOutNondeterministic rule can also handle it, so we can add this test before #7593 .

[cloud-fan]

to make it more clear, now this PR is aimed to bug fix, i.e. fix nondeterministic expression handling for local projection and sort. Leave #7593 to optimization, i.e. remove still-need-evaluate expressions from Sort.

[SparkQA]

Test build #38467 has finished for PR 7674 at commit 8f57ede.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #38469 has finished for PR 7674 at commit e3814fb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

lazy val here - otherwise a failure in the ctor could wipe out the entire test suite and then test cases won't show up as a failed tests.

[rxin]

@cloud-fan I'm confused -- what did this change do? Is it the change in the analyzer that fixes the bug? If yes, can you add a unit test to the analyzer rule, in addition to the test case in the DataFrameSuite?

[cloud-fan]

hi @rxin, I want to make this PR focus on bug fix, i.e. fix nondeterministic expression handling for local projection and sort. Leave #7593 to optimization, i.e. remove still-need-evaluate expressions from Sort. Is that OK?

[SparkQA]

Test build #38490 has finished for PR 7674 at commit e0a6bd6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please.

[SparkQA]

Test build #110 has finished for PR 7674 at commit e0a6bd6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #38495 has finished for PR 7674 at commit e0a6bd6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please.

[SparkQA]

Test build #38509 has finished for PR 7674 at commit e0a6bd6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #111 has finished for PR 7674 at commit e0a6bd6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #38528 has finished for PR 7674 at commit 888934f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

I'm going to merge this. Would be great if @marmbrus or @yhuai can review this post-hoc still.