[SQL]fix spark sql hive tests time out issue

[scwf]

When set TestHive.cacheTables = true then
correlationoptimizer14 will be pending which lead to time out, maybe there are some issue bebind this. This PR is a quick fix for the hive test issue.

[scwf]

@marmbrus, can you test this

[AmplabJenkins]

Can one of the admins verify this patch?

[marmbrus]

ok to test

[SparkQA]

QA tests have started for PR 2566 at commit f489758.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2566 at commit f489758.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20948/

[scwf]

unset TestHive.cacheTables = true made mapjoin_filter_on_outerjoin failed, so just ignore correlationoptimizer14 now to avoid time out.

[SparkQA]

QA tests have started for PR 2566 at commit 86f4d3b.

• This patch merges cleanly.

[marmbrus]

It my tests it seems like it hangs on different test cases each run. We should probably just roll back #1689 if that is the culprit. I'm trying this in #2565 to verify that that fixes things. Thanks for looking into this :)

[SparkQA]

QA tests have finished for PR 2566 at commit 86f4d3b.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class IndexedRecordToJavaConverter extends Converter[IndexedRecord, JMap[String, Any]]
  • class AvroWrapperToJavaConverter extends Converter[Any, Any]

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20950/

[SparkQA]

QA tests have started for PR 2566 at commit 86f4d3b.

• This patch merges cleanly.

[SparkQA]

QA tests have started for PR 2566 at commit 86f4d3b.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2566 at commit 86f4d3b.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have finished for PR 2566 at commit 86f4d3b.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 2566 at commit 86f4d3b.

• This patch merges cleanly.

[SparkQA]

QA tests have started for PR 2566 at commit 86f4d3b.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2566 at commit 86f4d3b.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have finished for PR 2566 at commit 86f4d3b.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[scwf]

@marmbrus, i tested the test case that hang on by bin/spark-sql, the result is it is ok when run in bin/spark-sql, but hang on when runing unit test by SBT.

[scwf]

sum up the things i see
1 when set TestHive.cacheTables = true in HiveCompatibilitySuite + merge #1689, correlationoptimizer14 hangs on
2 unset TestHive.cacheTables , there is no time out issue but mapjoin_filter_on_outerjoin failed
3 time out test case correlationoptimizer14 is ok in bin/spark-sql without anything changed

[marmbrus]

Thanks again for looking into this! Tracking down the PR that broke things was really helpful!

Can we close this issue now that the other PR has been reverted?

[scwf]

ok . There may be a potential bug in sql(cache table in-memory) to lead this, we can diagnose it by merging #1689.

[marmbrus]

Can you elaborate on what you think the bug is? Perhaps discuss with @erikerlandson.

[scwf]

I did not find the root cause, i debug this problem and it hang on in https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L298
it seems r.dependencies is in Infinite loop when in cache table case