SPARK-2749 [BUILD]. Spark SQL Java tests aren't compiling in Jenkins' Maven builds; missing junit:junit dep

[srowen]

The Maven-based builds in the build matrix have been failing for a few days:

https://amplab.cs.berkeley.edu/jenkins/view/Spark/

On inspection, it looks like the Spark SQL Java tests don't compile:

https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-pre-YARN/hadoop.version=1.0.4,label=centos/244/consoleFull

I confirmed it by repeating the command vs master:

mvn -Dhadoop.version=1.0.4 -Dlabel=centos -DskipTests clean package

The problem is that this module doesn't depend on JUnit. In fact, none of the modules do, but com.novocode:junit-interface (the SBT-JUnit bridge) pulls it in, in most places. However this module doesn't depend on com.novocode:junit-interface

Adding the junit:junit dependency fixes the compile problem. In fact, the other modules with Java tests should probably depend on it explicitly instead of happening to get it via com.novocode:junit-interface, since that is a bit SBT/Scala-specific (and I am not even sure it's needed).

[SparkQA]

QA tests have started for PR 1660. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17447/consoleFull

[SparkQA]

QA results for PR 1660:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17447/consoleFull

[srowen]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 1660. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17452/consoleFull

[SparkQA]

QA results for PR 1660:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17452/consoleFull

[srowen]

I'm pretty sure the test failure is spurious, as it's the same java.net.BindException: Address already in use that made other PRs today fail and then pass. It's not a test compilation problem, which should be about the only thing this could cause. I'll give Jenkins another chance.

[srowen]

Jenkins, retest this please.

[yhuai]

Do we need to include com.novocode:junit-interface in SQL? Based on https://spark-project.atlassian.net/browse/SPARK-66, seems SBT needs it to execute Java tests. But, after checking our SBT build (https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT-with-YARN/SPARK_HADOOP_VERSION=2.3.0,label=centos/234/consoleFull), seems those SQL java test were actually executed.

[marmbrus]

test this please

[SparkQA]

QA tests have started for PR 1660. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17479/consoleFull

[SparkQA]

QA results for PR 1660:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17479/consoleFull

[rxin]

Ok I'm merging this. Thanks.