Hadoop agnostic builds

[jey]

This PR allows one Spark binary to target multiple Hadoop versions. It also moves YARN support into a separate artifact. This is the follow-up to PR #803.

CC: @mateiz, @mridulm, @tgravescs

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/609/

[AmplabJenkins]

Thank you for submitting this pull request.

Unfortunately, the automated tests for this request have failed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/613/

[jey]

Jenkins, retest this please.

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/623/

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/627/

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/658/

[mateiz]

Thanks for putting this together, Jey. I've merged it manually due to a small conflict.

[pwendell]

@mateiz @jey - It's very unfortunate that this got merged without any documentation or notification to developers. This will affect many downstream things (tests, anyone running off of master, or building things on top of master, the ec2 scripts, etc). Also, some of the existing documentation, such as docs/building-with-maven are now invalid and tell users to do the wrong thing. Could one of you send an e-mail to the dev list explaining what this means for people that consume Spark master? Also, please fix the existing docs ASAP and ideally add new docs explaining how to use this.

[rxin]

Have you tried running mvn package?

I am getting the following error:

*** RUN ABORTED ***
java.lang.NoSuchMethodError: spark.scheduler.cluster.ClusterTaskSetManager.(Lspark/scheduler/cluster/ClusterScheduler;Lspark/scheduler/TaskSet;)V
at spark.scheduler.DummyTaskSetManager.(ClusterSchedulerSuite.scala:30)
at spark.scheduler.ClusterSchedulerSuite.createDummyTaskSetManager(ClusterSchedulerSuite.scala:111)
at spark.scheduler.ClusterSchedulerSuite$$anonfun$1.apply$mcV$sp(ClusterSchedulerSuite.scala:146)
at spark.scheduler.ClusterSchedulerSuite$$anonfun$1.apply(ClusterSchedulerSuite.scala:134)
at spark.scheduler.ClusterSchedulerSuite$$anonfun$1.apply(ClusterSchedulerSuite.scala:134)
at org.scalatest.FunSuite$$anon$1.apply(FunSuite.scala:1265)
at org.scalatest.Suite$class.withFixture(Suite.scala:1974)
at spark.scheduler.ClusterSchedulerSuite.withFixture(ClusterSchedulerSuite.scala:108)
at org.scalatest.FunSuite$class.invokeWithFixture$1(FunSuite.scala:1262)
at org.scalatest.FunSuite$$anonfun$runTest$1.apply(FunSuite.scala:1271)
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM .......................... SUCCESS [2.028s]
[INFO] Spark Project Core ................................ FAILURE [7:24.910s]
[INFO] Spark Project Bagel ............................... SKIPPED
[INFO] Spark Project Streaming ........................... SKIPPED
[INFO] Spark Project ML Library .......................... SKIPPED
[INFO] Spark Project Examples ............................ SKIPPED
[INFO] Spark Project Tools ............................... SKIPPED
[INFO] Spark Project REPL ................................ SKIPPED
[INFO] Spark Project REPL binary packaging ............... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7:27.541s

[mateiz]

Did you do mvn clean and sbt clean? Sounds like an old build issue.

[mateiz]

But yes I agree with Patrick on the docs -- I shouldn't have merged this without looking at that and trying Shark as well, so we know what will break there. Sorry about that.

[jey]

@pwendell: Agreed, I'll send am email to the list ASAP and submit a patch for the docs shortly.

@rxin: As Matei said, that sounds like your classpath is contaminated with old build artifacts.

[rxin]

I tried running mvn dependency:tree after I removed .m2 and .ivy2 and sbt clean and mvn clean. Got the following error

[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM .......................... SUCCESS [0.706s]
[INFO] Spark Project Core ................................ SUCCESS [1.026s]
[INFO] Spark Project Bagel ............................... FAILURE [0.038s]
[INFO] Spark Project Streaming ........................... SKIPPED
[INFO] Spark Project ML Library .......................... SKIPPED
[INFO] Spark Project Examples ............................ SKIPPED
[INFO] Spark Project Tools ............................... SKIPPED
[INFO] Spark Project REPL ................................ SKIPPED
[INFO] Spark Project REPL binary packaging ............... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.574s
[INFO] Finished at: Tue Aug 20 19:03:59 PDT 2013
[INFO] Final Memory: 10M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-bagel: Could not resolve dependencies for project org.spark-project:spark-bagel:jar:0.8.0-SNAPSHOT: Could not find artifact org.spark-project:spark-core:jar:0.8.0-SNAPSHOT -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]

[jey]

@rxin, it's my understanding that this is "normal" for running dependency:tree on the Maven build before packaging. I think after performing mvn -DskipTests package you'll be able to run mvn dependency:tree, mvn package, mvn test, etc.

[jey]

@rxin: actually, apparently Maven in its infinite wisdom requires mvn install before mvn dependency:tree will work: http://stackoverflow.com/a/1905927

[rxin]

alright thanks @jey. that worked (although a little bit convoluted...)