[SPARK-7042] [BUILD] use the standard akka artifacts with hadoop-2.x

[kostya-sh]

Both akka 2.3.x and hadoop-2.x use protobuf 2.5 so only hadoop-1 build needs
custom 2.3.4-spark akka version that shades protobuf-2.5

This change also updates akka version (for hadoop-2.x profiles only) to the
latest 2.3.11 as akka-zeromq_2.11 is not available for akka 2.3.4.

This partially fixes SPARK-7042 (for hadoop-2.x builds)

[srowen]

OK to test

[srowen]

Jenkins, add to whitelist

[SparkQA]

Test build #33728 has finished for PR 6492 at commit dc195b0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I'm inclined to pull this in if in fact it works with Scala 2.11 this time around, as reported on the JIRA. 2.3.11 is claimed to be a drop in replacement for earlier 2.3.x: http://akka.io/news/2015/05/12/akka-2.3.11-released.html Tests pass. OK to try out for master? @pwendell

[pwendell]

Yep LGTM

[srowen]

@kostya-sh I merged this, but it didn't auto-close due to a sync problem. You can close this PR.

[kostya-sh]

@srowen, thank you. Closing the pull request now.

[nchammas]

Heads up: This PR appears to have broken Spark when built against Hadoop 1. I've been having trouble coming up with a simple repro, but spark-perf tests succeed on preceding commits and fail starting precisely on this one.

For the spark-perf tests, Spark is being built as follows:

./make-distribution.sh --skip-java-test -Dhadoop.version=1.2.1 -Phadoop-1

[kostya-sh]

@nchammas, what errors are you getting?

[kostya-sh]

@nchammas, also what version of maven did you use to build Spark?

A quick test shows that at least maven-3.1.1 is broken (by not taking active profile into account when resolving transitive dependencies) and maven-3.3.3 (the latest) seems to work.

3.1.1 (note that that both 2.3.11 and 2.3.4-spark akka versions are referenced):

> mvn dependency:tree -Phadoop-1  | grep akka
[INFO] +- org.spark-project.akka:akka-remote_2.10:jar:2.3.4-spark:compile
[INFO] |  +- org.spark-project.akka:akka-actor_2.10:jar:2.3.4-spark:compile
[INFO] +- org.spark-project.akka:akka-slf4j_2.10:jar:2.3.4-spark:compile
[INFO] +- org.spark-project.akka:akka-testkit_2.10:jar:2.3.4-spark:test
[INFO] |  +- com.typesafe.akka:akka-remote_2.10:jar:2.3.11:compile
[INFO] |  |  +- com.typesafe.akka:akka-actor_2.10:jar:2.3.11:compile
[INFO] |  +- com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11:compile
...

3.3.3 (only 2.3.4-spark akka version)

[INFO] +- org.spark-project.akka:akka-remote_2.10:jar:2.3.4-spark:compile
[INFO] |  +- org.spark-project.akka:akka-actor_2.10:jar:2.3.4-spark:compile
[INFO] +- org.spark-project.akka:akka-slf4j_2.10:jar:2.3.4-spark:compile
[INFO] +- org.spark-project.akka:akka-testkit_2.10:jar:2.3.4-spark:test
[INFO] |  +- org.spark-project.akka:akka-remote_2.10:jar:2.3.4-spark:compile
[INFO] |  |  +- org.spark-project.akka:akka-actor_2.10:jar:2.3.4-spark:compile
[INFO] |  +- org.spark-project.akka:akka-slf4j_2.10:jar:2.3.4-spark:compile
[INFO] |  +- org.spark-project.akka:akka-remote_2.10:jar:2.3.4-spark:compile
[INFO] |  |  +- org.spark-project.akka:akka-actor_2.10:jar:2.3.4-spark:compile
[INFO] |  +- org.spark-project.akka:akka-slf4j_2.10:jar:2.3.4-spark:compile
...

[zsxwing]

3.2.5 has the same issue, too

mvn -Phadoop-1 dependency:tree | grep akka
[INFO] +- org.spark-project.akka:akka-remote_2.10:jar:2.3.4-spark:compile
[INFO] |  +- org.spark-project.akka:akka-actor_2.10:jar:2.3.4-spark:compile
[INFO] +- org.spark-project.akka:akka-slf4j_2.10:jar:2.3.4-spark:compile
[INFO] +- org.spark-project.akka:akka-testkit_2.10:jar:2.3.4-spark:test
[INFO] |  +- com.typesafe.akka:akka-remote_2.10:jar:2.3.11:compile
[INFO] |  |  +- com.typesafe.akka:akka-actor_2.10:jar:2.3.11:compile
[INFO] |  +- com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11:compile
[INFO] |  +- com.typesafe.akka:akka-remote_2.10:jar:2.3.11:compile
[INFO] |  |  +- com.typesafe.akka:akka-actor_2.10:jar:2.3.11:compile
[INFO] |  +- com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11:compile

[srowen]

So let's require Maven 3.3 -- problem solved!

I'm going to just say it: what about dropping Hadoop 1.x support in Spark 1.5? Java 6 support is going away too.

[nchammas]

@kostya-sh @zsxwing - I was using Maven 3.2.5. Lemme try with 3.3 and confirm that that fixes things.

The top level error was this:

15/06/09 17:02:10 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.remote.default-remote-dispatcher-6] shutting down ActorSystem [sparkDriver]
java.lang.VerifyError: class akka.remote.WireFormats$AkkaControlMessage overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
...
15/06/09 17:02:20 ERROR SparkContext: Error initializing SparkContext.
java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds]

@srowen - Do you want to start a discussion on the dev list about both requiring Maven 3.3 and dropping Hadoop 1 support (perhaps in separate threads)?

[nchammas]

OK, I confirmed that using Maven 3.3.3 resolves my issue.