[test-maven][test-hadoop1.0][SPARK-11236][CORE] Update Tachyon dependency 0.7.1 -> 0.8.1 #9395

calvinjia · 2015-11-01T10:07:23Z

This is a reopening of #9204 which failed hadoop1 sbt tests.

With the original PR, a classpath issue would occur due to the MIMA plugin pulling in hadoop-2.2 dependencies regardless of the hadoop version when building the oldDeps project. These affect the hadoop1 sbt build because they are placed in lib_managed and Tachyon 0.8.0's default hadoop version is 2.2.

calvinjia · 2015-11-01T10:08:45Z

@yhuai
Could you take a look? Thanks!

yhuai · 2015-11-01T15:10:03Z

cc @pwendell @srowen This is the new pr that upgrades Tachyon. I reverted the original one, which caused hadoop 1 test failures.

yhuai · 2015-11-01T15:11:30Z

btw, the profiles that we used in hadoop 1 tests are -Phadoop-1 -Dhadoop.version=1.2.1 -Pkinesis-asl -Phive-thriftserver -Phive.

SparkQA · 2015-11-01T16:30:38Z

Test build #44756 has finished for PR 9395 at commit 6802ecd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * logInfo(s\"Hive class not found $e\")\n * logDebug(\"Hive class not found\", e)\n

calvinjia · 2015-11-01T17:26:32Z

@yhuai
Thanks for the the response! The last PR would pass with those parameters, but and would only fail if the entire dev/runtests cycle was ran (specifically if MIMA ran before). I've tested locally that the issues do not occur when running dev/runtests with this change.

pwendell · 2015-11-02T02:18:01Z

@calvinjia can you add "[test-hadoop1.0]" to the title of this PR and then retest it? That will run the tests with hadoop 1. See more info here:

https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools

calvinjia · 2015-11-02T02:20:53Z

@pwendell could you trigger a build for me? Or should I just make an empty commit?

pwendell · 2015-11-02T02:21:09Z

Jenkins, test this please.

calvinjia · 2015-11-02T02:21:18Z

Thanks!

SparkQA · 2015-11-02T04:00:08Z

Test build #44788 has finished for PR 9395 at commit 6802ecd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

calvinjia · 2015-11-02T04:12:30Z

@pwendell

It seems like the previous issue is gone. The only error (which seems to be triggered by an oom / jvm crash?) that occurs is:

Exception in thread "Thread-58" Exception in thread "Thread-62" java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at java.net.SocketInputStream.read(SocketInputStream.java:210)
    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at sbt.React.react(ForkTests.scala:114)
    at sbt.ForkTests$$anonfun$mainTestTask$1$Acceptor$2$.run(ForkTests.scala:74)
    at java.lang.Thread.run(Thread.java:745)
java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1$React.react(Framework.scala:945)
    at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1.run(Framework.scala:934)
    at java.lang.Thread.run(Thread.java:745)

It doesn't seem like its related to this change.

yhuai · 2015-11-02T04:32:05Z

test this please

calvinjia · 2015-11-02T05:39:53Z

@yhuai Thanks for the retest. I'm not sure if this will go away by re-running or if there is something up with Jenkins/Spark master branch. It seems like the current Spark-Master-SBT build is not happy on hadoop1.0 for the same reason. (See: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3911/AMPLAB_JENKINS_BUILD_PROFILE=hadoop1.0,label=spark-test/consoleFull)

SparkQA · 2015-11-02T06:08:53Z

Test build #44790 has finished for PR 9395 at commit 6802ecd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-11-02T06:11:57Z

Hm - can you try also adding [test-maven] - might be better to test with
maven.

On Sun, Nov 1, 2015 at 9:40 PM, Calvin Jia [email protected] wrote:

@yhuai https://github.com/yhuai Thanks for the retest. I'm not sure if
this will go away by re-running or if there is something up with
Jenkins/Spark master branch. It seems like the current Spark-Master-SBT
build is not happy on hadoop1.0 for the same reason. (See:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3911/AMPLAB_JENKINS_BUILD_PROFILE=hadoop1.0,label=spark-test/consoleFull
)

—
Reply to this email directly or view it on GitHub
#9395 (comment).

calvinjia · 2015-11-02T06:23:41Z

@pwendell Sure, I've updated it.

pwendell · 2015-11-02T06:23:46Z

Jenkins, test this please.

SparkQA · 2015-11-02T06:54:54Z

Test build #44798 has finished for PR 9395 at commit 6802ecd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

calvinjia · 2015-11-02T07:03:41Z

@pwendell
It seems like the maven tests failed with:

java.lang.IllegalArgumentException: Logging directory specified is not a directory: file:/home/jenkins/workspace/SparkPullRequestBuilder/core/target/tmp/spark-ab60a5c8-38dd-4340-a4e9-109cf8dcf18e/

This doesn't seem like something that would be introduced by this change?

calvinjia · 2015-11-02T07:38:38Z

@pwendell

It seems like the Spark-Master-Maven build running for hadoop1.0 is also failing with the same problem (see: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/HADOOP_VERSION=1.2.1,label=spark-test/4631/).

calvinjia · 2015-11-02T21:28:28Z

@pwendell @yhuai
Is the only blocker for this PR the stability of hadoop1.0 tests in Spark master? Thanks!

pwendell · 2015-11-03T00:33:32Z

Hey @calvinjia seems okay to merge since this is just triggering other failures. Can you explain more though how this works around the mima issue - the patch seems the same as #9204. Did something change between Tachyon 0.8.0 and 0.8.1?

calvinjia · 2015-11-03T00:49:29Z

@pwendell
We changed the default hadoop version from 2.2. MIMA pulls in hadoop2.2 jars into the lib_managed directory when it builds oldDeps. In spark.utils.ShutdownHookManager, we do some reflection to see if we are using Hadoop 2.x, and it will get a false positive leading to the failures. (See: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L221)

pwendell · 2015-11-03T01:01:38Z

Got it- thanks I can merge it then.

calvinjia · 2015-11-03T01:05:10Z

@pwendell @yhuai
Thanks for the pointers and reviewing this PR.

calvinjia added 6 commits October 21, 2015 11:43

Update Tachyon dependency from 0.7.1 -> 0.8.0.

890d48c

Update make distribution to use Tachyon 0.8.0.

876a5bf

Style fix.

f7edb37

Merge branch 'master' into spark-11236

a187415

Merge branch 'master' into spark-11236

3dad39f

Update Tachyon version.

6802ecd

calvinjia changed the title ~~[SPARK-11236][CORE] Update Tachyon dependency 0.7.1 -> 0.8.1~~ [test-hadoop1.0][SPARK-11236][CORE] Update Tachyon dependency 0.7.1 -> 0.8.1 Nov 2, 2015

calvinjia changed the title ~~[test-hadoop1.0][SPARK-11236][CORE] Update Tachyon dependency 0.7.1 -> 0.8.1~~ [test-maven][test-hadoop1.0][SPARK-11236][CORE] Update Tachyon dependency 0.7.1 -> 0.8.1 Nov 2, 2015

asfgit closed this in 476f434 Nov 3, 2015

[test-maven][test-hadoop1.0][SPARK-11236][CORE] Update Tachyon dependency 0.7.1 -> 0.8.1 #9395

[test-maven][test-hadoop1.0][SPARK-11236][CORE] Update Tachyon dependency 0.7.1 -> 0.8.1 #9395

Uh oh!

Conversation

calvinjia commented Nov 1, 2015

Uh oh!

calvinjia commented Nov 1, 2015

Uh oh!

yhuai commented Nov 1, 2015

Uh oh!

yhuai commented Nov 1, 2015

Uh oh!

SparkQA commented Nov 1, 2015

Uh oh!

calvinjia commented Nov 1, 2015

Uh oh!

pwendell commented Nov 2, 2015

Uh oh!

calvinjia commented Nov 2, 2015

Uh oh!

pwendell commented Nov 2, 2015

Uh oh!

calvinjia commented Nov 2, 2015

Uh oh!

SparkQA commented Nov 2, 2015

Uh oh!

calvinjia commented Nov 2, 2015

Uh oh!

yhuai commented Nov 2, 2015

Uh oh!

calvinjia commented Nov 2, 2015

Uh oh!

SparkQA commented Nov 2, 2015

Uh oh!

pwendell commented Nov 2, 2015

Uh oh!

calvinjia commented Nov 2, 2015

Uh oh!

pwendell commented Nov 2, 2015

Uh oh!

SparkQA commented Nov 2, 2015

Uh oh!

calvinjia commented Nov 2, 2015

Uh oh!

calvinjia commented Nov 2, 2015

Uh oh!

calvinjia commented Nov 2, 2015

Uh oh!

pwendell commented Nov 3, 2015

Uh oh!

calvinjia commented Nov 3, 2015

Uh oh!

pwendell commented Nov 3, 2015

Uh oh!

calvinjia commented Nov 3, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants