[WIP][SPARK-18896][TESTS] Update to ScalaTest 3.0.1 #16309

jaceklaskowski · 2016-12-16T09:13:18Z

What changes were proposed in this pull request?

Fixes ScalaCheck warning by upgrading to the latest 1.13.4 version.

Warning: Unknown ScalaCheck args provided: -oDF

The reason is due to a bug in ScalaCheck as reported in typelevel/scalacheck#212 and fixed in typelevel/scalacheck@df435a5 that is available in ScalaCheck 1.13.4.

Spark uses ScalaCheck 1.12.5 which is behind the latest 1.12.6 released on Nov 1 (not to mention 1.13.4).

How was this patch tested?

Local build. Waiting for Jenkins.

HyukjinKwon · 2016-12-16T09:26:17Z

pom.xml

Hi @jaceklaskowski, I just wonder if it is safe assuming from the comment beside, 1.13.0 appears incompatible with scalatest 2.2.6. I just read https://github.com/rickynils/scalacheck/issues/217 BTW.

Thanks. The local build has passed so I'm now waiting for Jenkins to say how it liked the change (or not).

Yes, I remember this didn't work. Jenkins runs SBT; are you checking Maven locally?
I'd like to update, sure, but want to make sure we don't trip over whatever I found before.

We could consider updating to scalatest 3.0.1 as well, in any event. As a test-only dependency it should be safe to more freely update it.

Yes, I did use maven for the entire local built and sbt for this particular DAGSchedulerSuite (that it all started from actually).

I've got little experience with Spark test suite and hence I'm relying on Jenkins to do the heavy-lifting and see how things may have changed since the comment.

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IncompatibleClassChangeError: Implementing class

You were both indeed right. Let me dig into it a bit more...

SparkQA · 2016-12-16T11:10:25Z

Test build #70243 has finished for PR 16309 at commit 9a18c10.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-12-16T11:24:05Z

Yup, doesn't work I'm afraid:

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IncompatibleClassChangeError: Implementing class

Does 1.12.6 fix it? otherwise, I wonder what upgrading scalatest does?

jaceklaskowski · 2016-12-16T11:39:36Z

Right. Testing ScalaTest 3.0.1 with the following sbt command:

sbt -Phadoop-2.3 -Pmesos -Pkinesis-asl -Pyarn -Phive-thriftserver -Phive test:package streaming-kafka-0-8-assembly/assembly streaming-flume-assembly/assembly streaming-kinesis-asl-assembly/assembly

Is this correct to be as close to Jenkins as possible?

jaceklaskowski · 2016-12-16T12:11:11Z

Had to introduce changes to the tests given "Expired deprecations" in ScalaTest 3.0.0.

SparkQA · 2016-12-16T12:30:49Z

Test build #70249 has finished for PR 16309 at commit d5053bd.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class NettyBlockTransferSecuritySuite extends SparkFunSuite with MockitoSugar with Matchers

jaceklaskowski · 2016-12-16T12:31:44Z

Just learnt about export SPARK_TESTING=1 to avoid some test failures.

jaceklaskowski · 2016-12-16T12:38:40Z

Learnt about ./dev/test-dependencies.sh --replace-manifest just now. (Where's this all described?)

SparkQA · 2016-12-16T12:50:47Z

Test build #70250 has finished for PR 16309 at commit f8e8fca.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-12-16T13:33:26Z

Test build #70251 has finished for PR 16309 at commit 0484efd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AnalysisSuite extends AnalysisTest with Matchers

jaceklaskowski · 2016-12-16T13:41:32Z

Think the latest test failures are somehow related to:

Please set the environment variable SCALACTIC_FILL_FILE_PATHNAMES to yes at compile time to enable this feature.

SCALACTIC_FILL_FILE_PATHNAMES is in https://github.com/scalatest/scalatest/blob/10b38d73e804546aaf3690e6496b65d984f2459f/scalactic-macro/src/main/scala/org/scalactic/source/PositionMacro.scala#L27-L30 and looks it should be enabled before sbt starts a build:

export SCALACTIC_FILL_FILE_PATHNAMES=yes

BTW, I'm using sbt -Phadoop-2.3 -Phive -Pyarn -Pmesos -Phive-thriftserver -Pkinesis-asl test locally and the tests are still being executed (for the past 40 mins or so) and green.

jaceklaskowski · 2016-12-16T19:35:51Z

Hey @srowen, any idea about the following error? I'd appreciate any hints to help me fix it.

[info] - can use a custom recovery mode factory (57 milliseconds)
Exception in thread "Thread-89" [info] - master/worker web ui available (225 milliseconds)
[info] - master/worker web ui available with reverseProxy (336 milliseconds)
java.lang.OutOfMemoryError: GC overhead limit exceeded
[info] - basic scheduling - spread out (36 milliseconds)
[info] - basic scheduling - no spread out (34 milliseconds)
	at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3054)
[info] - basic scheduling with more memory - spread out (33 milliseconds)
	at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2874)
	at java.io.ObjectInputStream.readString(ObjectInputStream.java:1639)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1342)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
	at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1$React.react(Framework.scala:809)
	at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1.run(Framework.scala:798)
	at java.lang.Thread.run(Thread.java:745)

and

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOf(Arrays.java:3181)
	at scala.reflect.internal.Scopes$class.newNestedScope(Scopes.scala:436)
	at scala.reflect.internal.SymbolTable.newNestedScope(SymbolTable.scala:16)
	at scala.tools.nsc.typechecker.Contexts$Context.makeNewScope(Contexts.scala:510)
	at scala.tools.nsc.typechecker.Typers$Typer.defDefTyper$1(Typers.scala:5026)
	at scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5308)
	at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)

srowen · 2016-12-16T20:40:04Z

That's a simple "out of memory". Did you give SBT 4-6GB?
It's no big deal to push changes to see what Jenkins does with them if it's getting hard to test locally. A few runs for a [WIP] PR is no big deal.

jaceklaskowski · 2016-12-16T21:06:20Z

I'm testing the changes locally with the following:

export SCALACTIC_FILL_FILE_PATHNAMES=yes
export SBT_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"

sbt -Phadoop-2.3 -Pmesos -Pkinesis-asl -Pyarn -Phive-thriftserver -Phive test:package streaming-kafka-0-8-assembly/assembly streaming-flume-assembly/assembly

sbt -Phadoop-2.3 -Phive -Pyarn -Pmesos -Phive-thriftserver -Pkinesis-asl -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest test

It's been running for the past one hour and looks fine so far. I faced some SQL UDF-related errors that didn't look like mine so had to re-test everything again. I however got no changes to push so...how can I re-trigger tests on Jenkins to see if it's something temporary? Is there some magic Jenkins trigger?

srowen · 2016-12-16T21:07:03Z

2G is not enough. I use 4G at least.

jaceklaskowski · 2016-12-17T11:48:07Z

Rebasing with master to trigger tests on Jenkins...(hoping this time they pass)

SparkQA · 2016-12-17T12:19:01Z

Test build #70306 has finished for PR 16309 at commit b3c8f48.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AnalysisSuite extends AnalysisTest with Matchers

jaceklaskowski · 2016-12-17T13:36:04Z

@srowen Please help as I'm stuck with the OutOfMemoryError: GC overhead limit exceeded error. Should Jenkins run the tests with 6g?

What's even more interesting is that the tests seem to have all passed according to the report in Test build #70306 has finished.

BTW, I'm running the tests on my laptop with the following settings export SBT_OPTS="-Xmx6g -XX:ReservedCodeCacheSize=512m" (as recommended).

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at scala.collection.TraversableLike$class.withFilter(TraversableLike.scala:655)
	at scala.collection.AbstractTraversable.withFilter(Traversable.scala:104)
	at scala.tools.nsc.backend.jvm.GenASM$JPlainBuilder$scoping$2$.getMerged(GenASM.scala:2067)
	at scala.tools.nsc.backend.jvm.GenASM$JPlainBuilder.genLocalVariableTable$1(GenASM.scala:2112)
	at scala.tools.nsc.backend.jvm.GenASM$JPlainBuilder.genCode(GenASM.scala:2753)
	at scala.tools.nsc.backend.jvm.GenASM$JPlainBuilder.genMethod(GenASM.scala:1471)

jaceklaskowski · 2016-12-17T14:15:01Z

The tests ran locally on my laptop have finished after...7431 s which is 2 hours (!)

[error] (sql/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] (streaming-kafka-0-10/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 7431 s, completed Dec 17, 2016 2:56:59 PM
[error] Not a valid command: Pkinesis-asl (similar: itunes-pause)
[error] Not a valid key: Pkinesis-asl
[error] Pkinesis-asl
[error]             ^

Not sure why I'm getting Not a valid key: Pkinesis-asl -- it looks like the profile might not have been picked up correctly (?)

SparkQA · 2016-12-17T14:25:27Z

Test build #3509 has finished for PR 16309 at commit b3c8f48.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AnalysisSuite extends AnalysisTest with Matchers

srowen · 2016-12-18T09:27:23Z

You probably have a typo -- Pkinesis-asl instead of -Pkinesis-asl locally. This error does not appear in Jenkins output. This still has the OutOfMemoryError and I'm not sure why. It's coming from the compiler. I am not sure why changing scalatest would impact compilation.

The only interesting thing I spy is that one of the traces involves scala XML code, and you updated to scala-xml_2.11-1.0.5.jar That sounds fine but I wonder if it's related? was this otherwise required as an updated? can we back it out to rule it out?

If it's resolvable, let's try to update to scalatest 3. if not, see if plugin 1.12.6 fixes the minor warning. If not, I'd punt on this for now. You could leave the JIRA open as "update to scalatest 3" for a later look then.

jaceklaskowski · 2016-12-18T09:35:39Z

The reason for the update to scala-xml_2.11-1.0.5.jar was that once I updated ScalaTest I got the issue from Jenkins that the dependency list changed. That's when I was told about ./dev/test-dependencies.sh --replace-manifest (see #16309 (comment) and https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70249/console).

You may be right that updating the jar could have impacted the XML issue Jenkins's reporting. Let me see if I can somehow relate them by:

Reviewing the changes between scala-xml_2.11-1.0.2.jar and scala-xml_2.11-1.0.5.jar
Looking into the XML not generated issue in the recent builds.

Thanks for the hint!

srowen · 2016-12-22T09:57:58Z

In order to finish this thought, let's either update scalatest to just 1.12.6 if that solves it, or do nothing. In the latter case, you could leave the JIRA open as an 'update to scalatest 3' JIRA for later, though I don't know how to track down why it fails.

jaceklaskowski · 2016-12-25T22:16:08Z

The changes between scala-xml 1.0.2 and 1.0.5 are in this view.

For reference: scala-xml releases

SparkQA · 2016-12-26T12:08:13Z

Test build #70598 has finished for PR 16309 at commit d57f03c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AnalysisSuite extends AnalysisTest with Matchers

Closes apache#15736 Closes apache#16309 Closes apache#16485 Closes apache#16502 Closes apache#16196 Closes apache#16498 Closes apache#12380 Closes apache#16764

HyukjinKwon reviewed Dec 16, 2016

View reviewed changes

jaceklaskowski force-pushed the SPARK-18896 branch from 0484efd to b3c8f48 Compare December 17, 2016 11:46

jaceklaskowski changed the title ~~[SPARK-18896][TESTS] Suppress ScalaCheck warning~~ [WIP][SPARK-18896][TESTS] Update to ScalaTest 3.0.1 Dec 18, 2016

jaceklaskowski added 4 commits December 26, 2016 12:12

[SPARK-18896] Suppress ScalaCheck warning

d4a8d1f

Upgrade to ScalaTest 3.0.1

6650e6e

Replacing dependency manifests

ce36939

Forgot to add the change

d57f03c

jaceklaskowski force-pushed the SPARK-18896 branch from b3c8f48 to d57f03c Compare December 26, 2016 11:12

srowen added a commit to srowen/spark that referenced this pull request Feb 2, 2017

Close stale PRs

4a54bb4

Closes apache#15736 Closes apache#16309 Closes apache#16485 Closes apache#16502 Closes apache#16196 Closes apache#16498 Closes apache#12380 Closes apache#16764

srowen mentioned this pull request Feb 2, 2017

[BUILD] Close stale PRs #16778

Closed

asfgit closed this in 20b4ca1 Feb 3, 2017

[WIP][SPARK-18896][TESTS] Update to ScalaTest 3.0.1 #16309

[WIP][SPARK-18896][TESTS] Update to ScalaTest 3.0.1 #16309

Uh oh!

Conversation

jaceklaskowski commented Dec 16, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon Dec 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaceklaskowski Dec 16, 2016

Choose a reason for hiding this comment

Uh oh!

srowen Dec 16, 2016

Choose a reason for hiding this comment

Uh oh!

jaceklaskowski Dec 16, 2016

Choose a reason for hiding this comment

Uh oh!

jaceklaskowski Dec 16, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 16, 2016

Uh oh!

srowen commented Dec 16, 2016

Uh oh!

jaceklaskowski commented Dec 16, 2016

Uh oh!

jaceklaskowski commented Dec 16, 2016

Uh oh!

SparkQA commented Dec 16, 2016

Uh oh!

jaceklaskowski commented Dec 16, 2016

Uh oh!

jaceklaskowski commented Dec 16, 2016

Uh oh!

SparkQA commented Dec 16, 2016

Uh oh!

SparkQA commented Dec 16, 2016

Uh oh!

jaceklaskowski commented Dec 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaceklaskowski commented Dec 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Dec 16, 2016

Uh oh!

jaceklaskowski commented Dec 16, 2016

Uh oh!

srowen commented Dec 16, 2016

Uh oh!

jaceklaskowski commented Dec 17, 2016

Uh oh!

SparkQA commented Dec 17, 2016

Uh oh!

jaceklaskowski commented Dec 17, 2016

Uh oh!

jaceklaskowski commented Dec 17, 2016

Uh oh!

SparkQA commented Dec 17, 2016

Uh oh!

srowen commented Dec 18, 2016

Uh oh!

jaceklaskowski commented Dec 18, 2016

Uh oh!

srowen commented Dec 22, 2016

Uh oh!

jaceklaskowski commented Dec 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Dec 26, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

HyukjinKwon Dec 16, 2016 •

edited

Loading

jaceklaskowski commented Dec 16, 2016 •

edited

Loading

jaceklaskowski commented Dec 16, 2016 •

edited

Loading

jaceklaskowski commented Dec 25, 2016 •

edited

Loading