[SPARK-28683][BUILD] Upgrade Scala to 2.12.10 #25404

wangyum · 2019-08-10T10:43:44Z

What changes were proposed in this pull request?

This PR upgrade Scala to 2.12.10.

Release notes:

Fix regression in large string interpolations with non-String typed splices
Revert "Generate shallower ASTs in pattern translation"
Fix regression in classpath when JARs have 'a.b' entries beside 'a/b'
Faster compiler: 5–10% faster since 2.12.8
Improved compatibility with JDK 11, 12, and 13
Experimental support for build pipelining and outline type checking

More details:
https://github.com/scala/scala/releases/tag/v2.12.10
https://github.com/scala/scala/releases/tag/v2.12.9

How was this patch tested?

Existing tests

SparkQA · 2019-08-10T12:01:05Z

Test build #108914 has finished for PR 25404 at commit 839c27e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-08-10T22:35:12Z

Retest this please.

dongjoon-hyun · 2019-08-10T22:47:00Z

The failures seems relevant, @wangyum . Could you check the failures?

[info] *** 212 TESTS FAILED ***

[info] - Sort Array *** FAILED *** (10 milliseconds)
[info]   Code generation of sort_array([2,1,3], true) failed:
[info]   java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String
[info]   java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String
...

SparkQA · 2019-08-10T23:46:51Z

Test build #108924 has finished for PR 25404 at commit 839c27e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-08-11T19:45:20Z

Wow that's a weird one, but I played with it and, bizarrely this change in the generate code makes it work:

      } else if ($o2 == null) {
        return -$sortOrder * $nullOrder;
      }

to

      } else if ($o2 == null) {
        return -$sortOrder * ${nullOrder.toString};
      }

I can only guess something goes wrong in string interpolation because nullOrder is a NullOrder and typedef for an Int, and is boxed to Integer somewhere, but something believes it is a String.

... and then I decided to search for open bugs and yep this is it:
scala/bug#11665

We may just need to wait for 2.12.10

wangyum · 2019-08-12T00:06:36Z

Thank you @srowen Let's wait for 2.12.10.

dongjoon-hyun · 2019-08-12T04:32:00Z

Hi, All. Let's reuse SPARK-28683 for 2.12.10. I reopened the JIRA issue for that.

SparkQA · 2019-09-11T05:09:44Z

Test build #110459 has finished for PR 25404 at commit e5811fd.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-11T05:19:15Z

sbt.ResolveException: unresolved dependency: com.typesafe.genjavadoc#genjavadoc-plugin_2.12.10;0.13: not found
	at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:320)
	at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
	at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)

Let's wait for genjavadoc-plugin_2.12.10: https://repo1.maven.org/maven2/com/typesafe/genjavadoc/

srowen · 2019-09-11T13:52:22Z

OK by me, once the javadoc plugin is available and tests pass. Looks like the problem with string interpolation was resolved.

dongjoon-hyun · 2019-09-11T15:07:48Z

Thank you, @wangyum and @srowen . Yes. Let's retry when it becomes ready.

dongjoon-hyun · 2019-09-12T22:23:55Z

Ur, it seems to be delayed due to the release manager issue. Thank you for pushing it, @wangyum !

Support Scala 2.12.10 lightbend/genjavadoc#194

wangyum · 2019-09-17T13:00:22Z

retest this please

SparkQA · 2019-09-17T13:08:51Z

Test build #110780 has finished for PR 25404 at commit e5811fd.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-17T13:23:07Z

genjavadoc 0.14 has released: https://repo1.maven.org/maven2/com/typesafe/genjavadoc/genjavadoc-plugin_2.12.10/0.14/

SparkQA · 2019-09-17T15:21:18Z

Test build #110781 has finished for PR 25404 at commit 075134d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-09-17T15:35:48Z

Finally! Thanks!

dongjoon-hyun · 2019-09-17T15:36:26Z

Retest this please.

SparkQA · 2019-09-17T15:39:38Z

Test build #110793 has started for PR 25404 at commit 075134d.

srowen · 2019-09-17T15:41:56Z

Well... that's weird. Lots of tests fail because they get slightly different answers, and almost all look like tests that depend on a seeded random number generator at some level.

Take https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110781/testReport/test.org.apache.spark/JavaAPISuite/sample/ for example.

  @Test
  public void sample() {
    List<Integer> ints = IntStream.iterate(1, x -> x + 1)
      .limit(20)
      .boxed()
      .collect(Collectors.toList());
    JavaRDD<Integer> rdd = sc.parallelize(ints);
    // the seeds here are "magic" to make this work out nicely
    JavaRDD<Integer> sample20 = rdd.sample(true, 0.2, 8);
    assertEquals(2, sample20.count());
    JavaRDD<Integer> sample20WithoutReplacement = rdd.sample(false, 0.2, 2);
    assertEquals(4, sample20WithoutReplacement.count());
  }

It's seeded, so ought to produce the same answer. This has worked the same way for a long time, giving that particular answer with that particular seed. It's pretty straightforwardly using a java.util.Random, seeded. What could have changed?

At the moment I'm wondering if somehow a compiler or collections change causes the input to be partitioned differently or iterated over differently. Still thinking about it and reviewing the (pretty minor) Scala changes.

One way or the other I don't think we can let the behavior change for a seeded sample, but not clear whether it's some subtle assumption Spark makes or something else going on.

SparkQA · 2019-09-18T09:40:10Z

Test build #110882 has finished for PR 25404 at commit da7b223.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-18T10:10:37Z

retest this please

SparkQA · 2019-09-18T13:35:47Z

Test build #110901 has finished for PR 25404 at commit da7b223.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-18T13:44:58Z

retest this please

srowen · 2019-09-18T13:47:03Z

@dongjoon-hyun ah OK, I see what you did now. By using the equivalent two-arg bytesHash method, the behavior didn't change. But the implementation of two-arg bytesHash didn't change in Scala 2.12.10 so it avoids a behavior change.

Looks promising that tests have passed here once.

dongjoon-hyun · 2019-09-18T15:45:31Z

Yes. Right, @srowen !

dongjoon-hyun · 2019-09-18T15:49:42Z

So, if the last Jenkins run passes, this PR looks ready for me.

SBT (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110854/console)
SBT/JDK11 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110901/testReport)
Maven/JDK8 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110911/testReport, Pending)

dongjoon-hyun · 2019-09-18T19:31:28Z

Hmm. The last test has been running 5 hr 43 min already. It seems that we will hit timeout soon. :(

dongjoon-hyun · 2019-09-18T19:40:19Z

The last run passed all Scala/Java UTs and moved to Python test.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  05:24 h
[INFO] Finished at: 2019-09-18T12:39:46-07:00
[INFO] ------------------------------------------------------------------------

kiszk · 2019-09-18T19:41:49Z

LGTM, pending Jenkins

dongjoon-hyun · 2019-09-18T19:58:34Z

I checked and noticed that the PR Builder timeout 430m. It looks safe. I was confused with the other Jenkins job configurations which are shorter than this.

SparkQA · 2019-09-18T20:26:50Z

Test build #110911 has finished for PR 25404 at commit da7b223.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-09-18T20:29:07Z

Hmm. It's weird because it's terminated before the configuration.

AmplabJenkins · 2019-09-18T20:29:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110911/
Test FAILed.

dongjoon-hyun · 2019-09-18T20:29:58Z

Anyway, this Scala change looks safe to me. I'll merge this to master. Thank you, @wangyum , @srowen , @kiszk !

Please make a backport to branch-2.4.

### What changes were proposed in this pull request? This PR backport #25404 to branch-2.4. ### Why are the changes needed? Backport from master. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing tests Closes #25839 from wangyum/SPARK-28683-branch-2.4. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

This PR upgrade Scala to **2.12.10**. Release notes: - Fix regression in large string interpolations with non-String typed splices - Revert "Generate shallower ASTs in pattern translation" - Fix regression in classpath when JARs have 'a.b' entries beside 'a/b' - Faster compiler: 5–10% faster since 2.12.8 - Improved compatibility with JDK 11, 12, and 13 - Experimental support for build pipelining and outline type checking More details: https://github.com/scala/scala/releases/tag/v2.12.10 https://github.com/scala/scala/releases/tag/v2.12.9 Existing tests Closes apache#25404 from wangyum/SPARK-28683. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

Upgrade Scala to 2.12.9

839c27e

dongjoon-hyun added the BUILD label Aug 10, 2019

wangyum closed this Aug 12, 2019

wangyum deleted the SPARK-28683 branch August 12, 2019 00:06

xerial mentioned this pull request Aug 13, 2019

Upgrade to Scala 2.12.9 wvlet/airframe#576

Closed

1 task

wangyum restored the SPARK-28683 branch September 11, 2019 04:47

wangyum changed the title ~~[SPARK-28683][BUILD] Upgrade Scala to 2.12.9~~ [SPARK-28683][BUILD] Upgrade Scala to 2.12.10 Sep 11, 2019

Upgrade to 2.12.10

e5811fd

wangyum reopened this Sep 11, 2019

Update genjavadoc to 0.14

075134d

wangyum changed the title ~~[SPARK-28683][BUILD][test-hadoop3.2][test-java11] Upgrade Scala to 2.12.10~~ [SPARK-28683][BUILD][test-hadoop3.2][test-maven] Upgrade Scala to 2.12.10 Sep 18, 2019

dongjoon-hyun approved these changes Sep 18, 2019

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-28683][BUILD][test-hadoop3.2][test-maven] Upgrade Scala to 2.12.10~~ [SPARK-28683][BUILD] Upgrade Scala to 2.12.10 Sep 18, 2019

dongjoon-hyun closed this in 8c3f27c Sep 18, 2019

wangyum deleted the SPARK-28683 branch September 19, 2019 00:26

wangyum mentioned this pull request Sep 19, 2019

[SPARK-28683][BUILD][2.4] Upgrade Scala to 2.12.10 #25839

Closed

[SPARK-28683][BUILD] Upgrade Scala to 2.12.10 #25404

[SPARK-28683][BUILD] Upgrade Scala to 2.12.10 #25404

Uh oh!

Conversation

wangyum commented Aug 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Aug 10, 2019

Uh oh!

dongjoon-hyun commented Aug 10, 2019

Uh oh!

dongjoon-hyun commented Aug 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Aug 10, 2019

Uh oh!

srowen commented Aug 11, 2019

Uh oh!

wangyum commented Aug 12, 2019

Uh oh!

dongjoon-hyun commented Aug 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Sep 11, 2019

Uh oh!

wangyum commented Sep 11, 2019

Uh oh!

srowen commented Sep 11, 2019

Uh oh!

dongjoon-hyun commented Sep 11, 2019

Uh oh!

dongjoon-hyun commented Sep 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangyum commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

wangyum commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

dongjoon-hyun commented Sep 17, 2019

Uh oh!

dongjoon-hyun commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

srowen commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 18, 2019

Uh oh!

wangyum commented Sep 18, 2019

Uh oh!

SparkQA commented Sep 18, 2019

Uh oh!

wangyum commented Sep 18, 2019

Uh oh!

srowen commented Sep 18, 2019

Uh oh!

dongjoon-hyun commented Sep 18, 2019

Uh oh!

dongjoon-hyun commented Sep 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Sep 18, 2019

Uh oh!

dongjoon-hyun commented Sep 18, 2019

Uh oh!

kiszk commented Sep 18, 2019

Uh oh!

dongjoon-hyun commented Sep 18, 2019

Uh oh!

SparkQA commented Sep 18, 2019

Uh oh!

dongjoon-hyun commented Sep 18, 2019

Uh oh!

AmplabJenkins commented Sep 18, 2019

wangyum commented Aug 10, 2019 •

edited

Loading

dongjoon-hyun commented Aug 10, 2019 •

edited

Loading

dongjoon-hyun commented Aug 12, 2019 •

edited

Loading

dongjoon-hyun commented Sep 12, 2019 •

edited

Loading

dongjoon-hyun commented Sep 18, 2019 •

edited

Loading