Skip to content

Support cross building for Scala 2.11 #3159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 34 commits into from
Closed

Support cross building for Scala 2.11 #3159

wants to merge 34 commits into from

Conversation

@pwendell
Copy link
Contributor

@pwendell pwendell commented Nov 7, 2014

Let's give this another go using a version of Hive that shades its JLine dependency.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23060 has started for PR 3159 at commit 37d972c.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23060 has finished for PR 3159 at commit 37d972c.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 7, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23060/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23062 has started for PR 3159 at commit fd849b0.

  • This patch merges cleanly.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 7, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23062/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23064 has started for PR 3159 at commit ad1b053.

  • This patch merges cleanly.

@pwendell
Copy link
Contributor Author

pwendell commented Nov 7, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23067 has started for PR 3159 at commit d7b5091.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23067 has finished for PR 3159 at commit d7b5091.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 7, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23067/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23064 has finished for PR 3159 at commit ad1b053.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 7, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23064/
Test FAILed.

@pwendell
Copy link
Contributor Author

pwendell commented Nov 7, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23070 has started for PR 3159 at commit d7b5091.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23070 has finished for PR 3159 at commit d7b5091.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 7, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23070/
Test FAILed.

@pwendell pwendell force-pushed the scala-2.11-prashant branch from d7b5091 to c75582e Compare November 8, 2014 01:25
@SparkQA
Copy link

SparkQA commented Nov 8, 2014

Test build #23080 has started for PR 3159 at commit c75582e.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 8, 2014

Test build #23080 has finished for PR 3159 at commit c75582e.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 8, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23080/
Test FAILed.

@pwendell
Copy link
Contributor Author

pwendell commented Nov 8, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Nov 8, 2014

Test build #23084 has started for PR 3159 at commit d61d290.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 8, 2014

Test build #23084 has finished for PR 3159 at commit d61d290.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 8, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23084/
Test FAILed.

@pwendell pwendell force-pushed the scala-2.11-prashant branch from d61d290 to 7ac13e9 Compare November 8, 2014 02:17
@pwendell
Copy link
Contributor Author

pwendell commented Nov 8, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Nov 8, 2014

Test build #23087 has started for PR 3159 at commit 7ac13e9.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 8, 2014

Test build #23087 has finished for PR 3159 at commit 7ac13e9.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 8, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23087/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 9, 2014

Test build #23108 has started for PR 3159 at commit 125ba46.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 10, 2014

Test build #23133 has finished for PR 3159 at commit 5e44005.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 10, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23133/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 10, 2014

Test build #23135 has finished for PR 3159 at commit 2990cd2.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 10, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23135/
Test FAILed.

@pwendell pwendell force-pushed the scala-2.11-prashant branch from 2990cd2 to a8c41eb Compare November 11, 2014 23:35
@SparkQA
Copy link

SparkQA commented Nov 11, 2014

Test build #23227 has started for PR 3159 at commit a8c41eb.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 11, 2014

Test build #23227 has finished for PR 3159 at commit a8c41eb.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 11, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23227/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 11, 2014

Test build #23228 has started for PR 3159 at commit f65d17d.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 11, 2014

Test build #23228 has finished for PR 3159 at commit f65d17d.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 11, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23228/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 12, 2014

Test build #23236 has started for PR 3159 at commit e93aa3e.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 12, 2014

Test build #23236 has finished for PR 3159 at commit e93aa3e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 12, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23236/
Test PASSed.

@asfgit asfgit closed this in daaca14 Nov 12, 2014
asfgit pushed a commit that referenced this pull request Nov 12, 2014
@ScrapCodes @pwendell
Support cross building for Scala 2.11
12f5633 
Let's give this another go using a version of Hive that shades its JLine dependency.

Author: Prashant Sharma <[email protected]>
Author: Patrick Wendell <[email protected]>

Closes #3159 from pwendell/scala-2.11-prashant and squashes the following commits:

e93aa3e [Patrick Wendell] Restoring -Phive-thriftserver profile and cleaning up build script.
f65d17d [Patrick Wendell] Fixing build issue due to merge conflict
a8c41eb [Patrick Wendell] Reverting dev/run-tests back to master state.
7a6eb18 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into scala-2.11-prashant
583aa07 [Prashant Sharma] REVERT ME: removed hive thirftserver
3680e58 [Prashant Sharma] Revert "REVERT ME: Temporarily removing some Cli tests."
935fb47 [Prashant Sharma] Revert "Fixed by disabling a few tests temporarily."
925e90f [Prashant Sharma] Fixed by disabling a few tests temporarily.
2fffed3 [Prashant Sharma] Exclude groovy from sbt build, and also provide a way for such instances in future.
8bd4e40 [Prashant Sharma] Switched to gmaven plus, it fixes random failures observer with its predecessor gmaven.
5272ce5 [Prashant Sharma] SPARK_SCALA_VERSION related bugs.
2121071 [Patrick Wendell] Migrating version detection to PySpark
b1ed44d [Patrick Wendell] REVERT ME: Temporarily removing some Cli tests.
1743a73 [Patrick Wendell] Removing decimal test that doesn't work with Scala 2.11
f5cad4e [Patrick Wendell] Add Scala 2.11 docs
210d7e1 [Patrick Wendell] Revert "Testing new Hive version with shaded jline"
48518ce [Patrick Wendell] Remove association of Hive and Thriftserver profiles.
e9d0a06 [Patrick Wendell] Revert "Enable thritfserver for Scala 2.10 only"
67ec364 [Patrick Wendell] Guard building of thriftserver around Scala 2.10 check
8502c23 [Patrick Wendell] Enable thritfserver for Scala 2.10 only
e22b104 [Patrick Wendell] Small fix in pom file
ec402ab [Patrick Wendell] Various fixes
0be5a9d [Patrick Wendell] Testing new Hive version with shaded jline
4eaec65 [Prashant Sharma] Changed scripts to ignore target.
5167bea [Prashant Sharma] small correction
a4fcac6 [Prashant Sharma] Run against scala 2.11 on jenkins.
80285f4 [Prashant Sharma] MAven equivalent of setting spark.executor.extraClasspath during tests.
034b369 [Prashant Sharma] Setting test jars on executor classpath during tests from sbt.
d4874cb [Prashant Sharma] Fixed Python Runner suite. null check should be first case in scala 2.11.
6f50f13 [Prashant Sharma] Fixed build after rebasing with master. We should use ${scala.binary.version} instead of just 2.10
e56ca9d [Prashant Sharma] Print an error if build for 2.10 and 2.11 is spotted.
937c0b8 [Prashant Sharma] SCALA_VERSION -> SPARK_SCALA_VERSION
cb059b0 [Prashant Sharma] Code review
0476e5e [Prashant Sharma] Scala 2.11 support with repl and all build changes.

(cherry picked from commit daaca14)
Signed-off-by: Patrick Wendell <[email protected]>
@witgo
Copy link
Contributor

witgo commented Nov 12, 2014

@pwendell @ScrapCodes
This patch has a bug:
./make-distribution.sh -Dhadoop.version=2.3.0-cdh5.0.1 -Dyarn.version=2.3.0-cdh5.0.1 -Phadoop-2.3 -Pyarn
./bin/spark-shell=>

java.lang.ClassNotFoundException: org.apache.spark.repl.Main
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:337)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

@srowen
Copy link
Member

srowen commented Nov 12, 2014

I see the same thing with a simple mvn -DskipTests clean package; mvn test. I think the assembly doesn't quite include the REPL classes now that they are moved, somehow?

@witgo witgo mentioned this pull request Nov 12, 2014
Closed
@adrian-wang adrian-wang mentioned this pull request Nov 25, 2014
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@pwendell @SparkQA @AmplabJenkins @witgo @srowen @ScrapCodes @FRosner