[SPARK-5794] [SQL] fix add jar #4586

adrian-wang · 2015-02-13T07:48:30Z

No description provided.

SparkQA · 2015-02-13T07:53:57Z

Test build #27427 has finished for PR 4586 at commit 275a8f7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-02-13T08:44:06Z

Test build #27428 has finished for PR 4586 at commit 85db611.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

adrian-wang · 2015-02-13T09:02:42Z

This is because Jenkins do not have hcatalog.jar at /home/jenkins/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/0.13.1/hive-hcatalog-core-0.13.1.jar

I can remove the test from white list if the rest of this PR looks good.

watermen · 2015-03-02T12:33:40Z

I also found this issue, I fetched your branch and when I ran select * from src join t1 on src.key =t1.a; in mapjoin_addjar.q, it also threw ERROR CliDriver: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe like before.
I debuged the code and found it can't setup into if (proc.isInstanceOf[Driver] || proc.isInstanceOf[SetProcessor] in SparkSQLCLIDriver, so it can't run the run(sqlContext: SQLContext) in case class AddJar. Do I understand right?

adrian-wang · 2015-03-03T00:42:02Z

@watermen , I re-compiled and it can step into that.

watermen · 2015-03-03T01:33:15Z

@adrian-wang, you mean it can step into if (proc.isInstanceOf[Driver] || proc.isInstanceOf[SetProcessor] or run(sqlContext: SQLContext)? Because add jar is instance of AddResourceProcessor.

adrian-wang · 2015-03-03T02:03:39Z

If you are using HiveContext, it will step into the else branch, and trigger by
ret = proc.run(cmd_1).getResponseCode
so that will be executed by hive. if you are using SqlContext, then the parser will call the add jar case class.

watermen · 2015-03-03T06:32:20Z

@adrian-wang yeah, I'm using HiveContext, and threw ERROR CliDriver: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDerun when running the test case in mapjoin_addjar.q. I know it will be executed by hive, so it's strange.

adrian-wang · 2015-03-03T06:33:04Z

@waterman do you have that jar in your maven local repository then?

adrian-wang · 2015-03-03T06:35:31Z

just rebased twice...

watermen · 2015-03-03T06:36:33Z

@adrian-wang I'm running the sql in spark-sql shell, and use add jar /home/qiyadong/GitHub/temp/hive-hcatalog-core-0.13.1.jar; to add jar. You mean I should check the jar in maven local repository when I run maven test?

adrian-wang · 2015-03-03T06:48:11Z

@watermen there must be something wrong. I just did a clean pick and verified that this works well. You may need to clean and rebuild your assembly jar.

adrian-wang · 2015-03-03T06:56:19Z

@watermen Ah, I think I know what's wrong now. I'll fix that, thanks for your reminding!

watermen · 2015-03-03T07:00:58Z

@adrian-wang Look forward to your commit.

SparkQA · 2015-03-03T07:44:33Z

Test build #28221 has finished for PR 4586 at commit 97cab24.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2015-03-03T07:51:55Z

Test build #28223 has finished for PR 4586 at commit 1ce40c1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-03-03T07:52:24Z

Test build #28222 has finished for PR 4586 at commit b42b15c.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2015-03-03T08:03:38Z

Test build #28225 has finished for PR 4586 at commit 2b46223.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

watermen · 2015-03-03T08:12:36Z

@adrian-wang I try if (proc.isInstanceOf[Driver] || proc.isInstanceOf[SetProcessor] || proc.isInstanceOf[AddResourceProcessor]) like you before, and the exception also threw. But bin/spark-sql --jars xxx.jar or set xxx.jar in bin/beeline can be run successfully.

watermen · 2015-03-03T08:57:59Z

@adrian-wang I think sparkContext.addJar as same as bin/spark-sql --jars xxx.jar, but the former is fail and the latter is success.

adrian-wang · 2015-03-05T04:56:21Z

@watermen I have test my code this time and it should work. I'm trying to figure out where the problem lies, can you try this code first?

SparkQA · 2015-03-05T06:15:55Z

Test build #28275 has finished for PR 4586 at commit 48f93c2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- logInfo("Added " + substituted + " to class path")

watermen · 2015-03-05T07:59:31Z

@adrian-wang I test it and it works.

chenghao-intel · 2015-03-05T12:43:22Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala

There is a SerDe jar at sql/hive/src/test/resources/data/files/TestSerDe.jar , Instead of whitelisting the mapjoin_addjar in HiveCompatibilitySuite.scala, we can create a new unit test in HiveQuerySuite for simplifying the unit testing. As setting the local maven repo like this seems a little bit hacky to me, it probably doesn't work if people modify the default location or the jar hive-hcatalog-core-*.jar doesn't exist in some cases.

pzzs · 2015-03-12T11:28:18Z

LGTM

the-sea · 2015-03-13T06:48:59Z

greate feature!

SparkQA · 2015-04-02T09:48:09Z

Test build #29600 has finished for PR 4586 at commit 0810e71.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- logInfo("Added " + path + " to class path")
This patch does not change any dependencies.

SparkQA · 2015-04-08T06:24:20Z

Test build #29840 has finished for PR 4586 at commit 9957d87.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-08T08:33:46Z

Test build #29847 has finished for PR 4586 at commit 6c707e8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-08T08:53:55Z

Test build #29845 has finished for PR 4586 at commit 32c4fb8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

adrian-wang · 2015-04-08T08:56:32Z

retest this please.

SparkQA · 2015-04-08T10:25:26Z

Test build #29851 has finished for PR 4586 at commit 6c707e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

adrian-wang · 2015-04-08T16:14:25Z

ping.
/cc @liancheng

marmbrus · 2015-04-09T01:46:20Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala

Could you put this jar somewhere else? The stuff inside of resources/data are often replaced completely with files from hive. Perhaps just in the top level resources dir.

SparkQA · 2015-04-09T04:07:57Z

Test build #29916 has finished for PR 4586 at commit efdd602.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

adrian-wang · 2015-04-10T14:00:59Z

ping @marmbrus

adrian-wang · 2015-04-12T07:37:45Z

ping @marmbrus

marmbrus · 2015-04-14T01:26:56Z

Thanks, merged to master.

yhuai · 2015-04-15T05:17:14Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala

@adrian-wang Why we need a Row with a value of 0 at here?

Hive would return a 0 for add jar command.

(I thought it may be better to comment at the original pr).

OK, I see. In future, let's make sure we also update the output if the result of a command is not an empty Seq (#5350 will change the schema for AddJar).

adrian-wang force-pushed the addjar branch from 97cab24 to b42b15c Compare March 3, 2015 06:25

adrian-wang force-pushed the addjar branch from b42b15c to 1ce40c1 Compare March 3, 2015 06:34

adrian-wang force-pushed the addjar branch from 1ce40c1 to 2b46223 Compare March 3, 2015 07:24

adrian-wang changed the title ~~[SPARK-5794] [SQL] support env argus in add jar, and set add jar ret to 0~~ [SPARK-5794] [SQL] fix add jar Mar 5, 2015

chenghao-intel reviewed Mar 5, 2015
View reviewed changes

chenghao-intel mentioned this pull request Mar 19, 2015

[SPARK-6392][SQL]Minor fix ClassNotFound exception when use spark cli to add jar #5079

Closed

adrian-wang mentioned this pull request Mar 23, 2015

[SPARK-6202] [SQL] enable variable substitution on test framework #4930

Closed

adrian-wang added 2 commits April 1, 2015 21:01

support env argus in add jar, and set add jar ret to 0

95a40da

fix classnotfound

1898309

adrian-wang force-pushed the addjar branch from 48f93c2 to 0810e71 Compare April 2, 2015 08:17

remove variable substitution

0810e71

use sessionstate classloader in makeRDDforTable

9957d87

adrian-wang changed the title ~~[SPARK-5794] [SQL] [WIP] fix add jar~~ [SPARK-5794] [SQL] fix add jar Apr 8, 2015

adrian-wang added 2 commits April 8, 2015 00:23

fix style and add a test

32c4fb8

restrict hive version for test

6c707e8

adrian-wang force-pushed the addjar branch from f8a6a96 to 6c707e8 Compare April 8, 2015 07:36

marmbrus reviewed Apr 9, 2015
View reviewed changes

move jar to another place

efdd602

asfgit closed this in b45059d Apr 14, 2015

yhuai reviewed Apr 15, 2015
View reviewed changes

srowen mentioned this pull request Oct 16, 2018

[SPARK-25760][SQL] Set AddJarCommand return empty #22747

Closed

[SPARK-5794] [SQL] fix add jar #4586

[SPARK-5794] [SQL] fix add jar #4586

Uh oh!

Conversation

adrian-wang commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

adrian-wang commented Feb 13, 2015

Uh oh!

watermen commented Mar 2, 2015

Uh oh!

adrian-wang commented Mar 3, 2015

Uh oh!

watermen commented Mar 3, 2015

Uh oh!

adrian-wang commented Mar 3, 2015

Uh oh!

watermen commented Mar 3, 2015

Uh oh!

adrian-wang commented Mar 3, 2015

Uh oh!

adrian-wang commented Mar 3, 2015

Uh oh!

watermen commented Mar 3, 2015

Uh oh!

adrian-wang commented Mar 3, 2015

Uh oh!

adrian-wang commented Mar 3, 2015

Uh oh!

watermen commented Mar 3, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

watermen commented Mar 3, 2015

Uh oh!

watermen commented Mar 3, 2015

Uh oh!

adrian-wang commented Mar 5, 2015

Uh oh!

SparkQA commented Mar 5, 2015

Uh oh!

watermen commented Mar 5, 2015

Uh oh!

chenghao-intel Mar 5, 2015

Choose a reason for hiding this comment

Uh oh!

pzzs commented Mar 12, 2015

Uh oh!

the-sea commented Mar 13, 2015

Uh oh!

SparkQA commented Apr 2, 2015

Uh oh!

SparkQA commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 8, 2015

Uh oh!

adrian-wang commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 8, 2015

Uh oh!

adrian-wang commented Apr 8, 2015

Uh oh!

marmbrus Apr 9, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 9, 2015

Uh oh!

adrian-wang commented Apr 10, 2015

Uh oh!