[SPARK-2234][SQL]Spark SQL basicOperators add Except operator #1151

YanjieGao · 2014-06-20T07:57:52Z

Hi all,
I want to submit a Except operator in basicOperators.scala
In SQL case.SQL support two table do except operator.
select * from table1
except
select * from table2
This operator support the substract function .Return an table with the elements from this that are not in other.This operator should limit the input SparkPlan Seq only has two member.The check will later support
JIRA:https://issues.apache.org/jira/browse/SPARK-2234

Reformat the code sytle as intent 4

Hi all, I want to submit a Except operator in basicOperators.scala In SQL case.SQL support two table do except operator. select * from table1 except select * from table2 This operator support the substract function .Return an table with the elements from `this` that are not in `other`.

AmplabJenkins · 2014-06-20T07:59:58Z

Can one of the admins verify this patch?

Reformat the name and anotation.Thanks rxin .If there are other code style problems ,i will try to reformat it .

concretevitamin · 2014-06-20T22:19:04Z

For this to take effects, the parser (SQLParser) needs to be changed to be able to parse EXCEPT & return the correct operators.

YanjieGao · 2014-06-21T14:01:18Z

Hi Zongheng, I try it ,and try add code like other operator. I don't know if i want to add this except operator ,do i need to add code or modify code in other scala files ? Thanks a lot

marmbrus · 2014-06-21T18:54:34Z

sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala

Limit lines to 100 chars.

marmbrus · 2014-06-21T19:00:59Z

Thanks for working on this!

A few remaining tasks:

Add a new logical operator in basicOperators.scala in catalyst/....
Hook that new logical operator into both parsers HiveQl and SqlParser.
Add a case in the BasicOperators Strategy to convert the logical version into the physical version.
Address the review comments.
Add a few tests in SQLQuerySuite
See if there are any new hive tests that we can whitelist in HiveCompatibilitySuite otherwise add a test in HiveQuerySuite.

Let me know if you have any questions :)

YanjieGao · 2014-06-22T00:48:17Z

Thanks a lot , It's very nice of you .I will work around it .And then add code in the other files .I have some problems about some syntax .I have sent a mail to you .Thanks a lot!

1delete sc 2change args to left right 3annotation limit less than 100 charactors

YanjieGao · 2014-06-23T01:57:27Z

Hi marmbrus
I update these files as your comment tips ,but i think i may make some mistakes in the code .Could you help me and give me some tips ?I will continue to work around it and debug it to make it better
Thanks a lot !

adrian-wang · 2014-06-23T04:21:19Z

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

Maybe you missed a '(' here

Thanks ,I have correct it

Subtract is conflict with Subtract(e1,e2) in this file so i change the name except in basic operator i rename it Subtract

AmplabJenkins · 2014-07-03T01:30:55Z

Merged build started.

AmplabJenkins · 2014-07-03T03:31:58Z

Merged build finished.

AmplabJenkins · 2014-07-03T03:31:59Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16301/

YanjieGao · 2014-07-03T03:52:17Z

Thanks a lot, Michael,I have modify the code . Merge build use two hours .But I saw the console test log error. I don't know if the new code is the main cause or the code could be merge?

liancheng · 2014-07-03T18:23:56Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala

Please undo all the indentation changes to these lines, and add a space before all }s.

AmplabJenkins · 2014-07-04T01:20:51Z

Merged build triggered.

AmplabJenkins · 2014-07-04T01:21:00Z

Merged build started.

AmplabJenkins · 2014-07-04T01:30:50Z

Merged build triggered.

AmplabJenkins · 2014-07-04T01:30:59Z

Merged build started.

YanjieGao · 2014-07-04T01:31:33Z

Thanks a lot all ,I have modify the code as your suggestion ,next time i will match the Spark coding style

AmplabJenkins · 2014-07-04T01:41:04Z

Merged build finished.

AmplabJenkins · 2014-07-04T01:41:05Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16332/

AmplabJenkins · 2014-07-04T03:03:09Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-04T03:03:09Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16331/

YanjieGao · 2014-07-04T03:04:39Z

Hi all , This pr all test has passed.

marmbrus · 2014-07-04T09:46:27Z

Thanks! I've merged this into master. We'll back port it into 1.0.X after the release is made.

YanjieGao · 2014-07-04T09:50:36Z

Hi Michael,I also have a similar pr #1150 [SPARK-2235][SQL]Spark SQL basicOperator add Intersect operator ,Could you help me to review it? Thanks !

Hi all, I want to submit a Except operator in basicOperators.scala In SQL case.SQL support two table do except operator. select * from table1 except select * from table2 This operator support the substract function .Return an table with the elements from `this` that are not in `other`.This operator should limit the input SparkPlan Seq only has two member.The check will later support JIRA:https://issues.apache.org/jira/browse/SPARK-2234 Author: Yanjie Gao <[email protected]> Author: YanjieGao <[email protected]> Author: root <root@node4.(none)> Author: gaoyanjie <[email protected]> Closes apache#1151 from YanjieGao/patch-6 and squashes the following commits: f19f899 [YanjieGao] add a new blank line in basicoperators.scala 2ff7d73 [YanjieGao] resolve the identation in SqlParser and SparkStrategies fdb5227 [YanjieGao] Merge remote branch 'upstream/master' into patch-6 9940d19 [YanjieGao] make comment less than 100c 09c7413 [YanjieGao] pr 1151 SqlParser add cache ,basic Operator rename Except and modify comment b4b5867 [root] Merge remote branch 'upstream/master' into patch-6 b4c3869 [Yanjie Gao] change SparkStrategies Sparkcontext to SqlContext 7e0ec29 [Yanjie Gao] delete multi test 7e7c83f [Yanjie Gao] delete conflict except b01beb8 [YanjieGao] resolve conflict sparkstrategies and basicOperators 4dc8166 [YanjieGao] resolve conflict fa68a98 [Yanjie Gao] Update joins.scala 8e6bb00 [Yanjie Gao] delete conflict except dd9ba5e [Yanjie Gao] Update joins.scala a0d4e73 [Yanjie Gao] delete skew join 60f5ddd [Yanjie Gao] update less than 100c 0e72233 [Yanjie Gao] update SQLQuerySuite on master branch 7f916b5 [Yanjie Gao] update execution/basicOperators on master branch a28dece [Yanjie Gao] Update logical/basicOperators on master branch a639935 [Yanjie Gao] Update SparkStrategies.scala 3bf7def [Yanjie Gao] update SqlParser on master branch 26f833f [Yanjie Gao] update SparkStrategies.scala on master branch 8dd063f [Yanjie Gao] Update logical/basicOperators on master branch 9847dcf [Yanjie Gao] update SqlParser on masterbranch d6a4604 [Yanjie Gao] Update joins.scala 424c507 [Yanjie Gao] Update joins.scala 7680742 [Yanjie Gao] Update SqlParser.scala a7193d8 [gaoyanjie] [SPARK-2234][SQL]Spark SQL basicOperators add Except operator apache#1151 5c8a224 [Yanjie Gao] update the line less than 100c ee066b3 [Yanjie Gao] Update basicOperators.scala 32a80ab [Yanjie Gao] remove except in HiveQl cf232eb [Yanjie Gao] update 1comment 2space3 left.out f1ea3f3 [Yanjie Gao] remove comment 7ea9b91 [Yanjie Gao] remove annotation 7f3d613 [Yanjie Gao] update .map(_.copy()) 670a1bb [Yanjie Gao] Update HiveQl.scala 3fe7746 [Yanjie Gao] Update SQLQuerySuite.scala a36eb0a [Yanjie Gao] Update basicOperators.scala 7859e56 [Yanjie Gao] Update SparkStrategies.scala 052346d [Yanjie Gao] Subtract is conflict with Subtract(e1,e2) aab3785 [Yanjie Gao] Update SQLQuerySuite.scala 4bf80b1 [Yanjie Gao] update subtract to except 4bdd520 [Yanjie Gao] Update SqlParser.scala 2d4bfbd [Yanjie Gao] Update SQLQuerySuite.scala 0808921 [Yanjie Gao] SQLQuerySuite a8a1948 [Yanjie Gao] SparkStrategies 1fe96c0 [Yanjie Gao] HiveQl.scala update 3305e40 [Yanjie Gao] SqlParser 7a98c37 [Yanjie Gao] Update basicOperators.scala cf5b9d0 [Yanjie Gao] Update basicOperators.scala 8945835 [Yanjie Gao] object SkewJoin extends Strategy 2b98962 [Yanjie Gao] Update SqlParser.scala dd32980 [Yanjie Gao] update1 68815b2 [Yanjie Gao] Reformat the code style 4eb43ec [Yanjie Gao] Update basicOperators.scala aa06072 [Yanjie Gao] Reformat the code sytle

* [CARMEL-6392] Log cancelled query * Fix operation event bug * [CARMEL-6392] Log cancelled query

YanjieGao added 2 commits June 20, 2014 13:15

Reformat the code sytle

aa06072

Reformat the code sytle as intent 4

YanjieGao changed the title ~~Spark SQL basicOperators add Ecept operator~~ Spark SQL basicOperators add Except operator Jun 20, 2014

Reformat the code style

68815b2

Reformat the name and anotation.Thanks rxin .If there are other code style problems ,i will try to reformat it .

marmbrus reviewed Jun 21, 2014
View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala Outdated

Copy link

Contributor

marmbrus Jun 21, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Limit lines to 100 chars.

update1

dd32980

1delete sc 2change args to left right 3annotation limit less than 100 charactors

YanjieGao changed the title ~~Spark SQL basicOperators add Except operator~~ [SPARK-2234][SQL]Spark SQL basicOperators add Except operator Jun 22, 2014

YanjieGao added 9 commits June 22, 2014 23:16

Update SqlParser.scala

2b98962

object SkewJoin extends Strategy

8945835

Update basicOperators.scala

cf5b9d0

Update basicOperators.scala

7a98c37

SqlParser

3305e40

HiveQl.scala update

1fe96c0

SparkStrategies

a8a1948

SQLQuerySuite

0808921

Update SQLQuerySuite.scala

2d4bfbd

Update SqlParser.scala

4bdd520

adrian-wang reviewed Jun 23, 2014
View reviewed changes

YanjieGao added 4 commits June 23, 2014 13:20

update subtract to except

4bf80b1

Update SQLQuerySuite.scala

aab3785

Subtract is conflict with Subtract(e1,e2)

052346d

Subtract is conflict with Subtract(e1,e2) in this file so i change the name except in basic operator i rename it Subtract

Update SparkStrategies.scala

7859e56

make comment less than 100c

9940d19

liancheng reviewed Jul 3, 2014
View reviewed changes

Merge remote branch 'upstream/master' into patch-6

fdb5227

resolve the identation in SqlParser and SparkStrategies

2ff7d73

add a new blank line in basicoperators.scala

f19f899

asfgit closed this in 5dadda8 Jul 4, 2014

wangyum pushed a commit that referenced this pull request May 26, 2023

[CARMEL-6392] Bug fix for hive operation event (#1151)

e3898e8

* [CARMEL-6392] Log cancelled query * Fix operation event bug * [CARMEL-6392] Log cancelled query

[SPARK-2234][SQL]Spark SQL basicOperators add Except operator #1151

[SPARK-2234][SQL]Spark SQL basicOperators add Except operator #1151

Uh oh!

Conversation

YanjieGao commented Jun 20, 2014

Uh oh!

AmplabJenkins commented Jun 20, 2014

Uh oh!

concretevitamin commented Jun 20, 2014

Uh oh!

YanjieGao commented Jun 21, 2014

Uh oh!

marmbrus Jun 21, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Jun 21, 2014

Uh oh!

YanjieGao commented Jun 22, 2014

Uh oh!

YanjieGao commented Jun 23, 2014

Uh oh!

adrian-wang Jun 23, 2014

Choose a reason for hiding this comment

Uh oh!

YanjieGao Jun 23, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

YanjieGao commented Jul 3, 2014

Uh oh!

liancheng Jul 3, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

YanjieGao commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

YanjieGao commented Jul 4, 2014

Uh oh!

marmbrus commented Jul 4, 2014

Uh oh!

YanjieGao commented Jul 4, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants