Skip to content

[SPARK-3654][SQL] Implement all extended HiveQL statements/commands with a separate parser combinator #2590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 6 commits into from
Closed

[SPARK-3654][SQL] Implement all extended HiveQL statements/commands with a separate parser combinator #2590

wants to merge 6 commits into from

Conversation

@ravipesala
Copy link
Contributor

@ravipesala ravipesala commented Sep 30, 2014

Created separate parser for hql. It preparses the commands like cache,uncache,add jar etc.. and then parses with HiveQl

@ravipesala
Created seperate parser for hql.It pre parses the commands like
ba26cd1 
cache,uncache,add jar etc.. and then parses with HiveQl
@AmplabJenkins
Copy link

AmplabJenkins commented Sep 30, 2014

Can one of the admins verify this patch?

@liancheng
Copy link
Contributor

liancheng commented Sep 30, 2014

ok to test

liancheng
liancheng reviewed Sep 30, 2014
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSqlParser.scala Outdated
Copy link
Contributor

@liancheng liancheng Sep 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this class takes over all HiveQL parsing work (although it delegates to an underlying Hive parser), it's not accurate to call it a "pre parser", maybe this:

A parser that recognizes all HiveQL constructs together with several Spark SQL specific extensions like CACHE TABLE and UNCACHE TABLE.

Copy link
Contributor Author

@ravipesala ravipesala Sep 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I updated as per your comment

@SparkQA
Copy link

SparkQA commented Sep 30, 2014

QA tests have started for PR 2590 at commit ba26cd1.

  • This patch merges cleanly.

@liancheng
Copy link
Contributor

liancheng commented Sep 30, 2014

Awesome! I just started working on this last weekend and you've already got done :) Left some minor comments. This generally LGTM.

@SparkQA
Copy link

SparkQA commented Sep 30, 2014

QA tests have finished for PR 2590 at commit ba26cd1.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ravipesala
Copy link
Contributor Author

ravipesala commented Sep 30, 2014

Thank you. I updated the code as per your comments, Please review it.

liancheng
liancheng reviewed Sep 30, 2014
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/ExtendedHiveQlParser.scala Outdated
Copy link
Contributor

@liancheng liancheng Sep 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please follow this instruction to organize imports, and remove unused ones.

@liancheng
Copy link
Contributor

liancheng commented Sep 30, 2014

Left some more minor styling issues. This looks great :)

@ravipesala
Copy link
Contributor Author

ravipesala commented Sep 30, 2014

Thank you for your comments. I updated code as per your comments.Please review.

@liancheng
Copy link
Contributor

liancheng commented Sep 30, 2014

LGTM.

marmbrus
marmbrus reviewed Oct 1, 2014
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/ExtendedHiveQlParser.scala Outdated
Copy link
Contributor

@marmbrus marmbrus Oct 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove extra newline.

@ravipesala
Copy link
Contributor Author

ravipesala commented Oct 2, 2014

Fixed code as per comments, please review

@ravipesala
Copy link
Contributor Author

ravipesala commented Oct 2, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Oct 2, 2014

QA tests have started for PR 2590 at commit bbca7dd.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Oct 2, 2014

QA tests have finished for PR 2590 at commit bbca7dd.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • protected case class Keyword(str: String)

@marmbrus
Copy link
Contributor

marmbrus commented Oct 3, 2014

Thanks! I've merged this to master.

@asfgit asfgit closed this in 1c90347 Oct 3, 2014
@liancheng liancheng mentioned this pull request Oct 7, 2014
Closed
asfgit pushed a commit that referenced this pull request Oct 10, 2014
@liancheng @marmbrus
[SPARK-3654][SQL] Unifies SQL and HiveQL parsers
edf02da 
This PR is a follow up of #2590, and tries to introduce a top level SQL parser entry point for all SQL dialects supported by Spark SQL.

A top level parser `SparkSQLParser` is introduced to handle the syntaxes that all SQL dialects should recognize (e.g. `CACHE TABLE`, `UNCACHE TABLE` and `SET`, etc.). For all the syntaxes this parser doesn't recognize directly, it fallbacks to a specified function that tries to parse arbitrary input to a `LogicalPlan`. This function is typically another parser combinator like `SqlParser`. DDL syntaxes introduced in #2475 can be moved to here.

The `ExtendedHiveQlParser` now only handle Hive specific extensions.

Also took the chance to refactor/reformat `SqlParser` for better readability.

Author: Cheng Lian <[email protected]>

Closes #2698 from liancheng/gen-sql-parser and squashes the following commits:

ceada76 [Cheng Lian] Minor styling fixes
9738934 [Cheng Lian] Minor refactoring, removes optional trailing ";" in the parser
bb2ab12 [Cheng Lian] SET property value can be empty string
ce8860b [Cheng Lian] Passes test suites
e86968e [Cheng Lian] Removes debugging code
8bcace5 [Cheng Lian] Replaces digit.+ to rep1(digit) (Scala style checking doesn't like it)
d15d54f [Cheng Lian] Unifies SQL and HiveQL parsers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ravipesala @AmplabJenkins @liancheng @SparkQA @marmbrus