[SPARK-5213] [SQL] Pluggable SQL Parser Support

[scwf]

based on #4015, we should not delete sqlParser from sqlcontext, that leads to mima failed. Users implement dialect to give a fallback for sqlParser and we should construct sqlParser in sqlcontext according to the dialect
protected[sql] val sqlParser = new SparkSQLParser(getSQLDialect().parse(_))

[SparkQA]

Test build #31518 has finished for PR 5827 at commit c19780b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class Dialect
  • class DialectException(msg: String, cause: Throwable) extends Exception(msg, cause)
• This patch does not change any dependencies.

[chenghao-intel]

Remove this comment? It's probably confused once this PR merged.

[scwf]

ok

[chenghao-intel]

Thank you @scwf for the fixing. :)

[scwf]

@chenghao-intel this sqlparser actually will not be used for now, place here just to fix mima test

[chenghao-intel]

I think we'd better keep it, not just for the mima test, but also for the sub class of Dialect. e.g. we have to specify the SparkSQLParser for HiveQLDialect.

[scwf]

agree to keep it, and in dialect parser we should not use SparkSQLParser. Dialect give a fallback(string -> logicalplan) and we call it in sqlParser

[scwf]

Retest this please

[SparkQA]

Test build #31566 has finished for PR 5827 at commit 0878bd1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class Dialect
  • class DialectException(msg: String, cause: Throwable) extends Exception(msg, cause)

[SparkQA]

Test build #31573 timed out for PR 5827 at commit 81b9737 after a configured wait of 150m.

[scwf]

retest this please

[SparkQA]

Test build #31605 has finished for PR 5827 at commit 81b9737.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class Dialect
  • class DialectException(msg: String, cause: Throwable) extends Exception(msg, cause)

[scwf]

@marmbrus any comment here?

[rxin]

@scwf are you guys using this feature? I'm thinking about just removing it in Spark 2.0.

@chenghao-intel who wanted it in the first place no longer needs it.

[scwf]

@rxin, yes we used this and we implements a new sqlparser based on this interface to support ANSI tpcds sql.

[rxin]

What's different from the one in Spark master now? It would be great to contribute the parser changes back now we have a full fledged parser in Spark, and going towards more ANSI compatibility is definitely on the roadmap.

[scwf]

@rxin Our parser is a extended version of the SqlParser, the main difference is that we add the support for subquery(both correlated and uncorrelated ),exists, in and some minor improvement such as grouping, top, cube/rollup. It support the tpcds generated ANSI sql syntax without any change.

I noticed that there are some PRs for these features, i will take a look at that PRs when i have time and see what i can do.

[rxin]

FYI we are going to remove this pluggability. It is extra overhead to maintain, and actually encourages projects to not contribute their improvements upstream, which is bad.

[scwf]

Actually we were trying to contribute this improvements, unfortunately the community do not want them for maintain(or compatibility with hive ql) reason in the past:).

I am glad that spark sql use a single parser such that people can make contributions and make it more and more powerful.

[rxin]

Yup thanks. That's why we are only removing it now :)