[SPARK-2986] [SQL] fixed: setting properties does not effect

[guowei2]

it seems that set command does not run by SparkSQLDriver. it runs on hive api.
user can not change reduce number by setting spark.sql.shuffle.partitions

but i think setting hive properties seems just a role to spark sql.

[AmplabJenkins]

Can one of the admins verify this patch?

[marmbrus]

Why do you want use hive code for configuration? We have our own configuration that already mirrors into HiveConf.

[guowei2]

what i want is to use our own configuration. but it use hive code for configuration now.

[chenghao-intel]

+@liancheng, @marmbrus I think this is good catch.SparkSQLCLIDriver will resort to SetProcessor(of Hive) for all of the set command currently, but with this PR, it will goes into the SparkCliDriver, and eventually resort to SetCommand (of SparkSQL).

@guowei2 , can you also add a unit test for that?

[liancheng]

@guowei2 Are you sure set spark.sql.shuffle.partitions=n doesn't work? Would you mind to provide steps to reproduce this issue? 'Cause I just tried it with the most recent master branch (HiveThriftServer2 + beeline):

Stage 0 and stage 3 indicate that the set command works.

[guowei2]

@liancheng it is ok on (HiveThriftServer2 + beeline).
it doesn't work on bin/spark-sql. so does on my develop IDE with running SparkSQLCLIDriver

[liancheng]

Oh I see where the problem is. So in the current master branch, SET commands never get executed by Spark SQL since SetProcessor is not intercepted in SparkSQLCLIDriver and passed to Spark SQL. Thanks @guowei2 @chenghao-intel!

[liancheng]

test this please

[marmbrus]

test this please

[SparkQA]

QA tests have started for PR 1904. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18452/consoleFull

[SparkQA]

QA results for PR 1904:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18452/consoleFull

[marmbrus]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 1904. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18490/consoleFull

[SparkQA]

QA results for PR 1904:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18490/consoleFull

[marmbrus]

Thanks! I've merged this to master and 1.1