[WIP] Fix `SET -v` not to raise exceptions for configs with default value `null` #16624

dongjoon-hyun · 2017-01-17T21:17:09Z

What changes were proposed in this pull request?

This is to investigate the current SET -v behavior.

How was this patch tested?

Add two test cases.

dongjoon-hyun · 2017-01-17T21:35:37Z

This is just to see the isolated result of SET -v. After resolving the failure, this will be close and be merged into #16579 .

SparkQA · 2017-01-17T21:36:51Z

Test build #71530 has finished for PR 16624 at commit 60953bf.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-01-17T21:38:33Z

Retest this please.

SparkQA · 2017-01-17T23:16:27Z

Test build #71532 has finished for PR 16624 at commit 60953bf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-01-17T23:18:54Z

Retest this please

SparkQA · 2017-01-18T01:09:31Z

Test build #71539 has finished for PR 16624 at commit 60953bf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-01-18T01:16:05Z

Finally, we get the expected failures. (The first MiMa failure and the second HiveSparkSubmitSuite was just flaky failures.)

 org.apache.spark.sql.SQLQuerySuite.`SET -v` collect test
 org.apache.spark.sql.SQLQuerySuite.`SET -v` key collect test

Hi, @gatorsmile and @srowen .

The current set -v command has a problem as we can see in these test failures. I will fix this first here.

gatorsmile · 2017-01-18T02:51:42Z

Sure, please fix it. Thanks!

SparkQA · 2017-01-18T07:37:03Z

Test build #71571 has finished for PR 16624 at commit 87de8da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-18T09:54:21Z

Test build #71582 has finished for PR 16624 at commit 7b7308d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-18T17:58:58Z

Test build #71608 has finished for PR 16624 at commit 4cb51e0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-18T21:50:34Z

Test build #71615 has finished for PR 16624 at commit 0ad4880.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-21T21:39:51Z

Test build #71774 has finished for PR 16624 at commit 047a1f2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-22T00:29:31Z

Test build #71776 has finished for PR 16624 at commit 3513e59.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-22T00:53:05Z

Test build #71778 has finished for PR 16624 at commit 042f025.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-22T01:22:54Z

Test build #71780 has finished for PR 16624 at commit 075f466.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-01-22T02:23:43Z

Retest this please

dongjoon-hyun · 2017-01-22T02:28:58Z

The final failure, HiveSparkSubmitSuite.dir, is irrelevant to this PR.

SparkQA · 2017-01-22T04:24:12Z

Test build #71791 has finished for PR 16624 at commit 075f466.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-01-22T05:44:36Z

Please update the PR description.

dongjoon-hyun · 2017-01-22T06:33:23Z

Hi, @gatorsmile .
I tested here and applied to #16579 back.
PR #16579 has two fixes. After merging #16579 , I'm going to close this one.

…a sorted order ## What changes were proposed in this pull request? This PR aims to fix the following two things. 1. `sql("SET -v").collect()` or `sql("SET -v").show()` raises the following exceptions for String configuration with default value, `null`. For the test, please see [Jenkins result](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71539/testReport/) and 60953bf in #16624 . ``` sbt.ForkMain$ForkError: java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException createexternalrow(input[0, string, false].toString, input[1, string, false].toString, input[2, string, false].toString, StructField(key,StringType,false), StructField(value,StringType,false), StructField(meaning,StringType,false)) :- input[0, string, false].toString : +- input[0, string, false] :- input[1, string, false].toString : +- input[1, string, false] +- input[2, string, false].toString +- input[2, string, false] ``` 2. Currently, `SET` and `SET -v` commands show unsorted result. We had better show a sorted result for UX. Also, this is compatible with Hive. **BEFORE** ``` scala> sql("set").show(false) ... |spark.driver.host |10.22.16.140 | |spark.driver.port |63893 | |spark.repl.class.uri |spark://10.22.16.140:63893/classes | ... |spark.app.name |Spark shell | |spark.driver.memory |4G | |spark.executor.id |driver | |spark.submit.deployMode |client | |spark.master |local[*] | |spark.home |/Users/dhyun/spark | |spark.sql.catalogImplementation|hive | |spark.app.id |local-1484333618945 | ``` **AFTER** ``` scala> sql("set").show(false) ... |spark.app.id |local-1484333925649 | |spark.app.name |Spark shell | |spark.driver.host |10.22.16.140 | |spark.driver.memory |4G | |spark.driver.port |64994 | |spark.executor.id |driver | |spark.jars | | |spark.master |local[*] | |spark.repl.class.uri |spark://10.22.16.140:64994/classes | |spark.sql.catalogImplementation|hive | |spark.submit.deployMode |client | ``` ## How was this patch tested? Jenkins with a new test case. Author: Dongjoon Hyun <[email protected]> Closes #16579 from dongjoon-hyun/SPARK-19218.

…a sorted order ## What changes were proposed in this pull request? This PR aims to fix the following two things. 1. `sql("SET -v").collect()` or `sql("SET -v").show()` raises the following exceptions for String configuration with default value, `null`. For the test, please see [Jenkins result](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71539/testReport/) and apache@60953bf in apache#16624 . ``` sbt.ForkMain$ForkError: java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException createexternalrow(input[0, string, false].toString, input[1, string, false].toString, input[2, string, false].toString, StructField(key,StringType,false), StructField(value,StringType,false), StructField(meaning,StringType,false)) :- input[0, string, false].toString : +- input[0, string, false] :- input[1, string, false].toString : +- input[1, string, false] +- input[2, string, false].toString +- input[2, string, false] ``` 2. Currently, `SET` and `SET -v` commands show unsorted result. We had better show a sorted result for UX. Also, this is compatible with Hive. **BEFORE** ``` scala> sql("set").show(false) ... |spark.driver.host |10.22.16.140 | |spark.driver.port |63893 | |spark.repl.class.uri |spark://10.22.16.140:63893/classes | ... |spark.app.name |Spark shell | |spark.driver.memory |4G | |spark.executor.id |driver | |spark.submit.deployMode |client | |spark.master |local[*] | |spark.home |/Users/dhyun/spark | |spark.sql.catalogImplementation|hive | |spark.app.id |local-1484333618945 | ``` **AFTER** ``` scala> sql("set").show(false) ... |spark.app.id |local-1484333925649 | |spark.app.name |Spark shell | |spark.driver.host |10.22.16.140 | |spark.driver.memory |4G | |spark.driver.port |64994 | |spark.executor.id |driver | |spark.jars | | |spark.master |local[*] | |spark.repl.class.uri |spark://10.22.16.140:64994/classes | |spark.sql.catalogImplementation|hive | |spark.submit.deployMode |client | ``` ## How was this patch tested? Jenkins with a new test case. Author: Dongjoon Hyun <[email protected]> Closes apache#16579 from dongjoon-hyun/SPARK-19218.

dongjoon-hyun mentioned this pull request Jan 21, 2017

[SPARK-19218][SQL] Fix SET command to show a result correctly and in a sorted order #16579

Closed

dongjoon-hyun added 2 commits January 21, 2017 11:04

Add a test cases for SET -v.

94b5ac8

add show test.

047a1f2

dongjoon-hyun added 3 commits January 21, 2017 14:44

fix

3513e59

Add reset to prevent other settings.

042f025

Fix all.

075f466

dongjoon-hyun changed the title ~~[WIP] Add two test cases for SET -v.~~ [WIP] Fix SET -v not to raise exceptions for configs with default value null Jan 22, 2017

dongjoon-hyun closed this Jan 23, 2017

dongjoon-hyun deleted the setv branch March 5, 2017 19:25

[WIP] Fix SET -v not to raise exceptions for configs with default value null #16624

[WIP] Fix SET -v not to raise exceptions for configs with default value null #16624

Uh oh!

Conversation

dongjoon-hyun commented Jan 17, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Jan 17, 2017

Uh oh!

SparkQA commented Jan 17, 2017

Uh oh!

dongjoon-hyun commented Jan 17, 2017

Uh oh!

SparkQA commented Jan 17, 2017

Uh oh!

dongjoon-hyun commented Jan 17, 2017

Uh oh!

SparkQA commented Jan 18, 2017

Uh oh!

dongjoon-hyun commented Jan 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 18, 2017

Uh oh!

SparkQA commented Jan 21, 2017

Uh oh!

SparkQA commented Jan 22, 2017

Uh oh!

SparkQA commented Jan 22, 2017

Uh oh!

SparkQA commented Jan 22, 2017

Uh oh!

dongjoon-hyun commented Jan 22, 2017

Uh oh!

dongjoon-hyun commented Jan 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Jan 22, 2017

Uh oh!

gatorsmile commented Jan 22, 2017

Uh oh!

dongjoon-hyun commented Jan 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[WIP] Fix `SET -v` not to raise exceptions for configs with default value `null` #16624

[WIP] Fix `SET -v` not to raise exceptions for configs with default value `null` #16624

dongjoon-hyun commented Jan 18, 2017 •

edited

Loading

dongjoon-hyun commented Jan 22, 2017 •

edited

Loading

dongjoon-hyun commented Jan 22, 2017 •

edited

Loading