Skip to content

[SPARK-19218][SQL] Fix SET command to show a result correctly and in a sorted order #16579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

[SPARK-19218][SQL] Fix SET command to show a result correctly and in a sorted order #16579

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c182c7d
[SPARK-19218][SQL] SET command should show a sorted result
dongjoon-hyun Jan 13, 2017
2ed6d87
Address comments.
dongjoon-hyun Jan 13, 2017
fa4914b
Revert to the approved commit and remove `set -v` test case to avoid …
dongjoon-hyun Jan 21, 2017
fb578d1
Add fix and recover `SET -v` test case.
dongjoon-hyun Jan 22, 2017
0214fad
Address comments
dongjoon-hyun Jan 22, 2017
528b0fd
Use Option.
dongjoon-hyun Jan 23, 2017
387ab59
Add `unregister` and use it.
dongjoon-hyun Jan 23, 2017
7061cd9
Remove redundant one.
dongjoon-hyun Jan 23, 2017
7879201
Add a missing clear() while splitting the testcases
dongjoon-hyun Jan 23, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,16 +79,17 @@ case class SetCommand(kv: Option[(String, Option[String])]) extends RunnableComm
// Queries all key-value pairs that are set in the SQLConf of the sparkSession.
case None =>
val runFunc = (sparkSession: SparkSession) => {
sparkSession.conf.getAll.map { case (k, v) => Row(k, v) }.toSeq
sparkSession.conf.getAll.toSeq.sorted.map { case (k, v) => Row(k, v) }
}
(keyValueOutput, runFunc)

// Queries all properties along with their default values and docs that are defined in the
// SQLConf of the sparkSession.
case Some(("-v", None)) =>
val runFunc = (sparkSession: SparkSession) => {
sparkSession.sessionState.conf.getAllDefinedConfs.map { case (key, defaultValue, doc) =>
Row(key, defaultValue, doc)
sparkSession.sessionState.conf.getAllDefinedConfs.sorted.map {
case (key, defaultValue, doc) =>
Row(key, Option(defaultValue).getOrElse("<undefined>"), doc)
}
}
val schema = StructType(
Expand Down
5 changes: 5 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ object SQLConf {
sqlConfEntries.put(entry.key, entry)
}

// For testing only
private[sql] def unregister(entry: ConfigEntry[_]): Unit = sqlConfEntries.synchronized {
sqlConfEntries.remove(entry.key)
}

private[sql] object SQLConfigBuilder {

def apply(key: String): ConfigBuilder = new ConfigBuilder(key).onCreate(register)
Expand Down
27 changes: 27 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -982,6 +982,33 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
spark.sessionState.conf.clear()
}

test("SPARK-19218 SET command should show a result in a sorted order") {
val overrideConfs = sql("SET").collect()
sql(s"SET test.key3=1")
sql(s"SET test.key2=2")
sql(s"SET test.key1=3")
val result = sql("SET").collect()
assert(result ===
(overrideConfs ++ Seq(
Row("test.key1", "3"),
Row("test.key2", "2"),
Row("test.key3", "1"))).sortBy(_.getString(0))
)
spark.sessionState.conf.clear()
}

test("SPARK-19218 `SET -v` should not fail with null value configuration") {
import SQLConf._
val confEntry = SQLConfigBuilder("spark.test").doc("doc").stringConf.createWithDefault(null)

try {
val result = sql("SET -v").collect()
assert(result === result.sortBy(_.getString(0)))
Copy link
Member

@viirya viirya Jan 23, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, i meant that you actually don't need a try {...} finally {...} here. you don't catch anything.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Jan 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see what you meant. Actually, previously, SET -v raises exceptions, so this case use try and catch. But, as you mentioned, now it's not.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Jan 23, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, IMO, it's needed to clean up spark.test if there occurs some regression for this case in the future. For the exceptions in that case, we needs that regression cause test cases failures, so catch is not used here.

Copy link
Member

@viirya viirya Jan 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but you don't catch anything actually? so if any regression in the future, is it different with a try or not? you still see an exception.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Jan 23, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but we need to clean up spark.test in order not to interfere the other test cases here.

Copy link
Member

@viirya viirya Jan 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is failed. isn't it??

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Jan 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:) The point is the other test cases are still running.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Jan 23, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, we are confused on terms.

  • You meant the other test statements.
  • I meant the other test cases

Copy link
Member

@viirya viirya Jan 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, i meant the final Jenkins test result is failed. nvm, i think it is still useful so we can better infer which test causes the failure if we don't interfere other tests.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Jan 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I understand. Thanks. :)

} finally {
Copy link
Member

@viirya viirya Jan 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: try ... finally seems redundant.

SQLConf.unregister(confEntry)
}
}

test("SET commands with illegal or inappropriate argument") {
spark.sessionState.conf.clear()
// Set negative mapred.reduce.tasks for automatically determining
Expand Down