[SPARK-19558][sql] Add config key to register QueryExecutionListeners automatically. #19309
Conversation
… automatically. This change adds a new SQL config key that is equivalent to SparkContext's "spark.extraListeners", allowing users to register QueryExecutionListener instances through the Spark configuration system instead of having to explicitly do it in code. The code used by SparkContext to implement the feature was refactored into a helper method in the Utils class, and SQL's ExecutionListenerManager was modified to use it to initialize listener declared in the configuration. Unit tests were added to verify all the new functionality.
|
Test build #82041 has finished for PR 19309 at commit
|
squito
left a comment
squito
left a comment
There was a problem hiding this comment.
lgtm, a small question on the UnsupportedOperationException handling.
|
|
||
| case e: InvocationTargetException => | ||
| e.getCause() match { | ||
| case _: UnsupportedOperationException => |
There was a problem hiding this comment.
I am a bit worried catching UnsupportedOperationException is a bit too general ... what if its thrown by something else deep inside the extension class? The user won't get to see what happened.
OTOH, I can't think of anything better to catch (doesn't seem worth a special exception type). Maybe log the entire exception at debug level?
There was a problem hiding this comment.
I wanted to avoid adding a special new exception for this. I'll add the log.
|
Test build #82287 has finished for PR 19309 at commit
|
|
Will review it this weekend. |
|
|
||
| test("register query execution listeners using configuration") { | ||
| val conf = new SparkConf(false) | ||
| .set(QUERY_EXECUTION_LISTENERS, Seq(classOf[CountingQueryExecutionListener].getName())) |
There was a problem hiding this comment.
Now, this test only has one single listener whose constructor has zero arg. Could you add one more with a SparkConf arg in the constructor?
There was a problem hiding this comment.
That's covered by UtilsSuite already.
|
retest this please |
|
Let me summarize the PR. Please correct me if anything is missing. Background Currently, our users can register a listener one by one during the executions by calling the SparkContext API:
Users also can specify them through the Conf In Spark SQL, QueryExecutionListener enables users to register QueryExecutionListener by calling the API
Proposal This PR is proposing to add a similar static SQLConf |
|
That's basically what the PR summary says. |
|
Test build #82597 has finished for PR 19309 at commit
|
|
LGTM |
|
Thanks! Merged to master. |
4 participants
This change adds a new SQL config key that is equivalent to SparkContext's
"spark.extraListeners", allowing users to register QueryExecutionListener
instances through the Spark configuration system instead of having to
explicitly do it in code.
The code used by SparkContext to implement the feature was refactored into
a helper method in the Utils class, and SQL's ExecutionListenerManager was
modified to use it to initialize listener declared in the configuration.
Unit tests were added to verify all the new functionality.