[SPARK-18694][SS]Add StreamingQuery.explain and exception to Python and fix StreamingQueryException #16125
Conversation
python/pyspark/sql/streaming.py
Outdated
| :return: the StreamingQueryException if the query was terminated by an exception, or None. | ||
| >>> sq = sdf.writeStream.format('memory').queryName('query_exception').start() | ||
| >>> str(sq.exception()) |
There was a problem hiding this comment.
Convert None to a string 'None'. Otherwise, REPL prints nothing.
|
Test build #69596 has finished for PR 16125 at commit
|
|
Test build #69597 has finished for PR 16125 at commit
|
python/pyspark/sql/streaming.py
Outdated
| ... pass # Ignore the error as we want to test the "exception" method | ||
| >>> sq.exception() | ||
| StreamingQueryException() | ||
| >>> sq.stop() |
There was a problem hiding this comment.
I am not sure if such a big test should be present as doc tests. please add this to the sql/tests.py.
| val numRowsUpdated: Long) { | ||
|
|
||
| /** The compact JSON representation of this progress. */ | ||
| def json: String = compact(render(jsonValue)) |
There was a problem hiding this comment.
why are these needed? the outer StreamingQueryProgress already converts to json correctly. is this just for consistency with SourceProgress?
There was a problem hiding this comment.
is this just for consistency with SourceProgress?
Yes. Other progress classes have these json methods.
| "incorrect start offset on exception") | ||
| AssertOnQuery(q => { | ||
| q.exception.get.startOffset === | ||
| q.committedOffsets.toOffsetSeq(Seq(inputData), "{}").toString && |
There was a problem hiding this comment.
This line would conflict with my PR #16113 , you may have to fix issues after my PR gets merged.
|
Test build #69607 has finished for PR 16125 at commit
|
| endOffset: String) { | ||
| this( | ||
| // scalastyle:off | ||
| s"""${classOf[StreamingQueryException].getName}: ${cause.getMessage} ${cause.getStackTrace.take(10).mkString("", "\n|\t", "\n")} |
There was a problem hiding this comment.
The class name here allows capture_sql_exception to convert Java StreamingQueryException to Python StreamingQueryException.
|
Merging this to master and 2.1 |
|
Merged to master, but 2.1 failed due to mima exclude conflict. can you make another PR for 2.1 @zsxwing thanks! |
…and fix StreamingQueryException (branch 2.1) ## What changes were proposed in this pull request? Backport #16125 to branch 2.1. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes #16153 from zsxwing/SPARK-18694-2.1.
…and fix StreamingQueryException ## What changes were proposed in this pull request? - Add StreamingQuery.explain and exception to Python. - Fix StreamingQueryException to not expose `OffsetSeq`. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes apache#16125 from zsxwing/py-streaming-explain.
…and fix StreamingQueryException ## What changes were proposed in this pull request? - Add StreamingQuery.explain and exception to Python. - Fix StreamingQueryException to not expose `OffsetSeq`. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes apache#16125 from zsxwing/py-streaming-explain.
3 participants
What changes were proposed in this pull request?
OffsetSeq.How was this patch tested?
Jenkins