Skip to content

[SPARK-18665][SQL] set statement state to "ERROR" after user cancel job #16099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

[SPARK-18665][SQL] set statement state to "ERROR" after user cancel job #16099

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e3a9b08
set statement state to error after user canceled job
cenyuhai Dec 1, 2016
196ab66
change statement state by the current state
cenyuhai Dec 2, 2016
ddb4128
fix bug when transition status from "CLOSED" to "ERROR"
cenyuhai Dec 6, 2016
501d121
fix bug when transition status from "CLOSED" to "FINISHED"
cenyuhai Dec 6, 2016
40894a1
fix error when job is canceled, but job is still running
cenyuhai Dec 6, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions ...rc/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,8 @@ private[hive] class SparkExecuteStatementOperation(
dataTypes = result.queryExecution.analyzed.output.map(_.dataType).toArray
} catch {
case e: HiveSQLException =>
HiveThriftServer2.listener.onStatementError(
statementId, e.getMessage, SparkUtils.exceptionString(e))
Copy link

@BruceXu1991 BruceXu1991 May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried your code. there seems have a bug.
e.getMessage may get a null value, which may cause NPE in ThriftServerPage.errorMessageCell(detail)

Copy link

@BruceXu1991 BruceXu1991 Feb 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR still occurs in Spark 2.2.1. A workaround is to use "String.valueOf(e.getMessage)" instead of "e.getMessage" to avoid of NPE problem.

Copy link
Contributor Author

@cenyuhai cenyuhai Feb 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, but My pr is closed by community...

Copy link
Contributor Author

@cenyuhai cenyuhai Feb 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please catch the exception in ThriftServerPage.errorMessageCell. Without my pr, that function still throw exception.

if (getStatus().getState() == OperationState.CANCELED) {
return
} else {
Expand All @@ -252,13 +254,35 @@ private[hive] class SparkExecuteStatementOperation(
case e: Throwable =>
val currentState = getStatus().getState()
logError(s"Error executing query, currentState $currentState, ", e)
setState(OperationState.ERROR)
HiveThriftServer2.listener.onStatementError(
statementId, e.getMessage, SparkUtils.exceptionString(e))
if (statementId != null) {
sqlContext.sparkContext.cancelJobGroup(statementId)
}
getStatus().getState() match {
case OperationState.INITIALIZED =>
setState(OperationState.CLOSED)
case OperationState.CANCELED =>
case OperationState.FINISHED =>
case OperationState.CLOSED =>
case OperationState.ERROR =>
case _ =>
setState(OperationState.ERROR)
}

throw new HiveSQLException(e.toString)
}
setState(OperationState.FINISHED)
HiveThriftServer2.listener.onStatementFinish(statementId)
getStatus().getState() match {
case OperationState.INITIALIZED =>
setState(OperationState.CLOSED)
case OperationState.CANCELED =>
case OperationState.FINISHED =>
case OperationState.CLOSED =>
case OperationState.ERROR =>
case _ =>
setState(OperationState.FINISHED)
}
}

override def cancel(): Unit = {
Expand Down