K8s support watch tpr #12
Conversation
| isObjectDeleted = true | ||
| logInfo("TPR Object deleted. Cleaning up") | ||
| stop() | ||
| case Success(_: WatchObject) => throw new SparkException("Unexpected response received") |
There was a problem hiding this comment.
I don't follow.
There was a problem hiding this comment.
Oh you mean to add a logInfo or logError? That can be added.
However, If you take a look at SparkJobResource.scala you'll see that the promise is completed with a WatchObject only when its deleted or with an exception when the source is exhausted.
That middle case where the WatchObject is returned and it's state is not deleted is just to exhaust the matches. Looking at it now, I think it should be safe to take it out.
There was a problem hiding this comment.
I was thinking at least include the WatchObject in the error message when you don't expect it so we can see what the unexpected value is
There was a problem hiding this comment.
Okay. Sounds good.
| logInfo("TPR Object deleted. Cleaning up") | ||
| stop() | ||
| case Success(_: WatchObject) => throw new SparkException("Unexpected response received") | ||
| case Failure(e: Throwable) => throw new SparkException(e.getMessage) |
There was a problem hiding this comment.
Yes. This call back is triggered when the source from the server is exhausted (failure case) or the resource being watched is deleted (success case).
On the success case, it stops the spark job by cleaning up the pods.
Pending question is: what is the right thing to do if the callback returns with a failure, right now I defaulted to throwing an error because this means the state of the SparkJob resource is now unknown and the watcher has stopped.
…ic + separating external deletion of resource
|
@iyanuobidele want to close this in favor of work on apache-spark-on-k8s#126? |
|
Sure. Sounds good. |
3 participants
What changes were proposed in this pull request?