Ctrl+C on blocking spark-submit doesn't stop running job

[ash211]

The driver and executors continue to run even though the submit process is Ctrl+C'd. I think this is different from YARN behavior but would need to check.

[mccheah]

I would imagine this is because exceptions aren't thrown in this scenario but we handle cleanup in caught exceptions. We might need a shutdown hook to do the final clean up but it's difficult to reason about how we could get the shutdown hook to find what needs to be cleaned up so far. Spark uses shutdown hooks for a variety of cleanup tasks.

[ssuchter]

I just checked on a YARN cluster and interrupting (Ctrl-C) a spark-submit run does *not* interrupt the job - it keeps running to completion. This is expected behavior on YARN - the submission client is merely a submission client, *not* a controller. We could choose to mimic this behavior or not, but I’d advocate mimicing it. Not because we necessarily want compatibility with YARN behavior, but rather because I think it’s a good behavior. Once a job is successfully submitted to the cluster, it’s nice to not have the SPOF of the submission client anymore - think about jobs that may run for hours or days…

[mccheah]

What if the submission were to be cancelled partway through but the application hasn't started yet? Should we tear down everything that's been done so far? In this case there wouldn't be an application that's running on the driver, but the submission REST server would have started on the pod and that REST server would never terminate.

I get the impression that we should register a shutdown hook to tear down the driver pod after first creating the pod, but before we finish submitting the application. After we're done bootstrapping the application we should remove the shutdown hook so that it can run to completion even if the submission client exits.

[ssuchter]

That sounds ideal - we should either successfully submit the job or submit nothing (aka clean up after ourself). At least we should attempt to do so… I think that in the YARN case, this is effectively true. I think the way it’s done is that some resources may be delivered first into HDFS (e.g. shared stuff like jars) but in a place that will be garbage collected with age. (not sure of what mechanism does that.) Then the app master is requested of YARN in *one API call* so that it either works or doesn’t. Thus, there is little cleanup to do in any case. But if the K8s implementation doesn’t have such a sequence that can be interrupted, it’s kind of incumbent on us to clean up after ourselves.

[ash211]

Given that Ctrl+C on a blocking job doesn't kill the job in YARN cluster mode, it shouldn't kill the job in k8s cluster mode either. We also improved k8s resource cleanup in #122 which is relate to this.

Closing