Flag for preserving driver pods that failed to start

[ash211]

Brainstorming some flag names for that:

--preserve-pods
--keep-driver-pod
--preserve-driver-pod

Would we want this flag to keep the pod around even if it launched and finished its job successfully? Maybe some folks will want to look at logs afterwards for debugging, though kubectl logs might preserve those anyway so no need to keep successful pods just for logs.

See discussion at #11 (comment)

[foxish]

Maybe the right thing to do is to not cleanup pods at all. The biggest issue is that we cannot reuse the app name unless we delete the succeeded/failed driver pod, but that can be solved simply by using an auto-generated suffix + app-name as the driver pod name. The successful and failed pods should get eventually cleaned up by the pod garbage collector, and keeping them around ensures we always have access to the logs. I think currently we trigger pod garbage collection at 12500 successful/failed pods.

[foxish]

The auto-generated suffix should have the submission timestamp, so that we can tell drivers apart. kubectl get pods -a shows the entire list of pods which are in failed/succeeded state. So, the action items if we go with this will be:

1. At the end of the spark job, exit driver container.
2. Clean up secret and service.

This would also make kubernetes/kubernetes#22680 important to complete in order to prevent DOSing small clusters.

[foxish]

@ash211 can we close this? We do not delete completed pods as of now, and that seems like the right thing to do as well.

[ash211]

@foxish we do delete executor pods, but leave driver pods as completed. This makes difficult for me to read logs from the deleted executor pods when debugging (to get the name since it doesn't show even with kubectl get pods). Seems like we should either delete both driver/executor pods or leave them both as completed.

Or can logs be read from deleted pods and we just need to make getting executor pod names easier?

[foxish]

@ash211 logs cannot be accessed from pods after they are deleted. They can be accessed from completed pods (ending in state succeeded/failed).

In my opinion, since we need access to stdout, the driver pods should always be preserved (in succeeded/failed state), till they are eventually cleaned up by the GC. If the executor logs are important for debugging, we should introduce a flag to not delete them in specific at the end of the SparkJob. Does that make sense?

[ash211]

Filed changing the way we stop executor pods as #125

This issue for leaving the driver pod around was filed quite a bit ago and I don't think is relevant anymore now that we are able to get logs with kubectl logs <pod-name> which was only introduced in #13

Closing