Spark on k8s to beta

[foxish]

This issue tracks all the items we need to do to get Spark support to beta, at which point we can start encouraging people to use it.
I'm putting together the items from the design doc. We can break these down further if there is a need.

Phase One: Static Allocation MVP (This is the phase to provide the MVP. It will have significant feature and security gaps but after completion should allow running a Spark job in k8s for a narrow use case.)

• Spark-submit support for cluster mode
• Static number of executors
• Only Java + Scala support
• Providing user code from both the client’s local disk and remote locations
• Basic unit/integration testing
• Documentation

== Alpha Release ==

Phase Two: Dynamic Allocation

• Shuffle Service Finalization
• Dynamic allocation support
• External shuffle service prototypes, both with the sidecar approach and the daemon set approach. Assess the two implementations, and decide between them.
• Resource staging server for hosting local files

Phase Three: Complete Core Spark Features

• Use K8s secrets to secure external shuffle service communication
• “Decent security” for data processed in Spark
• Hooks in the scheduler back-end for the kube layer to request scale-up or scale-down aka “custom controller support” (do we still needs this?)
• Shuffle data protected at rest from neighbor processes

Beyond Beta

Phase Four: Future K8s Features

• Job Management UI (similar to YARN’s ResourceManager scheduler view)
• Support for remaining language bindings (Python, R)
• Integration with k8s Third Party Resources
• Isolation (malicious jobs can’t DOS neighbor jobs)
• Fair sharing / queueing mechanism
• Protection against disk exhaustion
• Protection against deadlock with all cluster resources running drivers and none running executors
• Spark-shell / client mode

[ash211]

I believe we're all aiming to make this PR cover the entirety of phase one: #1 Can we focus our code review efforts on that PR?

[luck02]

Can someone explain what this means: Support for remaining language bindings (Python, R)

Are we missing any python support currently? Any docs I can read? I'm just in the process of attempting to submit a simple-ish python job and if that's just not gonna work it would help to know :D
--- edit and
I guess the answer is yes. I'm reading the design doc to try why that is. On the surface it seems a bit odd as I would have imagined k8s and spark languages wouldn't interact meaningfully.

[mccheah]

We have python support now, but it would be wise for us to do some more rigorous testing on it. We do not have support for R.

Edit: To get Python support you'll need to be working from the head of the branch-2.1-kubernetes branch. It's not in a previous release.

[ifilonenko]

@luck02 I will make sure to write-up more documentation if it wasn't clear, but you run a PySpark job this way: https://github.com/apache-spark-on-k8s/spark/blob/branch-2.1-kubernetes/docs/running-on-kubernetes.md#running-pyspark

[ash211]

@foxish probably can close this now that the beta is out?