SparkJob scheduler for Kubernetes

[foxish]

extending thoughts in #133 (comment)

Currently, we schedule pods in k8s, but in the eventual state we want to get to, we should be able to reserve resources and schedule entire SparkJobs (where the resource requirement of the SparkJob is the sum of the minimum resources requested by its individual components). The first step to explore this idea would be to write a custom scheduler for SparkJobs which exhibits this particular behavior. The scheduler can fit in under the multi-scheduler paradigm.

[iyanuobidele]

I like where this is going. Luckily, I have a working prototype of this I wrote a couple months ago. It runs as a pod and essentially uses the special scheduler pod annotation to pick pods and assign them to bestfit (based on some heuristics) nodes. We can even customize this even further. If this is something we might like to explore, I'll talk to the right channels and see if it's something we can share with the group.

[iyanuobidele]

/cc @ccarrizo @khrisrichardson

[mccheah]

How would this work with dynamic allocation, where we would prefer that the job is as elastic as possible? I suppose in the dynamic allocation case we would like to be able to also scale the reserved amount. Or, in this case, are we only reserving the minimum resource requirement?