Brief build and usage docs for PR #1

[erikerlandson]

A readme file, maybe under the kubernetes subdir, with brief description for building with the kube package, building images, spark-submit options and configuration settings.

[erikerlandson]

cc/ @mccheah

[ash211]

Love this idea -- the docs will show us how complicated the setup is and help us start cutting down difficulty for new folks to onboard and try this out.

@mccheah want to take a stab?

Oh, and I think this will work best if we have PR #1 be a mirror of the one we intend to send upstream, and most other PRs build on top of that one (such as the tar+compress PR in #2).

So for this docs addition, let's open a new PR into the branch from #1 so we can review the first draft of the docs first.

[mccheah]

We can include instructions for the basic spark-submit usage from the command line. Documenting how to execute the integration tests should probably wait until we have a better idea of how we're going to approach that in general.

[foxish]

We had something like this in our prototype in https://github.com/foxish/spark/blob/k8s-support/resource-managers/kubernetes/README.md. I guess something similar to that might do.

[kimoonkim]

Hi, I am an engineer from Pepperdata.

We have an Ansible playbook code that installs a snapshot package built from https://github.com/foxish. This includes filling out spark-defaults.conf with common options so our spark-submit command lines are shorter. We install the package on our developer Macbooks and lets developers launch spark jobs on our kubernetes cluster.

Do you guys think this could be useful? If yes, I can clean up the Ansible code a bit and send a pull request. If Ansible playbook is too much, I can also translate it back into manual steps and add them to the readme.

[ash211]

Thanks @kimoonkim ! Exciting to see that you've been starting to put some usage on this codebase.

I think it'd definitely be helpful to see the Ansible code as a good show of what the steps are to get this set up. I don't know if the script will end up making it into the eventual Apache release (that's more their decision) but laying the steps out will be good for everyone to see regardless. I find that enumerating install instructions is the best way to start trimming down complexity of install so this makes sense.

Can you please send a PR into the k8s-support-alternate-incremental branch from PR 1, which is forming the basis for what will go to Apache?

Welcome to the efforts to get Spark on Kubernetes!

[kimoonkim]

Thanks @ash211 for the encouraging words.

I am happy to find this project gaining momentum and I already enjoy running Spark jobs on our k8s cluster using this code base. Thanks everyone for putting this all together!

I'll clean up the Ansible code a bit and send a PR to the k8s-support-alternate-incremental branch.

[erikerlandson]

@kimoonkim I was thinking it might be cleanest to have a separate repo on this org, for the ansible playbooks. I'll create one, and they can live there. I have a feeling it will be hard to make them an "official" place to live in the spark repo proper.

[erikerlandson]

@kimoonkim @ash211

I have a feeling it might be awkward finding a place for ansible playbooks to live in the main Spark repo. But it will be cool to have a place for them to live, so I created a new repo on the organization for them:

https://github.com/apache-spark-on-k8s/ansible

You should have push privs for this.

[erikerlandson]

It occurs to me we might do the same thing for docker-files and such. Create a docker repo for them. Does that seem like a plausible idea?

[mccheah]

I envisioned the docker files to live in the main repo, mainly because I'd expect a common dev workflow to be to check out Spark from github and try building the Docker files off of the local git repository. We might be able to configure a Maven goal to do this, for example.

[erikerlandson]

That still seems sensible to me, since the docker images are built from the spark build. We got some push-back on the docker images, so I've been thinking about a possible plan-B.

[kimoonkim]

@erikerlandson Yes. Separate repo for Ansible playbooks sounds good to me. I saw your invite. Thanks for creating the repo.