Documentation plan for Spark

[foxish]

As discussed in the SIG meeting earlier today, we need documentation coverage which is much more comprehensive to ensure that people have a good kick-the-tyres experience when using Spark on Kubernetes.
This will especially be a concern after our documentation change goes live.

Various broad categories of items that come to mind:

• High level overview of Kubernetes as a cluster scheduler backend for Spark
• Running a SparkJob from the beta release
  • Explaining various commandline options (mostly similar to this)
  • Cloud provider specific instructions
    • [ ] Firewall rules and setup on GKE, AWS etc (Discuss that GKE firewall should open the NodePort range to use Spark-on-K8s  #245)
    • GKE instructions
  • On-premise deployments
  • Setting up a shuffle service DaemonSet
  • Setting up a submission server
• Workflow
  • Building custom images for drivers and executors
  • Using the file submission server and local submissions
  • Using an external storage mechanism (File Server/S3/GCS/external HDFS cluster)
• Running a development build
  • Developer workflow
  • Integration tests
• Debugging a SparkJob
  • Using kubectl to identify memory, disk or other issues causing failure
  • Using the Spark UI and dashboard
• Best Practices and FAQ
• Reporting issues and submitting PRs

Thoughts on any items we've missed? I expect that each of the top-level items will be pages of their own.
cc @apache-spark-on-k8s/contributors

[tnachen]

I don't know if adding using the Spark UI is anything K8s specific, is this planning to be added into the Spark documentation or a standalone docs site somewhere?

[kimoonkim]

I don't know if adding using the Spark UI is anything K8s specific

Mostly no. But, in K8s, one needs to expose the driver UI port as nodePort using a config key.

[foxish]

Any thoughts on how we should present this documentation? IMO, the markdown files are getting pretty unwieldy, and we should either be generating html documentation from them, or creating html docs.
cc @ash211

[ash211]

@foxish probably can close this now that we have docs up on the separate repo and a fair amount of investment made in them?