Spark shell [WIP]

[foxish]

With these changes, Spark shell "almost" works.

./bin/spark-submit --master=k8s://<apiserver-ip:port> --deploy-mode=cluster --class=org.apache.spark.repl.Main \
--conf spark.kubernetes.driver.docker.image=foxish/spark-driver:vtry11 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.1.0-kubernetes-0.1.0-rc1 \
--conf spark.driver.extraJavaOptions="-Dscala.usejavacp=true"

A kubectl attach to the driver pod shows:

2017-04-01 01:53:31 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://10.0.0.111:4040
Spark context available as 'sc' (master = k8s://https://35.184.141.248, app id = org-apache-spark-repl-main-1491011566260).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0-k8s-0.1.0-SNAPSHOT
      /_/
         
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.

scala> :quit
2017-04-01 01:53:49 INFO  KubernetesSparkRestServer$KubernetesSubmitRequestServlet:54 - Spark application complete. Shutting down submission server...
2017-04-01 01:53:49 INFO  ServerConnector:306 - Stopped ServerConnector@4f905baf{HTTP/1.1}{org-apache-spark-repl-main-1491011566260:7077}

I think a separate process actually forks off for spark-shell which is why the rest submission driver immediately stops and marks the driver as succeeded. To detect when the driver can stop running, we probably need to watch for subprocesses to exit as well.

[lins05]

I wonder if it would lead to confusion: spark repl is used with "client" deploy mode with all other cluster managers, while here we use it with "cluster" deploy mode. IMHO it's better to follow the convention.

Anyway, I think this patch is exciting!

[ash211]

this nop resource I think is the same concept as Spark already has in SparkLauncher#NO_RESOURCE

Seems like this bit of your PR could be slightly modified to address #213 ?

[ash211]

I agree with @lins05 that the strong expectation for Spark users running spark-shell is that the driver is running locally to their laptop and they'll be able to access local files on their laptop through the shell, e.g. with sc.textFile("/Users/aash/myfile.txt")

There are still situations where it's useful to have the shell running in cluster mode like this via the attach though (e.g. data is stored in a file/object store of some sort rather than on local disk) though.

Getting shell working seamlessly would be a big win for spark on k8s for sure.