[SPARK-23529][K8s] Support mounting hostPath volumes for executors

[madanadit]

What changes were proposed in this pull request?

This PR introduces a new config spark.kubernetes.executor.volumes taking a values of the form hostPath:containerPath[:ro|rw][,...]; where hostPath is the path for the executor pod volume, containerPath is the mount path and ro is read-only mode.

The use case is to enable short-circuit writes to distributed storage on k8s. The Alluxio File System uses domain sockets to enable short-circuit writes from the client to worker memory when co-located on the same host machine. A directory, lets say /tmp/domain on the host, is mounted on the Alluxio worker container as well as the Alluxio client ( = Spark executor) container. The worker creates a domain socket /tmp/domain/d and if the client container mounts the same directory, it can write directly to the Alluxio worker w/o passing through network stack. The end result is faster data access when data is local.

How was this patch tested?

Manual testing on a k8s v1.8 cluster. Unit tests added to ExecutorPodFactorySuite.

[madanadit]

@foxish Can you take a look?. Thanks!

[foxish]

ok to test

[SparkQA]

Test build #88960 has finished for PR 20989 at commit 17cfdf3.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1981/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1981/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2014/

[SparkQA]

Test build #89000 has finished for PR 20989 at commit cb789ff.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2014/

[SparkQA]

Test build #89003 has finished for PR 20989 at commit d9f46d3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2016/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2016/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2017/

[SparkQA]

Test build #89004 has finished for PR 20989 at commit 3d8858a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• public class SchemaColumnConvertNotSupportedException extends RuntimeException
• class QueryExecutionException(message: String, cause: Throwable = null)

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2017/

[madanadit]

@foxish The failure seems to be because of SparkRemoteFileTest missing from branch-2.3 (only present in master branch). Which branch do you recommend targeting this PR towards?

[felixcheung]

hi there - you should always open PR on master. we could backport to "older releases" as needed, or ask you to open a separate PR that is based on branch-2.3 (like this one you have now)

[madanadit]

Thanks @felixcheung!. I've re targeted to master in this PR. Closing this one.