Support multiple directories for spark.local.dir

[weiting-chen]

This is a feature request to support multiple directories for spark.local.dir setup.
spark.local.dir use "/tmp" as the default setting (link).
In Spark-on-Yarn, most customers usually use multiple directories for spark.local.dir setup. This way can help to get better performance.
In Spark-on-K8s, it indicates to root volume and cannot be modified by default. Since Spark-on-K8s running application and creating containers by request. This feature must create storage(PV) by request as well as configure directories in spark conf before launching the Spark applications.

One related feature implementation(kubernetes/enhancements#121) from kubernetes.
We may need to wait for this feature implemented.

[ash211]

@weiting-chen do you need PV storage specifically, or would the EmptyDir from #486 work for you?

I don't think you need persistence in static allocation mode, and dynamic allocation requires an external shuffle service which stores data in spark.kubernetes.shuffle.dir, not in spark.local.dir

[weiting-chen]

Yes, #486 is enough for static mode.
Use PV storage doesn't make sense in spark.local.dir since the data is temporary and its life cycle comes with the executor pod.

[tangzhankun]

@ash211 @weiting-chen What determined the medium of emptydir by default?