This repository was archived by the owner on Jan 9, 2020. It is now read-only.
This repository was archived by the owner on Jan 9, 2020. It is now read-only.
Use ln instead of cp in the driver's Dockerfile #295
Description
Right now we copy spark.files to the driver's working directory with cp, which copies the bytes on disk. Possibly we could use ln to speed up this copy and use fewer bytes on disk (and save time on driver launch).
This is probably only a concern with large spark.files so deferring until this becomes an issue for someone.
See https://github.com/apache-spark-on-k8s/spark/pull/294/files#r118126984