[SPARK-6313] Add config option to disable file locks/fetchFile cache to ...

[nemccarthy]

...support NFS mounts.

This is a work around for now with the goal to find a more permanent solution.
https://issues.apache.org/jira/browse/SPARK-6313

[AmplabJenkins]

Can one of the admins verify this patch?

[JoshRosen]

Jenkins, this is ok to test.

[SparkQA]

Test build #28652 has started for PR 5036 at commit 5de7eb4.

• This patch merges cleanly.

[AmplabJenkins]

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28652/
Test FAILed.

[shaneknapp]

jenkins, test this please

[SparkQA]

Test build #28655 has started for PR 5036 at commit 5de7eb4.

• This patch merges cleanly.

[SparkQA]

Test build #28655 has finished for PR 5036 at commit 5de7eb4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28655/
Test PASSed.

[JoshRosen]

As a workaround for the NFS issue, this looks good to me. I have a couple of minor grammar / documentation nits, which I'll comment on below, but I can fix those up myself on merge if you don't get around to it first.

[JoshRosen]

I'd reword this slightly in order to match the style of similar boolean options. How about this:

If set to true (default), file fetching will use a local cache that is shared by executors that belong to the same application, which can improve task launching performance when running many executors on the same host. If set to false, these caching optimizations will be disabled and all executors will fetch their own copies of files. This optimization may be disabled in order to use Spark local directories that reside on NFS filesystems (see SPARK-6313 for more details).

[nemccarthy]

@JoshRosen Thanks for the feedback. Updated as suggested.

[SparkQA]

Test build #28682 has started for PR 5036 at commit 2eaaf42.

• This patch merges cleanly.

[SparkQA]

Test build #28682 has finished for PR 5036 at commit 2eaaf42.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28682/
Test PASSed.

[JoshRosen]

LGTM, so I'm going to merge this into master (1.4.0), branch-1.3 (1.3.1), and branch-1.2 (1.2.2) (since this cache was first introduced in 1.2.0).