SPARK-1860: Do not cleanup application work/ directories by default

[aarondav]

This causes an unrecoverable error for applications that are running for longer
than 7 days that have jars added to the SparkContext, as the jars are cleaned up
even though the application is still running.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[pwendell]

Sure - might be good to have it off by default. /cc @velvia.

[pwendell]

@aarondav what about just not cleaning up the data if the app is still running? In the future we should probably assess the TTL based on the finish time of the app, not the start time.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15035/

[aarondav]

This patch is intended as a hotfix in the hopes that it can make it into the 1.0 release. Avoiding cleaning up running applications seems like the better solution in general, but is out of scope of this PR.

[pwendell]

I agree, okay let's just pull in this fix and we can hopefully patch the bigger issue later.

[andrewor14]

LGTM

[pwendell]

btw - regarding the branch name - I don't think this was too shitty of a default. I'd actually like to have this on by default if we can get it into working order, because otherwise users will only find out once it's too late that they are out of disk space :P

[velvia]

@aarondav @pwendell I agree we don't want to clean up currently running apps, but also that this should be default to on when its fixed. Maybe its as simple as checking last modified time of the directory.

[aarondav]

Just a little miffed because it took some time to figure out why our executors suddenly started failing with jar-not-found errors :)

I'd prefer a full solution; last modified time runs into an issue if the executor lies dormant for a week. You might say, "that's unlikely", but I'd say, "it'll happen to someone, and they'll be a little miffed." The worker should have enough state to figure out which executors are currently active, though I'm not sure if the problem is made more difficult by multi-worker scenarios.