[SPARK-4973][CORE] Local directory in the driver of client-mode continues remaining even if application finished when external shuffle is enabled

[sarutak]

When we enables external shuffle service, local directories in the driver of client-mode continue remaining even if application has finished.
I think local directories for drivers should be deleted.

[SparkQA]

Test build #24834 has finished for PR 3811 at commit d99718e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Does this define a new system property just for deployment mode? This logic looks like it is applied even when external shuffle service is not enabled. Why is the driver behavior special here?

[sarutak]

If we run client-mode including local-mode, driver runs on client and executors doesn't run on client so no one shared the local directories of the driver.

[SparkQA]

Test build #25233 has finished for PR 3811 at commit 88feecd.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Why does it matter whether it's client mode or cluster mode? I'd say it should always clean it if it's the driver (which doesn't get killed throughout the application).

[sarutak]

@andrewor14 Ah, exactly. The driver is not killed until the end of the application. I'll fix my previous changes.

[andrewor14]

LGTM merging into master when tests pass

[SparkQA]

Test build #25253 has finished for PR 3811 at commit ad944ab.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

retest this please

[SparkQA]

Test build #25257 has finished for PR 3811 at commit ad944ab.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.