[SPARK-2439] Actually close FileSystem in Master

[andrewor14]

This was intended, but never materialized in code...

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

QA tests have started for PR 1363. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16521/consoleFull

[SparkQA]

QA results for PR 1363:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16521/consoleFull

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16521/

[mridulm]

On the contrary, we should remove all filesystem.close() in spark : since shutdown hooks commit/flush pending messages to hdfs.
filesystem object's are typically global and shared/cached across all uses.

[andrewor14]

Yes this is why I closed this. See #1365