SPARK-6338 [CORE] Use standard temp dir mechanisms in tests to avoid orphaned temp files

[srowen]

Use Utils.createTempDir() to replace other temp file mechanisms used in some tests, to further ensure they are cleaned up, and simplify

[SparkQA]

Test build #28614 has started for PR 5029 at commit 1a12efa.

• This patch merges cleanly.

[SparkQA]

Test build #28614 has finished for PR 5029 at commit 1a12efa.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28614/
Test FAILed.

[SparkQA]

Test build #28629 has started for PR 5029 at commit 5713d45.

• This patch merges cleanly.

[SparkQA]

Test build #28629 has finished for PR 5029 at commit 5713d45.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28629/
Test PASSed.

[SparkQA]

Test build #28647 has started for PR 5029 at commit 57609e4.

• This patch merges cleanly.

[SparkQA]

Test build #28647 has finished for PR 5029 at commit 57609e4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28647/
Test PASSed.

[sryza]

What's the advantage of a parent directory created with createTempDir when we're already using File.createTempFile?

[srowen]

createTempDir registers it for deletion at shutdown.

[sryza]

Is this necessary now that we're using createTempDir?

[vanzin]

This is redundant now.

[sryza]

It seems like in many places we're explicitly deleting the files in addition to using createTempDir. Is that necessary? Otherwise, LGTM.

[vanzin]

Redundant?

[srowen]

I'm torn on this, since it leaves temp files lying around while tests run, and, doesn't delete them if there's a hard crash. But the extra line of code is annoying everywhere and not done consistently. Hm.. Maybe the test superclass can manage a temp dir and delete it between tests. If anyone's interested in that, I'll go there. Otherwise maybe best to leave it as-is and not add recursive deletes where there were none before.

(A few of these changes are to make the temp file go in a temp dir instead of working dir, but most are indeed just about standardization and better guaranteeing cleanup)

[vanzin]

I was going to suggest a "TempDirectory" trait or something to manage a temp dir per test. If scalatest runs a single JVM for all tests, then it might be worth it, but it would be a very noisy change.

[srowen]

True, there is not much consistency in whether it is also deleted explicitly after the test. I think I was mostly keeping that as it was in each case

[vanzin]

Redundant?

[vanzin]

LGTM (btw you're my hero).

[srowen]

OK, I basically left the code as-is with respect to recursive deletion. Yeah adding a new trait to every test might be too much for the moment. I think this zaps a few minor actual problems (stray temp files) and helps clean up more stuff in more cases, and that leaves us with little practical problem.

[SparkQA]

Test build #28718 has started for PR 5029 at commit 9004081.

• This patch merges cleanly.

[SparkQA]

Test build #28718 has finished for PR 5029 at commit 9004081.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28718/
Test PASSed.

[marmbrus]

SQL changes LGTM. I'll note that at the top of TestHive there is more code that calls getTempFilePath. I'm not actually sure if that is using the correct code path or not, but it looks like its trying to do its own cleanup.

[srowen]

Let me see if I can also refactor that bit of code in TestHive too, yeah.

[SparkQA]

Test build #28871 has started for PR 5029 at commit 4a212fa.

• This patch merges cleanly.

[SparkQA]

Test build #28871 has finished for PR 5029 at commit 4a212fa.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28871/
Test FAILed.

[SparkQA]

Test build #28884 has started for PR 5029 at commit 27b740a.

• This patch merges cleanly.

[SparkQA]

Test build #28884 has finished for PR 5029 at commit 27b740a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28884/
Test PASSed.