[SPARK-8126] [BUILD] Use custom temp directory during build.

[vanzin]

Even with all the efforts to cleanup the temp directories created by
unit tests, Spark leaves a lot of garbage in /tmp after a test run.
This change overrides java.io.tmpdir to place those files under the
build directory instead.

After an sbt full unit test run, I was left with > 400 MB of temp
files. Since they're now under the build dir, it's much easier to
clean them up.

Also make a slight change to a unit test to make it not pollute the
source directory with test data.

[vanzin]

This reverts the revert and fixes the issue the original change caused.

[JoshRosen]

Can you explain what the root problem was? Why did this depend on test ordering?

[vanzin]

The root problem was that the tmp directory was not created (I added code to both builds to explicitly do it). I guess depending on which tests run first the directory might be there (some tests use guava's Files.createTempDirectory, maybe that one creates the dir if it isn't there).

[SparkQA]

Test build #34297 timed out for PR 6674 at commit 0f8ad41 after a configured wait of 175m.

[vanzin]

Jenkins, retest this please.

[yhuai]

cc @marmbrus

[marmbrus]

The problem with doing it this way is that buildLocation is just using the current directory as the build root, which is not always a valid assumption. We should instead remove that val and use projectRoot.value anyplace that we need the root spark directory.

[vanzin]

That should affect also the val spark =... line above, right? So my particular change shouldn't have made anything worse than before.

[marmbrus]

Yeah, sorry I should have been more clear. This was a general comment about how we are doing things in the SBT build and not something that should have blocked merging this PR.

[SparkQA]

Test build #34317 has finished for PR 6674 at commit 0f8ad41.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Looks good to me. Do we need any additional testing before trying to merge it again? I could merge to master and then later backport if it seems OK for a few days.

[andrewor14]

I've triggered a few more test runs for this patch. Hopefully this time it's not flaky.

[SparkQA]

Test build #885 has finished for PR 6674 at commit 0f8ad41.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #889 has finished for PR 6674 at commit 0f8ad41.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #888 timed out for PR 6674 at commit 0f8ad41 after a configured wait of 175m.

[SparkQA]

Test build #887 timed out for PR 6674 at commit 0f8ad41 after a configured wait of 175m.

[SparkQA]

Test build #886 timed out for PR 6674 at commit 0f8ad41 after a configured wait of 175m.

[srowen]

Jenkins, retest this please

[SparkQA]

Test build #34397 has finished for PR 6674 at commit 0f8ad41.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I'm going to merge this into master and see how it goes, then to 1.4 and see what happens, then 1.3.

[srowen]

master looks OK and has all day. Moving on to 1.4.

[srowen]

All of the 1.4 builds have succeeded since this patch, some a few times. The exception is: https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.4-Maven-with-YARN/ This succeeded after, then failed, and the failure in the Kafka suite looks unrelated since it doesn't involve a temp file. I'm declaring victory and moving to 1.3.