[SPARK-5845][Shuffle] Time to cleanup spilled shuffle files not included in shuffle write time

[ilganeli]

I've added a timer in the right place to fix this inaccuracy.

[SparkQA]

Test build #28438 has started for PR 4965 at commit 9434b50.

• This patch merges cleanly.

[SparkQA]

Test build #28438 has finished for PR 4965 at commit 9434b50.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28438/
Test FAILed.

[SparkQA]

Test build #28449 has started for PR 4965 at commit b946d08.

• This patch merges cleanly.

[ilganeli]

retest this please

[SparkQA]

Test build #28449 has finished for PR 4965 at commit b946d08.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28449/
Test FAILed.

[srowen]

CC @kayousterhout
Just checking, this is meant to be in nanos and not milliseconds?

[ilganeli]

Hi Sean - other usages of writeMetrics.incShuffleWriteTime also use nanoTime(). Please see BlockObjectWriter::callWithTiming() and ExternalSorter::writePartitionedFile.

[ilganeli]

retest this please

[SparkQA]

Test build #28475 has started for PR 4965 at commit 3e059b0.

• This patch merges cleanly.

[SparkQA]

Test build #28475 has finished for PR 4965 at commit 3e059b0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28475/
Test PASSed.

[srowen]

OK. As a matter of style I think it would be better to...

context.taskMetrics.shuffleWriteMetrics.foreach(
          _.incShuffleWriteTime(System.nanoTime - startTime))

Which is what ExternalSorter does. This looks like the correct bit to time.

[SparkQA]

Test build #28538 has started for PR 4965 at commit bfabf88.

• This patch merges cleanly.

[SparkQA]

Test build #28538 has finished for PR 4965 at commit bfabf88.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28538/
Test PASSed.