SPARK-10611 Clone Configuration for each task for NewHadoopRDD

[mingyukim]

This patch attempts to fix the Hadoop Configuration thread safety issue for NewHadoopRDD in the same way SPARK-2546 fixed the issue for HadoopRDD.

[srowen]

Looks like a plausible variation of #2684 -- cc @JoshRosen

[SparkQA]

Test build #1755 has finished for PR 8763 at commit d516aae.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

If HADOOP-10456 is still an issue then I think that we need to be locking on the same lock in both NewHadoopRDD and HadoopRDD. Therefore, I'd be in favor of moving this lock to SparkHadoopUtil or a similar location, then updating both RDD implementations to use that lock.

[mingyukim]

HADOOP-10456 is fixed in Hadoop 2.4.1 (https://issues.apache.org/jira/browse/HADOOP-10456), so people will still hit the issue in some configurations.

The concurrency issue with the Configuration/JobConf instantiations seems to be per object, not per class. (i.e. happens when two threads try to copy the same conf object, not when two threads try to instantiate Configuration/JobConf with two separate conf objects.) The lock could've even be allocated per NewHadoopRDD instance or HadoopRDD instance.

So, I think this should be safe, although I can certainly move this to SparkHadoopUtil for clarity. Please let me know what you think!

[JoshRosen]

Ah, right. In the original version of the fix for SPARK-1097, we actually synchronized on the Configuration being cloned. It looks like the HadoopRDD object lock was added in a followup patch to avoid a deadlock caused by this synchronization: #1409

[sryza]

This seems like it could have a pretty serious perf impact. Are we able to do some benchmarking to assess this?

[sryza]

Oh, nevermind, sorry, this is off by default.

[mccheah]

Jenkins, test this please

[mingyukim]

Can someone kick off the build? Also, please let me know if there's any other comments!

[SparkQA]

Test build #1769 has finished for PR 8763 at commit 4ba2acd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

LGTM, since this is off by default. I'm going to merge this into master, but let me know how far back you think this fix should be backported.

[mingyukim]

Thanks @JoshRosen. Backporting this to Spark 1.5 would be good enough for me. Would that be reasonable?

[JoshRosen]

SGTM; I'll pull it in now for inclusion in 1.5.1.

[mingyukim]

Thanks!