[SPARK-13906] Ensure that there are at least 2 dispatcher threads.

[yonran]

What changes were proposed in this pull request?

Force at least two dispatcher-event-loop threads. Since SparkDeploySchedulerBackend (in AppClient) calls askWithRetry to CoarseGrainedScheduler in the same process, there the driver needs at least two dispatcher threads to prevent the dispatcher thread from hanging.

How was this patch tested?

Manual.

Author: Yonathan Randolph [email protected]

[srowen]

@zsxwing do you have an opinion?
I was thinking the max would be applied just to availableProcessors(), but, yeah maybe a good idea to not even let the user set it to 1. Is there any legitimate reason to do that? if we can think of one then this can more conservatively apply the max(2,..) to just the default value.

[yonran]

It may be legitimate to set spark.rpc.netty.dispatcher.numThreads=1 for processes other than the driver (sparkMaster, sparkWorker).

[srowen]

OK then I'd move the max inside, so that the property value can still override it to 1 if needed.

[zsxwing]

I'm ok with max(2, availableProcessors).

However, the root cause is there are blocking calls in the event loops but not enough threads. This could happen in other places (such as netty, akka). Ideally, we should avoid blocking calls in all event loops. However, it's hard to figure out all of them in the huge code bases :(

[yonran]

Changed to math.max(2, Runtime.getRuntime.availableProcessors())

[srowen]

Jenkins test this please

[SparkQA]

Test build #53208 has finished for PR 11728 at commit 597f0e6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master