[SPARK-12276][Core] Prevent RejectedExecutionException by checking if ThreadPoolExecutor is shutdown and its capacity

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-12276

We noticed that it is possible to throw RejectedExecutionException when submitting thread in AppClient. The error is like following. We should add some checks to prevent it.

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2077082c rejected from java.util.concurrent.ThreadPoolExecutor@66b9915a[Running, pool size = 1, active threads     = 0, queued tasks = 0, completed tasks = 1]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:96)
...

[SparkQA]

Test build #47575 has finished for PR 10267 at commit d8fc88d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

you don't have to change the method signature here if you just to a flatten within this method

[andrewor14]

@viirya @zsxwing what is the root cause of this issue? It seems arbitrary to just cut off an attempt if the pool size is already at the capacity. I would think that we should still make the attempt, but how can we exceed the thread pool capacity if our pool size is already masterRpcAddresses.size?

[zsxwing]

I think this is already fixed by #10108. Could you try the master or 1.6 branch?

[andrewor14]

Yes, I forgot about that patch. This should not be a problem anymore, and this isn't the correct fix even if it is. @viirya would you mind closing this?

[viirya]

I am not sure if that patch solves this problem. As there is still the bug reported. Maybe it is just before the patch is committed. I would close this first. Thanks.