[SPARK-12101][Core]Fix thread pools that cannot cache tasks in Worker and AppClient

[zsxwing]

SynchronousQueue cannot cache any task. This issue is similar to #9978. It's an easy fix. Just use the fixed ThreadUtils.newDaemonCachedThreadPool.

[zsxwing]

The previous codes are right. Just use newDaemonCachedThreadPool to make codes simple.

[srowen]

This should be attached to SPARK-11999 right?

[zsxwing]

This should be attached to SPARK-11999 right?

RC1 is coming out, so I created a new ticket.

[srowen]

Hm, what do you mean? no RC has been announced

[zsxwing]

RC1 is cut. See bf52584

[srowen]

@pwendell this should probably be announced on the list when you start or else we might have a very small chance of a race condition in tagging/merging things.

I get the point that your first half of the fix could conceivably end up in 1.6.0 and not the other, but is that a good thing? you made a second JIRA anyway I see, so one of them needs to be associated here.

I think we have a larger problem, as ever in Spark, if people are just merging fixes right up to and well past the deadline. We were supposed to be cutting RCs 3 weeks ago, but it arbitrarily happened now.

[pwendell]

Hey @marmbrus is actually managing the RC - it just has my name on it because some automated tooling uses my account. Ping @marmbrus.

[pwendell]

BTW @srowen, some protocol for announcing these is probably a good idea to avoid races. I think we haven't suffered from races in the past, but mostly out of luck.

[zsxwing]

I get the point that your first half of the fix could conceivably end up in 1.6.0 and not the other, but is that a good thing? you made a second JIRA anyway I see, so one of them needs to be associated here.

Oops. Sorry. Forgot to add the JIRA number to the title.

[marmbrus]

Sorry if this seemed arbitrary, I cut the release as soon as all the blocker issues were resolved. In the future I'll announce on the dev list first.

[SparkQA]

Test build #47079 has finished for PR 10108 at commit 2b9f4da.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

the #%d doesn't do anything here right? We don't call format on this string

[andrewor14]

actually, this is setting a format so it does do something. Maybe we should add it back in the new code

[zsxwing]

ThreadUtils.namedThreadFactory will do the similar thing except the format is -%d. I think it's better to make all threads have the same format in Spark.

[andrewor14]

oh I see, it's a prefix that we're passing in here

[andrewor14]

This patch itself looks good pending 1 minor comment.

[andrewor14]

Merging into master 1.6