[SPARK-7874][MESOS] Don’t allocate more than spark.cores.max in fine-grained mode.

[dragos]

Currently only coarse-grained mode observes spark.cores.max. The fine-grained mode should follow the same limit, and not go above the defined maximum cores.

There's some duplication of logic between the coarse-grained and fine-grained mode. I created SPARK-10444 to fix that.

Rework of #8671

[dragos]

@tnachen @andrewor14 please have a look

[SparkQA]

Test build #43394 has finished for PR 9027 at commit 5817d68.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[nraychaudhuri]

LGTM

[dragos]

Up to date with master, which has a fix in #9282 that was blocking people from testing this one.

@tnachen please have a look

[SparkQA]

Test build #44926 has finished for PR 9027 at commit 960755d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

[dragos]

@tnachen can you please have a look? I addressed all your points from #8671

[SparkQA]

Test build #45878 has finished for PR 9027 at commit 68cee1c.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * public class JavaGradientBoostingClassificationExample\n * public class JavaGradientBoostingRegressionExample\n * public class JavaRandomForestClassificationExample\n * public class JavaRandomForestRegressionExample\n

[andrewor14]

@dragos can you fix the style here? Other than that, is this ready for review?

[tnachen]

I think we've consistently put the function parameters that are longer than 100 in newlines so far, can you move d: SchedulerDriver to the next line?

[tnachen]

@dragos just left some comments on this PR, sorry I wasn't able to take a deep look earlier. I think there are some open issues that needs to be addressed first. Take a look and let me know what you think.

[dragos]

Thanks @tnachen, but since this didn't make it in 1.6, and fine-grained mode is slated for removal, I don't think it makes sense to work on it anymore.

[blbradley]

@dragos Where can you see that fine-grained mode is slated for removal? All I see is #9795.

[dragos]

Discussions on the dev mailing list.

[blbradley]

@dragos Link?

[dragos]

The discussions started on Nov. 19 2015, "Removing the Mesos fine-grained mode". but I only have it in my inbox. I'm sure there's some archive somewhere, but I don't have a link at hand.