[SPARK-2140] Enabling YARN memory overhead calculation from heap of the JVM

[copester]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[witgo]

There is always returns args.amMemory.

[mateiz]

Jenkins, this is ok to test

[SparkQA]

QA tests have started for PR 1906 at commit a2abc51.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1906 at commit a2abc51.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tgravescs]

@copester Any chance you can upmerge this and update based on the jira conversation. I think the issue is actually with yarn alpha version now. The yarn alpha version of this shouldn't subtract out the memoryOverhead since its using that directly as the heap size and adding in the memory overhead when it requests the AM container ( memoryResource.setMemory(args.amMemory + memoryOverhead)).

We also don't really need to do the logic there of rounding based on the getMinimumResourceCapability because Yarn will handle that for us. So I think we can actually remove the calculateAMMemory routines and just use args.amMemory.

[tgravescs]

I assume this is replaced by #2253. Can you close this one?