[SPARK-1930] The Container is running beyond physical memory limits, so as to be killed #894

witgo · 2014-05-27T16:39:14Z

No description provided.

AmplabJenkins · 2014-05-27T16:42:58Z

Merged build triggered.

AmplabJenkins · 2014-05-27T16:43:05Z

Merged build started.

AmplabJenkins · 2014-05-27T17:23:32Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-27T17:23:32Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15228/

mridulm · 2014-05-27T19:17:29Z

The constant xxx mb overhead is to account for things like off vm overheads, interned strings, other native overheads, etc.
These are fairly small and reasonably constant.
Making it a function of vm max heap is not advisable particularly when There is no direct correlation between both.

Worst case, make it configurable constant - not dependent on Xmx

mridulm · 2014-05-27T19:18:16Z

Btw, same applies for both master and workers (though values should probably be different)

tgravescs · 2014-05-27T20:59:46Z

I agree with mridulm, I don't think we should change it. It looks like you just requested to small of a container. Am I missing something that applies directly to this 384MB?

If we do change it then I would prefer to see this constant removed all together and just have the user specify what they want. MR is an example of this and if you are going from MR to spark this 384MB special size is a bit confusing.

AmplabJenkins · 2014-05-28T03:47:58Z

Merged build triggered.

AmplabJenkins · 2014-05-28T03:48:07Z

Merged build started.

AmplabJenkins · 2014-05-28T04:28:16Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-28T04:28:17Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15242/

pwendell · 2014-05-28T05:06:50Z

Hey @tgravescs, one thing that could affect this is PySpark. In that case there are python VM's spawned by the executor which could increase the total memory used. Will YARN track the memory usage of sub-processes when deciding on allocation limits?

mridulm · 2014-05-28T05:17:06Z

The entire process tree is tracked ...
Note that yarn allocates in multiples of memory slots and kills only when
the container requirement is violated.
On 28-May-2014 10:36 am, "Patrick Wendell" [email protected] wrote:

Hey @tgravescs https://github.com/tgravescs, one thing that could
affect this is PySpark. In that case there are python VM's spawned by the
executor which could increase the total memory used. Will YARN track the
memory usage of sub-processes when deciding on allocation limits?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/894#issuecomment-44366137
.

sryza · 2014-05-28T05:23:32Z

Agree with @tgravescs and @mridulm that a constant overhead makes more sense.

@pwendell YARN includes the memory usage of subprocesses in its calculation.

Making the overhead configurable probably makes sense. PySpark could add a fixed amount, and users might want to add more if they're allocating direct byte buffers. Some compression codecs allocate direct byte buffers, so if we want to get fancy, we could take that in to account.

I'm opposed to removing the 384 altogether. Having had to explain 2 bajillion times that two MR configs need to be updated every time one wants to increase task memory, I've really appreciated that Spark handles this automatically.

AmplabJenkins · 2014-05-28T06:22:59Z

Merged build triggered.

AmplabJenkins · 2014-05-28T06:23:04Z

Merged build started.

AmplabJenkins · 2014-05-28T07:03:31Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-28T07:03:32Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15246/

AmplabJenkins · 2014-05-28T13:42:58Z

Merged build triggered.

AmplabJenkins · 2014-05-28T13:43:07Z

Merged build started.

witgo · 2014-05-28T13:44:11Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala

@mridulm @pwendell @sryza @tgravescs
Here's the default value of memoryOverhead should be changed dynamically calculated, right?

AmplabJenkins · 2014-05-28T14:41:34Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-28T14:41:34Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15252/

AmplabJenkins · 2014-06-13T15:49:23Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15764/

AmplabJenkins · 2014-06-13T16:27:05Z

Merged build triggered.

AmplabJenkins · 2014-06-13T16:27:14Z

Merged build started.

AmplabJenkins · 2014-06-13T17:10:35Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-13T17:10:35Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15767/

tgravescs · 2014-06-16T15:07:54Z

docs/running-on-yarn.md

can you make this text say:

The amount of off heap memory (in megabytes) to be allocated per executor. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc.

AmplabJenkins · 2014-06-16T15:19:46Z

Merged build triggered.

AmplabJenkins · 2014-06-16T15:19:51Z

Merged build started.

AmplabJenkins · 2014-06-16T16:03:28Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-16T16:03:28Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15814/

…so as to be killed Author: witgo <[email protected]> Closes #894 from witgo/SPARK-1930 and squashes the following commits: 564307e [witgo] Update the running-on-yarn.md 3747515 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1930 172647b [witgo] add memoryOverhead docs a0ff545 [witgo] leaving only two configs a17bda2 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1930 478ca15 [witgo] Merge branch 'master' into SPARK-1930 d1244a1 [witgo] Merge branch 'master' into SPARK-1930 8b967ae [witgo] Merge branch 'master' into SPARK-1930 655a820 [witgo] review commit 71859a7 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1930 e3c531d [witgo] review commit e16f190 [witgo] different memoryOverhead ffa7569 [witgo] review commit 5c9581f [witgo] Merge branch 'master' into SPARK-1930 9a6bcf2 [witgo] review commit 8fae45a [witgo] fix NullPointerException e0dcc16 [witgo] Adding configuration items b6a989c [witgo] Fix container memory beyond limit, were killed (cherry picked from commit cdf2b04) Signed-off-by: Thomas Graves <[email protected]>

…so as to be killed Author: witgo <[email protected]> Closes apache#894 from witgo/SPARK-1930 and squashes the following commits: 564307e [witgo] Update the running-on-yarn.md 3747515 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1930 172647b [witgo] add memoryOverhead docs a0ff545 [witgo] leaving only two configs a17bda2 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1930 478ca15 [witgo] Merge branch 'master' into SPARK-1930 d1244a1 [witgo] Merge branch 'master' into SPARK-1930 8b967ae [witgo] Merge branch 'master' into SPARK-1930 655a820 [witgo] review commit 71859a7 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1930 e3c531d [witgo] review commit e16f190 [witgo] different memoryOverhead ffa7569 [witgo] review commit 5c9581f [witgo] Merge branch 'master' into SPARK-1930 9a6bcf2 [witgo] review commit 8fae45a [witgo] fix NullPointerException e0dcc16 [witgo] Adding configuration items b6a989c [witgo] Fix container memory beyond limit, were killed

Co-authored-by: Egor Krivokon <>

Fix container memory beyond limit, were killed

b6a989c

witgo changed the title ~~[SPARK-1930] Container memory beyond limit, were killed~~ [SPARK-1930] Container is running beyond physical memory limits May 27, 2014

witgo changed the title ~~[SPARK-1930] Container is running beyond physical memory limits~~ [SPARK-1930] The Container is running beyond physical memory limits, so as to be killed May 27, 2014

Adding configuration items

e0dcc16

witgo changed the title ~~[SPARK-1930] The Container is running beyond physical memory limits, so as to be killed~~ [WIP][SPARK-1930] The Container is running beyond physical memory limits, so as to be killed May 28, 2014

fix NullPointerException

8fae45a

review commit

9a6bcf2

witgo reviewed May 28, 2014
View reviewed changes

add memoryOverhead docs

172647b

tgravescs reviewed Jun 16, 2014
View reviewed changes

witgo added 2 commits June 16, 2014 23:14

Merge branch 'master' of https://github.com/apache/spark into SPARK-1930

3747515

Update the running-on-yarn.md

564307e

asfgit closed this in cdf2b04 Jun 16, 2014

witgo deleted the SPARK-1930 branch June 17, 2014 02:04

nishkamravi2 mentioned this pull request Jul 13, 2014

Modify default YARN memory_overhead-- from an additive constant to a multiplier #1391

Closed

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

MapR [SPARK-947] Error when using Spark SQL with derby db (apache#894)

5748cae

Co-authored-by: Egor Krivokon <>

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

MapR [SPARK-947] Error when using Spark SQL with derby db (apache#894)

156dfae

Co-authored-by: Egor Krivokon <>

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

MapR [SPARK-947] Error when using Spark SQL with derby db (apache#894)

7e3c85a

Co-authored-by: Egor Krivokon <>

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

MapR [SPARK-947] Error when using Spark SQL with derby db (apache#894)

da1dad2

Co-authored-by: Egor Krivokon <>

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

MapR [SPARK-947] Error when using Spark SQL with derby db (apache#894)

1289cd0

Co-authored-by: Egor Krivokon <>

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

MapR [SPARK-947] Error when using Spark SQL with derby db (apache#894)

1937f48

Co-authored-by: Egor Krivokon <>

mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025

MapR [SPARK-947] Error when using Spark SQL with derby db (apache#894)

231375c

Co-authored-by: Egor Krivokon <>

[SPARK-1930] The Container is running beyond physical memory limits, so as to be killed #894

[SPARK-1930] The Container is running beyond physical memory limits, so as to be killed #894

Uh oh!

Conversation

witgo commented May 27, 2014

Uh oh!

AmplabJenkins commented May 27, 2014

Uh oh!

AmplabJenkins commented May 27, 2014

Uh oh!

AmplabJenkins commented May 27, 2014

Uh oh!

AmplabJenkins commented May 27, 2014

Uh oh!

mridulm commented May 27, 2014

Uh oh!

mridulm commented May 27, 2014

Uh oh!

tgravescs commented May 27, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

pwendell commented May 28, 2014

Uh oh!

mridulm commented May 28, 2014

Uh oh!

sryza commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

witgo May 28, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

tgravescs Jun 16, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 16, 2014

Uh oh!

AmplabJenkins commented Jun 16, 2014

Uh oh!

AmplabJenkins commented Jun 16, 2014

Uh oh!

AmplabJenkins commented Jun 16, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development