Skip to content

[SPARK-22292][Mesos] Added spark.mem.max support for Mesos #19510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

[SPARK-22292][Mesos] Added spark.mem.max support for Mesos #19510

wants to merge 4 commits into from

Conversation

@windkit
Copy link
Contributor

@windkit windkit commented Oct 17, 2017

What changes were proposed in this pull request?

To limit the amount of resources a spark job accept from Mesos, currently we can only use spark.cores.max to limit in terms of cpu cores.
However, when we have big memory executors, it would consume all the resources.

This PR added spark.mem.max option for Mesos

How was this patch tested?

Added Unit Test

skonto
skonto reviewed Oct 18, 2017
View reviewed changes
...scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala Outdated
Copy link
Contributor

@skonto skonto Oct 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove space

@skonto
Copy link
Contributor

skonto commented Oct 18, 2017

@windkit there is an open issue here: https://issues.apache.org/jira/browse/SPARK-22133
Could you please add documentation of the new property along with the old ones.

skonto
skonto reviewed Oct 18, 2017
View reviewed changes
...main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala Outdated
Copy link
Contributor

@skonto skonto Oct 18, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we defend against minimum values and fail fast? For example default executor memory is 1.4MB. We could calculate the value returned by MesosSchedulerUtils.executorMemory. I don't think these values calculated in canLaunchTask ever change. Same applies for cpus.

Copy link
Contributor Author

@windkit windkit Oct 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skonto
For cpus, I think we can compare with minCoresPerExecutor
For mem, calling the MesosSchedulerUtils.executorMemory to get the minimum requirement.

Then at here, we parse the option, check the minimum and if it is too small, throw exception?

Copy link
Contributor

@skonto skonto Oct 20, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exception I think would be ok, the idea if something is never going to work let the user know, especially for the novice user. In general, the minimum would be a warning if we dont want an exception thrown.

Copy link
Contributor Author

@windkit windkit Oct 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I add the check with this PR or a separate one?

Copy link

@ArtRand ArtRand Oct 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to have a check similar to

spark/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala

Line 73 in 06df34d

// Maximum number of cores to acquire
but for memory, so that we know we'll "land" on the maximum?

@windkit
Copy link
Contributor Author

windkit commented Oct 20, 2017

@skonto sure I can add those in, can you point to me where the documentation source code is?

@skonto
Copy link
Contributor

skonto commented Oct 20, 2017

@windkit https://github.com/apache/spark/blob/e2fea8cd6058a807ff4841b496ea345ff0553044/docs/running-on-mesos.md

@skonto
Copy link
Contributor

skonto commented Oct 20, 2017

@susanxhuynh pls review.

@windkit windkit mentioned this pull request Oct 23, 2017
Closed
@susanxhuynh
Copy link
Contributor

susanxhuynh commented Oct 26, 2017

@windkit Trying to understand the need for this config ... could you accomplish the same thing by setting spark.cores.max, spark.executor.cores, and spark.executor.memory? Could you give an example scenario where this is needed?

susanxhuynh
susanxhuynh reviewed Oct 26, 2017
View reviewed changes
docs/running-on-mesos.md Outdated
Copy link
Contributor

@susanxhuynh susanxhuynh Oct 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: 'park'

docs/running-on-mesos.md Outdated
Copy link
Contributor

@susanxhuynh susanxhuynh Oct 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: 'park'

docs/running-on-mesos.md Outdated
Copy link
Contributor

@susanxhuynh susanxhuynh Oct 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: 'park'

docs/running-on-mesos.md Outdated
Copy link
Contributor

@susanxhuynh susanxhuynh Oct 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add "across the cluster (not from each machine)". And, something about there is no maximum if this property is not set.

docs/running-on-mesos.md Outdated
Copy link
Contributor

@susanxhuynh susanxhuynh Oct 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion: "Duration for which unused resources are considered declined, when maximum number of cores spark.cores.max has been reached."
@ArtRand Is this the documentation you had in mind in https://issues.apache.org/jira/browse/SPARK-22133 ? Is this enough information for a non-Mesos expert to set this?

@windkit
Copy link
Contributor Author

windkit commented Nov 1, 2017

@susanxhuynh Thanks for reviewing.
I want to use both spark.mem.max and spark.cores.max to limit resource one task can use within the cluster.
Now I am setting up a common cluster for several users, they are allowed to configure spark.executor.cores and spark.executor.memory according to their need. I then need a limit for both cpu cores and memory.

asfgit pushed a commit that referenced this pull request Nov 8, 2017
@windkit @srowen
[SPARK-22133][DOCS] Documentation for Mesos Reject Offer Configurations
6447d7b 
## What changes were proposed in this pull request?
Documentation about Mesos Reject Offer Configurations

## Related PR
#19510 for `spark.mem.max`

Author: Li, YanKit | Wilson | RIT <[email protected]>

Closes #19555 from windkit/spark_22133.
@skonto
Copy link
Contributor

skonto commented May 19, 2018

@windkit Could you update the PR? @susanxhuynh I think this is fine to merge, are you ok?

@AmplabJenkins
Copy link

AmplabJenkins commented Jun 9, 2018

Can one of the admins verify this patch?

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 16, 2018

gentle ping @windkit

@HyukjinKwon HyukjinKwon mentioned this pull request Jul 16, 2018
Closed
@asfgit asfgit closed this in 1a4fda8 Jul 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

3 more reviewers

@skonto skonto skonto left review comments

@ArtRand ArtRand ArtRand left review comments

@susanxhuynh susanxhuynh susanxhuynh left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@windkit @skonto @susanxhuynh @AmplabJenkins @HyukjinKwon @ArtRand