[SPARK-10471] [CORE] [MESOS] prevent getting offers for unmet constraints #8639

felixb · 2015-09-07T08:30:23Z

this change rejects offers for slaves with unmet constraints for 120s to mitigate offer starvation.
this prevents mesos to send us these offers again and again.
in return, we get more offers for slaves which might meet our constraints.
and it enables mesos to send the rejected offers to other frameworks.

dragos · 2015-09-08T17:27:24Z

You should probably modify the fine-grained scheduler in the same way.

tnachen · 2015-09-09T08:15:15Z

core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala

Can we make this configurable? Also please comment on the unit as well.

tnachen · 2015-09-09T08:20:46Z

I think the change makes sense, we're planning to add dynamic attribute changes on the slave but that's not merged yet in Mesos. as @dragos mentioned please add this to coarse grain mode too.
And once you make this a configuration please also update the spark on mesos docs in docs folder .

felixb · 2015-09-10T06:18:59Z

I made the duration configurable. Still need to add it to fine grained scheduler.

andrewor14 · 2015-09-10T18:08:13Z

ok to test

tnachen · 2015-09-10T18:15:30Z

docs/running-on-mesos.md

There is also a configurations.md that you should add this too.

@tnachen, none of the Yarn or Mesos specific settings are listed in there.

Also, typo: for for.

SparkQA · 2015-09-10T20:56:46Z

Test build #42277 has finished for PR 8639 at commit bb79444.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

felixb · 2015-09-11T08:23:57Z

Added the same logic for fine grained scheduler.
All mesos configuration is only available in running-on-mesos.md. So we skipped adding it to configurations.md

SparkQA · 2015-09-11T08:36:13Z

Test build #42320 has finished for PR 8639 at commit 5acfd65.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-11T09:00:30Z

Test build #42321 has finished for PR 8639 at commit 7626d45.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-11T11:55:21Z

Test build #42324 has finished for PR 8639 at commit 66a1a73.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

felixb · 2015-09-15T10:17:26Z

I just rebased to the current upstream/master.
Any hint, why the tests keep failing?

SparkQA · 2015-09-15T12:35:01Z

Test build #42483 has finished for PR 8639 at commit ce84b1a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dragos · 2015-09-15T12:40:08Z

They're probably just flaky.

felixb · 2015-09-16T07:07:42Z

fixed typo.

SparkQA · 2015-09-16T09:53:47Z

Test build #42527 has finished for PR 8639 at commit 9e00071.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SleepyThread · 2015-09-17T11:54:39Z

@tnachen @andrewor14 friendly reminder..

tnachen · 2015-09-17T15:04:30Z

There is a big HTML table in the bottom of this file, can you also add it
to that list?

On Thu, Sep 17, 2015 at 4:55 AM, Akash Mishra [email protected]
wrote:

@tnachen https://github.com/tnachen @andrewor14
https://github.com/andrewor14 friendly reminder..

—
Reply to this email directly or view it on GitHub
#8639 (comment).

felixb · 2015-09-18T05:22:25Z

added to table of parameters.

SparkQA · 2015-09-18T08:02:49Z

Test build #42646 has finished for PR 8639 at commit 58aaa79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SleepyThread · 2015-09-23T08:52:53Z

@tnachen @andrewor14 friendly reminder..

andrewor14 · 2015-10-16T23:01:28Z

core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala

please use conf.getTimeAsSeconds instead, in which case the default value would be "120s"

andrewor14 · 2015-10-16T23:32:20Z

@felixb sorry for slipping. This looks pretty good. Thanks for taking the time to fix this. Once you address the comments I will merge this.

felixb · 2015-10-19T05:21:31Z

I worked in all your comments.

SparkQA · 2015-10-19T05:37:13Z

Test build #43912 has finished for PR 8639 at commit 69c3e52.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-10-19T07:49:52Z

Test build #43914 has finished for PR 8639 at commit 72a2855.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-10-19T23:27:30Z

core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala

please keep this protected

…ints this change rejects offers for slaves with unmet constraints for 120s to mitigate offer starvation. this prevents mesos to send us these offers again and again. in return, we get more offers for slaves which might meet our constraints. and it enables mesos to send the rejected offers to other frameworks.

SparkQA · 2015-10-20T11:13:34Z

Test build #43971 has finished for PR 8639 at commit 785e4ae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixb · 2015-10-25T11:50:42Z

Is there anything else I can do?

andrewor14 · 2015-11-09T21:28:08Z

retest this please

andrewor14 · 2015-11-09T21:35:43Z

LGTM merging into master and 1.6. Thanks for your work and patience!

this change rejects offers for slaves with unmet constraints for 120s to mitigate offer starvation. this prevents mesos to send us these offers again and again. in return, we get more offers for slaves which might meet our constraints. and it enables mesos to send the rejected offers to other frameworks. Author: Felix Bechstein <[email protected]> Closes #8639 from felixb/decline_offers_constraint_mismatch. (cherry picked from commit 5039a49) Signed-off-by: Andrew Or <[email protected]>

SparkQA · 2015-11-10T00:28:04Z

Test build #45416 has finished for PR 8639 at commit 785e4ae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Similar to #8639 This change rejects offers for 120s when reached `spark.cores.max` in coarse-grained mode to mitigate offer starvation. This prevents Mesos to send us offers again and again, starving other frameworks. This is especially problematic when running many small frameworks on the same Mesos cluster, e.g. many small Sparks streaming jobs, and cause the bigger spark jobs to stop receiving offers. By rejecting the offers for a long period of time, they become available to those other frameworks. Author: Sebastien Rainville <[email protected]> Closes #10924 from sebastienrainville/master.

Similar to #8639 This change rejects offers for 120s when reached `spark.cores.max` in coarse-grained mode to mitigate offer starvation. This prevents Mesos to send us offers again and again, starving other frameworks. This is especially problematic when running many small frameworks on the same Mesos cluster, e.g. many small Sparks streaming jobs, and cause the bigger spark jobs to stop receiving offers. By rejecting the offers for a long period of time, they become available to those other frameworks. Author: Sebastien Rainville <[email protected]> Closes #10924 from sebastienrainville/master. (cherry picked from commit eb019af) Signed-off-by: Andrew Or <[email protected]>

tnachen reviewed Sep 9, 2015
View reviewed changes

felixb force-pushed the decline_offers_constraint_mismatch branch from c1efb1f to bb79444 Compare September 10, 2015 06:17

tnachen reviewed Sep 10, 2015
View reviewed changes

felixb force-pushed the decline_offers_constraint_mismatch branch from bb79444 to 5acfd65 Compare September 11, 2015 08:22

felixb force-pushed the decline_offers_constraint_mismatch branch from 5acfd65 to 7626d45 Compare September 11, 2015 08:55

felixb force-pushed the decline_offers_constraint_mismatch branch from 7626d45 to 66a1a73 Compare September 11, 2015 09:31

felixb force-pushed the decline_offers_constraint_mismatch branch from 66a1a73 to ce84b1a Compare September 15, 2015 10:16

felixb force-pushed the decline_offers_constraint_mismatch branch from ce84b1a to 9e00071 Compare September 16, 2015 07:07

dragos mentioned this pull request Sep 16, 2015

[SPARK-10618] [Mesos] Refactoring scheduling condition and adding test for same #8771

Closed

felixb force-pushed the decline_offers_constraint_mismatch branch from 9e00071 to 58aaa79 Compare September 18, 2015 05:22

andrewor14 reviewed Oct 16, 2015
View reviewed changes

felixb force-pushed the decline_offers_constraint_mismatch branch from 58aaa79 to 69c3e52 Compare October 19, 2015 05:20

felixb force-pushed the decline_offers_constraint_mismatch branch from 69c3e52 to 72a2855 Compare October 19, 2015 05:42

andrewor14 reviewed Oct 19, 2015
View reviewed changes

core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala Outdated

Copy link

Contributor

andrewor14 Oct 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please keep this protected

felixb force-pushed the decline_offers_constraint_mismatch branch from 72a2855 to 785e4ae Compare October 20, 2015 08:29

asfgit closed this in 5039a49 Nov 9, 2015

sebastienrainville mentioned this pull request Jan 26, 2016

[SPARK-13001] [CORE] [MESOS] Prevent getting offers when reached max cores #10924

Closed

[SPARK-10471] [CORE] [MESOS] prevent getting offers for unmet constraints #8639

[SPARK-10471] [CORE] [MESOS] prevent getting offers for unmet constraints #8639

Uh oh!

Conversation

felixb commented Sep 7, 2015

Uh oh!

dragos commented Sep 8, 2015

Uh oh!

tnachen Sep 9, 2015

Choose a reason for hiding this comment

Uh oh!

tnachen commented Sep 9, 2015

Uh oh!

felixb commented Sep 10, 2015

Uh oh!

andrewor14 commented Sep 10, 2015

Uh oh!

tnachen Sep 10, 2015

Choose a reason for hiding this comment

Uh oh!

dragos Sep 15, 2015

Choose a reason for hiding this comment

Uh oh!

dragos Sep 15, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 10, 2015

Uh oh!

felixb commented Sep 11, 2015

Uh oh!

SparkQA commented Sep 11, 2015

Uh oh!

SparkQA commented Sep 11, 2015

Uh oh!

SparkQA commented Sep 11, 2015

Uh oh!

felixb commented Sep 15, 2015

Uh oh!

SparkQA commented Sep 15, 2015

Uh oh!

dragos commented Sep 15, 2015

Uh oh!

felixb commented Sep 16, 2015

Uh oh!

SparkQA commented Sep 16, 2015

Uh oh!

SleepyThread commented Sep 17, 2015

Uh oh!

tnachen commented Sep 17, 2015

Uh oh!

felixb commented Sep 18, 2015

Uh oh!

SparkQA commented Sep 18, 2015

Uh oh!

SleepyThread commented Sep 23, 2015

Uh oh!

andrewor14 Oct 16, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Oct 16, 2015

Uh oh!

felixb commented Oct 19, 2015

Uh oh!

SparkQA commented Oct 19, 2015

Uh oh!

SparkQA commented Oct 19, 2015

Uh oh!

andrewor14 Oct 19, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 20, 2015

Uh oh!

felixb commented Oct 25, 2015

Uh oh!

andrewor14 commented Nov 9, 2015

Uh oh!

andrewor14 commented Nov 9, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!