[SPARK-7007][core] Add a metric source for ExecutorAllocationManager #5589

jerryshao · 2015-04-20T09:03:43Z

Add a metric source to expose the internal status of ExecutorAllocationManager to better monitoring the resource usage of executors when dynamic allocation is enable. Please help to review, thanks a lot.

SparkQA · 2015-04-20T10:57:31Z

Test build #30580 has finished for PR 5589 at commit a6d5ec5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

jerryshao · 2015-04-20T12:36:44Z

Jenkins, retest this please.

SparkQA · 2015-04-20T14:23:08Z

Test build #30587 has finished for PR 5589 at commit a6d5ec5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-27T21:03:18Z

Test build #30999 has finished for PR 5589 at commit a6d5ec5.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.
This patch adds the following new dependencies:
- activation-1.1.jar
- aopalliance-1.0.jar
- avro-1.7.7.jar
- breeze-macros_2.10-0.11.2.jar
- breeze_2.10-0.11.2.jar
- commons-cli-1.2.jar
- commons-codec-1.10.jar
- commons-compress-1.4.1.jar
- commons-io-2.1.jar
- commons-lang-2.5.jar
- commons-math3-3.4.1.jar
- gmbal-api-only-3.0.0-b023.jar
- grizzly-framework-2.1.2.jar
- grizzly-http-2.1.2.jar
- grizzly-http-server-2.1.2.jar
- grizzly-http-servlet-2.1.2.jar
- grizzly-rcm-2.1.2.jar
- guice-3.0.jar
- hadoop-annotations-2.2.0.jar
- hadoop-auth-2.2.0.jar
- hadoop-client-2.2.0.jar
- hadoop-common-2.2.0.jar
- hadoop-hdfs-2.2.0.jar
- hadoop-mapreduce-client-app-2.2.0.jar
- hadoop-mapreduce-client-common-2.2.0.jar
- hadoop-mapreduce-client-core-2.2.0.jar
- hadoop-mapreduce-client-jobclient-2.2.0.jar
- hadoop-mapreduce-client-shuffle-2.2.0.jar
- hadoop-yarn-api-2.2.0.jar
- hadoop-yarn-client-2.2.0.jar
- hadoop-yarn-common-2.2.0.jar
- hadoop-yarn-server-common-2.2.0.jar
- ivy-2.4.0.jar
- jackson-annotations-2.4.0.jar
- jackson-core-2.4.4.jar
- jackson-databind-2.4.4.jar
- jackson-jaxrs-1.8.8.jar
- jackson-module-scala_2.10-2.4.4.jar
- jackson-xc-1.8.8.jar
- javax.inject-1.jar
- javax.servlet-3.0.0.v201112011016.jar
- javax.servlet-3.1.jar
- javax.servlet-api-3.0.1.jar
- jaxb-api-2.2.2.jar
- jaxb-impl-2.2.3-1.jar
- jersey-client-1.9.jar
- jersey-core-1.9.jar
- jersey-grizzly2-1.9.jar
- jersey-guice-1.9.jar
- jersey-json-1.9.jar
- jersey-server-1.9.jar
- jersey-test-framework-core-1.9.jar
- jersey-test-framework-grizzly2-1.9.jar
- jettison-1.1.jar
- jetty-util-6.1.26.jar
- jodd-core-3.6.3.jar
- management-api-3.0.0-b012.jar
- protobuf-java-2.4.1.jar
- snappy-java-1.1.1.7.jar
- spark-bagel_2.10-1.4.0-SNAPSHOT.jar
- spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
- spark-core_2.10-1.4.0-SNAPSHOT.jar
- spark-graphx_2.10-1.4.0-SNAPSHOT.jar
- spark-launcher_2.10-1.4.0-SNAPSHOT.jar
- spark-mllib_2.10-1.4.0-SNAPSHOT.jar
- spark-network-common_2.10-1.4.0-SNAPSHOT.jar
- spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
- spark-repl_2.10-1.4.0-SNAPSHOT.jar
- spark-sql_2.10-1.4.0-SNAPSHOT.jar
- spark-streaming_2.10-1.4.0-SNAPSHOT.jar
- stax-api-1.0.1.jar
- xz-1.0.jar
This patch removes the following dependencies:
- breeze-macros_2.10-0.3.1.jar
- breeze_2.10-0.10.jar
- commons-codec-1.5.jar
- commons-el-1.0.jar
- commons-io-2.4.jar
- commons-lang-2.4.jar
- commons-math3-3.1.1.jar
- hadoop-client-1.0.4.jar
- hadoop-core-1.0.4.jar
- hsqldb-1.8.0.10.jar
- jackson-annotations-2.3.0.jar
- jackson-core-2.3.0.jar
- jackson-databind-2.3.0.jar
- jblas-1.2.3.jar
- snappy-java-1.1.1.6.jar
- spark-bagel_2.10-1.3.0-SNAPSHOT.jar
- spark-catalyst_2.10-1.3.0-SNAPSHOT.jar
- spark-core_2.10-1.3.0-SNAPSHOT.jar
- spark-graphx_2.10-1.3.0-SNAPSHOT.jar
- spark-mllib_2.10-1.3.0-SNAPSHOT.jar
- spark-network-common_2.10-1.3.0-SNAPSHOT.jar
- spark-network-shuffle_2.10-1.3.0-SNAPSHOT.jar
- spark-repl_2.10-1.3.0-SNAPSHOT.jar
- spark-sql_2.10-1.3.0-SNAPSHOT.jar
- spark-streaming_2.10-1.3.0-SNAPSHOT.jar

andrewor14 · 2015-04-28T19:22:41Z

@jerryshao this looks good. It will allow us to plot these variables in very interesting ways. Note that this relies on internal variables that are subject to change. For instance, #5536 gets rid of numExecutorsPending. Maybe we should add a comment at the top to indicate that this is unstable across Spark versions.

andrewor14 · 2015-04-28T19:23:50Z

core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala

Would you mind putting this as an inner class at the bottom (like ExecutorAllocationManagerListener) instead?

andrewor14 · 2015-04-28T19:24:01Z

Also it seems that you need to rebase this one.

jerryshao · 2015-04-29T01:19:54Z

Hi @andrewor14 , thanks a lot for your comments, I will change the code accordingly.

SparkQA · 2015-04-29T04:15:42Z

Test build #31218 has finished for PR 5589 at commit 08ff59e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

jerryshao · 2015-04-29T04:26:57Z

Hi @andrewor14 , I've changed the code according to your comments, please take a look at this, thanks a lot.

sryza · 2015-04-29T20:42:37Z

core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala

The class is already private[spark], so we don't need to declare this again here.

SparkQA · 2015-04-30T02:58:00Z

Test build #31364 has finished for PR 5589 at commit 6cb0c37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

andrewor14 · 2015-05-02T01:46:29Z

@jerryshao I believe this doesn't compile anymore since I merged #5074. Would you mind rebasing this?

sryza · 2015-05-02T02:02:00Z

@jerryshao no specific concerns about the implementation. If @andrewor14 is ok with the inconsistency, I'm fine with it too.

andrewor14 · 2015-05-02T03:37:25Z

Yeah it's actually going to be a little tricky to bring this outside because this accesses a whole bunch of internal variables. I even think that we should add a big comment at the top of the source class to say this is experimental and there are no backward compatibility guarantees at all.

jerryshao · 2015-05-02T05:25:59Z

@andrewor14 @sryza , thanks a lot for your comments, I just rebase the codes, please review it.

SparkQA · 2015-05-02T07:05:56Z

Test build #31656 has finished for PR 5589 at commit 104d155.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ShuffleHandle(val shuffleId: Int) extends Serializable
- class SaslEncryption
- static class EncryptedMessage extends AbstractReferenceCounted implements FileRegion
- class SaslRpcHandler extends RpcHandler
- public class SaslServerBootstrap implements TransportServerBootstrap
- public class SparkSaslClient implements SaslEncryptionBackend
- public class SparkSaslServer implements SaslEncryptionBackend
- public class ByteArrayWritableChannel implements WritableByteChannel
- class DataFrameStatFunctions(object):

andrewor14 · 2015-05-05T16:43:39Z

Looks great. Merging into master and 1.4 thanks @sryza @jerryshao

Add a metric source to expose the internal status of ExecutorAllocationManager to better monitoring the resource usage of executors when dynamic allocation is enable. Please help to review, thanks a lot. Author: jerryshao <[email protected]> Closes #5589 from jerryshao/dynamic-allocation-source and squashes the following commits: 104d155 [jerryshao] rebase and address the comments c501a2c [jerryshao] Address the comments d237ba5 [jerryshao] Address the comments 2c3540f [jerryshao] Add a metric source for ExecutorAllocationManager (cherry picked from commit 9f1f9b1) Signed-off-by: Andrew Or <[email protected]>

Add a metric source to expose the internal status of ExecutorAllocationManager to better monitoring the resource usage of executors when dynamic allocation is enable. Please help to review, thanks a lot. Author: jerryshao <[email protected]> Closes apache#5589 from jerryshao/dynamic-allocation-source and squashes the following commits: 104d155 [jerryshao] rebase and address the comments c501a2c [jerryshao] Address the comments d237ba5 [jerryshao] Address the comments 2c3540f [jerryshao] Add a metric source for ExecutorAllocationManager

andrewor14 reviewed Apr 28, 2015
View reviewed changes

jerryshao force-pushed the dynamic-allocation-source branch from a6d5ec5 to 08ff59e Compare April 29, 2015 02:32

sryza reviewed Apr 29, 2015
View reviewed changes

jerryshao added 3 commits May 2, 2015 12:58

Add a metric source for ExecutorAllocationManager

2c3540f

Address the comments

d237ba5

Address the comments

c501a2c

jerryshao force-pushed the dynamic-allocation-source branch from 6cb0c37 to 104d155 Compare May 2, 2015 05:22

rebase and address the comments

104d155

asfgit closed this in 9f1f9b1 May 5, 2015

[SPARK-7007][core] Add a metric source for ExecutorAllocationManager #5589

[SPARK-7007][core] Add a metric source for ExecutorAllocationManager #5589

Uh oh!

Conversation

jerryshao commented Apr 20, 2015

Uh oh!

SparkQA commented Apr 20, 2015

Uh oh!

jerryshao commented Apr 20, 2015

Uh oh!

SparkQA commented Apr 20, 2015

Uh oh!

SparkQA commented Apr 27, 2015

Uh oh!

andrewor14 commented Apr 28, 2015

Uh oh!

andrewor14 Apr 28, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Apr 28, 2015

Uh oh!

jerryshao commented Apr 29, 2015

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

jerryshao commented Apr 29, 2015

Uh oh!

sryza Apr 29, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

andrewor14 commented May 2, 2015

Uh oh!

sryza commented May 2, 2015

Uh oh!

andrewor14 commented May 2, 2015

Uh oh!

jerryshao commented May 2, 2015

Uh oh!

SparkQA commented May 2, 2015

Uh oh!

andrewor14 commented May 5, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants