[SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service #22498

pgandhi999 · 2018-09-20T15:16:24Z

Recently, the ability to expose the metrics for YARN Shuffle Service was added as part of SPARK-18364. We need to add some metrics to be able to determine the number of active connections as well as open connections to the external shuffle service to benchmark network and connection issues on large cluster environments.

What changes were proposed in this pull request?

Added two more shuffle server metrics for Spark Yarn shuffle service: numRegisteredConnections which indicate the number of registered connections to the shuffle service and numActiveConnections which indicate the number of active connections to the shuffle service at any given point in time.

How was this patch tested?

If these metrics are outputted to a file, we get something like this:

1533674653489 default.shuffleService: Hostname=server1.abc.com, openBlockRequestLatencyMillis_count=729, openBlockRequestLatencyMillis_rate15=0.7110833548897356, openBlockRequestLatencyMillis_rate5=1.657808981793011, openBlockRequestLatencyMillis_rate1=2.2404486061620474, openBlockRequestLatencyMillis_rateMean=0.9242558551196706,
numRegisteredConnections=35,
blockTransferRateBytes_count=2635880512, blockTransferRateBytes_rate15=2578547.6094160094, blockTransferRateBytes_rate5=6048721.726302424, blockTransferRateBytes_rate1=8548922.518223226, blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5, registerExecutorRequestLatencyMillis_count=5, registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836, registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426, registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6, registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053, numActiveConnections=35

Added shuffle server metrics for Spark Yarn shuffle service. I have made my changes on top of Andrew Ash's PR and have additionally added two more metrics on top of them: numRegisteredConnections which indicate the number of registered connections to the shuffle service and numActiveConnections which indicate the number of active connections to the shuffle service at any given point in time. If these metrics are outputted to a file, we get something like this: 1533674653489 default.shuffleService: Hostname=openqe26blue-n9.blue.ygrid.yahoo.com, openBlockRequestLatencyMillis_count=729, openBlockRequestLatencyMillis_rate15=0.7110833548897356, openBlockRequestLatencyMillis_rate5=1.657808981793011, openBlockRequestLatencyMillis_rate1=2.2404486061620474, openBlockRequestLatencyMillis_rateMean=0.9242558551196706, numRegisteredConnections=35, blockTransferRateBytes_count=2635880512, blockTransferRateBytes_rate15=2578547.6094160094, blockTransferRateBytes_rate5=6048721.726302424, blockTransferRateBytes_rate1=8548922.518223226, blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5, registerExecutorRequestLatencyMillis_count=5, registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836, registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426, registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6, registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053, numActiveConnections=35

tgravescs · 2018-09-20T20:54:48Z

ok to test

SparkQA · 2018-09-20T21:00:33Z

Test build #96380 has finished for PR 22498 at commit cf74f36.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class ShuffleMetrics implements MetricSet
public class YarnShuffleServiceMetrics implements MetricsSource

SparkQA · 2018-09-21T02:00:50Z

Test build #96382 has finished for PR 22498 at commit 1ac18d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2018-09-21T14:00:18Z

@pgandhi999 thanks for the pr, since you have additional changes and there are additional changes that were made by the original author here: palantir#236 Perhaps for now we should review the lowest common denominator one and get that in and we can put additional metrics on top of it in separate prs.

pgandhi999 · 2018-09-21T14:31:07Z

@tgravescs Fine with me, thank you.

…RK-18364

SparkQA · 2018-10-04T23:04:30Z

Test build #96950 has finished for PR 22498 at commit edd355e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-06T01:56:04Z

Test build #97013 has finished for PR 22498 at commit 70472a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-12-11T23:48:19Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java

+
+      // register metrics on the block handler into the Node Manager's metrics system.
+      blockHandler.getAllMetrics().getMetrics().put("numRegisteredConnections",
+              shuffleServer.getRegisteredConnections());


nit: indented too far, here and in others.

vanzin · 2018-12-11T23:49:00Z

retest this please

SparkQA · 2018-12-12T04:19:31Z

Test build #99996 has finished for PR 22498 at commit 70472a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-12-12T18:42:36Z

@tgravescs it's not clear from your last comment whether you wanted more things to be added here? otherwise I was planning to merge it.

tgravescs · 2018-12-12T20:45:20Z

no the other ones can be done separately. Sorry I haven't had time to review so thanks for doing it @vanzin.

vanzin · 2018-12-13T00:50:25Z

...on/network-common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java


+  @Override
+  public void channelRegistered(ChannelHandlerContext ctx) {
+    transportContext.getRegisteredConnections().inc();


I think you need to call super.blah in these overrides so that other handlers in the pipeline also get the notification.

Calling superclass methods from overrided methods and fixing indentation.

pgandhi999 · 2018-12-20T17:11:05Z

Sorry, I was on vacation so getting back to the PR today. Thank you for reviewing the PR @vanzin . I have pushed the required changes.

SparkQA · 2018-12-20T17:22:04Z

Test build #100337 has finished for PR 22498 at commit 3c5ef99.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

…RK-18364 [SPARK-25642] : Upmerging with master branch

SparkQA · 2018-12-20T21:37:39Z

Test build #100340 has finished for PR 22498 at commit b5559e3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-12-20T21:40:48Z

retest this please

vanzin · 2018-12-20T21:41:36Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java

+      Counter c = (Counter) metric;
+      long counterValue = c.getCount();
+      metricsRecordBuilder.addGauge(new ShuffleServiceMetricsInfo(name, "Number of " +
+              "connections to shuffle service " + name), counterValue);


indentation

Done. Thank you.

SparkQA · 2018-12-21T04:21:51Z

Test build #100345 has finished for PR 22498 at commit b5559e3.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-12-21T18:45:11Z

Test build #100362 has finished for PR 22498 at commit e2414ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin

Merging to master. I'll remove the import during merge.

vanzin · 2018-12-21T19:12:58Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java


 import java.io.File;
 import java.io.IOException;
+import java.lang.reflect.Method;


Thank you so much. Sorry about that.

…gistered connections as well as the number of active connections to YARN Shuffle Service Recently, the ability to expose the metrics for YARN Shuffle Service was added as part of [SPARK-18364](apache#22485). We need to add some metrics to be able to determine the number of active connections as well as open connections to the external shuffle service to benchmark network and connection issues on large cluster environments. Added two more shuffle server metrics for Spark Yarn shuffle service: numRegisteredConnections which indicate the number of registered connections to the shuffle service and numActiveConnections which indicate the number of active connections to the shuffle service at any given point in time. If these metrics are outputted to a file, we get something like this: 1533674653489 default.shuffleService: Hostname=server1.abc.com, openBlockRequestLatencyMillis_count=729, openBlockRequestLatencyMillis_rate15=0.7110833548897356, openBlockRequestLatencyMillis_rate5=1.657808981793011, openBlockRequestLatencyMillis_rate1=2.2404486061620474, openBlockRequestLatencyMillis_rateMean=0.9242558551196706, numRegisteredConnections=35, blockTransferRateBytes_count=2635880512, blockTransferRateBytes_rate15=2578547.6094160094, blockTransferRateBytes_rate5=6048721.726302424, blockTransferRateBytes_rate1=8548922.518223226, blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5, registerExecutorRequestLatencyMillis_count=5, registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836, registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426, registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6, registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053, numActiveConnections=35 Closes apache#22498 from pgandhi999/SPARK-18364. Authored-by: pgandhi <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>

…gistered connections as well as the number of active connections to YARN Shuffle Service Recently, the ability to expose the metrics for YARN Shuffle Service was added as part of [SPARK-18364](apache#22485). We need to add some metrics to be able to determine the number of active connections as well as open connections to the external shuffle service to benchmark network and connection issues on large cluster environments. Added two more shuffle server metrics for Spark Yarn shuffle service: numRegisteredConnections which indicate the number of registered connections to the shuffle service and numActiveConnections which indicate the number of active connections to the shuffle service at any given point in time. If these metrics are outputted to a file, we get something like this: 1533674653489 default.shuffleService: Hostname=server1.abc.com, openBlockRequestLatencyMillis_count=729, openBlockRequestLatencyMillis_rate15=0.7110833548897356, openBlockRequestLatencyMillis_rate5=1.657808981793011, openBlockRequestLatencyMillis_rate1=2.2404486061620474, openBlockRequestLatencyMillis_rateMean=0.9242558551196706, numRegisteredConnections=35, blockTransferRateBytes_count=2635880512, blockTransferRateBytes_rate15=2578547.6094160094, blockTransferRateBytes_rate5=6048721.726302424, blockTransferRateBytes_rate1=8548922.518223226, blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5, registerExecutorRequestLatencyMillis_count=5, registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836, registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426, registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6, registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053, numActiveConnections=35 Closes apache#22498 from pgandhi999/SPARK-18364. Authored-by: pgandhi <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]> (cherry picked from commit 8dd29fe)

…gistered connections as well as the number of active connections to YARN Shuffle Service Recently, the ability to expose the metrics for YARN Shuffle Service was added as part of [SPARK-18364](apache#22485). We need to add some metrics to be able to determine the number of active connections as well as open connections to the external shuffle service to benchmark network and connection issues on large cluster environments. Added two more shuffle server metrics for Spark Yarn shuffle service: numRegisteredConnections which indicate the number of registered connections to the shuffle service and numActiveConnections which indicate the number of active connections to the shuffle service at any given point in time. If these metrics are outputted to a file, we get something like this: 1533674653489 default.shuffleService: Hostname=server1.abc.com, openBlockRequestLatencyMillis_count=729, openBlockRequestLatencyMillis_rate15=0.7110833548897356, openBlockRequestLatencyMillis_rate5=1.657808981793011, openBlockRequestLatencyMillis_rate1=2.2404486061620474, openBlockRequestLatencyMillis_rateMean=0.9242558551196706, numRegisteredConnections=35, blockTransferRateBytes_count=2635880512, blockTransferRateBytes_rate15=2578547.6094160094, blockTransferRateBytes_rate5=6048721.726302424, blockTransferRateBytes_rate1=8548922.518223226, blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5, registerExecutorRequestLatencyMillis_count=5, registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836, registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426, registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6, registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053, numActiveConnections=35 Closes apache#22498 from pgandhi999/SPARK-18364. Authored-by: pgandhi <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]> (cherry picked from commit 8dd29fe) (cherry picked from commit 7a6d4d476a81419be0561d0cb98ad5b89de0782d) Change-Id: I682129f3c372aa83e382f159423b440344eee266 (cherry picked from commit 905ab9e0b4f67b226ef5ce4e3b2088d0d5cd43aa)

[SPARK-18364] : Fixing Scalastyle Tests

1ac18d9

Merge branch 'master' of https://github.com/pgandhi999/spark into SPA…

edd355e

…RK-18364

pgandhi999 changed the title ~~[SPARK-18364] : Expose metrics for YarnShuffleService~~ [SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service Oct 4, 2018

[SPARK-18364] : Fixing Code to pass Unit tests

70472a2

vanzin reviewed Dec 11, 2018

View reviewed changes

vanzin reviewed Dec 13, 2018

View reviewed changes

[SPARK-18364] : Addressing Reviews December 20, 2018

3c5ef99

Calling superclass methods from overrided methods and fixing indentation.

Merge branch 'master' of https://github.com/pgandhi999/spark into SPA…

b5559e3

…RK-18364 [SPARK-25642] : Upmerging with master branch

vanzin reviewed Dec 20, 2018

View reviewed changes

[SPARK-25642] : Fixing indentation

e2414ca

vanzin reviewed Dec 21, 2018

View reviewed changes

asfgit closed this in 8dd29fe Dec 21, 2018

[SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service #22498

[SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service #22498

Uh oh!

Conversation

pgandhi999 commented Sep 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

tgravescs commented Sep 20, 2018

Uh oh!

SparkQA commented Sep 20, 2018

Uh oh!

SparkQA commented Sep 21, 2018

Uh oh!

tgravescs commented Sep 21, 2018

Uh oh!

pgandhi999 commented Sep 21, 2018

Uh oh!

SparkQA commented Oct 4, 2018

Uh oh!

SparkQA commented Oct 6, 2018

Uh oh!

vanzin Dec 11, 2018

Choose a reason for hiding this comment

Uh oh!

vanzin commented Dec 11, 2018

Uh oh!

SparkQA commented Dec 12, 2018

Uh oh!

vanzin commented Dec 12, 2018

Uh oh!

tgravescs commented Dec 12, 2018

Uh oh!

vanzin Dec 13, 2018

Choose a reason for hiding this comment

Uh oh!

pgandhi999 commented Dec 20, 2018

Uh oh!

SparkQA commented Dec 20, 2018

Uh oh!

SparkQA commented Dec 20, 2018

Uh oh!

vanzin commented Dec 20, 2018

Uh oh!

vanzin Dec 20, 2018

Choose a reason for hiding this comment

Uh oh!

pgandhi999 Dec 21, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 21, 2018

Uh oh!

SparkQA commented Dec 21, 2018

Uh oh!

vanzin left a comment

Choose a reason for hiding this comment

Uh oh!

vanzin Dec 21, 2018

Choose a reason for hiding this comment

Uh oh!

pgandhi999 Dec 21, 2018

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pgandhi999 commented Sep 20, 2018 •

edited

Loading