[SPARK-10997] [core] Add "client mode" to netty rpc env. #9210

vanzin · 2015-10-22T00:44:20Z

"Client mode" means the RPC env will not listen for incoming connections.
This allows certain processes in the Spark stack (such as Executors or
tha YARN client-mode AM) to act as pure clients when using the netty-based
RPC backend, reducing the number of sockets needed by the app and also the
number of open ports.

Client connections are also preferred when endpoints that actually have
a listening socket are involved; so, for example, if a Worker connects
to a Master and the Master needs to send a message to a Worker endpoint,
that client connection will be used, even though the Worker is also
listening for incoming connections.

With this change, the workaround for SPARK-10987 isn't necessary anymore, and
is removed. The AM connects to the driver in "client mode", and that connection
is used for all driver <-> AM communication, and so the AM is properly notified
when the connection goes down.

"Client mode" means the RPC env will not listen for incoming connections. This allows certain processes in the Spark stack (such as Executors or tha YARN client-mode AM) to act as pure clients when using the netty-based RPC backend, reducing the number of sockets needed by the app and also the number of open ports. Client connections are also preferred when endpoints that actually have a listening socket are involved; so, for example, if a Worker connects to a Master and the Master needs to send a message to a Worker endpoint, that client connection will be used, even though the Worker is also listening for incoming connections. With this change, the workaround for SPARK-10987 isn't necessary anymore, and is removed. The AM connects to the driver in "client mode", and that connection is used for all driver <-> AM communication, and so the AM is properly notified when the connection goes down. This change also removes the workaround for SPARK-10987.

vanzin · 2015-10-22T00:44:42Z

network/yarn/pom.xml

note: this is unrelated but sbt started complaining about this dependency while testing this change.

vanzin · 2015-10-22T00:48:43Z

/cc @rxin @zsxwing

I tested this on a real YARN cluster with dynamic allocation (= also used the external shuffle service), everything looks fine.

SparkQA · 2015-10-22T03:37:28Z

Test build #44110 has finished for PR 9210 at commit 48f3c55.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

vanzin · 2015-10-22T04:46:24Z

retest this please

SparkQA · 2015-10-22T07:16:22Z

Test build #44127 has finished for PR 9210 at commit 48f3c55.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

zsxwing · 2015-10-22T14:41:11Z

core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala

nit: _address and _name can be val.

zsxwing · 2015-10-22T15:02:47Z

Just a nit. Otherwise LGTM

SparkQA · 2015-10-23T00:42:08Z

Test build #44185 has finished for PR 9210 at commit da4d824.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

vanzin · 2015-10-23T00:45:41Z

retest this please

SparkQA · 2015-10-23T02:45:20Z

Test build #44187 has finished for PR 9210 at commit da4d824.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

rxin · 2015-10-23T03:41:32Z

core/src/main/scala/org/apache/spark/SparkEnv.scala

can you add some comment on what this is checking and when this can happen

Conflicts: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala

SparkQA · 2015-10-23T07:26:12Z

Test build #44200 has finished for PR 9210 at commit 418b472.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

SparkQA · 2015-10-23T07:51:57Z

Test build #44199 has finished for PR 9210 at commit 11d9092.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

SparkQA · 2015-10-24T22:01:01Z

Test build #44298 has finished for PR 9210 at commit acce31f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

vanzin · 2015-10-28T08:21:08Z

hi there, any remaining feedback here?

vanzin · 2015-11-01T23:12:18Z

retest this please

SparkQA · 2015-11-02T01:41:30Z

Test build #44778 has finished for PR 9210 at commit acce31f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

vanzin · 2015-11-02T01:42:42Z

hmm, akka test... it's failed for me in unrelated changes, also. retest this please.

SparkQA · 2015-11-02T04:08:46Z

Test build #44787 has finished for PR 9210 at commit acce31f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage\n

vanzin · 2015-11-02T18:26:15Z

Merging this to master.

zsxwing · 2015-11-02T18:37:03Z

Just took another look. LGTM

vanzin reviewed Oct 22, 2015
View reviewed changes

network/yarn/pom.xml

Copy link

Contributor Author

vanzin Oct 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: this is unrelated but sbt started complaining about this dependency while testing this change.

zsxwing reviewed Oct 22, 2015
View reviewed changes

core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala Outdated

Copy link

Member

zsxwing Oct 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: _address and _name can be val.

Feedback.

da4d824

rxin reviewed Oct 23, 2015
View reviewed changes

Marcelo Vanzin added 2 commits October 22, 2015 21:44

Feedback.

11d9092

Merge branch 'master' into SPARK-10997

418b472

Conflicts: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala

vanzin mentioned this pull request Oct 23, 2015

SPARK-11286 Make Outbox stopped exception singleton #9254

Closed

Restore a comment.

acce31f

asfgit closed this in 71d1c90 Nov 2, 2015

vanzin deleted the SPARK-10997 branch November 4, 2015 17:14

andrewor14 mentioned this pull request Dec 11, 2015

[SPARK-11131] [core] Fix race in worker registration protocol. #9138

Closed

[SPARK-10997] [core] Add "client mode" to netty rpc env. #9210

[SPARK-10997] [core] Add "client mode" to netty rpc env. #9210

Uh oh!

Conversation

vanzin commented Oct 22, 2015

Uh oh!

vanzin Oct 22, 2015

Choose a reason for hiding this comment

Uh oh!

vanzin commented Oct 22, 2015

Uh oh!

SparkQA commented Oct 22, 2015

Uh oh!

vanzin commented Oct 22, 2015

Uh oh!

SparkQA commented Oct 22, 2015

Uh oh!

zsxwing Oct 22, 2015

Choose a reason for hiding this comment

Uh oh!

zsxwing commented Oct 22, 2015

Uh oh!

SparkQA commented Oct 23, 2015

Uh oh!

vanzin commented Oct 23, 2015

Uh oh!

SparkQA commented Oct 23, 2015

Uh oh!

rxin Oct 23, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 23, 2015

Uh oh!

SparkQA commented Oct 23, 2015

Uh oh!

SparkQA commented Oct 24, 2015

Uh oh!

vanzin commented Oct 28, 2015

Uh oh!

vanzin commented Nov 1, 2015

Uh oh!

SparkQA commented Nov 2, 2015

Uh oh!

vanzin commented Nov 2, 2015

Uh oh!

SparkQA commented Nov 2, 2015

Uh oh!

vanzin commented Nov 2, 2015

Uh oh!

zsxwing commented Nov 2, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants