[SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system #5719

zuxqoj · 2015-04-27T11:25:14Z

No description provided.

…rkYarnAM actor system

srowen · 2015-04-27T11:37:51Z

Does this need to be documented too?

zuxqoj · 2015-04-27T11:45:50Z

I think we should document it.
I will update running-on-yarn.md also

…rkYarnAM actor system

zuxqoj · 2015-04-27T11:59:17Z

added "spark.yarn.am.port" in running-on-yarn.md

srowen · 2015-04-27T12:12:32Z

@zsxwing @vanzin what do you think?

SparkQA · 2015-04-27T13:45:08Z

Test build #718 has finished for PR 5719 at commit 456a592.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

tgravescs · 2015-04-27T14:05:43Z

docs/running-on-yarn.md

I think we should be more specific in the description of this. an application master could have many ports open, what is this one for. I think either put in name or atleast put in description.

Also kind of confusing is that in other places we use the spark.yarn.am.* namespace to refer only to the yarn-client application master. If this applies to both yarn-client and yarn-cluster, the naming is kind of inconsistent.

@sryza what do you suggest.

Unless I'm missing something this is only used in yarn client mode. This the communication channel (rpc) between AM on cluster and driver on the client side. It looks like the rest of spark uses things just like spark.executor.port, spark.driver.port, etc.. So this kind of follows that convention. Personally I would like it more descriptive but since that is kind of convention I'm ok with it.

@WangTaoTheTonic I am not 100% sure but I think it does communicate with driver also, in runAMEndpoint method it is creating RpcEndpoint to communicate with driver. Can someone please confirm, I will update description accordingly.

vanzin · 2015-04-27T17:13:37Z

So, isn't a single port here kind of limiting? If you're unlucky and two AMs start on the same node, one of them will fail. Wouldn't it be better to have a port range here?

zuxqoj · 2015-04-27T17:24:51Z

@vanzin user is expected to specify different port number for different application.

vanzin · 2015-04-27T17:26:10Z

That is not very user-friendly at all.

zsxwing · 2015-04-27T17:29:39Z

@vanzin, Rpc.create will call Utils.startServiceOnPort to find another port for port conflicts.

vanzin · 2015-04-27T17:33:43Z

Rpc.create will call Utils.startServiceOnPort to find another port for port conflicts.

I see. In that case the documentation added in this change is wrong, since the actual port will be anything between x and x + spark.port.maxRetries.

tgravescs · 2015-04-27T17:36:42Z

There is also a separate jira to add support for port ranges. SPARK-4449

tgravescs · 2015-04-27T18:25:57Z

ok to test

…ystem

WangTaoTheTonic · 2015-04-29T07:47:57Z

docs/running-on-yarn.md

Is this channel to communicate with driver either?

this is not the channel used to communicate with executors.
This channel is used in yarn client mode to communicate between the spark driver running on a gateway and the application master (which is doing container requests) running on yarn.

It also looks like I was wrong my previous comment. It actually is also used in yarn cluster mode. Its used for the dynamic executor feature, where it is handling the kill from the scheduler backend.

SparkQA · 2015-04-29T08:45:53Z

Test build #31258 has finished for PR 5719 at commit 9de5330.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

tgravescs · 2015-05-01T20:24:31Z

this is odd, I don't see our comments showing up out here. @zuxqoj can you look at the Files changed tab and update based on the comments

@tgravescs

…ystem updated config description as suggested by @tgravescs

srowen · 2015-05-02T10:01:54Z

LGTM pending tests

SparkQA · 2015-05-02T10:14:38Z

Test build #31661 has finished for PR 5719 at commit 5117258.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-05-02T22:19:15Z

docs/running-on-yarn.md

Nits: yarn -> YARN, spark -> Spark, 'application master' -> 'Application Master'. Is this really the most accurate description of the AM role? it's where you find the Spark UI, right? the rest of this info isn't so useful to callers. It's just the port where the AM listens.

@srowen

…ystem Incorporating @srowen feedback

SparkQA · 2015-05-03T10:20:05Z

Test build #31688 has finished for PR 5719 at commit 5574ff7.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-05-03T12:13:27Z

Jenkins, retest this please.

SparkQA · 2015-05-03T13:58:41Z

Test build #31690 has finished for PR 5719 at commit 5574ff7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…system Author: shekhar.bansal <[email protected]> Closes #5719 from zuxqoj/master and squashes the following commits: 5574ff7 [shekhar.bansal] [SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system 5117258 [shekhar.bansal] [SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system 9de5330 [shekhar.bansal] [SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system 456a592 [shekhar.bansal] [SPARK-6653][yarn] New configuration property to specify port for sparkYarnAM actor system 803e93e [shekhar.bansal] [SPARK-6653][yarn] New configuration property to specify port for sparkYarnAM actor system (cherry picked from commit fc8feaa) Signed-off-by: Sean Owen <[email protected]>

…system Author: shekhar.bansal <[email protected]> Closes apache#5719 from zuxqoj/master and squashes the following commits: 5574ff7 [shekhar.bansal] [SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system 5117258 [shekhar.bansal] [SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system 9de5330 [shekhar.bansal] [SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system 456a592 [shekhar.bansal] [SPARK-6653][yarn] New configuration property to specify port for sparkYarnAM actor system 803e93e [shekhar.bansal] [SPARK-6653][yarn] New configuration property to specify port for sparkYarnAM actor system

[SPARK-6653][yarn] New configuration property to specify port for spa…

803e93e

…rkYarnAM actor system

[SPARK-6653][yarn] New configuration property to specify port for spa…

456a592

…rkYarnAM actor system

tgravescs reviewed Apr 27, 2015
View reviewed changes

[SPARK-6653][yarn] New config to specify port for sparkYarnAM actor s…

9de5330

…ystem

WangTaoTheTonic reviewed Apr 29, 2015
View reviewed changes

[SPARK-6653][yarn] New config to specify port for sparkYarnAM actor s…

5117258

…ystem updated config description as suggested by @tgravescs

srowen reviewed May 2, 2015
View reviewed changes

[SPARK-6653][yarn] New config to specify port for sparkYarnAM actor s…

5574ff7

…ystem Incorporating @srowen feedback

asfgit closed this in fc8feaa May 5, 2015

[SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system #5719

[SPARK-6653][yarn] New config to specify port for sparkYarnAM actor system #5719

Uh oh!

Conversation

zuxqoj commented Apr 27, 2015

Uh oh!

srowen commented Apr 27, 2015

Uh oh!

zuxqoj commented Apr 27, 2015

Uh oh!

zuxqoj commented Apr 27, 2015

Uh oh!

srowen commented Apr 27, 2015

Uh oh!

SparkQA commented Apr 27, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vanzin commented Apr 27, 2015

Uh oh!

zuxqoj commented Apr 27, 2015

Uh oh!

vanzin commented Apr 27, 2015

Uh oh!

zsxwing commented Apr 27, 2015

Uh oh!

vanzin commented Apr 27, 2015

Uh oh!

tgravescs commented Apr 27, 2015

Uh oh!

tgravescs commented Apr 27, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 29, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

srowen commented May 2, 2015

Uh oh!

SparkQA commented May 2, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 3, 2015

Uh oh!

srowen commented May 3, 2015

Uh oh!

SparkQA commented May 3, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants