Skip to content

[SPARK-4449][Core] Specify port range in spark #3314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-4449][Core] Specify port range in spark #3314

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5f5fda8
https -> http in pom
scwf Nov 17, 2014
4203b7f
start port in specified rang ports
scwf Nov 17, 2014
1773b12
Revert "https -> http in pom"
scwf Nov 17, 2014
cd1a88d
fix 'else' branch and doc update
scwf Nov 18, 2014
da05d64
inclusive-exclusive issue fix
scwf Nov 22, 2014
fa5bcbb
update with apache master
scwf Mar 19, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1837,26 +1837,31 @@ private[spark] object Utils extends Logging {
* Attempt to start a service on the given port, or fail after a number of attempts.
* Each subsequent attempt uses 1 + the port used in the previous attempt (unless the port is 0).
*
* @param startPort The initial port to start the service on.
* @param port The initial port to start the service on.
* @param maxRetries Maximum number of retries to attempt.
* A value of 3 means attempting ports n, n+1, n+2, and n+3, for example.
* @param startService Function to start service on a given port.
* This is expected to throw java.net.BindException on port collision.
* @param conf A SparkConf used to get the maximum number of retries when binding to a port.
* @param serviceName Name of the service.
*/
def startServiceOnPort[T](
startPort: Int,
port: Int,
Copy link

@XuefengWu XuefengWu Nov 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this api, startPort means it would try from this port. I think startPort is a better name.

Copy link
Contributor Author

@scwf scwf Nov 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have used startPort and endPort for range, Using port here also make sense

startService: Int => (T, Int),
conf: SparkConf,
serviceName: String = ""): (T, Int) = {
val serviceString = if (serviceName.isEmpty) "" else s" '$serviceName'"
val maxRetries = portMaxRetries(conf)
val startPort = conf.getInt("spark.port.min", 1024)
val endPort = conf.getInt("spark.port.max", 65536)
for (offset <- 0 to maxRetries) {
// Do not increment port if startPort is 0, which is treated as a special port
val tryPort = if (startPort == 0) {
startPort
// specific a random port between 'spark.port.min' and 'spark.port.max'
// if port is 0
val tryPort = if (port == 0) {
(startPort + Math.random() * (endPort - startPort + 1)).toInt
} else {
// If the new port wraps around, do not try a privilege port
((startPort + offset - 1024) % (65536 - 1024)) + 1024
((port + offset - 1024) % (65536 - 1024)) + 1024
}
try {
val (service, port) = startService(tryPort)
Expand Down
14 changes: 14 additions & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,20 @@ of the most common options to set are:
Logs the effective SparkConf as INFO when a SparkContext is started.
</td>
</tr>
<tr>
<td><code>spark.port.min</code></td>
<td>1024</td>
<td>
Min port for spark(UI, HttpServer. ConnectionManager, Akka)
</td>
</tr>
<tr>
<td><code>spark.port.max</code></td>
<td>65536</td>
<td>
Max port for spark(UI, HttpServer. ConnectionManager, Akka)
</td>
</tr>
<tr>
<td><code>spark.master</code></td>
<td>(none)</td>
Expand Down