Skip to content

Update CommandUtils.scala #157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

Update CommandUtils.scala #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e9d7fb7
Update CommandUtils.scala
baishuo Mar 16, 2014
679a82a
Update CommandUtils.scala
baishuo Mar 17, 2014
8cd5fca
Update CommandUtils.scala
baishuo Mar 17, 2014
90c1588
Update monitoring.md
baishuo Mar 17, 2014
3519e51
Update CommandUtils.scala
baishuo Mar 17, 2014
a7093de
Update monitoring.md
baishuo Mar 17, 2014
9f933c6
Update monitoring.md
baishuo Mar 17, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,14 @@ object CommandUtils extends Logging {
Seq(sparkHome + "/bin/compute-classpath" + ext),
extraEnvironment=command.environment)

Seq("-cp", classPath) ++ libraryOpts ++ workerLocalOpts ++ userOpts ++ memoryOpts
val debugflag = System.getProperty("spark.excutor.debug", "false").toBoolean
if (!debugflag) {
Seq("-cp", classPath) ++ libraryOpts ++ workerLocalOpts ++ userOpts ++ memoryOpts
}
else {
val debugInfo = "-Xrunjdwp:transport=dt_socket,address=" +System.getProperty("spark.excutor.debug.port", "18000") +",server=y,suspend=y"
Seq("-Xdebug",debugInfo, "-cp", classPath) ++ libraryOpts ++ workerLocalOpts ++ userOpts ++ memoryOpts
}
}

/** Spawn a thread that will redirect a given stream to a file */
Expand Down
4 changes: 4 additions & 0 deletions docs/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,7 @@ can provide fine-grained profiling on individual nodes.
* JVM utilities such as `jstack` for providing stack traces, `jmap` for creating heap-dumps,
`jstat` for reporting time-series statistics and `jconsole` for visually exploring various JVM
properties are useful for those comfortable with JVM internals.

#debug the Excutor process

Since the excutor process is started by ProcessBuilder, if we wang to trace the code, we can modify the CommandUtils.scala, and rebuild the package. We can set spark.excutor.debug=true(default is false) to turn on this function and set spark.excutor.debug.port=xxxx （default is 18000）to specify the port for remote-debugging . The both option are set as jvm arguments when we start one worker process. please refer https://github.com/apache/spark/pull/157 for detail