Skip to content

Super minor: Close inputStream in SparkSubmitArguments #914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Super minor: Close inputStream in SparkSubmitArguments #914

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,16 +381,19 @@ private[spark] class SparkSubmitArguments(args: Seq[String]) {
object SparkSubmitArguments {
/** Load properties present in the given file. */
def getPropertiesFromFile(file: File): Seq[(String, String)] = {
require(file.exists(), s"Properties file ${file.getName} does not exist")
require(file.exists(), s"Properties file $file does not exist")
require(file.isFile(), s"Properties file $file is not a normal file")
Copy link
Contributor

@rxin rxin May 31, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does symlink work here?

Copy link
Member

@srowen srowen May 31, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isFile() is false for symlinks. It may be more conservative to require !file.isDirectory(), since it seems valid to point to a symlinked config file.

Copy link
Contributor Author

@aarondav aarondav May 31, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File.isFile() returns true for symlinks which point to files.

$ touch testfile
$ ln -s testfile testlink
$ scala
scala> new java.io.File("testlink").isFile()
res0: Boolean = true

Additionally, since the docs aren't 100% clear and I couldn't find a solid answer from Google, I checked both the UnixFileSystem and WindowsFileSystem. The former uses stat which resolves symbolic links. The latter will set isFile to true if and only if it is not a directory, so symlinks would be included.

Copy link
Member

@srowen srowen May 31, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, my test was silly, as I realize I had it pointed at a directory actually.

val inputStream = new FileInputStream(file)
val properties = new Properties()
try {
val properties = new Properties()
properties.load(inputStream)
properties.stringPropertyNames().toSeq.map(k => (k, properties(k).trim))
} catch {
case e: IOException =>
val message = s"Failed when loading Spark properties file ${file.getName}"
val message = s"Failed when loading Spark properties file $file"
throw new SparkException(message, e)
} finally {
inputStream.close()
}
properties.stringPropertyNames().toSeq.map(k => (k, properties(k).trim))
}
}