Skip to content

Commit 3fe3365

Browse files
sidedoorleftroaddongjoon-hyun
authored andcommitted
[SPARK-32172][CORE] Use createDirectory instead of mkdir
### What changes were proposed in this pull request? Use Files.createDirectory() to create local directory instead of File.mkdir() in DiskBlockManager. Many times, we will see such error log information like "Failed to create local dir in xxxxxx". But there is no clear information indicating why the directory creation failed. When Files.createDirectory() fails to create a local directory, it can give specific error information for subsequent troubleshooting(also throws IOException). ### Why are the changes needed? Throw clear error message when creating directory fails. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? `DiskBlockManagerSuite` Closes apache#28997 from sidedoorleftroad/SPARK-32172. Authored-by: sidedoorleftroad <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 1d18096 commit 3fe3365

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
package org.apache.spark.storage
1919

2020
import java.io.{File, IOException}
21+
import java.nio.file.Files
2122
import java.util.UUID
2223

2324
import org.apache.spark.SparkConf
@@ -69,8 +70,8 @@ private[spark] class DiskBlockManager(conf: SparkConf, deleteFilesOnStop: Boolea
6970
old
7071
} else {
7172
val newDir = new File(localDirs(dirId), "%02x".format(subDirId))
72-
if (!newDir.exists() && !newDir.mkdir()) {
73-
throw new IOException(s"Failed to create local dir in $newDir.")
73+
if (!newDir.exists()) {
74+
Files.createDirectory(newDir.toPath)
7475
}
7576
subDirs(dirId)(subDirId) = newDir
7677
newDir

0 commit comments

Comments
 (0)