Skip to content

Commit bc37c97

Browse files
yanboliangliancheng
authored andcommitted
[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful
Do the same check as #4610 for ParquetRelation2. Author: Yanbo Liang <[email protected]> Closes #5107 from yanboliang/spark-5821-parquet and squashes the following commits: 7092c8d [Yanbo Liang] ParquetRelation2 CTAS should check if delete is successful
1 parent 25e271d commit bc37c97

File tree

1 file changed

+14
-5
lines changed

1 file changed

+14
-5
lines changed

sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -611,13 +611,22 @@ private[sql] case class ParquetRelation2(
611611
val destinationPath = new Path(paths.head)
612612

613613
if (overwrite) {
614-
try {
615-
destinationPath.getFileSystem(conf).delete(destinationPath, true)
616-
} catch {
617-
case e: IOException =>
614+
val fs = destinationPath.getFileSystem(conf)
615+
if (fs.exists(destinationPath)) {
616+
var success: Boolean = false
617+
try {
618+
success = fs.delete(destinationPath, true)
619+
} catch {
620+
case e: IOException =>
621+
throw new IOException(
622+
s"Unable to clear output directory ${destinationPath.toString} prior" +
623+
s" to writing to Parquet table:\n${e.toString}")
624+
}
625+
if (!success) {
618626
throw new IOException(
619627
s"Unable to clear output directory ${destinationPath.toString} prior" +
620-
s" to writing to Parquet file:\n${e.toString}")
628+
s" to writing to Parquet table.")
629+
}
621630
}
622631
}
623632

0 commit comments

Comments
 (0)