Skip to content

Commit 7536e28

Browse files
steveloughransquito
authored andcommitted
[SPARK-20038][SQL] FileFormatWriter.ExecuteWriteTask.releaseResources() implementations to be re-entrant
## What changes were proposed in this pull request? have the`FileFormatWriter.ExecuteWriteTask.releaseResources()` implementations set `currentWriter=null` in a finally clause. This guarantees that if the first call to `currentWriter()` throws an exception, the second releaseResources() call made during the task cancel process will not trigger a second attempt to close the stream. ## How was this patch tested? Tricky. I've been fixing the underlying cause when I saw the problem [HADOOP-14204](https://issues.apache.org/jira/browse/HADOOP-14204), but SPARK-10109 shows I'm not the first to have seen this. I can't replicate it locally any more, my code no longer being broken. code review, however, should be straightforward Author: Steve Loughran <[email protected]> Closes #17364 from steveloughran/stevel/SPARK-20038-close.
1 parent 8ddf0d2 commit 7536e28

File tree

1 file changed

+10
-4
lines changed

1 file changed

+10
-4
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -324,8 +324,11 @@ object FileFormatWriter extends Logging {
324324

325325
override def releaseResources(): Unit = {
326326
if (currentWriter != null) {
327-
currentWriter.close()
328-
currentWriter = null
327+
try {
328+
currentWriter.close()
329+
} finally {
330+
currentWriter = null
331+
}
329332
}
330333
}
331334
}
@@ -459,8 +462,11 @@ object FileFormatWriter extends Logging {
459462

460463
override def releaseResources(): Unit = {
461464
if (currentWriter != null) {
462-
currentWriter.close()
463-
currentWriter = null
465+
try {
466+
currentWriter.close()
467+
} finally {
468+
currentWriter = null
469+
}
464470
}
465471
}
466472
}

0 commit comments

Comments
 (0)