Commit 7536e28
[SPARK-20038][SQL] FileFormatWriter.ExecuteWriteTask.releaseResources() implementations to be re-entrant
## What changes were proposed in this pull request?
have the`FileFormatWriter.ExecuteWriteTask.releaseResources()` implementations set `currentWriter=null` in a finally clause. This guarantees that if the first call to `currentWriter()` throws an exception, the second releaseResources() call made during the task cancel process will not trigger a second attempt to close the stream.
## How was this patch tested?
Tricky. I've been fixing the underlying cause when I saw the problem [HADOOP-14204](https://issues.apache.org/jira/browse/HADOOP-14204), but SPARK-10109 shows I'm not the first to have seen this. I can't replicate it locally any more, my code no longer being broken.
code review, however, should be straightforward
Author: Steve Loughran <[email protected]>
Closes #17364 from steveloughran/stevel/SPARK-20038-close.1 parent 8ddf0d2 commit 7536e28
File tree
1 file changed
+10
-4
lines changed- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources
1 file changed
+10
-4
lines changedLines changed: 10 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
324 | 324 | | |
325 | 325 | | |
326 | 326 | | |
327 | | - | |
328 | | - | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
329 | 332 | | |
330 | 333 | | |
331 | 334 | | |
| |||
459 | 462 | | |
460 | 463 | | |
461 | 464 | | |
462 | | - | |
463 | | - | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
464 | 470 | | |
465 | 471 | | |
466 | 472 | | |
| |||
0 commit comments