-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-11974][CORE]Not all the temp dirs had been deleted when the JVM exits #9951
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
What's the problem with the original approach? |
|
It can not delete all element of shutdownDeletePaths. |
|
The part I don't get is --- where are we deleting elements of shutdownDeletePaths? |
|
|
Ah ok - can you update the change to add a line of comment saying we need to materialize the paths to delete because deleteRecursively removes items from shutdownDeletePaths as we are traversing through it? |
//we need to materialize the paths to delete because deleteRecursively removes items from //shutdownDeletePaths as we are traversing through it.
|
Ok, get it @rxin |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need a space after // - otherwise this will fail the style check
|
Is it OK? @rxin |
|
Test build #2108 has finished for PR 9951 at commit
|
|
I've merged this. Thanks. |
…VM exits
deleting the temp dir like that
```
scala> import scala.collection.mutable
import scala.collection.mutable
scala> val a = mutable.Set(1,2,3,4,7,0,8,98,9)
a: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98)
scala> a.foreach(x => {a.remove(x) })
scala> a.foreach(println(_))
98
```
You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection
Author: Zhongshuai Pei <[email protected]>
Closes #9951 from DoingDone9/Bug_RemainDir.
(cherry picked from commit 6b78157)
Signed-off-by: Reynold Xin <[email protected]>
…VM exits
deleting the temp dir like that
```
scala> import scala.collection.mutable
import scala.collection.mutable
scala> val a = mutable.Set(1,2,3,4,7,0,8,98,9)
a: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98)
scala> a.foreach(x => {a.remove(x) })
scala> a.foreach(println(_))
98
```
You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection
Author: Zhongshuai Pei <[email protected]>
Closes #9951 from DoingDone9/Bug_RemainDir.
(cherry picked from commit 6b78157)
Signed-off-by: Reynold Xin <[email protected]>
…VM exits
deleting the temp dir like that
```
scala> import scala.collection.mutable
import scala.collection.mutable
scala> val a = mutable.Set(1,2,3,4,7,0,8,98,9)
a: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98)
scala> a.foreach(x => {a.remove(x) })
scala> a.foreach(println(_))
98
```
You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection
Author: Zhongshuai Pei <[email protected]>
Closes #9951 from DoingDone9/Bug_RemainDir.
(cherry picked from commit 6b78157)
Signed-off-by: Reynold Xin <[email protected]>
deleting the temp dir like that
You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection