Skip to content

[SPARK-4883][Shuffle] Add a name to the directoryCleaner thread #3734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-4883][Shuffle] Add a name to the directoryCleaner thread #3734

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cc74727
Add a name to the directoryCleaner thread
zsxwing Dec 18, 2014
e6f2b61
Fix the name
zsxwing Dec 20, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions ...k/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
import org.apache.spark.network.buffer.ManagedBuffer;
import org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo;
import org.apache.spark.network.util.JavaUtils;
import org.apache.spark.network.util.NettyUtils;
import org.apache.spark.network.util.TransportConf;

/**
Expand All @@ -49,7 +50,7 @@
* the Executor's memory, unlike the IndexShuffleBlockManager.
*/
public class ExternalShuffleBlockManager {
private final Logger logger = LoggerFactory.getLogger(ExternalShuffleBlockManager.class);
private static final Logger logger = LoggerFactory.getLogger(ExternalShuffleBlockManager.class);

// Map containing all registered executors' metadata.
private final ConcurrentMap<AppExecId, ExecutorShuffleInfo> executors;
Expand All @@ -60,8 +61,9 @@ public class ExternalShuffleBlockManager {
private final TransportConf conf;

public ExternalShuffleBlockManager(TransportConf conf) {
// TODO: Give this thread a name.
this(conf, Executors.newSingleThreadExecutor());
this(conf, Executors.newSingleThreadExecutor(
Copy link
Contributor

@aarondav aarondav Dec 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use NettyUtils.createThreadFactory() (could move that to JavaUtils too, less important).

Copy link
Member Author

@zsxwing zsxwing Dec 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be daemon or not? If it's daemon, it may not have enough time to delete all files when the service is exiting.

Copy link
Contributor

@aarondav aarondav Dec 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this should be daemon, I think, otherwise it may prevent the JVM from exiting even when it's not doing anything. If we wanted to prevent the service from exiting before cleanup has completed, I would rather block in a stop() method until the queue of directories to delete has been drained, but I do not think this too important. Unlike Spark Executors, the graceful termination of this service should be uncorrelated with buildup of shuffle data.

Copy link
Member Author

@zsxwing zsxwing Dec 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. Use NettyUtils.createThreadFactor now

// Add `spark` prefix because it will run in NM in Yarn mode.
NettyUtils.createThreadFactory("spark-shuffle-directory-cleaner")));
}

// Allows tests to have more control over when directories are cleaned up.
Expand Down