Skip to content

[SPARK-27338][Core] Fix deadlock in UnsafeExternalSorter.SpillableIterator when locking both UnsafeExternalSorter.SpillableIterator and TaskMemoryManager #24265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-27338][Core] Fix deadlock in UnsafeExternalSorter.SpillableIterator when locking both UnsafeExternalSorter.SpillableIterator and TaskMemoryManager #24265

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a9de27
Fix deadlock while freeing page
Mar 28, 2019
229cfae
Addressed review comments
Apr 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 23 additions & 11 deletions core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -573,19 +573,31 @@ public boolean hasNext() {

@Override
public void loadNext() throws IOException {
synchronized (this) {
loaded = true;
if (nextUpstream != null) {
// Just consumed the last record from in memory iterator
if (lastPage != null) {
freePage(lastPage);
lastPage = null;
MemoryBlock pageToFree = null;
Copy link
Contributor

@beliefer beliefer Apr 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your PR seems good!

try {
synchronized (this) {
loaded = true;
if (nextUpstream != null) {
// Just consumed the last record from in memory iterator
if(lastPage != null) {
// Do not free the page here, while we are locking `SpillableIterator`. The `freePage`
// method locks the `TaskMemoryManager`, and it's a bad idea to lock 2 objects in
// sequence. We may hit dead lock if another thread locks `TaskMemoryManager` and
// `SpillableIterator` in sequence, which may happen in
// `TaskMemoryManager.acquireExecutionMemory`.
pageToFree = lastPage;
Copy link
Contributor

@cloud-fan cloud-fan Apr 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add some comment to explain it? You can just copy it from https://github.com/apache/spark/pull/24269/files#diff-027299fb14327ddcaba457f81ecff32cR583

Copy link
Contributor Author

@venkata91 venkata91 Apr 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure @cloud-fan, will add and update the PR

Copy link
Contributor Author

@venkata91 venkata91 Apr 3, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done @cloud-fan . I didn't know that you also filed a PR simultaneously for the same issue. :) Thanks for graciously closing your PR and accepting this PR.

lastPage = null;
}
upstream = nextUpstream;
nextUpstream = null;
}
upstream = nextUpstream;
nextUpstream = null;
numRecords--;
upstream.loadNext();
}
} finally {
if (pageToFree != null) {
freePage(pageToFree);
}
numRecords--;
upstream.loadNext();
}
}

Expand Down