Fix deadlock in windows profiler #34454
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The deadlock here happens when the main thread is trying to register a new JIT object while the profiler is triggered. The main thread will acquire the objectmap lock and then get suspended. Then the profiler thread will attempt to stackwalk, but in order to do that effectively, it needs to look at the object map to find the unwind info for the function on the stack. We don't have the same problem on linux because the main thread runs the backtracing. On linux and mac the registration process works differently and on linux at least the backtracing happens on a different thread so we may not have this problem. However, #13294 says that it is also a problem on OS X. We should keep an eye out for it. For the moment, just try to fix this by terminating the stack unwind when we fail to acquire the lock. That's not ideal, because it reduces the quality of the profiler info, but only in situations where we would previous deadlock. This entire code needs a rewrite, but for now, I'm just hoping to get CI to stopping deadlocking on us.
Keno
force-pushed
the
kf/win32profdeadlock
branch
from
7b77430 to
8faec67
Compare
Member
Author
|
How many of these do we have? |
KristofferC
pushed a commit
that referenced
this pull request
Apr 11, 2020
The deadlock here happens when the main thread is trying to register a new JIT object while the profiler is triggered. The main thread will acquire the objectmap lock and then get suspended. Then the profiler thread will attempt to stackwalk, but in order to do that effectively, it needs to look at the object map to find the unwind info for the function on the stack. We don't have the same problem on linux because the main thread runs the backtracing. On linux and mac the registration process works differently and on linux at least the backtracing happens on a different thread so we may not have this problem. However, #13294 says that it is also a problem on OS X. We should keep an eye out for it. For the moment, just try to fix this by terminating the stack unwind when we fail to acquire the lock. That's not ideal, because it reduces the quality of the profiler info, but only in situations where we would previous deadlock. This entire code needs a rewrite, but for now, I'm just hoping to get CI to stopping deadlocking on us.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The deadlock here happens when the main thread is trying to register
a new JIT object while the profiler is triggered. The main thread
will acquire the objectmap lock and then get suspended. Then the profiler
thread will attempt to stackwalk, but in order to do that effectively,
it needs to look at the object map to find the unwind info for the function
on the stack. We don't have the same problem on linux because the main
thread runs the backtracing. On linux and mac the registration process
works differently and on linux at least the backtracing happens on a
different thread so we may not have this problem. However, #13294 says
that it is also a problem on OS X. We should keep an eye out for it.
For the moment, just try to fix this by terminating the stack unwind
when we fail to acquire the lock. That's not ideal, because it reduces
the quality of the profiler info, but only in situations where we would
previous deadlock. This entire code needs a rewrite, but for now, I'm
just hoping to get CI to stopping deadlocking on us.