Change EventTimer to reset its Timer without requiring a reset channel #329
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sending multiple messages in rapid succession (or concurrently) causes deadlock.
While building an Acceptor implementation, I experienced the symptoms of a deadlock when the server happened to receive two requests in quick succession. The messages would be processed on the same goroutine as the FromApp callback and a response sent back using SendToTarget. The session thread got stuck in the second call to SendToTarget.
To debug the issue, I created an Initiator (which runs in a separate process on the same machine) that generates many requests:
Surprisingly the Initiator would also deadlock quickly and stop printing "after send".
The deadlock occurs between the session thread and the EventTimer thread (for the stateTimer heartbeat timer).
Here is a diagram of the two threads:
In summary, the session thread's select happens to enter the case for messageEvent, but there
is a race with the timer thread which enters the case for executing the timer action.
The timer's action is to send to an unbuffered channel that can only get read when the session thread is in a different select case than it is currently in.
If there happens to be more than 1 message to send, then the session thread will call EventTimer.Reset more than once, which will guarantee that the reset channel will become full and now these threads are deadlocked.
The deadlock won't happen if EventTimer.Reset simply resets the timer in a nonblocking fashion.
Past commit b75fbea#diff-8dc0bfb75b5413b6f4ed2d880e648ad9 moved from using time.After to the current implementation.
This PR changes the EventTimer to instead use time.Timer directly, without the need for a reset channel, as any thread can call Reset on it to a future time, without the possibility of getting blocked.