Fix memory/breaker leaks for outbound responses #76474
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Outbound responses would not get the expected `decRef`, resulting in memory and/or circuit breaker leaks. In particular, the `GetCcrRestoreFileChunkResponse` expects this, causing a leak when a follower bootstraps. Relates elastic#65921
henningandersen
added
>bug
:Distributed Coordination/Network
elasticmachine
added
the
Team:Distributed (Obsolete)
Collaborator
|
Pinging @elastic/es-distributed (Team:Distributed) |
original-brownbear
suggested changes
Aug 13, 2021
Contributor
original-brownbear
left a comment
original-brownbear
left a comment
There was a problem hiding this comment.
Thanks for spotting this Henning!
unfortunately, the tests are failing now. There must be more to this than just a failure in the general case as this would definitely fail existing tests if it would leak all the time.
We must fail to invoke a listener/decrement in some exceptional case as now we're failing tests with:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=50, name=elasticsearch[leader1][transport_worker][T#2], state=RUNNABLE, group=TGRP-RestartIndexFollowingIT]
Caused by:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([3BDF36A18D3FAC5D]:0)
at org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:52)
at org.elasticsearch.common.bytes.ReleasableBytesReference.close(ReleasableBytesReference.java:90)
at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:314)
at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:217)
at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:29)
at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:128)
at org.elasticsearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:140)
at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:409)
at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:235)
at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:163)
at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120)
at java.base/java.lang.Thread.run(Thread.java:834)
original-brownbear
approved these changes
Aug 13, 2021
Contributor
original-brownbear
left a comment
original-brownbear
left a comment
There was a problem hiding this comment.
LGTM after discussing on another channel this looks just fine :)
Contributor
Author
|
Let us do one more round of randomized test: |
Contributor
Author
|
@elasticmachine test this please |
henningandersen
added
auto-backport
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Aug 14, 2021
Outbound responses would not get the expected `decRef`, resulting in memory and/or circuit breaker leaks. In particular, the `GetCcrRestoreFileChunkResponse` expects this, causing a leak when a follower bootstraps. Relates elastic#65921
henningandersen
added a commit
that referenced
this pull request
Aug 15, 2021
Outbound responses would not get the expected `decRef`, resulting in memory and/or circuit breaker leaks. In particular, the `GetCcrRestoreFileChunkResponse` expects this, causing a leak when a follower bootstraps. Relates #65921
henningandersen
added a commit
that referenced
this pull request
Aug 15, 2021
Outbound responses would not get the expected `decRef`, resulting in memory and/or circuit breaker leaks. In particular, the `GetCcrRestoreFileChunkResponse` expects this, causing a leak when a follower bootstraps. Relates #65921
tlrx
added a commit
that referenced
this pull request
Nov 4, 2022
…1289) In #76474 we fixed a circuit breaker leak in TransportActionProxy by incrementing a reference on the TransportResponse that is later decremented by the OutboundHandler. This works well for all cases except when the request targets the node which is also the proxy node. In that case the reference is incremented but will never be decremented as the local execution (using TransportService#localNodeConnection and DirectResponseChannel) bypasses the OutboundHandler. This change fixes the ref counting by also decrementing the TransportResponse in DirectResponseChannel. This will also have the consequence to correctly decrement used bytes of the request circuit breaker when GetCcrRestoreFileChunkResponse are executed on a node that is also a proxy node.
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Nov 4, 2022
…astic#91289) In elastic#76474 we fixed a circuit breaker leak in TransportActionProxy by incrementing a reference on the TransportResponse that is later decremented by the OutboundHandler. This works well for all cases except when the request targets the node which is also the proxy node. In that case the reference is incremented but will never be decremented as the local execution (using TransportService#localNodeConnection and DirectResponseChannel) bypasses the OutboundHandler. This change fixes the ref counting by also decrementing the TransportResponse in DirectResponseChannel. This will also have the consequence to correctly decrement used bytes of the request circuit breaker when GetCcrRestoreFileChunkResponse are executed on a node that is also a proxy node.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Nov 4, 2022
…1289) (#91315) In #76474 we fixed a circuit breaker leak in TransportActionProxy by incrementing a reference on the TransportResponse that is later decremented by the OutboundHandler. This works well for all cases except when the request targets the node which is also the proxy node. In that case the reference is incremented but will never be decremented as the local execution (using TransportService#localNodeConnection and DirectResponseChannel) bypasses the OutboundHandler. This change fixes the ref counting by also decrementing the TransportResponse in DirectResponseChannel. This will also have the consequence to correctly decrement used bytes of the request circuit breaker when GetCcrRestoreFileChunkResponse are executed on a node that is also a proxy node.
lchqlchq
pushed a commit
to lchqlchq/elasticsearch
that referenced
this pull request
Aug 23, 2023
lchqlchq
pushed a commit
to lchqlchq/elasticsearch
that referenced
this pull request
Aug 23, 2023
…elastic#76536" This reverts commit 411e5e7
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Outbound responses would not get the expected
decRef, resulting inmemory and/or circuit breaker leaks. In particular, the
GetCcrRestoreFileChunkResponseexpects this, causing a leak whena follower bootstraps.
Relates #65921