Fix issue where pages aren't released

[Tim-Brooks]

This is related to #27422. Right now when we send a write to the netty
transport, we attach a listener to the future. When you submit a write
on the netty event loop and the event loop is shutdown, the onFailure
method is called. Unfortunately, netty then tries to notify the listener
which cannot be done without dispatching to the event loop. In this
case, the dispatch fails and netty logs and error and does not tell us.

This commit checks that netty is still not shutdown after sending a
message. If netty is shutdown, we complete the listener.

[Tim-Brooks]

I also opened an issue with netty for this.

[s1monw]

LGTM in general & good catch. I left one request

[s1monw]

do we protect this from double invocation? I think we should just make sure we only invoke once. can you wrap it in here and ensure that? maybe just use NotifyOnceListener

[Tim-Brooks]

Well - the only place we call this is from internalSendMessage.

    private void internalSendMessage(TcpChannel channel, BytesReference message, SendMetricListener listener) {
        try {
            channel.sendMessage(message, listener);
        } catch (Exception ex) {
            // call listener to ensure that any resources are released
            listener.onFailure(ex);
            onException(channel, ex);
        }
    }

And SendMetricListener is a NotifyOnceListener. So incidentally - yes it is always a notify once listener. Do you want me to codify that by changing the signature of TcpChannel#sendMessage to:

/**
     * Sends a tcp message to the channel. The listener will be executed once the send process has been
     * completed.
     *
     * @param reference to send to channel
     * @param listener to execute upon send completion
     */
    void sendMessage(BytesReference reference, NotifyOnceListener<Void> listener);

???

[Tim-Brooks]

@s1monw

[s1monw]

it's find in this case! LGTM

[jasontedor]

LGTM. Thanks for opening the issue against Netty too.