[CI] Intermittent test failures with pages not released

[Tim-Brooks]

We have been getting intermittent failures related to pages not being released.

Throwable #1: java.lang.RuntimeException: 1 pages have not been released
at __randomizedtesting.SeedInfo.seed([877D649F05E62A39:CC4ADB0AA57D7FFD]:0)
at org.elasticsearch.common.util.MockPageCacheRecycler.ensureAllPagesAreReleased(MockPageCacheRecycler.java:52)
at org.elasticsearch.test.ESTestCase.checkStaticState(ESTestCase.java:398)
at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:291)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Throwable
at org.elasticsearch.common.util.MockPageCacheRecycler.wrap(MockPageCacheRecycler.java:69)
at org.elasticsearch.common.util.MockPageCacheRecycler.bytePage(MockPageCacheRecycler.java:118)
at org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:478)
at org.elasticsearch.common.util.MockBigArrays.newByteArray(MockBigArrays.java:98)
at org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:490)
at org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:61)
at org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:47)
at org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:43)
at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1162)
at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1153)
at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:63)
at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:57)
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)
at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:286)
at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:281)
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)

I think I have figured out what the issue is. When I look in the test logs I see the following log from netty:

[2017-11-17T03:00:03,545][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down
java.util.concurrent.RejectedExecutionException: event executor terminate
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[netty-common-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[netty-common-4.1.13.Final.jar:4.1.13.Final]
09at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[netty-common-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[netty-common-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760) ~[netty-common-4.1.13.Final.jar:4.1.13.Final]
09at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428) ~[netty-common-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113) ~[netty-common-4.1.13.Final.jar:4.1.13.Final]

Sorry the formatting is terrible. But it looks like netty will fail to schedule the listener to be executed if the event loop is shutdown. Instead it just logs the issue. So no one ever calls the listener. The should not impact real world usage as it only happens at shutdown. But we will need to figure out some resolution for tests.

[Tim-Brooks]

@jaymode @jasontedor

[Tim-Brooks]

This should be closed by #27459.