[CI] SslIntegrationTests fail with Netty buffer leak

[danielmitterdorfer]

Build failure link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/208/console

Build failure:

   > Expected: an empty collection
   >      but: <[LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
   > WARNING: 4 leak records were discarded because the leak record count is limited to 4. Use system property io.netty.leakDetection.maxRecords to increase the limit.
   > Recent access records: 4
   > #4:
   >    io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:732)
[...]
   >    io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
   >    java.base/java.lang.Thread.run(Thread.java:834)
   > #3:
   >    io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:702)
   >    io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:927)
   >    io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:762)
   >    io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:731)
   >    io.netty.handler.ssl.SslHandler.flush(SslHandler.java:712)
[...]
   >    java.base/java.lang.Thread.run(Thread.java:834)
   > #2:
   >    io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:702)
   >    io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:759)
   >    io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:731)
   >    io.netty.handler.ssl.SslHandler.flush(SslHandler.java:712)
[...]
   >    java.base/java.lang.Thread.run(Thread.java:834)
   > #1:
   >    io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:588)
   >    io.netty.handler.ssl.SslHandler$SslHandlerCoalescingBufferQueue.composeFirst(SslHandler.java:1831)
   >    io.netty.channel.AbstractCoalescingBufferQueue.remove(AbstractCoalescingBufferQueue.java:154)
   >    io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:752)
   >    io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:731)
   >    io.netty.handler.ssl.SslHandler.flush(SslHandler.java:712)
[...]
   >    io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
   >    java.base/java.lang.Thread.run(Thread.java:834)
   > Created at:
   >    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
   >    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
   >    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:172)
   >    io.netty.handler.ssl.SslHandler$SslHandlerCoalescingBufferQueue.composeFirst(SslHandler.java:1830)
   >    io.netty.channel.AbstractCoalescingBufferQueue.remove(AbstractCoalescingBufferQueue.java:154)
   >    io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:752)
   >    io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:731)
   >    io.netty.handler.ssl.SslHandler.flush(SslHandler.java:712)
[...]
   >    java.base/java.lang.Thread.run(Thread.java:834)]>

Reproduces locally with:

./gradlew :x-pack:plugin:security:test -Dtests.seed=28197402351DB64D -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslIntegrationTests -Dtests.method="testThatUnconfiguredCiphersAreRejected" -Dtests.security.manager=true -Dtests.locale=fr-MC -Dtests.timezone=SystemV/MST7

Note that one other test failed as well in the same test run but as the underlying cause appears to be the same I just created one ticket.

Here are all reproduction lines:

REPRODUCE WITH: ./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=28197402351DB64D \
  -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslIntegrationTests \
  -Dtests.method="testThatTransportClientUsingSSLv3ProtocolIsRejected" \
  -Dtests.security.manager=true \
  -Dtests.locale=fr-MC \
  -Dtests.timezone=SystemV/MST7

REPRODUCE WITH: ./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=28197402351DB64D \
  -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslIntegrationTests \
  -Dtests.method="testThatUnconfiguredCiphersAreRejected" \
  -Dtests.security.manager=true \
  -Dtests.locale=fr-MC \
  -Dtests.timezone=SystemV/MST7

[elasticmachine]

Pinging @elastic/es-security