[CI] NoHandlerIT#testNoHandlerRespectsAcceptHeader() fails with Netty buffer leak

[danielmitterdorfer]

Build failure link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+release-tests/11/console

build failure:

1> [2018-07-30T11:04:30,062][ERROR][i.n.u.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
  1> Recent access records: 4
  1> #4:
  1> 	Hint: 'cors' will handle the message from this point.
  1> 	io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.touch(AdvancedLeakAwareCompositeByteBuf.java:36)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.touch(HttpObjectAggregator.java:374)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:454)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:404)
  1> 	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
[...]
  1> 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
  1> 	java.lang.Thread.run(Thread.java:748)
  1> #3:
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.release(HttpObjectAggregator.java:386)
  1> 	io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
  1> 	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:90)
  1> 	io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
[...]
  1> 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
  1> 	java.lang.Thread.run(Thread.java:748)
  1> #2:
  1> 	io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.retain(AdvancedLeakAwareCompositeByteBuf.java:36)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.retain(HttpObjectAggregator.java:362)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.retain(HttpObjectAggregator.java:442)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.retain(HttpObjectAggregator.java:404)
  1> 	io.netty.util.ReferenceCountUtil.retain(ReferenceCountUtil.java:36)
[...]
  1> 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
  1> 	java.lang.Thread.run(Thread.java:748)
  1> #1:
  1> 	Hint: 'encoder_compress' will handle the message from this point.
  1> 	io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.touch(AdvancedLeakAwareCompositeByteBuf.java:36)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.touch(HttpObjectAggregator.java:374)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:454)
  1> 	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:404)
  1> 	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
[...]
  1> 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
  1> 	java.lang.Thread.run(Thread.java:748)
  1> Created at:
  1> 	io.netty.buffer.AbstractByteBufAllocator.compositeHeapBuffer(AbstractByteBufAllocator.java:207)
  1> 	io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:197)
  1> 	io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:255)
[...]
  1> 	io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
[...]
  1> 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
  1> 	java.lang.Thread.run(Thread.java:748)

reproduction line:

./gradlew :qa:smoke-test-http:integTestRunner \
  -Dtests.seed=FF1FE06FB78C0B31 \
  -Dtests.class=org.elasticsearch.http.NoHandlerIT \
  -Dtests.method="testNoHandlerRespectsAcceptHeader" \
  -Dtests.security.manager=true \
  -Dtests.locale=en-IN \
  -Dtests.timezone=Etc/GMT+9

(does not reproduce locally)

[elasticmachine]

Pinging @elastic/es-core-infra

[original-brownbear]

@tbrooks8 you can probably just back port your fix to 6.4 here right? :)

[Tim-Brooks]

I merged that work into 6.4 here #32505.