[CI] Aggregation mixed cluster failure

[martijnvg]

A serialization failure, because of an unknown named writeable in MixedClusterClientYamlTestSuiteIT test {p0=search.aggregation/330_auto_date_histogram/basic}

Failure:

[2020-03-30T14:31:28,724][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  "stash" : {
    "body" : {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : "Unknown NamedWriteable [org.elasticsearch.search.aggregations.PipelineAggregationBuilder][]",
            "stack_trace" : "org.elasticsearch.ElasticsearchException$1: Unknown NamedWriteable [org.elasticsearch.search.aggregations.PipelineAggregationBuilder][]
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)
	at org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:167)
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:626)
	at org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)
	at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)
    ...
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalArgumentException: Unknown NamedWriteable [org.elasticsearch.search.aggregations.PipelineAggregationBuilder][]
	at org.elasticsearch.common.io.stream.NamedWriteableRegistry.getReader(NamedWriteableRegistry.java:112)
	at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:45)
	at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:39)
	at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.<init>(AggregatorFactories.java:257)
	at org.elasticsearch.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:989)
	at org.elasticsearch.search.builder.SearchSourceBuilder.<init>(SearchSourceBuilder.java:204)
	at org.elasticsearch.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:989)
	at org.elasticsearch.search.internal.ShardSearchRequest.<init>(ShardSearchRequest.java:167)
	at org.elasticsearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:56)
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:187)
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:130)
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:107)
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:681)
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:136)
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:110)
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:68)

Build url: https://gradle-enterprise.elastic.co/s/wdpjhc2oh534q/tests/77udqutc4yt34-blndfefc2cef2

Reproduce line: ./gradlew ':qa:mixed-cluster:v7.8.0#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/330_auto_date_histogram/basic}" -Dtests.seed=2DD4185E797F2251 -Dtests.security.manager=true -Dtests.locale=he -Dtests.timezone=Asia/Ujung_Pandang -Dcompiler.java=13

This does not reproduce for me locally.

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

[benwtrent]

Many builds have failed due to this test over the past 24 hours.

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(_source),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'%22FAILURE%20MixedClusterClientYamlTestSuiteIT%20test%20%7Bp0%3Dsearch.aggregation%2F330_auto_date_histogram%2Fbasic%7D%22'),sort:!(process.time-start,desc))

All are BWC tests with 7.8.

Muting in #54404

[polyfractal]

/cc @nik9000 related to recent pipeline work?

[nik9000]

Maybe! I'll take a look now.

[nik9000]

The test is actually new in the pipeline agg work. We didn't have any yaml tests for auto_date_histogram in the past.

[nik9000]

I've opened a PR that should fix this. It re-enables the test and fixes a serialization bug that feels like a likely suspect. I have no idea why this doesn't reproduce more consistently.