[CI] RestClientSingleHostIntegTests.testGetWithBody failed on master

[dimitris-athanasiou]

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=debian/2325/console

No reproduce line was printed.

Failures:

11:07:12 Suite: org.elasticsearch.client.RestClientSingleHostIntegTests
11:07:12 ERROR   0.01s J0 | RestClientSingleHostIntegTests.testGetWithBody <<< FAILURES!
11:07:12    > Throwable #1: org.apache.http.ConnectionClosedException: Connection closed
11:07:12    > 	at __randomizedtesting.SeedInfo.seed([BA869D479D2D8EDD:60456C0CDC8F44A]:0)
11:07:12    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:718)
11:07:12    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
11:07:12    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
11:07:12    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.bodyTest(RestClientSingleHostIntegTests.java:321)
11:07:12    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.bodyTest(RestClientSingleHostIntegTests.java:312)
11:07:12    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.testGetWithBody(RestClientSingleHostIntegTests.java:216)
11:07:12    > 	at java.lang.Thread.run(Thread.java:748)
11:07:12    > Caused by: org.apache.http.ConnectionClosedException: Connection closed
11:07:12    > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:344)
11:07:12    > 	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
11:07:12    > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
11:07:12    > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
11:07:12    > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
11:07:12    > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
11:07:12    > 	... 1 more

[elasticmachine]

Pinging @elastic/es-core-infra

[javanna]

I had a look, saw that there's a couple more recent failures that are similar. But it doesn't repro for me and I can't quite see at first glance what the problem could be, nor can I reproduce it when running multiple iterations of this test.

[markharwood]

Another example in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=linux/5/console

 Suite: org.elasticsearch.client.RestClientSingleHostIntegTests
 ERROR   0.02s J0 | RestClientSingleHostIntegTests.testEncodeParams <<< FAILURES!
    > Throwable #1: java.io.IOException: Connection reset by peer
    > 	at __randomizedtesting.SeedInfo.seed([8B1467B354CAACAB:353358761BB80C2A]:0)
    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:728)
    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:177)
    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.testEncodeParams(RestClientSingleHostIntegTests.java:221)
    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    > 	at java.base/java.lang.Thread.run(Thread.java:844)
    > Caused by: java.io.IOException: Connection reset by peer
    > 	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    > 	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    > 	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:283)
    > 	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:250)

Again no reproduce line in logs and would not reproduce locally.
Mock HTTP server closing the connection?

[bleskes]

another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=linux/24/console

16:55:54 Suite: org.elasticsearch.client.RestClientSingleHostIntegTests
16:55:54 ERROR   0.01s J0 | RestClientSingleHostIntegTests.testHeaders <<< FAILURES!
16:55:54    > Throwable #1: java.io.IOException: Connection reset by peer
16:55:54    > 	at __randomizedtesting.SeedInfo.seed([B67B0BD19953E74F:7CAB3B8E1061062F]:0)
16:55:54    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:728)
16:55:54    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
16:55:54    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
16:55:54    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:177)
16:55:54    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.testHeaders(RestClientSingleHostIntegTests.java:177)
16:55:54    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:55:54    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:55:54    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:55:54    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
16:55:54    > 	at java.base/java.lang.Thread.run(Thread.java:844)
16:55:54    > Caused by: java.io.IOException: Connection reset by peer
16:55:54    > 	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
16:55:54    > 	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
16:55:54    > 	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:283)
16:55:54    > 	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:250)

it looks like the http server the test sets up crashed, no info as to why (I logged onto the server and couldn't find anything telling in the build folder)

[spinscale]

another one here https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=amazon/20/console

Suite: org.elasticsearch.client.RestClientSingleHostIntegTests
ERROR   0.01s J0 | RestClientSingleHostIntegTests.testDeleteWithBody <<< FAILURES!
   > Throwable #1: java.io.IOException: Connection reset by peer
   >    at __randomizedtesting.SeedInfo.seed([67C3230DCE755C5:8B6D096CBE2B9D58]:0)
   >    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:728)
   >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
   >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
   >    at org.elasticsearch.client.RestClientSingleHostIntegTests.bodyTest(RestClientSingleHostIntegTests.java:321)
   >    at org.elasticsearch.client.RestClientSingleHostIntegTests.bodyTest(RestClientSingleHostIntegTests.java:312)
   >    at org.elasticsearch.client.RestClientSingleHostIntegTests.testDeleteWithBody(RestClientSingleHostIntegTests.java:207)
   >    at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.io.IOException: Connection reset by peer
   >    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
   >    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
   >    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
   >    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
   >    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
   >    at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:204)
   >    at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136)
   >    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241)
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   >    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   >    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   >    ... 1 more

Wild assumption: I do somewhat remember that the default HttpServer does not have an executor configured (meaning it does not use one and uses the thread it used to bind to the port), so if anyone else connects to that port, I dont know how the HttpServer reacts.

[jdconrad]

Another one: (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=debian/1039/consoleFull)

[talevy]

this has failed again:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux/2422/console

I cannot reproduce, but here would be the command:

./gradlew :client:rest:test   -Dtests.seed=D840989E2D6365FD   -Dtests.class=org.elasticsearch.client.RestClientSingleHostIntegTests   -Dtests.method="testDeleteWithBody"

It has failed 12 times in the last 30 days. I am AwaitsFix'ing it for master and 6.x.
It has also failed on 6.3, but I am not sure whether it warrants muting it there, if others disagree
I'll add that too

UPDATE: nevermind. cannot use AwaitsFix here since the client tests don't have the lucene test framework. The leading theory is that this is an infra issue, but not clear why

UPDATE2: I do not believe this has to do with any specific one of these test methods, but the suite in general. testDeleteWithBody,testHeaders,testEncodeParams,testGetWithBody have all failed on CI recently. @jasontedor (thanks!) pointed out that one suspicious test to blame may be testManyAsyncRequests.