[CI] GoogleCloudStorageBlobContainerRetriesTests.testReadLargeBlobWithRetries fails reproducibly

[mark-vieira]

We've encountered two failures of GoogleCloudStorageBlobContainerRetriesTests.testReadLargeBlobWithRetries. This failure reproduces locally for me.

org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainerRetriesTests > testReadLargeBlobWithRetries FAILED
    com.google.cloud.storage.StorageException: Connection closed prematurely: bytesRead = 2097151, Content-Length = 2097152
        at __randomizedtesting.SeedInfo.seed([8062D6DCDCB1069C:FAFB08A56F33DF3]:0)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:227)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:690)
        at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127)
        at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124)
        at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
        at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
        at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
        at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123)
        at org.elasticsearch.repositories.gcs.GoogleCloudStorageRetryingInputStream$1.lambda$read$0(GoogleCloudStorageRetryingInputStream.java:83)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44)
        at org.elasticsearch.repositories.gcs.GoogleCloudStorageRetryingInputStream$1.read(GoogleCloudStorageRetryingInputStream.java:83)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
        at org.elasticsearch.repositories.gcs.GoogleCloudStorageRetryingInputStream.read(GoogleCloudStorageRetryingInputStream.java:126)
        at java.base/java.io.InputStream.read(InputStream.java:205)
        at org.elasticsearch.common.io.Streams.doCopy(Streams.java:86)
        at org.elasticsearch.common.io.Streams.copy(Streams.java:79)
        at org.elasticsearch.common.io.Streams.copy(Streams.java:61)
        at org.elasticsearch.common.io.Streams.readFully(Streams.java:252)
        at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainerRetriesTests.testReadLargeBlobWithRetries(GoogleCloudStorageBlobContainerRetriesTests.java:228)

        Caused by:
        java.io.IOException: Connection closed prematurely: bytesRead = 2097151, Content-Length = 2097152
            at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.throwIfFalseEOF(NetHttpResponse.java:204)
            at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:166)
            at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
            at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49)
            at com.google.api.client.util.IOUtils.copy(IOUtils.java:87)
            at com.google.api.client.util.IOUtils.copy(IOUtils.java:59)
            at com.google.api.client.http.HttpResponse.download(HttpResponse.java:386)
            at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:684)
            ... 20 more
REPRODUCE WITH: ./gradlew ':plugins:repository-gcs:test' --tests "org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainerRetriesTests.testReadLargeBlobWithRetries" -Dtests.seed=8062D6DCDCB1069C -Dtests.security.manager=true -Dtests.locale=zh -Dtests.timezone=MET -Dcompiler.java=13

master - https://gradle-enterprise.elastic.co/s/bcwgezylwlugc/tests/avmwsupzrdux6-q3hb3jtfyd7iq
7.x - https://gradle-enterprise.elastic.co/s/cr2apsim5ep4y/tests/avmwsupzrdux6-q3hb3jtfyd7iq

Looks like we are barfing after reading exactly 2MB of data. Seems a bit suspect.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Discovery-Plugins)

[mark-vieira]

Since this reproduces it's certain to continue failing. I've muted in master (297bfd5e486db95b2c1f74a4ba337d209c59c657) and 7.x (4bce998).

[mark-vieira]

Shortly afterwards I saw a different failure, which doesn't reproduce but it looks strangely similar (connections being prematurely closed):

https://gradle-enterprise.elastic.co/s/as6wfs4gowp5a/tests/avmwsupzrdux6-lvbo7kvcj6ksq

[2020-02-20T13:17:26,295][ERROR][o.e.r.g.GoogleCloudStorageBlobStoreRepositoryTests] [[Thread-6]] Exception when handling request /127.0.0.1:33344 GET /download/storage/v1/b/bucket/o/foobar?alt=media
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
	at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:?]
	at sun.net.httpserver.Request$WriteStream.write(Request.java:391) ~[?:?]
	at sun.net.httpserver.FixedLengthOutputStream.write(FixedLengthOutputStream.java:78) ~[?:?]
	at sun.net.httpserver.PlaceholderOutputStream.write(ExchangeImpl.java:444) ~[?:?]
	at org.elasticsearch.common.bytes.AbstractBytesReference.writeTo(AbstractBytesReference.java:62) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
	at fixture.gcs.GoogleCloudStorageHttpHandler.handle(GoogleCloudStorageHttpHandler.java:149) ~[gcs-fixture-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
	at org.elasticsearch.repositories.blobstore.ESMockAPIBasedRepositoryIntegTestCase.lambda$wrap$4(ESMockAPIBasedRepositoryIntegTestCase.java:238) ~[framework-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) ~[?:1.8.0_241]
	at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83) ~[?:?]
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82) ~[?:1.8.0_241]
	at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675) ~[?:?]
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) ~[?:1.8.0_241]
	at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647) ~[?:?]
	at org.elasticsearch.repositories.blobstore.ESMockAPIBasedRepositoryIntegTestCase.lambda$startHttpServer$0(ESMockAPIBasedRepositoryIntegTestCase.java:83) ~[framework-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
	at sun.net.httpserver.ServerImpl$Dispatcher.handle(ServerImpl.java:431) [?:1.8.0_241]
	at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:396) [?:1.8.0_241]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]

[mark-vieira]

Thanks, @original-brownbear!