Reduce allocations when draining HTTP requests bodies in repository tests

[tlrx]

This PR adjusts the total time that a Google Cloud Storage request can take in the GCS repository tests from 50 sec to 120 seconds (see https://gradle-enterprise.elastic.co/s/zz5bstxyoqunk/tests/avmwsupzrdux6-dp22mjd7itzv6 for a test failure).

@original-brownbear I still think that instead of increasing the timeout we should configure an executor service in the HTTP handler, as GCS requests take more time than S3/Azure to be processed (most of them are multipart encoded).

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[tlrx]

@elasticmachine run elasticsearch-ci/1
(unrelated test failure)

[original-brownbear]

Hmm it seems to me something else but slowness in the HttpServer must be going on here?
I just profiled this test a little and it doesn't seem like we're spending a lot of time on the http server's IO loop in the failing test here.

What we are doing though is allocating a ton of buffers on it during the error simulations, maybe that's where we're burning so much time when under memory pressure?

How about adding this optimization:

--- a/test/framework/src/main/java/org/elasticsearch/repositories/blobstore/ESMockAPIBasedRepositoryIntegTestCase.java
+++ b/test/framework/src/main/java/org/elasticsearch/repositories/blobstore/ESMockAPIBasedRepositoryIntegTestCase.java
@@ -26,7 +26,6 @@ import org.elasticsearch.action.admin.indices.forcemerge.ForceMergeResponse;
 import org.elasticsearch.cluster.metadata.IndexMetaData;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.SuppressForbidden;
-import org.elasticsearch.common.io.Streams;
 import org.elasticsearch.common.network.InetAddresses;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.mocksocket.MockHttpServer;
@@ -37,6 +36,7 @@ import org.junit.Before;
 import org.junit.BeforeClass;
 
 import java.io.IOException;
+import java.io.InputStream;
 import java.net.InetAddress;
 import java.net.InetSocketAddress;
 import java.util.Map;
@@ -165,8 +165,11 @@ public abstract class ESMockAPIBasedRepositoryIntegTestCase extends ESBlobStoreR
             }
         }
 
+        private static final byte[] BUFFER = new byte[1024];
+
         protected void handleAsError(final HttpExchange exchange) throws IOException {
-            Streams.readFully(exchange.getRequestBody());
+            final InputStream inputStream = exchange.getRequestBody();
+            while (inputStream.read(BUFFER) >= 0);
             exchange.sendResponseHeaders(HttpStatus.SC_INTERNAL_SERVER_ERROR, -1);
             exchange.close();
         }

to dump the bytes more efficiently than through actually instantiating a 1MB or so buffer on every run and see if that helps?
I don't see where we'd need another executor from profiling here and also don't think we should up the timeout here, if it's not GC pressure something else must be off here unless you were able to find some massive CPU usage on the http server dispatcher thread?

[tlrx]

Thanks @original-brownbear. I'm not sure the allocating are the cause of the read timeouts, but your suggestion is a good one anyway so I pushed 71fd095 👍

We'll see if it has any impacts on the test failures. I also agree that increasing the timeout is not a good thing to do unless we have exhausted all other source of trouble.

[original-brownbear]

LGTM, thanks Tanguy :)

Let's see if this helps (at least it should be an improvement, it's even a speedup on my local box where things are completely quiet otherwise) and otherwise try and find ways of addting debuging that would track down the source of slowness maybe?

[tlrx]

Thanks Armin!