org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStoreRepositoryTests testSnapshotWithLargeSegmentFiles

[jakelandis]

Reproduce with (does not repro locally):

./gradlew ':plugins:repository-gcs:test' --tests "org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles" -Dtests.seed=79146B9E4DC8DB20 -Dtests.security.manager=true -Dtests.locale=ro -Dtests.timezone=America/La_Paz -Dcompiler.java=13 -Druntime.java=8	

Error

 java.lang.AssertionError: Only index blobs should remain in repository but found [indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__U-yvyHGGSC-HG3-rYgESAQ, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__2W1fsDR7RHqVaHUbR2ljDg, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__PYCKH3IuR8OvsesbZ53Q4A, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__ChKaNwG-QDeOLEFMBmV4hg, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__4BrkUvaDSoaRxda8zaYzmw, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__8TyJLropTGCbwEdpGgkCgg, snap-6BvhIzAVS8Wk0yEzY1yHHw.dat, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__TYjSMqKWSAyimBORtvz2hQ, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__nr1hKmAYS8mwQoYGfONLxA, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__AIvhEKrQTT6rQ95Q_a6Mcw, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/snap-6BvhIzAVS8Wk0yEzY1yHHw.dat, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__OYx13Q6MT329W_IM0oSGsg, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__lc7ep5qgSfGUi5XOU2GgVA, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__FnKJqi01RL6xd-BqebZ9qw, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__iv8fkpEiRkGaiSfcDb6cTg, indices/U3Ra9xA9QYOIf2JnrS9qiw/meta-6BvhIzAVS8Wk0yEzY1yHHw.dat, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__45KqhxnDQJyJbNxg5bOdBw, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__zLdkon40Q5aMTM5YjMsXWQ, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__UAbVvyV5QHq0RsZGVRdphg, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__jXvlBf1lTn6Ct1d2xyEeHg, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__ED_68lu_Qci0brGcdBWWqA, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__0j2RqpNKRKqy6RNbv4iniA, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__SqeeF3m4TkqcEJrcZfx0XA, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__6j7bvik8RDqSNmZIHaHdlw, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__gj5mlfNKQmC_sqRVWkQh4A, meta-6BvhIzAVS8Wk0yEzY1yHHw.dat, indices/U3Ra9xA9QYOIf2JnrS9qiw/0/__YCpLvzOYR6m3ZMbSCjvCuw]	
 
    Expected: a collection with size <0>	
         but: collection size was <27>	
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)	
        at org.junit.Assert.assertThat(Assert.java:956)	
        at org.elasticsearch.repositories.blobstore.ESMockAPIBasedRepositoryIntegTestCase.tearDownHttpServer(ESMockAPIBasedRepositoryIntegTestCase.java:112)

Build scan : https://gradle-enterprise.elastic.co/s/5agjwd5uxd4g6

90 day history

Suspected related issues (via comments from prior failures)

• Reduce allocations when draining HTTP requests bodies in repository tests #48541
• 732fc4d

Note - it seems this happens (exclusively?) on 7.5/7.6/7.x and sometimes, but not always have a SocketTimeout too. For example (different build scan then above) https://gradle-enterprise.elastic.co/s/4z2vxrxrohjmq/console-log?anchor=7206

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[ywelsch]

New failure on 7.6: https://gradle-enterprise.elastic.co/s/prwylyxj4k5ja