[CI] S3BlobStoreRepositoryTests.testMultipleSnapshotAndRollback fails in master and 7.x

[henningandersen]

Failed on 7.x with:

java.lang.AssertionError: 

Expected: <7>
     but: was <4>
Close stacktrace
at __randomizedtesting.SeedInfo.seed([D6D6F325AB1133B1:E88211DBE45C8808]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.repositories.blobstore.ESBlobStoreRepositoryIntegTestCase.assertSuccessfulRestore(ESBlobStoreRepositoryIntegTestCase.java:317)
at org.elasticsearch.repositories.blobstore.ESBlobStoreRepositoryIntegTestCase.assertSuccessfulRestore(ESBlobStoreRepositoryIntegTestCase.java:312)
at org.elasticsearch.repositories.blobstore.ESBlobStoreRepositoryIntegTestCase.testMultipleSnapshotAndRollback(ESBlobStoreRepositoryIntegTestCase.java:208)

and from master:

org.elasticsearch.snapshots.SnapshotException: [qyqtw:ujgavvr-4/3MRHMgvaRAGThWsmmkmONA] failed to write metadata for snapshotOpen stacktrace
Caused by: java.io.IOException: Unable to upload object [meta-3MRHMgvaRAGThWsmmkmONA.dat] using a single uploadClose stacktrace
at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:338)
at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:106)
at java.security.AccessController.doPrivileged(AccessController.java:551)
at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48)
at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:104)
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.lambda$write$1(ChecksumBlobStoreFormat.java:187)
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.writeTo(ChecksumBlobStoreFormat.java:219)
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.write(ChecksumBlobStoreFormat.java:185)

looks like underlying cause in first case could be:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: 7cc42b0e-f8ac-780b-dfd2-e44d1842cdf3; S3 Extended Request ID: null)

Example failures:

7.x
master

Were able to reproduce this using:

./gradlew :plugins:repository-s3:test --tests "org.elasticsearch.repositories.s3.S3BlobStoreRepositoryTests.testMultipleSnapshotAndRollback" -Dtests.seed=7EACA78186296F35 -Dtests.security.manager=true -Dtests.locale=ur-IN -Dtests.timezone=America/Menominee -Dcompiler.java=12 -Druntime.java=12

Two tries gave me one of each type of failure from above.

Looks like it started failing more frequently on August 31st.

[elasticmachine]

Pinging @elastic/es-distributed

[henningandersen]

Most likely highly related to #46218

[henningandersen]

This 7.x build has a slight variant of the error:

https://gradle-enterprise.elastic.co/s/mcwywstujemwq/console-log?task=:plugins:repository-s3:test

org.elasticsearch.repositories.s3.S3BlobStoreRepositoryTests > testMultipleSnapshotAndRollback FAILED
--
RepositoryException[[oqrye] failed to delete snapshot [g-3/59cuGY7xR4OziRJpu3OOmA]]; nested: NotSerializableExceptionWrapper[amazon_s3_exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: 97ae6d51-094f-7983-3035-792120c3a5e1; S3 Extended Request ID: null)];
 
Caused by:
NotSerializableExceptionWrapper[amazon_s3_exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: 97ae6d51-094f-7983-3035-792120c3a5e1; S3 Extended Request ID: null)]

[henningandersen]

Another variation here:
https://gradle-enterprise.elastic.co/s/azmiwaigyxnrw/console-log?task=:plugins:repository-s3:test

org.elasticsearch.repositories.s3.S3BlobStoreRepositoryTests > testMultipleSnapshotAndRollback FAILED
--
RepositoryException[[fewmz] Could not determine repository generation from root blobs]; nested: IOException[Exception when listing blobs by prefix [index-]]; nested: NotSerializableExceptionWrapper[amazon_s3_exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: b837ee39-aaa6-2ab5-da20-4148309c3ccb; S3 Extended Request ID: null)];
 
Caused by:
java.io.IOException: Exception when listing blobs by prefix [index-]
 
Caused by:
NotSerializableExceptionWrapper[amazon_s3_exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: b837ee39-aaa6-2ab5-da20-4148309c3ccb; S3 Extended Request ID: null)]

[tlrx]

Thanks @henningandersen. I suspect that the tests sometimes pick high random values (for the number of snapshot to execute, or the number of indices to snapshot, for example) that generate more requests (because more objects to snapshot/list/restore). And because the tests generate random server errors, it is possible that at some point the S3 client reaches its internal number of erroneous errors that are acceptable and start failing request where the test expect these request to be retried.

Any type of request can fail, so sometimes it's a delete, sometimes a list, sometimes an upload.