Reproducible Failure in S3BlobStoreRepositoryTests.testMultipleSnapshotAndRollback

[original-brownbear]

./gradlew :plugins:repository-s3:test --tests "org.elasticsearch.repositories.s3.S3BlobStoreRepositoryTests.testMultipleSnapshotAndRollback" -Dtests.seed=F901373721E2E900 -Dtests.security.manager=true -Dtests.locale=en-SX -Dtests.timezone=IET -Dcompiler.java=12 -Druntime.java=11

fails every time (current master) with the following:

  1> Caused by: java.io.IOException: Unable to upload object [indices/IqtnNK-PRzmvOdry1yZ_0Q/3/__DnIqPmsWRtOdfmhPzc36bQ] using a single upload
  1>    at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:338) ~[main/:?]
  1>    at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:106) ~[main/:?]
  1>    at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
  1>    at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[main/:?]
  1>    at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:104) ~[main/:?]
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:1284) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1056) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    ... 5 more
  1> Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: fb9a4c3c-2fc7-7033-6b91-483335f735c8; S3 Extended Request ID: null)
  1>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[aws-java-sdk-core-1.11.562.jar:?]
  1>    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920) ~[aws-java-sdk-s3-1.11.562.jar:?]
  1>    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866) ~[aws-java-sdk-s3-1.11.562.jar:?]
  1>    at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:389) ~[aws-java-sdk-s3-1.11.562.jar:?]
  1>    at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5800) ~[aws-java-sdk-s3-1.11.562.jar:?]
  1>    at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1789) ~[aws-java-sdk-s3-1.11.562.jar:?]
  1>    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1749) ~[aws-java-sdk-s3-1.11.562.jar:?]
  1>    at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$17(S3BlobContainer.java:335) ~[main/:?]
  1>    at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[main/:?]
  1>    at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
  1>    at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[main/:?]
  1>    at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:334) ~[main/:?]
  1>    at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:106) ~[main/:?]
  1>    at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
  1>    at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[main/:?]
  1>    at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:104) ~[main/:?]
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:1284) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1056) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    ... 5 more

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

@tlrx can you take a look here, it seem sto me there is some problem with the new test infra here?

[original-brownbear]

Closing this as a duplicate of #46219 then (the other one has nicer links :))