AzureBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles fails on 7.x with Internal Server Error

[dakrone]

The failure looks like it's coming from the Azure plugin:

  1> [2019-10-01T16:32:40,341][WARN ][o.e.r.VerifyNodeRepositoryAction] [node_sd1] [zcsoxknmm] failed to verify repository
  1> org.elasticsearch.repositories.RepositoryVerificationException: [zcsoxknmm] store location [container] is not accessible on the node [{node_sd1}{38kFyoC3RuSBjX_rgphzjQ}{cO84XnveTPafOEcEqSWK3w}{127.0.0.1}{127.0.0.1:39435}{di}]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:1211) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:126) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$400(VerifyNodeRepositoryAction.java:49) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:158) [elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:153) [elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) [elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
  1> Caused by: java.io.IOException: Can not write blob data-38kFyoC3RuSBjX_rgphzjQ.dat
  1> 	at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:104) ~[main/:?]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:1208) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
  1> 	... 11 more
  1> Caused by: com.microsoft.azure.storage.StorageException: Internal Server Error
  1> 	at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87) ~[azure-storage-8.0.0.jar:?]
  1> 	at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:313) ~[azure-storage-8.0.0.jar:?]
  1> 	at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:185) ~[azure-storage-8.0.0.jar:?]
  1> 	at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadFullBlob(CloudBlockBlob.java:938) ~[azure-storage-8.0.0.jar:?]
  1> 	at com.microsoft.azure.storage.blob.CloudBlockBlob.upload(CloudBlockBlob.java:768) ~[azure-storage-8.0.0.jar:?]
  1> 	at org.elasticsearch.repositories.azure.AzureStorageService.lambda$writeBlob$15(AzureStorageService.java:333) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:64) ~[main/:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_221]
  1> 	at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:63) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureStorageService.writeBlob(AzureStorageService.java:332) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:120) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:102) ~[main/:?]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:1208) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]

This didn't reproduce for me with:

./gradlew ':plugins:repository-azure:test' --tests "org.elasticsearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles" -Dtests.seed=163A499DF6578E5A -Dtests.security.manager=true -Dtests.locale=fr-BE -Dtests.timezone=Europe/Zaporozhye -Dcompiler.java=12 -Druntime.java=8

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+EAR/158/console
https://gradle-enterprise.elastic.co/s/snmdwrgcmvbja

[original-brownbear]

Another similar failure in a different test https://gradle-enterprise.elastic.co/s/4cmspe62pxaly I think