Snapshots continuously failing with blob [pending-index-293] already exists, cannot overwrite

[alexbrasetvik]

Got a 5.0.0-cluster logging this and failing to take snapshots. It should probably overwrite or gracefully handle the fact that he file exists.

[2016-11-10T13:04:33,885][WARN ][org.elasticsearch.snapshots.SnapshotsService] [found-snapshots:scheduled-1478783056-instance-0000000006/BWJzkt8QS6WPFBT8D5Ox4w] failed to finalize snapshot
org.elasticsearch.repositories.RepositoryException: [found-snapshots] failed to update snapshot in repository
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:544) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:878) [elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:444) [elasticsearch-5.0.0.jar:5.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.nio.file.FileAlreadyExistsException: blob [pending-index-293] already exists, cannot overwrite
	at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.writeBlob(S3BlobContainer.java:105) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:872) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeIndexGen(BlobStoreRepository.java:782) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:540) ~[elasticsearch-5.0.0.jar:5.0.0]
	... 5 more

[abeyad]

@alexbrasetvik do you have additional logs you can share from the node? this looks like a genuine issue - we purposely prevented overwriting of blobs and reworked snapshots to use UUIDs so we don't have to worry about overwrites, so this is likely a bug related to that.