-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Description
Continuing from #39656 here:
Currently deleting snapshots can be a very slow process if the delete entails removing a large number of blobs from the repository (i.e. when one or more indices that consistent of many files becomes unreferenced and has to be deleted).
The current implementation of the blob store only offers synchronous and sequential deletes of these blobs, which means that one needs at least as many network calls as the number of to be deleted blobs.
In addition to being potentially very slow, this also might incur higher than necessary cost for blob stores cloud solutions that bill for each API request.
There are at least two possible improvements to be made to performance:
- Make the delete API non-blocking so blobs can be removed in parallel where possible. (Suggested implementation in Async Snapshot Repository Deletes #40144)
- Use bulk deletion APIs where possible (e.g. S3's bulk delete)
I think we should do the latter here and adjust org.elasticsearch.common.blobstore.BlobContainer#deleteBlob to accept a list of blobs to delete and leverage bulk delete APIs in implementations where possible.
Note: Improving the performance of deletes will also lower the likelihood of running into stale snapshot blobs that aren't referenced by any meta data (see #13159 ).