Expunge deletes after shrink

[inqueue]

Describe the feature:
Looking at the Shrink API, the target index consumes twice the amount of disk space as the source index due to unsearchable segments left behind by the API. It would be an easy win to free up disk space by executing a force merge with only_expunge_deletes=true after completing the shrink operation. Could these options be considered?

• Add a parameter to _shrink to trigger segment removal merged segments; i.e., call /_force_merge?only_expunge_deletes=true.
• Document a clean up step to execute force merge with only_expunge_deletes=true.
• Both options above.

Elasticsearch version (bin/elasticsearch --version):

Version: 6.2.2, Build: 10b1edd/2018-02-16T19:01:30.685723Z, JVM: 1.8.0_51

[elasticmachine]

Pinging @elastic/es-distributed

[s1monw]

hey @inqueue thanks for opening this. I first need to clarify a couple of things:

• the index you show doesn't have any deleted documents so an call that expunges deletes won't help
• the reason that there is more space used is that we kick of a merge after the shrink but don't wait for it to finish. We could but this would mean we need to block for a long time and we don't want that since this is during a recovery.
• a call to force merge with /_force_merge?only_expunge_deletes=true is blocking and will cause a commit (flush in ES terms) after it's done. The commit / flush is what will free up the merged segments since lucene won't select any segments to force merge (no deletes are present).
• if a shard is idle we will force a commit (5 min no indexing) or we commit after a period of indexing. This will cause the old segments to go away. All we need is to wait for the flush to happen.
• if the merge finishes after the shard is idle we will cause a flush once the last merge is done

given that I think we can close this issue since we are already doing what's needed?

[s1monw]

@inqueue I added testcases for this unit and integration and they both pass. I am not sure what you are seeing here.

[s1monw]

@inqueue I am going to close this for now.