Incremental restores treat segments containing soft-deleted docs as "different"

[DaveCTurner]

Today in restores and peer recoveries we treat any .liv files as per-commit rather than per-segment since they may change independently of the rest of the segment as documents are deleted:

elasticsearch/server/src/main/java/org/elasticsearch/index/store/Store.java

Lines 963 to 969 in a862dff

	if (IndexFileNames.SEGMENTS.equals(segmentId) ||
	DEL_FILE_EXTENSION.equals(extension) || LIV_FILE_EXTENSION.equals(extension)) {
	// only treat del files as per-commit files fnm files are generational but only for upgradable DV
	perCommitStoreFiles.add(meta);
	} else {
	perSegment.computeIfAbsent(segmentId, k -> new ArrayList<>()).add(meta);
	}

This means that a restore or peer recovery that only deletes documents need only copy the corresponding .liv files. However today we soft-delete documents instead which updates their doc values, not their liveness markers, which means we fully restore/recover any segments whose set of deleted documents have changed.

Relates #55013
Relates https://discuss.elastic.co/t/227184
Relates https://issues.apache.org/jira/browse/LUCENE-9324

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Store)

[yjchen-tw]

any update for this issue?
it seems to have a PR #55239 to fix this but have been postponed from v7.9.0 to v7.14.0
we suffer from large network traffic during restore due to newly deleted documents in large segments
would be happy to know if this can be fixed in recent release