Use soft-deletes to maintain document history

[dnhatn]

With the introduction of soft-delete in Lucene, a history of a document can be maintained. This meta issue tracks work on migrating from hard-deletes to soft-deletes.

Cut over

• Replace hard-updates by soft-updates (Use soft-update to maintain document history #29458)
• Replace hard-deletes by soft-deletes (Replace hard-deletes by soft-deletes to maintain document history #29549)
• Index no-op (Add tombstone document into Lucene for Noop #30226)
• Index stale operations (i.e. out of order) on replicas (Index stale operations to Lucene to have complete history #29679)

Retention and source

• Use SoftDeletesRetentionMergePolicy to maintain only the recent history (Introduce soft-deletes retention policy based on global checkpoint #30335)
• Special merge strategy when _source is disabled (Use a _recovery_source if source is omitted or modified #31106)
• Do not keep delete tombstones in version map. Delete tombstones can be queried from Lucene (to be done in Per doc replica rollbacks #31637)

Translog

• Add API allows querying operations given a range of seq# ([CCR] Read changes from Lucene instead of translog #30120)
• Use maxTimestamp from engine for autoGeneratedIdTimestamp (migrated to Introduce cross-cluster replication #30086)
• Replace translog by Lucene history in peer-recovery (Use Lucene soft-deletes in peer recovery #30522)
• Replace translog by Lucene history in primary-replica resync (Use Lucene history in primary-replica resync #33178)
• Only sends operations after the local checkpoint in peer-recovery (File-based recovery with soft-deletes should send only operations after the local checkpoint #33190)

TBD

• Avoid having multiple docs for the same stale operation. We currently defer the dedup until the search time but we might revisit this decision to do it at the index time(to be done in Per doc replica rollbacks #31637).

Misc

• Store the reason of noop in its document tombstone (Store the reason of noop in its document tombstone #30570)
• Use exact numDocs in synced-flush (Use exact numDocs in synced-flush and metadata snapshot #30228)
• Randomize soft-deletes settings (TEST: Randomize soft-deletes settings #31585)
• Do not expose hard-deleted documents (Do not expose hard-deleted docs in Lucene history #32333)

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

👍