Use sequences numbers for Compare and Set operations

[bleskes]

Elasticsearch currently supports CAS operations (needed for optimistic concurrency) using a document _version field. Sadly this is not water tight and can go wrong if a primary fails between the read and the CAS write operation. To remedy this we need to move to using a document's _seq_no and _primary_term fields as an identifier for CAS operations.

To do so we will add two parameters to write operations - _expected_seq_no and _expected_primary_term. When set, those parameters will require the document to have exactly the expected values in the respective fields for a successful write.

Note that this can co-exists with the the current _version based method, which we can phase out or make optional later on.

Relates #10708

[dnhatn]

Hi @jasontedor, I am thinking to work on this. What do you think? Thank you.

[jasontedor]

@dnhatn I think we can get there, but let’s work our ways towards it.

[DaveCTurner]

Just registering my interest in this too @dnhatn. Please ping me if you're discussing this.

[dnhatn]

@DaveCTurner Sure.

[jasontedor]

Let’s return to this one in the future after we have addressed some other issues first.