Resync should not send operations without sequence number

[dnhatn]

Primary-replica resync in a mixed-cluster between 6.x and 5.6 can send operations without sequence number to a replica which already processed operations with sequence number. This leads to the failure of that replica for we trip the sequence number assertion when writing resync operations without sequence number to translog.

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@dnhatn I think I'm missing something. The resync is in charge of verifying operations above the global checkpoint. We currently starting to sync from the global checkpoint + 1 , which seems to only be negative if the global checkpoint is uknown on the new primary. I think we talked before (although I don't remember the details) about bootstrapping the global checkpoint in these cases. I think the right solution is there (and also asserting we always have a starting sequence number for a resync). Is there any reason for you to go down the route of explicit filtering rather than checking the starting sequence number?

[dnhatn]

@bleskes Thank you for looking :)

Bootstrapping the global checkpoint would work too - I can make that change. I did not have any preference for my implementation, merely followed the pattern we did in #27580. Moreover, if we have an assigned global checkpoint then we will never use operations without seqno; therefore I think we should exclude them from translog snapshot.

[bleskes]

Bootstrapping the global checkpoint would work too - I can make that change. I did not have any preference for my implementation, merely followed the pattern we did in #27580

I'm not sure I follow the reference to the PR as it's about flushing, but that aside - anything we can do to make the situation as "normal" as possible the better it is. Under normal operation (I.e., no BWC mode) we always have a global checkpoint, so that's why I favor that direction.

[ywelsch]

There is another related issue here: https://discuss.elastic.co/t/extremely-large-translog-files-per-shard-in-elasticsearch-6-2-4/174041/5 as a gcp of UNASSIGNED_SEQ_NO can cause a resync to send the full translog, which can cause high loads during a rolling upgrade.

Bootstrapping the global checkpoint would work too - I can make that change.

I'm a little uncomfortable making that change in 6.7.1, as it might have a broader impact, especially in the presence of 5.x indices with no sequence numbers. I wonder if we can have the more contained change here for now to be backported to 6.7 and follow-up with a more comprehensive investigation of getting rid of UNASSIGNED_SEQ_NO in various places as we know that in 7.x all docs will have sequence numbers. WDYT @bleskes?

[bleskes]

WDYT @bleskes?
+1

[dnhatn]

@bleskes and @ywelsch This is ready again. Can you please have another look? Thank you!

[dnhatn]

@ywelsch I pushed 61ad34d to use a weaker form which won't send operations without sequence numbers. Can you have another look? Thank you!

[ywelsch]

LGTM

[dnhatn]

Thanks @bleskes and @ywelsch.