Rolling upgrade tests with 3 nodes

[ywelsch]

TL;DR: I'm adding a BWC layer in #24841 that will only be triggered in a mixed-version cluster with at least 3 nodes and an index with at least 3 shard copies (i.e. index.number_of_replicas > 1). Q: Should we change our existing qa/rolling-upgrade tests to run with 3 nodes?

Details: When a primary fails and a replica is promoted to primary, the new primary will trigger a primary-replica resync, making sure that all the replicas are realigned with the new primary. This is a transport action that will only go into 6.0, not into 5.x. As such I added a BWC layer to the transport action that it will not send these resync requests to 5.6 nodes (i.e. replicas on 5.6 nodes won't be realigned with the new 6.0 primary, same as we do today in < 6.0).

[dakrone]

+1, I think 3 nodes is a good way to test, it may shake out things that we don't see with primary promotion (such as things similar to #25277)

[cbuescher]

Discussed in FixitFriday, sound like a good idea in general but we want to get @rjernst opinion about this too since it touches on build issues as well.

[rjernst]

I don't think a bwc integration test is the place to find issues. We won't have randomization here, and the intent of these tests is to directly test integration of components in the system with how they behave in a backward compatible way. Testing internal behavior of how the system operates should be done with unit tests IMO.

[rjernst]

I chatted with @ywelsch and we came to an agreement. I am ok with modifying the the rolling restart test to use 3 nodes, under the premise that a user doing a rolling restart with 2 nodes is not realistic, and the entire point of these tests is to be "representative" of a basic user. With 3 nodes, it also makes sense to have an index with both 1 and 2 replicas test through the rolling restart. What I would like to not see is any test mentioning "primary failover" or anything like that. And if we ever catch something at this level, a unit test should be added, as finding issues with the rolling restart tests is a last resort.

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@ywelsch I presume we still want to do this (the test still uses 2 nodes as far as I can tell). I vaguely remember you trying and it ended up being more difficult, but I'm not sure. If that's true, can you share your experiences? Otherwise it might be a good opportunity for @vladimirdolzhenko do learn the qa tests universe.

[ywelsch]

The tricky bit is to make the Gradle changes. Setting this up for a 2 node cluster is already tricky, see https://github.com/elastic/elasticsearch/blob/master/qa/rolling-upgrade/build.gradle
Doing it for a 3 node cluster is even harder as it requires more state transitions, which are tricky to model with Gradle dependencies and easy to get wrong (and very hard to debug). Unless @vladimirdolzhenko wants to become a Gradle expert, I don't recommend this as a ramp-up task.