[CI] Rolling upgrade tests fail to find dynamic hosts list

[romseygeek]

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=fedora/54/console

14:54:51 > Task :x-pack:qa:rolling-upgrade:with-system-key:v6.6.0#oneThirdUpgradedTestCluster#wait FAILED
14:54:51 :x-pack:qa:rolling-upgrade:with-system-key:v6.6.0#oneThirdUpgradedTestCluster#wait (Thread[Task worker for ':' Thread 11,5,main]) completed. Took 40.016 secs.

The task fails because a cluster doesn't form.

The log files for the cluster nodes all contain the following line:

[2018-11-12T14:53:27,628][WARN ][o.e.d.z.FileBasedUnicastHostsProvider] [node-1] expected, but did not find, a dynamic hosts list at [/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/fedora/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.6.0#oldClusterTestCluster node1/elasticsearch-6.6.0-SNAPSHOT/config/unicast_hosts.txt]

However, that file is there, and contains the correct list of IPv6 addresses.

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

I'll take a look :)

[romseygeek]

I think this has caused quite a few failures today, but it's only possible to tell by logging into the workers, and they keep getting shut down and re-provisioned for other builds.

[original-brownbear]

@romseygeek yea we redid the way the discovery works in these in #35178 and apparently introduced some instability. I'll look into this in a few hours at the very latest.