Form multi-node clusters in packaging tests

[DaveCTurner]

Today we only start a single node in our packaging tests. However, an important part of our out-of-the-box experience is that it's possible to start multiple nodes on a single host and have them form a cluster with very little extra configuration.

We currently test this in SettingsBasedHostProviderIT#testClusterFormsByScanningPorts but this is unreliable (see #34781) because it's not always possible to grab two free ports that are close enough together for the port scanning to work.

It seems that one easy way to do this would be to extract multiple installations using this:

elasticsearch/qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Archives.java

Line 66 in 0f1a5ec

public static Installation installArchive(Distribution distribution, Path fullInstallPath, String version) {

Then start them using this:

elasticsearch/qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Archives.java

Line 257 in 0f1a5ec

public static void runElasticsearch(Installation installation) throws IOException {

It may also be desirable to start multiple nodes from the same installation, either by setting max_local_storage_nodes or by giving them different data paths, since these are all valid ways to start multi-node clusters on a single host.

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

Having this be part of packaging tests so it is tested for rpm and deb as well would be good. Even better would be finding a way to have the "instructions" on what needs to be changed be used in the test. Maybe @nik9000 has ideas on that; something similar to doc snippets?