Test cloud based repository plugins using real services backends

[tlrx]

Disclaimer: this issue concerns the repository plugins that use cloud-based services like Amazon S3, Azure Storage and Google Cloud Storage in order to store snapshot files.

These plugins used to be tested automatically against the real service they implement but it is not the case anymore. While some developers complained that the integration tests failed too often because of service outage/disconnections, we still think that we must have these automatized tests if we want to catch more bugs and add more features.

We recently added integration tests that use fixtures which emulate service backends (#28788, #29296, #29347). After discussing with @rjernst we could move these tests in QA sub projects under each repository plugin project. We could change the build.gradle files so that the required credentials for each service are pulled out from environment variables. If the env vars are supplied then the REST integration tests are executed against the real service backend. Otherwise the fixtures will be used to execute the tests.

• Move repository-s3 fixture tests to repository-s3/qa/amazon-s3 project and allow testing against external service (Move repository-s3 fixture tests to QA third party project #29372)
• Move repository-gcs fixture tests to repository-gcs/qa/google-cloud-storage project and allow testing against external service (Move repository-gcs fixture test to QA project and allow testing against external service #30208)
• Move azure fixture and tests under qa/third-party/microsoft-azure-storage and use env vars to retrieve the container and account names (Move repository-azure fixture test to QA project and allow testing using external service #30253)

Note: The fs and hdfs plugins are already correctly tested I think. The url could have integration tests but it can be treated as a separate issue. Same for discovery plugins that could benefit from fixture based integration tests and also real integration tests, but I'd like to address those in a separate issue.

Related to #20862

[elasticmachine]

Pinging @elastic/es-distributed

[rjernst]

Having the ability to test against the real services sounds great (given sufficient setup information passed into the test). One ask: it would be nice to have the qa/third-party tests work "normally" so we know eg the rest test files work, and "swap out" the fixture for the real service when running the test when credentials/etc are passed in. Your idea to use the rest tests from the plugin itself sound good, but I worry the setup of the test in gradle would be left untested until we actually pass in the credentials. What if the fixture based tests were all under qa/third-party?

[tlrx]

Thanks @rjernst, that's the feedback I wanted. I updated the description of the issue and moving REST tests and fixtures under qa/third-party sounds like a good idea.