Stop accessing System Indices directly in REST tests

[gwbrown]

As part of formalizing System Indices in Elasticsearch (#50251), we will be restricting access to System Indices via the REST interface, except by means of purpose-built APIs. For example, accessing .watches directly will not be allowed - all interaction with the .watches index via the REST API should use the purpose-built Watcher APIs.

If it is necessary to access System Indices via the REST API today, an API should be added to fulfill that need rather than using generic APIs to do so.

There are a number of places in our REST tests which access System Indices directly. These tests will be modified by #60945 to avoid failures by consuming the deprecation warning as necessary, but that is not a long-term solution. The tests below should be evaluated to determine if there is a necessary API that is currently missing, and if so, add that API and convert the test to use it. If not, we should determine an alternate method for testing the functionality which does not directly access System Indices via the REST API.

Tests/test-related methods which currently access System Indices directly via the REST API:

Test Framework

• ESRestTestCase.wipeAllIndices - @williamrandolph

Tasks

• delete_by_query/80_slices.yml - Multiple slices with wait_for_completion=false
• delete_by_query/80_slices.yml - Multiple slices with rethrottle
• reindex/80_slices.yml - Multiple slices with wait_for_completion=false
• reindex/80_slices.yml - Multiple slices with rethrottle
• update_by_query/80_slices.yml - Multiple slices with wait_for_completion=false
• update_by_query/80_slices.yml - Multiple slices with rethrottle
• rolling_upgrade .. upgraded_cluster/10_basic.yml - Find a task result record from the old cluster

Transforms

• TransformUsageIT.testUsage
• TransformConfigurationIndexIT.testDeleteConfigurationLeftOver
• TranformInternalIndexIT.testUpdateDeletesOldTransformConfig
• TransformRestTestCase.wipeTransforms
• TranformRobustnessIT.beEvilAndDeleteTheTransformIndex
• rolling-upgrade .. upgraded_cluster/80_transform_jobs_crud.yml - Test index mappings for latest internal index and audit index

ML

• MlMappingsUpgradeIT.assertUpgradedConfigMappings
• IndexMappingTemplateAsserter.assertMlMappingsMatchTemplates
• InferenceIngestIT.cleanUpData
• ml/calendar_crud.yml - Test delete calendar deletes events
• ml/custom_all_field.yml - Test wildcard job id
• ml/filter_crud.yml - setup ([ML] Remove direct access to system index from filter_crud REST test #63111)
• ml/filter_crud.yml - Test get all filter given index exists but no mapping for filter_id ([ML] Remove direct access to system index from filter_crud REST test #63111)
• ml/inference_stats_crud.yml - setup
• ml/inference_crud.yml - setup
• ml/index_layout.yml - Test force close does not create state
• ml/index_layout.yml - Test CRUD on two jobs in shared index

Watcher

• WatcherRestTestCase.stopWatcher Remove more usages of .watches system index. #65835
• WatcherYamlSuiteTestCase.stopWatcher Remove more usages of .watches system index. #65835
• MonitoringWithWatcherIT.cleanExporters Remove more usages of .watches system index. #65835
• MonitoringWithWatcherIT.assertTotalWatchCount Add query watches api to retrieve multiple watches. #64582
• FullClusterRestartIT.testWatcher Remove usage of .watches system index in full cluster restart. #65884
• watcher/activate_watch/10_basic.yml - Test activate watch api Use watcher get api instead of .watches index. #64199
• watcher/ack_watch/10_basic.yml - Test ack watch api Use watcher get api instead of .watches index. #64199
• watcher/delete_watch/10_basic.yml - Test delete watch api Use watcher get api instead of .watches index. #64199
• watcher/get_watch/10_basic.yml - Test get watch api Use watcher get api instead of .watches index. #64199
• watcher/put_watch/80_put_get_watch_with_passwords.yml - Test putting a watch with a redacted password with old seq no returns an error Use watcher get api instead of .watches index. #64199
• managing_watches.asciidoc Remove more usages of .watches system index. #65835

Security

• FullClusterRestartIT.testSecurityNativeRealm
• users/10_basic.yml - Test put user with password hash

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[elasticmachine]

Pinging @elastic/es-security (:Security/Security)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Watcher)

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Task Management)

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)