[CI] Internal data frame indices can cause unrelated docs tests to fail

[droberts195]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.2+intake/84/console failed due to the .data-frame-notifications-1 index existing in a docs test that didn't expect it to exist:

org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT > test {yaml=reference/api-conventions/line_345} FAILED
    java.lang.AssertionError: Failure at [reference/api-conventions:275]: $body didn't match expected value:
                             $body: 
                            metadata: 
                               indices: 
                                 index-1: 
                                     state: same [open]
                                 index-2: 
                                     state: same [open]
                                 index-3: 
                                     state: same [open]
             .data-frame-notifications-1: unexpected but found [{state=open}]

        Caused by:
        java.lang.AssertionError: $body didn't match expected value:
                                 $body: 
                                metadata: 
                                   indices: 
                                     index-1: 
                                         state: same [open]
                                     index-2: 
                                         state: same [open]
                                     index-3: 
                                         state: same [open]
                 .data-frame-notifications-1: unexpected but found [{state=open}]

I imagine we had similar problems previously with ML internal indices leaking into unrelated docs tests. We should look at how that was solved and implement something similar for data frame internal indices.

[elasticmachine]

Pinging @elastic/ml-core

[elasticmachine]

Pinging @elastic/es-docs

[talevy]

Assuming this is due to pending tasks, do you think it can be resolved the same way Rollup Jobs are in ESRestTestCase#wipeCluster?

[droberts195]

I had a look through the main data frame transforms API reference documentation and testing is skipped for all the console snippets, so the problem cannot be originating there. This means it must be coming from the data frame transform HLRC docs tests.

do you think it can be resolved the same way Rollup Jobs are in ESRestTestCase#wipeCluster?

DataFrameTransformDocumentationIT.cleanUpTransforms() is supposed to be doing something very similar, albeit only for the HLRC docs tests rather than all REST tests. However, it requires that the names of transforms to clean up are registered by adding them to the transformsToClean list during each test. It looks like there are some tests in DataFrameTransformDocumentationIT that don't do this. @davidkyle it looks like you wrote a lot of the tests so please can you have a look and see whether it's a mistake that some tests don't add to transformsToClean or if it's really not necessary for those tests?

[davidkyle]

A couple of recent failures:
https://scans.gradle.com/s/elkcdyg2v3smm
https://gradle.com/s/n3nomp3njwit6

In all cases the error occurs in the test following put-transform.asciidoc line 67.

Data Frames have a mechanism to audit notable events such as the creation of a data frame by indexing a document to .data-frame-notifications-1. PUT Data Frame Transform does this async after the data frame is created

elasticsearch/x-pack/plugin/data-frame/src/main/java/org/elasticsearch/xpack/dataframe/action/TransportPutDataFrameTransformAction.java

Line 248 in 4e392e1

auditor.info(config.getId(), "Created data frame transform.");

In the failures this async indexing occurs after the test has finished and after the test teardown which deletes the indices, so the index is re-created post teardown and leaks into the next test.

XPackRestIT#cleanup uses ESRestTestCase#waitForPendingTasks to handle this problem but that may not be the best solution here.

I'll mute put-transformL67 for now until we find good fix.