[CI] MlJobIT.testDeleteJob and testDeleteJobAsync fail with index_not_found_exception

[astefan]

This one comes from an intake job. Cannot reproduce it. Full log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/793/consoleText

org.elasticsearch.xpack.ml.integration.MlJobIT > testDeleteJobAsync FAILED
    org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:42583], URI [/_cat/indices], status line [HTTP/1.1 404 Not Found]
    {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.ml-annotations-read]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":".ml-annotations-read","index":".ml-annotations-read"}],"type":"index_not_found_exception","reason":"no such index [.ml-annotations-read]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":".ml-annotations-read","index":".ml-annotations-read"},"status":404}
        at __randomizedtesting.SeedInfo.seed([F0352FCC3A079DA4:26F0FE2820C6C5DE]:0)
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
        at org.elasticsearch.xpack.ml.integration.MlJobIT.testDeleteJobAsync(MlJobIT.java:421)
REPRODUCE WITH: ./gradlew :x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner --tests "org.elasticsearch.xpack.ml.integration.MlJobIT.testDeleteJobAsync" -Dtests.seed=F0352FCC3A079DA4 -Dtests.security.manager=true -Dtests.locale=es-ES -Dtests.timezone=Europe/Belgrade -Dcompiler.java=12 -Druntime.java=11

[elasticmachine]

Pinging @elastic/ml-core

[mayya-sharipova]

Another same test failure on master happened today.
Doesn't reproduce for me as well.

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/241/console
Build Scans: https://gradle-enterprise.elastic.co/s/j4a3uqqaircp4

org.elasticsearch.client.ResponseException: 
method [GET], host [http://127.0.0.1:38147], URI [/_cat/indices], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.ml-annotations-read]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":".ml-annotations-read","index":".ml-annotations-read"}],"type":"index_not_found_exception","reason":"no such index [.ml-annotations-read]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":".ml-annotations-read","index":".ml-annotations-read"},"status":404}

[benwtrent]

#43286

Was merged recently and it significantly changed the internal logic for _cat/indices this MAY be part of the cause, but not 100% sure.

[cbuescher]

Similar looking issue today (although this failure is in MlJobIT.testDeleteJobAfterMissingIndex, not in the method mentioned in the issue title, but the symptom looks close enough) in a build on 7.3 branch:
Build Scans: https://gradle-enterprise.elastic.co/s/nnmkvcz6wpiqk

org.elasticsearch.client.ResponseException
: 
method [GET], host [http://127.0.0.1:42808], URI [/_cat/indices], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.ml-notifications]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":".ml-notifications","index":".ml-notifications"}],"type":"index_not_found_exception","reason":"no such index [.ml-notifications]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":".ml-notifications","index":".ml-notifications"},"status":404}

[2019-09-09T14:59:21,544][INFO ][o.e.x.m.i.MlJobIT        ] [testDeleteJobAfterMissingIndex] before test
[2019-09-09T14:59:21,963][INFO ][o.e.x.m.i.MlJobIT        ] [testDeleteJobAfterMissingIndex] after test
REPRODUCE WITH: ./gradlew :x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner --tests "org.elasticsearch.xpack.ml.integration.MlJobIT.testDeleteJobAfterMissingIndex" -Dtests.seed=4281757EE28B37E2 -Dtests.security.manager=true -Dtests.locale=en-NZ -Dtests.timezone=Etc/GMT-2 -Dcompiler.java=12 -Druntime.java=8

[benwtrent]

@tlrx I am not sure if this indicates a larger issue or not. It does not make sense to me that _cat/indices should ever throw a 404 when no index patterns are supplied to the API.

[tlrx]

@benwtrent Can you please provide a reproducing scenario with cat/indices that returns a 404?

[benwtrent]

@tlrx the scenario are these tests. They have failed due to this error.

.ml-notifications is an index that gets automatically created via an index template, and may be in the middle of being created while _cat/indices is called.

The best I can figure out is this:

• Dynamically create an index by indexing a document
• While that is occurring call _cat/indices with no parameters (i.e. get all current indices)
• The API will then periodically return this error.

I am not sure how to reliably re-create an internal race condition.

[mark-vieira]

We've had a few instances of a similar looking failure for the testDeleteJob test as well in the last couple of days.

https://gradle-enterprise.elastic.co/s/i4jrzz5jxishg/tests/zcquf3hc3eoda-negkewcdisf2a
https://gradle-enterprise.elastic.co/s/6kfpqmgpqenjq/tests/zcquf3hc3eoda-negkewcdisf2a
https://gradle-enterprise.elastic.co/s/wu7dcruehjevo/tests/zcquf3hc3eoda-negkewcdisf2a
https://gradle-enterprise.elastic.co/s/knpzzuexwfamq/tests/zcquf3hc3eoda-jxxxxzbenydmq
https://gradle-enterprise.elastic.co/s/emsz5rngrvvbo/tests/zcquf3hc3eoda-f6v5qmftscc6k

[droberts195]

This failure is happening a lot now. It seems to me that the only way to make progress is to find out exactly where the index_not_found_exception comes from within all the various actions the _cat/indices API calls. I opened #46739 as a way of doing this. Then the next time there's a failure on the master branch we'll know which of the inner calls caused it, and then hopefully we can reason about what the underlying problem is and how to fix it.