Multindex expressions expands to missing indices with Security

[albertzaharovits]

The ES Security Plugin does wildcard expansion in IndicesAndAliasesResolver on the coordinating node of the request. During this process, it rewrites the request to not include wildcards but only concrete indices. There are a few known problems with this approach, for example see #45171 (comment).

This issue is acknowledging another limitation of this approach. The wildcard expansion and the actual handling of the request for the expanded concrete indices could happen on different cluster state versions. For example, a wildcard expression is expanded to an index that is subsequently removed, but before the actual handling of the request takes place. This will generate an index missing exception (assuming ignore_unavailable=false). The same result is not possible with Security turned off because wildcard expansion and request handling work on the same cluster state version.

Causes #45652

[elasticmachine]

Pinging @elastic/es-security

[droberts195]

The code I used to investigate this problem is on commit droberts195@3fe500d of the cat_indices_404_repro branch in my fork.

Run ./runner.sh >& runner.log from the top level of the repo and eventually the problem will occur and you can then edit x-pack/plugin/ml/qa/native-multi-node-tests/build/testclusters/*/logs/integTest.log to examine the server logs.

(You probably think I'm crazy to not just run ./gradlew :x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner --tests org.elasticsearch.xpack.ml.integration.MlJobIT -Dtests.iters=100. I did try this, and never managed to reproduce the problem with it. This leads me to think that once hotspot has optimised the code the problem doesn't occur. I also tried a while loop in the shell script and that didn't reproduce the problem either, hence the many duplicated lines. I cannot think of a good explanation for why copy and paste was more successful than a while loop at the shell script level. I guess the problem is just extremely intermittent and sensitive to timing.)

[droberts195]

#81901 contains another way to reproduce this problem.