[7.x] [ML] Make ML indices hidden when the node becomes master (#77416)

[przemekwitek]

Backports the following commits to 7.x:

• [ML] Make ML indices hidden when the node becomes master ([ML] Make ML indices hidden when the node becomes master #77416)

[przemekwitek]

run elasticsearch-ci/bwc

[przemekwitek]

run elasticsearch-ci/bwc

[droberts195]

It's too late to debug this for 7.16 now - closing.

[droberts195]

I had a look at why the BWC tests failed for this PR. It was because the job results aliases against the .ml-anomalies-shared index lost their filters when they were made hidden, so ended up returning documents from other jobs when they were used for searches.

This is a bug that will affect the same functionality in version 8.0 if a user upgrades to 8.0 in multiple steps, for example 6.8 -> 7.16 -> 8.0. The only reason the 8.0 PR passed CI is that we don't test that multiple upgrade scenario in CI.

I have opened #80148 to fix this for 8.0.

[droberts195]

I've realised that the best way to test #80148 on real indices is on the 7.16 branch, where the BWC tests will upgrade indices that really were created by 6.8 instead of just fake attempts at simulating these. Therefore I am temporarily opening this PR, not to merge it, but to exercise the code from #80148 that will eventually be committed to 8.0.

[przemekwitek]

I had a look at why the BWC tests failed for this PR. It was because the job results aliases against the .ml-anomalies-shared index lost their filters when they were made hidden, so ended up returning documents from other jobs when they were used for searches.

Great job on getting to the root cause. Unfortunately I was not aware of these filters and could not find the issue :/

I have opened #80148 to fix this for 8.0.

Thanks. I assigned this to myself.

[przemekwitek]

Therefore I am temporarily opening this PR, not to merge it, but to exercise the code from #80148 that will eventually be committed to 8.0.

But that code (committed to 8.0) will not be in 7.16 right? So will BWC tests really exercise this path?

[droberts195]

But that code (committed to 8.0) will not be in 7.16 right? So will BWC tests really exercise this path?

They won't on 8.0. But they will on this PR, hence why I cherry-picked the change to it.

I'll run BWC tests 4 times on this PR, and if they pass all 4 times then that gives confidence that the fix works. There are two jobs in the BWC tests, so getting the model size stats for a random job will get the correct ones 50% of the time, hence the need to run several iterations.

If this PR proves that the fix works with a real pre-7.7 index then the next step will be to add a fake old alias with a filter to #80148 so that the code at least gets covered a little in CI even though it won't be completely real coverage. It would be great if you could think about where and how to write that test. The place where the filter gets added to the job's read alias in the production code is:

elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/job/persistence/JobResultsProvider.java

Line 331 in 12ad399

.filter(QueryBuilders.termQuery(Job.ID.getPreferredName(), job.getId()))

Finally, this PR can be closed without merging it.

[droberts195]

@przemekwitek I think you just force pushed this PR branch to remove the change I cherry-picked from #80148.

To make the BWC tests pass for this PR you'll need to add back the changes from ca8297a. Otherwise the tests are just going to fail in the same way they did a week or two ago.

[przemekwitek]

run elasticsearch-ci/bwc

[przemekwitek]

run elasticsearch-ci/bwc

[przemekwitek]

run elasticsearch-ci/bwc

[przemekwitek]

run elasticsearch-ci/bwc

[droberts195]

It looks like we had 5 consecutive successful runs of the BWC tests, so this PR has done its job of confirming that the fix works with real 6.8 indices and aliases.

The next step is to add a test to #80148 that tries to recreate the filtered alias scenario as best it can.

[przemekwitek]

Re-opening in order to perform a few more tests.

[droberts195]

run elasticsearch-ci/bwc

[droberts195]

run elasticsearch-ci/bwc

[droberts195]

run elasticsearch-ci/bwc

[droberts195]

This functionality won't go into 7.16.