HLREST: Machine learning tests leave ML state after they've finished

[nik9000]

This build failed because the ML tests leave the cluster in a state where it recreates the .ml-notifcations. This interferes with other tests that expect an empty cluster. I think the right thing to do to fix this is to open source the ml state cleaner and move it into the test framework and use it in the rest client tests.

[elasticmachine]

Pinging @elastic/es-core-infra

[dimitris-athanasiou]

We've been aware we'd have to clean up. We were waiting to have the basic APIs to do so using the HLRC itself. We need #32960 first. Once that's in, I'll add the cleaner and re-enable the tests.

[nik9000]

Awesome!

[elasticmachine]

Pinging @elastic/ml-core

[dimitris-athanasiou]

I've been thinking about which is a better option: reuse MlRestTestStateCleaner.java from server or write a new cleaner in the client library. While it is nice to reuse the server cleaner, it will only be a temporary solution. We are heading towards a hard split between the client and the server side, so we will not be able to keep reusing it. However, it might be a good idea to do that at first as it allows us to implement the datafeed APIs while adding tests for them. Then, we can move back the MlRestTestStateCleaner and write a client one using the HLRC.

Actually, to avoid the churn in the server side, we can simply duplicate the MlRestTestStateCleaner temporarily, then remove it and introduce the HLRC cleaner. That way we don't churn at all the server side.

I'll go for this approach unless there is a strong downside I missed. Any thoughts @benwtrent , @davidkyle ?