[CI] Various MlWithSecurityIT tests failing on Windows with incorrect auth failures

[droberts195]

There have been several cases in the last few days of the x-pack/qa/smoke-test-ml-with-security suite failing on Windows due to auth failures for users who ought to have permissions to call the endpoints they're calling.

An example from master is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/1675/console

action [cluster:monitor/xpack/ml/job/results/overall_buckets/get] is unauthorized for user [ml_admin]

An example from 6.x is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-windows-compatibility/1099/console

action [cluster:monitor/main] is unauthorized for user [ml_user]

I suspect this is a side effect of a security change, as no changes have recently been made to the ML permissions. The Windows builds have been repeatedly failing for around 2 weeks due to various problems, so the security change could be up to 2 weeks old. But its effects on ML would have been masked by failures earlier in the build.

[elasticmachine]

Pinging @elastic/ml-core

[elasticmachine]

Pinging @elastic/es-security

[droberts195]

#30794 was merged on 24th May, and is present in 6.4 but not 6.3 which would be consistent with these spurious auth failures affecting CI for 6.4 but not 6.3.

(#30787 was also merged on 24th May, but into 6.3 too, so probably not the cause of this problem as 6.3 builds are succeeding.)

[mayya-sharipova]

Recent failures of MlWithSecurityUserRoleIT tests on Windows on master are in: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/1678/console

An example of one of them:

REPRODUCE WITH: gradlew :x-pack:qa:smoke-test-ml-with-security:integTestRunner \
  -Dtests.seed=9B4D2B8ED6F5BF3E \
  -Dtests.class=org.elasticsearch.smoketest.MlWithSecurityUserRoleIT \
  -Dtests.method="test {yaml=ml/start_stop_datafeed/Test start given field without mappings}" \
  -Dtests.security.manager=true \
  -Dtests.locale=fi-FI \
  -Dtests.timezone=Australia/Adelaide \
  -Dtests.rest.blacklist=ml/ml_classic_analyze/Test analyze API with an analyzer that does what we used to do in native code,ml/calendar_crud/Test get calendar given missing,ml/calendar_crud/Test cannot create calendar with name _all,ml/calendar_crud/Test PageParams with ID is invalid,ml/calendar_crud/Test post calendar events given empty events,ml/calendar_crud/Test put calendar given id contains invalid chars,ml/calendar_crud/Test delete event from non existing calendar,ml/calendar_crud/Test delete job from non existing calendar,ml/custom_all_field/Test querying custom all field,ml/datafeeds_crud/Test delete datafeed with missing id,ml/datafeeds_crud/Test put datafeed referring to missing job_id,ml/datafeeds_crud/Test put datafeed with invalid query,ml/datafeeds_crud/Test put datafeed with security headers in the body,ml/datafeeds_crud/Test update datafeed with missing id,ml/delete_job_force/Test cannot force delete a non-existent job,ml/delete_model_snapshot/Test delete snapshot missing snapshotId,ml/delete_model_snapshot/Test delete snapshot missing job_id,ml/delete_model_snapshot/Test delete with in-use model,ml/filter_crud/Test create filter api with mismatching body ID,ml/filter_crud/Test get filter API with bad ID,ml/filter_crud/Test invalid param combinations,ml/filter_crud/Test non-existing filter,ml/get_datafeed_stats/Test get datafeed stats given missing datafeed_id,ml/get_datafeeds/Test get datafeed given missing datafeed_id,ml/jobs_crud/Test cannot create job with existing categorizer state document,ml/jobs_crud/Test cannot create job with existing quantiles document,ml/jobs_crud/Test cannot create job with existing result document,ml/jobs_crud/Test cannot create job with model snapshot id set,ml/jobs_crud/Test cannot decrease model_memory_limit below current usage,ml/jobs_crud/Test get job API with non existing job id,ml/jobs_crud/Test put job after closing results index,ml/jobs_crud/Test put job after closing state index,ml/jobs_crud/Test put job with inconsistent body/param ids,ml/jobs_crud/Test put job with time field in analysis_config,ml/jobs_crud/Test job with categorization_analyzer and categorization_filters,ml/jobs_get/Test get job given missing job_id,ml/jobs_get_result_buckets/Test mutually-exclusive params,ml/jobs_get_result_buckets/Test mutually-exclusive params via body,ml/jobs_get_result_categories/Test with invalid param combinations,ml/jobs_get_result_categories/Test with invalid param combinations via body,ml/jobs_get_result_overall_buckets/Test overall buckets given missing job,ml/jobs_get_result_overall_buckets/Test overall buckets given non-matching expression and not allow_no_jobs,ml/jobs_get_result_overall_buckets/Test overall buckets given top_n is 0,ml/jobs_get_result_overall_buckets/Test overall buckets given top_n is negative,ml/jobs_get_result_overall_buckets/Test overall buckets given invalid start param,ml/jobs_get_result_overall_buckets/Test overall buckets given invalid end param,ml/jobs_get_result_overall_buckets/Test overall buckets given bucket_span is smaller than max job bucket_span,ml/jobs_get_stats/Test get job stats given missing job,ml/jobs_get_stats/Test no exception on get job stats with missing index,ml/job_groups/Test put job with empty group,ml/job_groups/Test put job with group that matches an job id,ml/job_groups/Test put job with group that matches its id,ml/job_groups/Test put job with id that matches an existing group,ml/job_groups/Test put job with invalid group,ml/ml_info/Test ml info,ml/post_data/Test Flush data with invalid parameters,ml/post_data/Test flushing and posting a closed job,ml/post_data/Test open and close with non-existent job id,ml/post_data/Test POST data with invalid parameters,ml/preview_datafeed/Test preview missing datafeed,ml/revert_model_snapshot/Test revert model with invalid snapshotId,ml/start_stop_datafeed/Test start datafeed job, but not open,ml/start_stop_datafeed/Test start non existing datafeed,ml/start_stop_datafeed/Test stop non existing datafeed,ml/update_model_snapshot/Test without description,ml/validate/Test invalid job config,ml/validate/Test job config is invalid because model snapshot id set,ml/validate/Test job config that is invalid only because of the job ID,ml/validate_detector/Test invalid detector

[jaymode]

Given this is related to authorization, this is unlikely related to #30794 as we would see a different type of failure. I'll see what I can dig up, but one issue I have is why are we only seeing this on windows?

[droberts195]

@jaymode you're correct that #30794 is not to blame for this - sorry I jumped to conclusions by looking for security PRs in the change list.

So far I've worked out that the breakage occured somewhere in between ad43231 and 946f519 on the 6.x branch, i.e. late on 22nd May or early on 23rd May and more than 20 commits before your security PRs went into 6.x. I'll try to narrow it down further.

[tvernum]

@droberts195 I wonder if it's due to bb8f1e8

That's windows only, and this smoke test has a number of setupCommand in the build.gradle

[droberts195]

You are correct @tvernum. The ML with security tests pass on commit 4d3e285 but all fail with auth failures on bb8f1e8 (which is the one immediately afterwards).