APM Server monitoring using apm_system role fails

[axw]

Describe the bug

The Monitoring APM Server section in the docs describes how to configure the server to send monitoring data to Elasticsearch, and recommends using the built-in apm_system user/role.

In 7.2 this fails with an error in the logs like this:

Failed to publish events: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/bulk] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:data/write/bulk] is unauthorized for user [apm_system]"},"status":403}

To Reproduce
Steps to reproduce the behavior:

1. Create a 7.2 stack with security enabled. Set a password for the apm_system user
2. Configure apm-server.yml according to https://www.elastic.co/guide/en/apm/server/7.2/monitoring.html, using the password from step 1.
3. Start the APM Server
4. Check the logs, see there's an error

Expected behavior

Monitoring should work using the apm_system user/role.

[axw]

This appears to be a regression in 7.2, relating to changes in the underlying Beats infrastructure.

The apm_system role has the same cluster-level privileges as beats_system, but the latter has some additional index-level privileges. These were added to beats_system in elastic/elasticsearch#40876; apm_system should presumably have been updated at the same time.

[cachedout]

I have opened elastic/elasticsearch#47302 to start work on this issue.

[simitt]

@cachedout can you confirm this issue was fixed with elastic/elasticsearch#47302 (comment)?

[cachedout]

@simitt I am still waiting for reviews on this: elastic/elasticsearch#47668