Move resources creation to init stage

[ycombinator]

Currently, the Monitoring plugin creates resources it needs such as index templates, ingest pipelines, and watches lazily via the exporter.

In the future, Metricbeat will collect monitoring data from stack products and ship it directly to the Monitoring ES cluster. That is, no exporter will be involved. As such, we want to now eagerly create the aforementioned resources', when the Monitoring plugin is initialized.

[elasticmachine]

Pinging @elastic/es-core-features

[ycombinator]

It's worth mentioning that in the longer term, something outside of Elasticsearch might take over the responsibility of creating these resources. However, for the immediate future, we want to implement what is stated in this issue description.

[cachedout]

cc: @elastic/stack-monitoring for visibility

[cachedout]

Today I tried to see what this would look like and I attempted to migrate some of the logic from the exporter to the plugin initialization. You can see some (obviously very rough) work on this in my fork here: https://github.com/cachedout/elasticsearch/tree/mon_mb

This does not presently function because it appears that the cluster_state is not fully initialized at the point that I am trying to create the templates. Below is the error which ES returns on startup with my changes:

[2019-11-12T18:50:53,419][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [runTask-0] fatal error in thread [main], exiting
java.lang.AssertionError: initial cluster state not set yet
        at org.elasticsearch.cluster.service.ClusterApplierService.state(ClusterApplierService.java:208) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.state(ClusterService.java:125) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.localNode(ClusterService.java:109) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.<init>(TransportMasterNodeAction.java:117) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:103) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:88) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:241) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:144) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:243) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:193) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:172) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:158) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$2(SecurityActionFilter.java:100) ~[?:?]
        at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:96) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:117) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:98) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:138) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:88) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:77) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1211) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.putTemplate(AbstractClient.java:1581) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:75) ~[?:?]
        at org.elasticsearch.xpack.monitoring.MonitoringService.putTemplate(MonitoringService.java:254) ~[?:?]
        at org.elasticsearch.xpack.monitoring.MonitoringService.lambda$preFlightResourceCreation$0(MonitoringService.java:285) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1507) ~[?:?]
        at org.elasticsearch.xpack.monitoring.MonitoringService.preFlightResourceCreation(MonitoringService.java:293) ~[?:?]
        at org.elasticsearch.xpack.monitoring.MonitoringService.doStart(MonitoringService.java:314) ~[?:?]
        at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.ArrayList.forEach(ArrayList.java:1507) ~[?:?]
        at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1085) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:657) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]

@jakelandis I could use some guidance here. I'm feeling a little out of my depth trying to move forward here. Is this something you could help me with?

[jakelandis]

@cachedout - you may be running into #48277 (not merged yet, but easy enough copy the code for a quick test)

[cachedout]

Hi @jakelandis. Thanks for the quick reply!

I applied the change in #48277 and it didn't have any effect and I observed the same stacktrace. Then I moved the logic up to the top of doStart() prior to the call to preFlightResourceCreation(); which would set up the template. As expected, since the cluster state isn't ready, it never reaches the call to preFlightResourceCreation().

Do you have a suggested approach? Is it expected that cluster state will not be available at this point during startup? Is it safe to block a plugin from starting until the state is available? (I assume this is frowned on.) I have to defer to you and the ES team at this point.

[cachedout]

I am wondering if it would be more sensible to split this up into two separate PRs.

PR1: Additional templates

This PR would simply add new templates which are prefixed by metrics-stack-<product> as discussed during a conversation with @elastic/stack-monitoring and @ycombinator at EAH. It would not move resource creation to the init stage.

PR2: Move to init stage

This would do as it says in the title. It would move all resource creation into the init stage of the plugin and away from the local exporter.

---

The reasoning for this is that after some investigation, it appears to me that the work which would be in PR2 is a significantly heavier lift and if we can accomplish this right away, it would unblock @ycombinator from starting on the last set of Metricbeat changes to use the new index naming and the rest of @elastic/stack-monitoring from following those changes on the Kibana side.

Thoughts?

[ycombinator]

++ to splitting up the work into two PRs. A couple things to note:

1. The templates in PR1 will be fairly trivial, as we don't yet know the exact field names we'll be using in these templates. Those will be determined as we work through the changes in Metricbeat + UI. So PR1 will probably just introduce "shell" templates which we'll flesh out in subsequent PRs.

2. As discussed at EAH, PR1 can be merged into master and backported to 7.x as it's an additive change. The same is true of PR2. And the work in PR2 sets us up to perform the various breaking changes we need for 8.0.0. So I would try and get PR2 in right after PR1; otherwise we'd risk having to scramble to get PR2 and all the breaking changes PRs into master at the last minute.

[cachedout]

I spoke to @jakelandis yesterday who kindly agreed to help out with the first phase. I promised him a list of assets that we're looking for and there has been an email discussion to confirm the details.

To re-cap that discussion, there should be four templates. Each template should conform to a naming scheme wherein the name of the monitored service is prefixed by metrics-stack-. For example, the elasticsearch template should be metrics-stack-elasticsearch. Concretely, this means the four templates should be:

metrics-stack-elasticsearch
metrics-stack-kibana
metrics-stack-beats
metrics-stack-logstash

There should also be a single ILM policy which applies to the four templates. That policy should, I believe, set the default retention to 7d, which is the current default retention level. This policy could really be named anything but it might be good to conform to the same naming scheme and use something like metrics-stack-policy.

[cachedout]

I just wanted to follow up on this. I believe the current status here is that at least in regard to PR1 that it is pending resolution on a couple of email discussion threads. Is that your impression as well? If so, it might be good, I think, to copy the questions that we're trying to resolve into this issue so that we don't splinter the conversation too much.