Improve how builtin resources are being installed

[martijnvg]

Today our builtin resources (component/composable index/legacy templates and ilm policies) that ship by default with Elasticsearch get installed and updated via IndexTemplateRegistry infrastructure. This installs or updates the builtin resources using the put transport actions that exist for templates and ilm policies by checking the cluster state on each update.

Reusing these put transport actions that are also used by the rest APIs have downsides.

• There is nothing preventing admin users from removing or updating the builtin resources. When this happens, the IndexTemplateRegistry detects this and reverts the change, but for a short moment, these builtin resources are removed or in an unexpected state.
• An info log line is printed for each template/policy being installed/updates, which is very verbose at cluster startup. This is because printing info log is hardcoded in the put transport actions.
• Each time a template/policy is installed or updated this then results in a separate cluster state update. This slows down initializing Elasticsearch quite a bit. Ideally installing/updating builtin resources can be performed in a single cluster state update.

With the development of the operator and plugin read only settings project, infrastructure will be added that we can reuse to install and update our builtin resources. This new infrastructure will allow builtin resources to be protected from modification via API. Any attempt to modify the builtin resources via API will be forbidden. Also the other mentioned downside will be addressed by reusing the new infrastructure for operator and plugin read only settings project.

At same time we should also formalise the notion of a package, which are templates and ilm policies (and other config resources) that have a dependency on each other and belong together. Part of the notion of a package will be defined in the operator and plugin read only settings project. The builtin resources that we ship by default should be defined as a package as well.

We will be needing a way for modules and plugins to be able to define the builtin resources that need to be installed/updated. This was previously done with the IndexTemplateRegistry infrastructure. This can be something lightweight that builds on top of the infrastructure built for the operator and plugin read only settings project. For example an interface (BuiltinPackage or PackageProvider) that plugins and modules can implement that is loaded via SPI.

Tasks

• Build infrastructure that allows for modules and plugins to define legacy, component, composable index templates and ilm policies. An interface should be added that allows modules/plugins to define a package. A component should added that loads all instances of this interface defined in modules/plugins and manages the installation of all the defined resources.
• Implement operator handlers for legacy, component and composable index templates. Currently the operator and plugin read only settings project infrastructure only has support for defining cluster settings and ilm templates as part of packages.
• Migrate stack module away from IndexTemplateRegistry
• Migrate watcher module away from IndexTemplateRegistry
• Migrate ilm module away from IndexTemplateRegistry
• Migrate monitoring module away from IndexTemplateRegistry
• Migrate fleet module away from IndexTemplateRegistry
• Migrate ml module away from IndexTemplateRegistry
• Migrate deprecation module away from IndexTemplateRegistry
• Remove IndexTemplateRegistry infrastructure.

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)