Establishing relationships between plugins and their persistent data

[imotov]

As elasticsearch plugins become more sophisticated, they rely more and more on the data that they persist in elasticsearch. This persistent data can be found in three places: plugin-specific indices, index templates that control these indices, and custom cluster state metadata components.

Currently, any plugin can freely create and modify any of these structures in the cluster and we cannot limit certain potentially dangerous operation (such as upgrade for example #24379 (comment)) to plugin's own data structures since no relationships between these structures and the plugin is maintained.

One possible solution that was discussed before is to formalize and unify the interface through which a plugin registers and upgrades custom cluster state and templates. At the same time the plugin might register the mask for its indices or we can come up with some index naming convention that could use to identify that an index "belongs" to a certain plugin.

We would like to discuss this and other possible solutions for this problem.

[droberts195]

I think it would also be good to consider how such a framework could allow the yaml test framework to be made more robust. At the moment plugins that rely on their index templates and/or custom cluster state to be set up before they run their yaml tests generally do this by creating a @Before method in a test class that extends ESClientYamlSuiteTestCase. Different developers do the checks in different ways, so it gets messy, and we can also get problems like the one described in #25100.

If part of the registration process for custom cluster state and index templates was to specify the mechanism to wait for a remote cluster to have fully set these up then some of the classes that extend ESClientYamlSuiteTestCase could be removed.

It would also be good to allow plugins that register custom cluster state to register a mechanism for cleaning up that state between tests. This would allow almost all plugin test classes that extend ESClientYamlSuiteTestCase to be removed.

[bleskes]

@imotov I think the idea is appealing and I can see a bit how we can add proper plugin API in the custom cluster state metadata. This is because we have some experience in that area. When it comes to indices I'm less sure of what would be a good plan. Can you may come up with some examples?

[imotov]

@bleskes we have applications/plugins that run as part of the elasticsearch cluster and generate potentially large amount of data. Examples of these apps are monitoring, ml, and watcher. This data falls into 2 categories: app configuration information (watches, for example) and reports of app execution (watch history). The current approach is to store this data in special indices. On one side this solution provides maximum flexibility - you have full power of elasticsearch to work with these indices. On the other side, this flexibility means that any user or plugin can modify these indices any way they want, we can have potential naming conflicts, users or other plugins may inadvertently make incompatible mapping or template changes.

[imotov]

We discussed it in FixItFriday and decided that it warrants a bigger separate discussion.