Add the ability to bundle multiple plugins into one

[jimczi]

Currently we don't allow multiple plugins in a single zip. It would be useful to be able to group multiple plugins in an uber plugin that would install them all in one command.
The current structure for a plugin zip is as follow:

<plugin_name>-<version>.zip
|____elasticsearch
| |____<plugin_name>
| | |____<plugin_files, plugin-descriptor.properties>

In order to allow multiple plugins in a single zip we could extend this format to accept:

<plugin_name>-<version>.zip
|____elasticsearch
| |____<uber_plugin_name>
| | |____uber_plugin-descriptor.properties
| | |____<plugin_name_1>
| | | |____<plugin_files, plugin-descriptor.properties>
| | |____<plugin_name_2>
| | | |____<plugin_files, plugin-descriptor.properties>

The uber_plugin-descriptor.properties would contain the name of the sub-plugins that should be installed.
The installation would create a directory named after uber_plugin_name and would put the jars and required files in a sub directory named after the sub plugin name. Each sub plugin would still be loaded as a single plugin, e.g. with a different class loader.

[dadoonet]

Thinking of it loud: I was wondering if we can just read plugin-descriptor.properties and if there is a property like uber-plugin: true, just read the existing subdirectories and consider them as plugins to install within /plugins/uber_plugin_name ?

So the ZIP structure would be:

<plugin_name>-<version>.zip
|____elasticsearch
| |____<uber_plugin_name>
| | |____plugin-descriptor.properties
| | |____<plugin_name_1>
| | | |____<plugin_files, plugin-descriptor.properties>
| | |____<plugin_name_2>
| | | |____<plugin_files, plugin-descriptor.properties>

After installation we would have:

/plugins/
|____<uber_plugin_name>
| |____plugin-descriptor.properties
| |____<plugin_name_1>
| | |____<plugin_files, plugin-descriptor.properties>
| |____<plugin_name_2>
| |____<plugin_files, plugin-descriptor.properties>

The PluginService would "just" have to be adapted to read this new structure when /plugins/uber_plugin_name/plugin-descriptor.properties contains uber-plugin: true.

But may it would introduce leniency though?

[jimczi]

Sorry the uber_plugin-descriptor.properties in the description is misleading. I don't think it should be named differently but as you said just using new parameters to handle the bundle of plugins.

[rjernst]

I think there are 2 things to discuss: the zip format, and the installed format.

For the zip format, your proposal seems ok (I would use a separate properties file name, so that we dont add leniency for all the required properties we have now).

For the installed format, I would like to keep this straightforward for how plugins are loaded. With the SPI work, we will have plugins depending on the names of other installed plugins. I would like to keep this as simple as possible in how they are loaded, so I don't think we should have to traverse an "uber plugin" to load all the plugins it has installed. Instead, I think the uber plugin can be handled completely in the plugin cli for installation/removal. That is, the inner plugins should exist in the plugins directory as normal plugins, and only in the plugin cli commands would they be reorganized to be "part of" the uber plugin eg when listing plugins, or asking to remove a plugin. I think this is important not only for simplicity in loading, but also b/c we will continue to have a global namespace for installed plugins. Otherwise it gets tricky with two uber plugins having the same name for an inner plugin, and what that means for plugin SPI deps.

[jimczi]

Instead, I think the uber plugin can be handled completely in the plugin cli for installation/removal.

+1, so the following uber-plugin:

<plugin_name>-<version>.zip
|____elasticsearch
| |____<uber_plugin_name>
| | |____uber-plugin-descriptor.properties
| | |____<plugin_name_1>
| | | |____<plugin_files, plugin-descriptor.properties, bin, config>
| | |____<plugin_name_2>
| | | |____<plugin_files, plugin-descriptor.properties>

... creates:

|____<plugins>
| |____<uber_plugin_name>
| | |____uber-plugin-descriptor.properties
| | ____<plugin_name_1>
| | | ____<plugin_files, plugin-descriptor.properties>
| | ____<plugin_name_2>
| | | ____<plugin_files, plugin-descriptor.properties>
| |____<bin>
| | ____<plugin_name_1>
| | | ____<bin_files>
| |____<config>
| | ____<plugin_name_1>
| | | ____<config_files>

The uber plugin creates an entry in the plugins directory but only for the descriptor. The file is used for remove and list plugins in order to re-group the inner plugins with their parent.
The sub plugins are installed as normal plugins. The installation fails if any sub-plugin name clashes with a previously installed plugin.
Should we allow bin and config for the uber-plugin as well ?

[rjernst]

Should we allow bin and config for the uber-plugin as well ?

I would not. Let's keep it simple. The uber plugin is simply a format for delivering multiple plugins.