Make Node Stats and Node Info pluggable

[williamrandolph]

In the core Elasticsearch code, we have endpoints that simply retrieve information about nodes without otherwise affecting the cluster. The /_nodes endpoint returns what we call "info": information that doesn’t change as long as the node is running. “Info” includes, for example, the version of the JVM we are running and the list of plugins that have been installed on the node. The /_nodes/stats endpoint, by contrast, returns “stats”: information that can change over the course of the node’s runtime. “Stats” include the current JVM heap size and pools, the CPU load of the underlying OS, and statistics about indexes.

Several of our plugins create endpoints that return info and stats. They do this by implementing the ActionPlugin class and adding Actions and RestHandlers.

• The X-Pack plugin returns info on the /_xpack endpoint and usage stats from /_xpack/usage.
• The CCR (cross-cluster replication) plugin has a /_ccr/stats endpoint
• The Enrich plugin has an /_enrich/_stats endpoint
• The EQL plugin has an /_eql/stats endpoint
• The Machine Learning plugin has an info endpoint at /_xpack/ml/info (deprecated) and /_ml/info; it’s possible that this might be more like a “stats” or “settings” endpoint.
• The SQL plugin has a /_sql/stats endpoint.

Note the inconsistencies among APIs here; a user has to read code or documentation to find this information. If this information were rolled into the /_nodes and /_nodes/stats endpoints.

We also have a /_cluster/stats endpoint aggregates certain stats from multiple nodes.

We should make our /_nodes and /_nodes/stats endpoints pluggable, so that a plugin could insert info or stats into the core info and stats routes by implementing a certain interface.

With that accomplished, we can make the /_cluster/stats pluggable as well. A plugin could optionally define how a stats response can aggregate over multiple stats responses.

One open question is whether all plugins’ stats or info should appear in the default response of /_nodes or /_nodes/stats, or whether we’d want to add a flag for including plugins or not.

This issue will serve as a meta issue for smaller chunks of work.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Plugins)

[williamrandolph]

Plan of work (subject to change):

The first section of work will be reducing the amount of "hard-coding" in the current code.

• NodesInfoRequest - internally, use lists of metrics to retrieve, not fields of booleans (changes wire compatibility) (Serialize NodesInfoRequest as a set of strings #53140)
• NodesInfoRequest - externally, use a general method to add and remove metrics rather than .os(true), .threadPool(false), etc. (Create set-based interface for NodesInfoRequest #53410)
• NodesStatsRequest - internally, use lists of metrics to retrieve, not fields of booleans (changes wire compatibility) (Serialize NodesStatsRequest as set of strings #53235)
• NodesStatsRequest - externally, use a general method to add and remove stats (Use set-based interface for NodesStatsRequest #53637)
• NodesInfoRequestBuilder and NodesStatsRequestBuilder - refactor to match requests objects (Refactor nodes stats request builders to match requests #54363)
• NodeInfo serialization - internally, make the NodeInfo object serialize and deserialize in a flexible way, rather than having an ordered sequence of Writeables (Give NodeInfo a more flexible serialization/deserialization style #54305)
• NodeStats serialization - internally, make the NodeStats object serialize and deserialize in a flexible way, rather than having an ordered sequence of Writeables
• NodeInfo interface - blocks of info should be retrieved by a getInfo() method rather than hardcoded getters, internally the class should be able to hold arbitrary info objects (NodeInfo response should use a collection rather than fields #54460)
• NodeStats interface - blocks of stats should be retrieved by a getStats() method rather than hardcoded getters, internally the class should be able to hold arbitrary stats objects
• Request validation should move from the RestNodesInfoAction and RestNodesStatsAction classes to the NodeService
• NodeService should use a map or a registry object to manage its list of services, rather than using class fields
• Services capable of returning "info" and "stats" should implement an interface with "info" and "stats" methods
• Consider a general approach to requesting a selection of fields within a metrics group (i.e., sub-metrics)
• The Plugin#createComponents method should identify info/stats services and make them available to NodeService

Another section of work will involve refactoring /_cluster/stats for pluggability. Note that /_cluster/stats also returns some cluster-level data. Adjusting this aspect of the end-point may be out of scope for this ticket. See discussion below for details.

• Refactor so that services or stats classes can define their own collation. Essentially, allow plugins to define reducers for stats blocks that produce a useful aggregation. (In many cases this is just summing or averaging, e.g. total heap size or average heap size; in some cases it may be collecting lists of strings, e.g. name of JVM distribution.)
• Have the /_cluster/stats endpoint call these reducers for all of the node-level stats that it aggregates.

We can do all of the above without changing the application's behavior from the user's perspective.

We will then need to guarantee that the app can return telemetry data that is guaranteed not to contain user data:

• Design an API call that will be guaranteed to return no user-data (no index names, no mapper names, etc.). Alternately, use the current _cluster/stats endpoint for this and design a different API call for stats-including-user-data.
• Implement some kind of filter in code that keeps user data from making its way into the API that's used for telemetry.

Finally, we will need to update the plugins.

• Consider updating the sql plugin to provide node-level stats

[jasontedor]

Note the inconsistencies among APIs here; a user has to read code or documentation to find this information. If this information were rolled into the /_nodes and /_nodes/stats endpoints.

For some of the examples listed (e.g., CCR, Enrich), the stats are not oriented at the node level, and I don't think they make sense at the node level, so I'm struggling to see how they would be rolled into the node endpoints.

I'm also not sure that it would be usability improvement to roll all of the feature stats (e.g., CCR, Enrich) into a single endpoint.

We also have a /_cluster/stats endpoint aggregates certain stats from multiple nodes.

This one will require some care because it's used in the telemetry endpoint.

[rjernst]

We also have a /_cluster/stats endpoint aggregates certain stats from multiple nodes.

This one will require some care because it's used in the telemetry endpoint.

While I do agree it will require some care, the fact it is used by telemetry is one of the major benefits I see here. I believe (please correct me if I am wrong) that telemetry must be explicitly updated when new features are added to grab the appropriate parts. Having the stats centralized would allow telemetry hitting a single endpoint, and plugins/modules can add to that without needing new endpoints.

For some of the examples listed (e.g., CCR, Enrich), the stats are not oriented at the node level

That is something we should consider. Is it correct then that they are based entirely on cluster state rather than node local information? If so that matches with a lot of the xpack usage api as well.

[jasontedor]

This one will require some care because it's used in the telemetry endpoint.

While I do agree it will require some care, the fact it is used by telemetry is one of the major benefits I see here. I believe (please correct me if I am wrong) that telemetry must be explicitly updated when new features are added to grab the appropriate parts. Having the stats centralized would allow telemetry hitting a single endpoint, and plugins/modules can add to that without needing new endpoints.

Let me expand on why it requires some care. In telemetry, we can not pull user-defined content (e.g., index names, enrich policy names, etc.). There are no exceptions to this. There's a tension then between the purpose of the /_cluster/stats endpoint as a useful stats endpoint, and its service as the telemetry endpoint. If we do this, it means we either have to:

• make the stats useless by not including user-defined content in them (i.e., the user can't see CCR follower stats at an index level)
• have a URI parameter that when enabled (e.g., ?telemetry=true) strips user-defined content from the endpoint; this means we effectively have to write two x-content methods for all of the stats that would be in this endpoint, taking a lot of care to ensure that we properly exclude user-defined content when the flag is enabled

This gets more challenging though in the context of third-party plugins that could register and add any content they want to this endpoint, and we have no way to ensure that they are properly observing the ?telemetry=true flag so that user-defined content does not end up in telemetry. Okay, we can solve that by saying we do not expose any third-party plugins stats in this endpoint when ?telemetry=true, so now we need an additional mechanism to distinguish between our plugins and third-party plugins.

So, we either have a useless for user-facing stats endpoint, or a complex endpoint.

[williamrandolph]

@jasontedor

To sum up your comments: there are two issues with this issue description so far. I described a distinction between "info" and "stats," but didn't make other important distinctions.

First, there's the difference between node-level stats, cluster-level stats that are aggregations of node-level stats, and stats that are reporting things that have nothing to do with nodes. CCR and Enrich stats are examples of that last category. Such non-node-related stats could be per-index, per-cluster, or per-some-other-abstraction.

Second, there's the difference between user-defined content and non-user-defined content. We have to avoid putting user-defined content in our telemetry data, so the REST call that returns telemetry data should not be pluggable.

Sound right?

Plugins: Per-cluster stats vs per-node stats

So here's another pass through the plugins I mentioned:

• X-Pack plugin: /_xpack returns information that's local to the node (build hash and enabled plugins), but the node checks Cluster State to get license information. /_xpack/usage return per-cluster stats. (info docs, usage docs)
• CCR plugin: /_ccr/stats returns a mix of per-cluster and per-index index stats. (docs)
• Enrich plugin: /_enrich/_stats returns a mix of per-cluster stats (executing_policies) and per-ingest-node stats (coordinator_stats) (docs)
• Machine-learning plugin: /_ml/info is per-cluster information, and is really more about machine learning configuration than it is about runtime stats. In other words, it's not relevant to this project. (docs)
• SQL plugin: /_sql/stats is undocumented. It returns a stats block per node, but doesn't provide node IDs. It returns counts of the number of times a SQL construct (e.g. LIMIT, HAVING) has been used, and number of queries by source (e.g. sql-cli, rest api, jdbc). It doesn't collate this information, which would make it awkward to use in a large cluster.
• EQL plugin: /_eql/stats is not only undocumented, but is currently unimplemented as well.

It looks to me as if none of our documented plugin "stats" endpoints are returning information that would cleanly plug into the per-node stats and info endpoints. However, it does seem as if the undocumented SQL endpoint could be completed or made more useful with a "pluggable node stats" framework.

If we decided not to make "per-cluster" stats pluggable, would it still be worth making node stats and info pluggable? I'm not sure. It might be handy for plugin writers, but I don't know if there's demand. The only problem it seems to solve for us is making the SQL stats endpoint more useful, but I don't even know if that's an endpoint that we want to make into an official part of our API contracts.

On the other hand, if we do want to make "per-cluster" stats pluggable, is it necessary to make per-node stats pluggable under the _nodes and _nodes/stats endpoints? It seems like it would be handy, but if the pluggability priority is X-pack info and usage, we could leave the per-node work for later.

Telemetry and user data

Protecting user data is so important that I wonder if we want two different endpoints: one for a hard-coded list of approved telemetry stats, and another one that can return any arbitrarily defined cluster stat. One advantage of an explicit filtering mechanism for telemetry is that it would be very obvious when a code change might affect what comes back in a telemetry response. Right now it's not clear from the code that there's any kind of restriction on what can be allowed in ClusterStatsIndices, ClusterStatsNodes, or their various inner *Stats.

[rjernst]

Regarding only allowing our plugins to add to telemetry (so that we trust they are not adding user data), we already have the list of official plugins and modules. Currently this is used in the plugin installer, but we can make this available through the plugin service and perhaps on the Plugin class to easily identify if a plugin is official.