TemplateUpgradeService runs updates under existing ThreadContext

[tvernum]

The TemplateUpgradeService has a high level flow of:

• receive ClusterChangedEvent
• check preconditions (global block, master node, etc)
• gather necessary updates from registered upgraders (plugins)
• apply template updates on the generic thread pool.

However

• the ClusterChangedEvent comes in with the same ThreadContext as the action that triggered the event (which might be a node join/leave, but it also might be a settings change or index create/delete over REST).
• The generic threadpool execute preserves the ThreadContext from the calling code.

Consequently, the template update runs with a ThreadContext that matches the original triggering action.
If X-Pack Security is enabled, that means that update which should run as _system might attempt to run as the user which authenticated to the Rest API. That user may not have privileges to perform that update.

[elasticmachine]

Pinging @elastic/es-core-infra

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

@tvernum I am removing the discuss label as I am not sure that discussion is needed about this issue. Please add it back and comment about what needs to be discussed if you still believe there is something to discuss.

[tvernum]

@ywelsch And I discussed this just after I raised it, and believed that it would be good to discuss this more widely.
We were of the view that cluster state events should be published in a system context (which they probably are on non-master nodes, but the master node shortcuts its own update event).