Graceful node shutdowns in ESIntegTestCase should integrate with ShutdownAwarePlugin

[DaveCTurner]

In #73690 we introduced a way for plugins to integrate with the node shutdown mechanism, but we bypass this mechanism when gracefully shutting nodes down in ESIntegTestCase. I think we should use the same shutdown mechanism in these tests as in production to ensure proper test coverage.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mosche]

@DaveCTurner Do you have any specific use case / integration test in mind here? It would be great to get some clarifications on the intend here. @rjernst and I chatted briefly about adding the shutdown module to integ tests, which is currently not present there. One of the questions is certainly if that's what you are looking for or if there's another solution (e.g. just using PluginShutdownService).

[DaveCTurner]

Nothing specific, it's more that our integ tests try to faithfully simulate a realistic cluster, including graceful node shutdowns, but the graceful node shutdowns we do in the tests today no longer reflect how we shut nodes down in production. There's a PR linked above where we had to do something special with a test that positively needs the real production behaviour, but more generally we should be exercising these code paths in our integ tests since it's much easier to work in a single JVM.

[mosche]

the graceful node shutdowns we do in the tests today no longer reflect how we shut nodes down in production

Is that just the case for serverless (by means of the sigterm plugin) or does this also apply to cloud?

@rjernst I suppose there's been good reasons to implement the sigterm plugin for serverless only. Is moving an option at all? It sounds like that plugin could be a flexible way on how to enable the behavior on a per test suite basis without reimplementing a similar TerminationHandlerProvider for tests only.

[rjernst]

For the purposes of invoking ShutdownAwarePlugin, I think using the PluginShutdownService explicitly in ESIntegTestCase (or more specifically InternalTestCluster) is fine. Sigterm termination is another thing entirely, and I think that logic should stay in place for now. It invokes the shutdown API and blocks, which I don't think we want to do in every integ test.

[mosche]

Just a brief update on this, I've tested the behavior of ESIntegTestCase based on integrating PluginShutdownService only. Though, I don't think this meets the expectations of this ticket.

While notifying plugins of node shutdowns in tests, cluster state remains untouched this way.
I couldn't find any tests that currently work around missing shutdown signals for plugins and confirm the value / relevance of this change.

Also, it's certainly not the solution with respect to the workaround added in #97959 (autoManageVotingExclusions).
I'm currently looking into this further to understand the minimum behavior required so IT tests pass if the workaround is disabled.