[SPARK-33249][CORE][UI] Add status plugin for live application

[kuwii]

What changes were proposed in this pull request?

• Add an interface called AppLiveStatusPlugin for live application to create extra listeners that can write a given store.
• Add a configuration called spark.appLiveStatusPlugins, to specify what AppLiveStatusPlugin should be loaded.
• In SparkContext, load AppLiveStatusPlugin using Utils.loadExtensions, then create and register listeners.

Why are the changes needed?

There are cases that developers may want to extend the current REST API of Web UI. In most of these cases, adding external module is a better option than directly editing the original Spark code.

For an external module, to extend the REST API of the Web UI, 2 things may need to be done:

• Add extra API to provide extra status info. This can be simply done by implementing another ApiRequestContext which will be automatically loaded.
• If the info can not be calculated from the original data in the store, add extra listeners to generate them.

For history server, there is an interface called AppHistoryServerPlugin, which is loaded based on SPI, providing a method to create listeners. In live application, the only way is spark.extraListeners based on Utils.loadExtensions. But this is not enough for the cases.

To let the API get the status info, the data need to be written to the AppStatusStore, which is the only store that an API can get by accessing ui.store or ui.sc.statusStore. But listeners created by Utils.loadExtensions only get a SparkConf in construction, and are unable to write the AppStatusStore.

So I think we still need plugin like AppHistorySever for live UI. For concerns like SPARK-22786, the plugin for live app can be separated from the history server one, and also loaded using Utils.loadExtensions with an extra configurations. So by default, nothing will be loaded.

Does this PR introduce any user-facing change?

Yes, a new configuration property called spark.appLiveStatusPlugins, indicating all AppLiveStatusPlugin to load.

How was this patch tested?

Tested using existed UT.

Also add tests for AppLiveStatusPlugin to make sure plugins in spark.appLiveStatusPlugins will be loaded and listeners will be created and registered.

[AmplabJenkins]

Can one of the admins verify this patch?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!