Backport [SPARK-5847] Allow for configuring MetricsSystem's prefix

[AnthonyTruchet]

This is a backport of #14270. Because the spark.internal.config system
does not exists in branch 1.6, a simpler substitution scheme for ${} in
the spark.metrics.namespace value, using only Spark configuration had to
be added to preserve the behaviour discussed in the tickets and tested.

This backport is contributed by Criteo SA under the Apache v2 licence.

What changes were proposed in this pull request?

Adding a new property to SparkConf called spark.metrics.namespace that allows users to
set a custom namespace for executor and driver metrics in the metrics systems.

By default, the root namespace used for driver or executor metrics is
the value of spark.app.id. However, often times, users want to be able to track the metrics
across apps for driver and executor metrics, which is hard to do with application ID
(i.e. spark.app.id) since it changes with every invocation of the app. For such use cases,
users can set the spark.metrics.namespace property to any given value
or to another spark configuration key reference like ${spark.app.name}
which is then used to populate the root namespace of the metrics system
(with the app name in our example). spark.metrics.namespace property can be set to any
arbitrary spark property key, whose value would be used to set the root namespace of the
metrics system. Non driver and executor metrics are never prefixed with spark.app.id, nor
does the spark.metrics.namespace property have any such affect on such metrics.

How was this patch tested?

Added new unit tests, modified existing unit tests.

[HyukjinKwon]

Please ignore AppVeyor test here. (Please refer #15022 (comment)) it seems it happens only in other branchs as they don't have appveyor.yml and try to refer the configuration in the web (although I have to test this) (tested - before/after). I'd be great if any of auhorized one sets the branch to test to master brunch only as described in https://github.com/apache/spark/blob/master/dev/appveyor-guide.md#specifying-the-branch-for-building-and-setting-the-build-schedule

[vanzin]

We don't generally backport features to maintenance releases. Pinging @marmbrus who I believe is currently doing RM for the 1.6 line.

[AnthonyTruchet]

I'm aware that features are not generally back-ported. The point is, for us this is a bug, preventing a deployment in production. We thus back-ported the fix internally and now propose to share it with the community as the work is already done.

[marmbrus]

Thanks for spending the time to backport this, but it does seem a little risky to include changes to the configuration system in a maintenance release. As such, I'd probably error on the side of caution and close this PR unless there are a lot of 1.6 users clamoring for this functionality.

[AnthonyTruchet]

This is perfectly understandable, Just in the same way that we err on the side of caution by not switching to 2.0 branch for prod just now :-)

[SparkQA]

Test build #3262 has finished for PR 15023 at commit c1c57fb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Thanks for understanding! I do hope you guys upgrade eventually, there's a lot of good stuff and 2.0.1 should be out in the near future. Please do report any issues you see :)