[ML] Review and upgrade static node settings and into dynamic cluster settings

[dimitris-athanasiou]

ML has accumulated some static node settings that make it hard for users to change. These settings require setting the setting to each node and restarting the node. For example, max.anomaly.records limits the number of records that our output per bucket to 500 by default. There has been users requesting increasing that limit. We should update that setting to be a dynamic cluster setting and deprecate the old one. At the same time, it would be good to review other similar settings.

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

This is something you could look at when you have a spare minute @benwtrent.

I think the settings to make dynamic and cluster-wide are:

• AutodetectBuilder.MAX_ANOMALY_RECORDS_SETTING - new setting should be xpack.ml.max_anomaly_records
• DataCountsReporter.ACCEPTABLE_PERCENTAGE_DATE_PARSE_ERRORS_SETTING - new setting should be xpack.ml.max_percent_date_errors
• DataCountsReporter.ACCEPTABLE_PERCENTAGE_OUT_OF_ORDER_ERRORS_SETTING - new setting should be xpack.ml.max_percent_out_of_order_errors

(If you spot any other good candidates let me know.)

In each case, add Property.Deprecated to the existing setting together with a @Deprecated annotation, and introduce the new setting with Property.Dynamic, Property.NodeScope with the old setting as a fallback. There's an example here:

elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/job/process/autodetect/AutodetectProcessManager.java

Line 105 in 8d50c10

Setting.intSetting("xpack.ml.max_open_jobs", MAX_RUNNING_JOBS_PER_NODE, 1, Property.NodeScope);

Also, because the new settings are going to be dynamic, you need to store the dynamic value in a variable somewhere, and register a callback to update it when the user updates the setting. There's an example here:

elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/action/TransportOpenJobAction.java

Line 695 in 8d50c10

.addSettingsUpdateConsumer(MachineLearning.CONCURRENT_JOB_ALLOCATIONS, this::setMaxConcurrentJobAllocations);