[ML] More advanced model snapshot retention policy

[droberts195]

Currently ML anomaly detector model snapshots are written every 3-4 hours and retained for a number of days (default 1) specified in the model_snapshot_retention_days setting of the job configuration.

Many users have a desire to store model snapshots for longer than the default of 1 day. However, this means that 6-8 potentially large model snapshots are retained for each extra day that is configured.

It would be nicer if the retention policy could be changed to something like "retain all model snapshots for 1 day, plus 1 model snapshot per day for 6 days before that, plus 1 model snapshot per week for 21 days before that".

This leads to some questions:

1. How could we allow such policies to be configured in a way that is backwards compatible with our current configuration setting?
2. Or, should we completely change the way model snapshot retention is configured as a breaking change in 8.0, and migrate the existing setting to the new format on upgrade to 8.x?

Once the configuration mechanism is decided it should not be very difficult to make the code changes necessary to action the configured policy.

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[droberts195]

The plan for 7.8 is:

• Change the default for model_snapshot_retention_days for new jobs from 1 to 10
• Introduce a new job setting, daily_model_snapshot_retention_after_days, defaulting to 1 for new jobs and model_snapshot_retention_days for pre-7.8 jobs
• For days that are older than model_snapshot_retention_days, continue to delete all model snapshots
• For days that are in between daily_model_snapshot_retention_after_days and model_snapshot_retention_days delete all but the first model snapshot for that day
• Continue to respect the retain setting of model snapshots to allow selected model snapshots to be retained indefinitely