[CI][ML] Rolling upgrade failure in '30_ml_jobs_crud/Test model memory limit is updated'

[davidkyle]

Several failures in the rolling upgrade tests with the same error

java.lang.AssertionError: Failure at [upgraded_cluster/30_ml_jobs_crud:66]: jobs.0.analysis_limits.model_memory_limit didn't match expected value:
jobs.0.analysis_limits.model_memory_limit: expected [130mb] but was [100mb]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+bwc-tests/180/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+bwc-tests/179/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+bwc-tests/8/console

[elasticmachine]

Pinging @elastic/ml-core

[davidkyle]

Muted on 6.6 7827b94, 6.x 6012e98

[droberts195]

These always seem to happen when upgrading from 6.1. The 30% discrepancy probably means we’ve accidentally changed which versions the 30% uplift in model memory limit applies to during the job-in-index work.

[droberts195]

The problem here is that we look at the job version to work out whether the model memory limit needs to be uplifted. But we also set the job version to current when migrating from cluster state to an index.

I think the solution is to add the model memory limit uplift logic into MlConfigMigrator.updateJobForMigration. This will guarantee that every job that is stored in the config index has had the uplift applied if appropriate. In the master branch the logic can be completely removed from TransportOpenJobAction.

[droberts195]

Fixes are #37125 for 6.6/6.x and #37126 for master.