upgraded_cluster/30_ml_jobs_crud fails with "No known job with id 'old-cluster-job'"

[DaveCTurner]

The following build failed: https://scans.gradle.com/s/crky52pgtdybm/console-log?task=:x-pack:qa:rolling-upgrade:v7.3.0%23upgradedClusterTestRunner

1> [2019-06-18T19:12:12,539][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] [p0=upgraded_cluster/30_ml_jobs_crud/Test get old cluster job's timing stats] before test
--
1> [2019-06-18T19:12:12,612][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
1>   "stash" : {
1>     "body" : {
1>       "error" : {
1>         "root_cause" : [
1>           {
1>             "type" : "resource_not_found_exception",
1>             "reason" : "No known job with id 'old-cluster-job'",
1>             "stack_trace" : "ResourceNotFoundException[No known job with id 'old-cluster-job']
1> 	at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)
1> 	at org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)
1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
1> 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)
1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
1> 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)
1> 	at org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)
1> 	at org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)
1> 	at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)
1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
1> 	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)
1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
1> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
1> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
1> 	at java.base/java.lang.Thread.run(Thread.java:834)
1> "
1>           }
1>         ],
1>         "type" : "resource_not_found_exception",
1>         "reason" : "No known job with id 'old-cluster-job'",
1>         "stack_trace" : "ResourceNotFoundException[No known job with id 'old-cluster-job']
1> 	at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)
1> 	at org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)
1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
1> 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)
1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
1> 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)
1> 	at org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)
1> 	at org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)
1> 	at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)
1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
1> 	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)
1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
1> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
1> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
1> 	at java.base/java.lang.Thread.run(Thread.java:834)
1> "
1>       },
1>       "status" : 404
1>     }
1>   }
1> }]
1> [2019-06-18T19:12:12,649][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] There are still tasks running after this test that might break subsequent tests [xpack/ml/job[c]].
1> [2019-06-18T19:12:12,650][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] [p0=upgraded_cluster/30_ml_jobs_crud/Test get old cluster job's timing stats] after test
2> REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:v7.3.0#upgradedClusterTestRunner --tests "org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.test {p0=upgraded_cluster/30_ml_jobs_crud/Test get old cluster job's timing stats}" -Dtests.seed=8DF4A47F62374ACB -Dtests.security.manager=true -Dtests.locale=kea-CV -Dtests.timezone=Asia/Nicosia -Dcompiler.java=12 -Druntime.java=11 -Dtests.rest.suite=upgraded_cluster
2> java.lang.AssertionError: Failure at [upgraded_cluster/30_ml_jobs_crud:114]: expected [2xx] status code but api [ml.get_job_stats] returned [404 Not Found] [{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"No known job with id 'old-cluster-job'","stack_trace":"ResourceNotFoundException[No known job with id 'old-cluster-job']\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)\n\tat org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)\n\tat org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"}],"type":"resource_not_found_exception","reason":"No known job with id 'old-cluster-job'","stack_trace":"ResourceNotFoundException[No known job with id 'old-cluster-job']\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)\n\tat org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)\n\tat org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"},"status":404}]
 
Caused by:
java.lang.AssertionError: expected [2xx] status code but api [ml.get_job_stats] returned [404 Not Found] [{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"No known job with id 'old-cluster-job'","stack_trace":"ResourceNotFoundException[No known job with id 'old-cluster-job']\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)\n\tat org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)\n\tat org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"}],"type":"resource_not_found_exception","reason":"No known job with id 'old-cluster-job'","stack_trace":"ResourceNotFoundException[No known job with id 'old-cluster-job']\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)\n\tat org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)\n\tat org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"},"status":404}]

I tried the given REPRODUCE WITH line locally but it did not reproduce this failure.

[elasticmachine]

Pinging @elastic/ml-core

[DaveCTurner]

Muted in 631c63c.

[przemekwitek]

AFAICT this is a duplicate of #43314 which I fixed yesterday with #43315.

I will unmute the test now

[DaveCTurner]

Thanks @przemekwitek, yes, that does look to be the same issue. I'll close this.

[DaveCTurner]

Oh wait, they're still muted. I'll let you close this once they're unmuted.

[przemekwitek]

They are muted as I'm waiting for the CI in #43358 to pass