DeleteExpiredDataIT.testDeleteExpiredDataWithStandardThrottle test fails for 5 days

[przemekwitek]

Build scan:

[7.7.1] https://gradle-enterprise.elastic.co/s/lre3ofg54upw2
[6.8.10] https://gradle-enterprise.elastic.co/s/eudojx2jyn3rm
[6.8.10] https://gradle-enterprise.elastic.co/s/f6pgvpd4zbxxa
[7.7.1] https://gradle-enterprise.elastic.co/s/buujiz2gy4qoq
https://gradle-enterprise.elastic.co/s/xddi3rkcdnyne

Repro line:

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.DeleteExpiredDataIT.testDeleteExpiredDataWithStandardThrottle" \
  -Dtests.seed=C7F615C834A73F48 \
  -Dtests.security.manager=true \
  -Dtests.locale=es-DO \
  -Dtests.timezone=Pacific/Kwajalein \
  -Dcompiler.java=14 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.DeleteExpiredDataIT.testDeleteExpiredDataWithStandardThrottle" \
  -Dtests.seed=C7F615C834A73F48 \
  -Dtests.security.manager=true \
  -Dtests.locale=es-DO \
  -Dtests.timezone=Pacific/Kwajalein \
  -Dcompiler.java=14 \
  -Druntime.java=8

Reproduces locally?:

No

Applicable branches:

7.x, 7.8

Failure history:

This is a new test.
Started failing on May 20th.
~ 1 failure daily since then

Failure excerpt:

From
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.8+multijob-unix-compatibility/os=centos-6&&immutable/38/console

08:12:19 » WARN ][o.e.x.m.d.DatafeedManager] [integTest-2] Datafeed for [realtime-job-auto-stop] has seen no data in [2] attempts, and never seen any data previously, so stopping...
08:12:19 » WARN ][o.e.p.AllocatedPersistentTask] [integTest-2] task datafeed-realtime-job-auto-stop-datafeed failed with an exception
08:12:19 »  org.elasticsearch.xpack.ml.datafeed.DatafeedJob$EmptyDataCountException: null
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedJob.run(DatafeedJob.java:398) ~[x-pack-ml-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedJob.runRealtime(DatafeedJob.java:188) ~[x-pack-ml-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$Holder.executeRealTime(DatafeedManager.java:428) ~[x-pack-ml-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$Holder.access$600(DatafeedManager.java:311) [x-pack-ml-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$3.doRun(DatafeedManager.java:235) [x-pack-ml-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
08:12:19 »  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
08:12:19 »  	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
08:12:19 » WARN ][o.e.p.PersistentTasksClusterService] [integTest-2] persistent task datafeed-realtime-job-auto-stop-datafeed failed
08:12:19 »  org.elasticsearch.xpack.ml.datafeed.DatafeedJob$EmptyDataCountException: null
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedJob.run(DatafeedJob.java:398) ~[?:?]
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedJob.runRealtime(DatafeedJob.java:188) ~[?:?]
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$Holder.executeRealTime(DatafeedManager.java:428) ~[?:?]
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$Holder.access$600(DatafeedManager.java:311) ~[?:?]
08:12:19 »  	at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$3.doRun(DatafeedManager.java:235) ~[?:?]
08:12:19 »  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
08:12:19 »  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
08:12:19 »  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
08:12:19 »  	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
08:12:19 » ERROR][o.e.x.m.p.l.CppLogMessageHandler] [integTest-2] [forecast-it-test-memory-status] [autodetect/470145] [CForecastRunner.cc@495] Forecast cannot be executed as model memory status is not OK
08:12:19 » ERROR][o.e.x.m.p.l.CppLogMessageHandler] [integTest-2] [forecast-it-test-no-data] [autodetect/470211] [CForecastRunner.cc@495] Forecast cannot be executed as job requires data to have been processed and modeled
08:12:19 » WARN ][o.e.x.m.d.DatafeedManager] [integTest-2] [test-lookback-only-given-empty-index] Datafeed lookback retrieved no data
08:12:19 » ERROR][o.e.x.m.p.l.CppLogMessageHandler] [integTest-2] [restore-model-snapshot-job] [autodetect/471151] [CForecastRunner.cc@495] Forecast cannot be executed as job requires data to have been processed and modeled
08:12:19 » WARN ][o.e.x.m.MlAssignmentNotifier] [integTest-2] [realtime-job-test-enable-upgrade-mode] No node found to start datafeed [realtime-job-test-enable-upgrade-mode-datafeed]. Reasons [persistent task cannot be assigned while upgrade mode is enabled.]
08:12:19 » WARN ][o.e.x.m.MlAssignmentNotifier] [integTest-2] [realtime-job-test-enable-upgrade-mode] No node found to start datafeed [realtime-job-test-enable-upgrade-mode-datafeed]. Reasons [cannot start datafeed [realtime-job-test-enable-upgrade-mode-datafeed], because the job's [realtime-job-test-enable-upgrade-mode] state is stale]
08:12:19 » WARN ][o.e.x.m.j.p.a.AutodetectProcessManager] [integTest-2] [realtime-job-test-enable-upgrade-mode] No quantiles could be found for a job with processed records
08:12:19 » WARN ][o.e.x.m.u.p.ResultsPersisterService] [integTest-2] [bulk-failure-retry-job] experienced failure that cannot be automatically retried. Bulk failure message [failure in bulk execution:
08:12:19 »   ↑ repeated 2 times ↑
08:12:19 » WARN ][o.e.x.m.d.DatafeedTimingStatsReporter] [integTest-2] [bulk-failure-retry-job] failed to report datafeed timing stats
08:12:19 »  org.elasticsearch.ElasticsearchException: bulk-failure-retry-job experienced failure that cannot be automatically retried. See logs for bulk failures

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[davidkyle]

The failure occurred in the test cleanup

[2020-05-25T17:56:47,559][INFO ][o.e.x.m.i.DeleteExpiredDataIT] [testDeleteExpiredDataWithStandardThrottle] before test |  
  | [2020-05-25T17:56:47,559][INFO ][o.e.x.m.i.DeleteExpiredDataIT] [testDeleteExpiredDataWithStandardThrottle] [DeleteExpiredDataIT#testDeleteExpiredDataWithStandardThrottle]: setting up test |  
  | [2020-05-25T17:56:48,085][INFO ][o.e.x.m.i.DeleteExpiredDataIT] [testDeleteExpiredDataWithStandardThrottle] [DeleteExpiredDataIT#testDeleteExpiredDataWithStandardThrottle]: all set up test |  
  | [2020-05-25T17:56:49,157][INFO ][o.e.x.m.i.DeleteExpiredDataIT] [testDeleteExpiredDataWithStandardThrottle] [DeleteExpiredDataIT#testDeleteExpiredDataWithStandardThrottle]: cleaning up after test |  
  | [2020-05-25T17:56:49,870][INFO ][o.e.x.m.i.DeleteExpiredDataIT] [testDeleteExpiredDataWithStandardThrottle] [DeleteExpiredDataIT#testDeleteExpiredDataWithStandardThrottle]: cleaned up after test |  
  | [2020-05-25T17:56:49,871][INFO ][o.e.x.m.i.DeleteExpiredDataIT] [testDeleteExpiredDataWithStandardThrottle] after test

org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

at __randomizedtesting.SeedInfo.seed([C7F615C834A73F48:CF9641CEBAFAD0D4]:0) |  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551)
....

Logs are attached but I can't see the reason for the exception or indeed where it came from

logs.zip