RunDataFrameAnalyticsIT.testStopOutlierDetectionWithEnoughDocumentsToScroll failures

[przemekwitek]

A failure from today:

https://gradle-enterprise.elastic.co/s/5bmqki2fxth7i

Reproduce lines:

REPRODUCE WITH: gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testStopOutlierDetectionWithEnoughDocumentsToScroll" \
  -Dtests.seed=AEF305B74CA6788B \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Europe/Istanbul \
  -Dcompiler.java=13

REPRODUCE WITH: gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testStopOutlierDetectionWithEnoughDocumentsToScroll" \
  -Dtests.seed=AEF305B74CA6788B \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Europe/Istanbul \
  -Dcompiler.java=13

Failure stack trace from one of the tests suggests a problem with MultiSearch query performed as part of TransportGetDataFrameAnalyticsStatsAction action:

org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT > testStopOutlierDetectionWithEnoughDocumentsToScroll FAILED
    org.elasticsearch.ElasticsearchException: all shards failed
        at __randomizedtesting.SeedInfo.seed([AEF305B74CA6788B:3E02F093CBFBFFB1]:0)
        at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:55)
        at org.elasticsearch.xpack.ml.action.TransportGetDataFrameAnalyticsStatsAction.lambda$searchStats$8(TransportGetDataFrameAnalyticsStatsAction.java:201)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
        at org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:97)
        at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:144)
        at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:138)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.finish(TransportMultiSearchAction.java:179)
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.handleResponse(TransportMultiSearchAction.java:165)
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.onFailure(TransportMultiSearchAction.java:158)
        at org.elasticsearch.client.node.NodeClient.lambda$executeLocally$1(NodeClient.java:97)
        at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:153)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:568)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:548)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:308)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:577)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:388)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:244)
        at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:406)
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1067)
        at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:242)
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:177)
        at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:240)
        at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:232)
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:136)
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102)
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:667)
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:834)

        Caused by:
        Failed to execute phase [query], all shards failed
            at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:548)
            ... 49 more

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[droberts195]

The problem is happening because we're trying to search the new .ml-stats index very soon after it is created, before the shards have been allocated.

I am pretty sure we have had this problem before with a different index and solved it in some way...

[mark-vieira]

This test has failed a dozen times in the past few days so I've muted it in master (a134ccc) and 7.x (34671be374b2b045d6af47e56e6eae1cb1024f45).

[droberts195]

I adjusted the PR title because this didn't just happen on Windows. For example:

• CentOS 8: https://gradle-enterprise.elastic.co/s/ekmadt3ijei4w
• Debian 9: https://gradle-enterprise.elastic.co/s/i4mxj6gvu6evu

It's a timing issue, not a platform issue.

[mark-vieira]

Here's another failure after unmuting and adding logging:

https://gradle-enterprise.elastic.co/s/5zdq3jww3gte6/tests/zcquf3hc3eoda-so3ayypsxka5i

[benwtrent]

Some interesting logs

The node running the task

[2020-03-03T19:12:03,661][INFO ][o.e.x.m.a.TransportStartDataFrameAnalyticsAction] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] Starting data frame analytics
[2020-03-03T19:12:03,743][DEBUG][o.e.x.m.d.DataFrameAnalyticsManager] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] Starting to create/update .ml-state index
[2020-03-03T19:12:03,948][DEBUG][o.e.x.m.d.DataFrameAnalyticsManager] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] created .ml-state index. Starting to create/update .ml-stats index
[2020-03-03T19:12:03,987][INFO ][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] Stopping task with force [false]
[2020-03-03T19:12:03,988][DEBUG][o.e.x.m.d.DataFrameAnalyticsTask] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] stop called with reason [stop_data_frame_analytics (api)]
[2020-03-03T19:12:03,988][DEBUG][o.e.x.m.d.DataFrameAnalyticsTask] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] doStop called with reason [stop_data_frame_analytics (api)] and reindexTaskId [null]
[2020-03-03T19:12:03,988][DEBUG][o.e.x.m.d.p.AnalyticsProcessManager] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] No process context to stop
[2020-03-03T19:12:03,988][DEBUG][o.e.x.m.d.DataFrameAnalyticsTask] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] Persisting progress
[2020-03-03T19:12:03,988][DEBUG][o.e.x.m.a.TransportGetDataFrameAnalyticsStatsAction] [integTest-2] Get stats for data frame analytics [test_stop_outlier_detection_with_enough_docs_to_scroll]
[2020-03-03T19:12:03,991][DEBUG][o.e.x.m.a.TransportGetDataFrameAnalyticsStatsAction] [integTest-2] Get stats for running task [test_stop_outlier_detection_with_enough_docs_to_scroll]
[2020-03-03T19:12:04,027][DEBUG][o.e.x.m.d.DataFrameAnalyticsTask] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] Successfully indexed progress document
[2020-03-03T19:12:04,204][DEBUG][o.e.x.m.d.DataFrameAnalyticsManager] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] created .ml-stats index
[2020-03-03T19:12:04,207][DEBUG][o.e.x.m.d.DataFrameAnalyticsTask] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] Last incomplete progress [reindexing, 0]
[2020-03-03T19:12:04,208][DEBUG][o.e.x.m.d.DataFrameAnalyticsManager] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] Starting job from state [FIRST_TIME]
[2020-03-03T19:12:04,245][ERROR][o.e.x.m.d.DataFrameAnalyticsTask] [integTest-2] [test_stop_outlier_detection_with_enough_docs_to_scroll] Could not update task state to [failed] with reason [[integTest-0][127.0.0.1:39101][cluster:admin/persistent/update_status]]
org.elasticsearch.transport.RemoteTransportException: [integTest-0][127.0.0.1:39101][cluster:admin/persistent/update_status]
Caused by: org.elasticsearch.ResourceNotFoundException: the task with id data_frame_analytics-test_stop_outlier_detection_with_enough_docs_to_scroll and allocation id 87 doesn't exist
	at org.elasticsearch.persistent.PersistentTasksClusterService$4.execute(PersistentTasksClusterService.java:234) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

The Node handling the API requests:

[2020-03-03T19:12:02,006][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [test-stop-outlier-detection-with-enough-docs-to-scroll] creating index, cause [api], templates [random_index_template], shards [7]/[0], mappings [_doc]
[2020-03-03T19:12:02,226][INFO ][o.e.c.r.a.AllocationService] [integTest-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test-stop-outlier-detection-with-enough-docs-to-scroll][2], [test-stop-outlier-detection-with-enough-docs-to-scroll][5]]]).
[2020-03-03T19:12:02,533][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [.ml-config] creating index, cause [auto(bulk api)], templates [random_index_template, .ml-config], shards [7]/[0], mappings [_doc]
[2020-03-03T19:12:02,534][INFO ][o.e.c.r.a.AllocationService] [integTest-0] updating number_of_replicas to [1] for indices [.ml-config]
[2020-03-03T19:12:02,727][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [.ml-annotations-6] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[2020-03-03T19:12:02,977][INFO ][o.e.x.m.MlInitializationService] [integTest-0] Created ML annotations index and aliases
[2020-03-03T19:12:03,153][INFO ][o.e.c.m.MetaDataMappingService] [integTest-0] [.ml-config/peQYne0tTdCex8uR7EEoSQ] update_mapping [_doc]
[2020-03-03T19:12:03,280][DEBUG][o.e.x.m.a.TransportGetDataFrameAnalyticsStatsAction] [integTest-0] Get stats for data frame analytics [test_stop_outlier_detection_with_enough_docs_to_scroll]
[2020-03-03T19:12:03,285][DEBUG][o.e.x.m.d.DataFrameAnalyticsTask] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] Last incomplete progress [reindexing, 0]
[2020-03-03T19:12:03,287][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] incompatible field [categorical_1]; types [keyword]; supported [boolean, byte, double, float, half_float, integer, long, scaled_float, short]
[2020-03-03T19:12:03,288][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] field [numeric_1] is compatible as it is numerical
[2020-03-03T19:12:03,288][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] field [numeric_2] is compatible as it is numerical
[2020-03-03T19:12:03,288][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] field [numeric_1] is compatible as it is numerical
[2020-03-03T19:12:03,288][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] field [numeric_2] is compatible as it is numerical
[2020-03-03T19:12:03,292][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] incompatible field [categorical_1]; types [keyword]; supported [boolean, byte, double, float, half_float, integer, long, scaled_float, short]
[2020-03-03T19:12:03,293][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] field [numeric_1] is compatible as it is numerical
[2020-03-03T19:12:03,293][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] field [numeric_2] is compatible as it is numerical
[2020-03-03T19:12:03,293][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] field [numeric_1] is compatible as it is numerical
[2020-03-03T19:12:03,293][DEBUG][o.e.x.m.d.e.ExtractedFieldsDetector] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] field [numeric_2] is compatible as it is numerical
[2020-03-03T19:12:03,295][DEBUG][o.e.x.m.d.e.DataFrameDataExtractor] [integTest-0] [memory_usage_estimation_43500] Data summary rows [1839]
[2020-03-03T19:12:03,362][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [.ml-notifications-000001] creating index, cause [auto(bulk api)], templates [.ml-notifications-000001], shards [1]/[1], mappings [_doc]
[2020-03-03T19:12:03,610][DEBUG][o.e.x.m.d.p.MemoryUsageEstimationProcessManager] [integTest-0] [memory_usage_estimation_43500] Closing process
[2020-03-03T19:12:03,611][DEBUG][o.e.x.m.d.p.MemoryUsageEstimationProcessManager] [integTest-0] [memory_usage_estimation_43500] Closed process
[2020-03-03T19:12:03,663][DEBUG][o.e.x.m.a.TransportStartDataFrameAnalyticsAction] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] waiting for task to start
[2020-03-03T19:12:03,734][DEBUG][o.e.x.m.a.TransportStartDataFrameAnalyticsAction] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] task is now started
[2020-03-03T19:12:03,743][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [integTest-0] Received request to stop data frame analytics [test_stop_outlier_detection_with_enough_docs_to_scroll]
[2020-03-03T19:12:03,746][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [integTest-0] Resolved data frame analytics to stop: [test_stop_outlier_detection_with_enough_docs_to_scroll]
[2020-03-03T19:12:03,756][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [.ml-state-000001] creating index, cause [api], templates [.ml-state], shards [1]/[1], mappings [_doc]
[2020-03-03T19:12:04,001][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [.ml-stats-000001] creating index, cause [api], templates [.ml-stats], shards [1]/[1], mappings [_doc]
[2020-03-03T19:12:04,133][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] analytics is stopped.
[2020-03-03T19:12:04,161][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [integTest-0] Received request to stop data frame analytics [*]
[2020-03-03T19:12:04,164][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [integTest-0] Resolved data frame analytics to stop: []
[2020-03-03T19:12:04,242][WARN ][o.e.p.PersistentTasksClusterService] [integTest-0] trying to update state on non-existing task data_frame_analytics-test_stop_outlier_detection_with_enough_docs_to_scroll
[2020-03-03T19:12:04,244][WARN ][o.e.p.PersistentTasksClusterService] [integTest-0] trying to update state on non-existing task data_frame_analytics-test_stop_outlier_detection_with_enough_docs_to_scroll
[2020-03-03T19:12:04,338][INFO ][o.e.c.r.a.AllocationService] [integTest-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ml-stats-000001][0]]]).
[2020-03-03T19:12:04,351][INFO ][o.e.x.m.a.TransportDeleteDataFrameAnalyticsAction] [integTest-0] [test_stop_outlier_detection_with_enough_docs_to_scroll] Deleted
[2020-03-03T19:12:04,564][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [.ml-config/peQYne0tTdCex8uR7EEoSQ] deleting index
[2020-03-03T19:12:04,564][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [.ml-stats-000001/qDiTUS2GTIKZqCNiYk08kw] deleting index
[2020-03-03T19:12:04,564][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [.ml-annotations-6/rvcNloIUT_-h3hG3aG9mNg] deleting index
[2020-03-03T19:12:04,564][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [test-stop-outlier-detection-with-enough-docs-to-scroll/1Hw1RWgpQGCGCK8bU11-rg] deleting index
[2020-03-03T19:12:04,564][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [.ml-state-000001/ejt62B14RQ62ccME116S6w] deleting index
[2020-03-03T19:12:04,564][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [.ml-notifications-000001/GgJYJFG9TFiEbalDD_RZTA] deleting index