[CI] MachineLearningLicensingTests testMachineLearningStopDatafeedActionNotRestricted failing with shardIndex is already set

[mark-vieira]

This has happened three times today, twice on master and once on 7.x.

WARNING: Uncaught exception in thread: Thread[elasticsearch[node_t0][search][T#6],5,TGRP-MachineLearningLicensingTests]
java.lang.AssertionError: shardIndex: 8 is already set

https://gradle-enterprise.elastic.co/s/yf7z2t4wmiu2a/tests/7bifs6bt3pims-itrx5zsacwiae
https://gradle-enterprise.elastic.co/s/mkkdmmvo6to64/tests/7bifs6bt3pims-itrx5zsacwiae
https://gradle-enterprise.elastic.co/s/mvhcps4bsvsdu/tests/7bifs6bt3pims-itrx5zsacwiae

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[mark-vieira]

Interestingly I a similar error in a completely different test, FieldSortIT:

https://gradle-enterprise.elastic.co/s/wu5v3wyltzbk2/tests/kyv2y2z3r4v7m-j4mq7ctqehcoa

[droberts195]

It’s been seen in the standalone ML QA tests to. There is speculation it’s due to #51708

[albertzaharovits]

Another one on 7.x https://gradle-enterprise.elastic.co/s/2dmqtvz7ojrec :

        org.elasticsearch.license.MachineLearningLicensingTests > testMachineLearningStopDatafeedActionNotRestricted FAILED	
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=227, name=elasticsearch[node_t0][search][T#4], state=RUNNABLE, group=TGRP-MachineLearningLicensingTests]	
	
        Caused by:	
        java.lang.AssertionError: shardIndex: 7 is already set	
            at __randomizedtesting.SeedInfo.seed([40766AD4C52C6E43]:0)	
            at org.elasticsearch.action.search.ArraySearchPhaseResults.consumeResult(ArraySearchPhaseResults.java:43)	
            at org.elasticsearch.action.search.SearchPhaseController$1.consumeResult(SearchPhaseController.java:745)	
            at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:471)	
            at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:67)	
            at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:235)	
            at org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:45)	
            at org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:29)	
            at org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:68)	
            at org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:36)	
            at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54)	
            at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:397)	
            at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1116)	
            at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1194)	
            at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1174)	
            at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)

[droberts195]

#52042 is a combined issue for all the tests that are failing with this same error. Nothing was changed in ML to cause this failure so it is highly likely that nothing can be done about it in ML code. I will leave this issue open as a way to navigate to #52042.

[droberts195]

Fixed by #52063