[CI] TransportMonitoringMigrateAlertsActionTests failing in CI

[mark-vieira]

Looks like this was a recently added test.

Build scan:
https://gradle-enterprise.elastic.co/s/ewewexqv5nt4e/tests/:x-pack:plugin:monitoring:test/org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests/testRepeatedLocalAlertsRemoval#1

Repro line:
./gradlew ':x-pack:plugin:monitoring:test' --tests "org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests.testRepeatedLocalAlertsRemoval" -Dtests.seed=D03188317B615BEA -Dtests.security.manager=true -Dtests.locale=nb-NO -Dtests.timezone=Asia/Choibalsan -Druntime.java=11

Reproduces locally?:
No.

Applicable branches:
Only master but has also failed in the backport pr

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.tags=CI&search.timeZoneId=America/Los_Angeles&tests.container=org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests&tests.sortField=FAILED&tests.unstableOnly=true

Failure excerpt:

org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests > testRepeatedLocalAlertsRemoval FAILED
    [.watches] org.elasticsearch.index.IndexNotFoundException: no such index [.watches]
        at __randomizedtesting.SeedInfo.seed([4C1D81BB141BBF19:5515C5AF4842E3AD]:0)
        at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:945)
        at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:882)
        at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:838)
        at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:217)
        at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:197)
        at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:90)
        at org.elasticsearch.action.support.master.info.TransportClusterInfoAction.checkBlock(TransportClusterInfoAction.java:49)
        at org.elasticsearch.action.support.master.info.TransportClusterInfoAction.checkBlock(TransportClusterInfoAction.java:35)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:132)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:109)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:57)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:88)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64)
        at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
        at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:60)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72)
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:207)
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107)
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89)
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:685)
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142)
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117)
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82)
        at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:296)
        at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:228)
        at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:40)
        at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:139)
        at org.elasticsearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:151)
        at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:420)
        at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246)
        at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:174)
        at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:131)
        at java.lang.Thread.run(Thread.java:834)

[mark-vieira]

@jbaiera looks like you recently added this test. Can you please take a look?

[jbaiera]

This looks like the test is failing while waiting on the .watches index to be set up. I'll add some extra test logic to give watcher an appropriate amount of time to set up.

[jbaiera]

With #66416, the failure rate has lowered, but is still occurring. Will investigate if there are better ways to ensure the .watches index is set up correctly

[mark-vieira]

@jbaiera if this is still regularly occuring can we please mute it until we have a fix in all the applicable branches.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[jbaiera]

closing this in favor of #66586