PointInTimeIT#testMultipleIndices failed (Unexpected ShardFailures)

[DaveCTurner]

Build scan: https://gradle-enterprise.elastic.co/s/alsc5n2wskfyc/console-log?task=:x-pack:plugin:core:internalClusterTest

Repro line:

REPRODUCE WITH: ./gradlew ':x-pack:plugin:core:internalClusterTest' --tests "org.elasticsearch.xpack.core.search.PointInTimeIT.testMultipleIndices" -Dtests.seed=9D7169CD5CBEFE35 -Dtests.security.manager=true -Dtests.locale=en-MT -Dtests.timezone=Etc/GMT-11 -Druntime.java=11

Reproduces locally?: Yes (after <5 iterations, doesn't fail reliably)

Applicable branches: master

Failure history: One previous similar-looking failure, in a PR branch: https://gradle-enterprise.elastic.co/s/3fzowvfm3t34i

Failure excerpt:

org.elasticsearch.xpack.core.search.PointInTimeIT > testMultipleIndices FAILED
    java.lang.AssertionError: Unexpected ShardFailures: [shard [[K3_-IIK9Q_m_2Q3INmxriA][index-2][7]], reason [org.elasticsearch.transport.RemoteTransportException: [node_s1][127.0.0.1:36963][indices:data/read/search[phase/dfs]]
    Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [7]
    	at org.elasticsearch.search.SearchService.findReaderContext(SearchService.java:648)
    	at org.elasticsearch.search.SearchService.getShard(SearchService.java:445)
    	at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:331)
    	at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$5(SearchTransportService.java:311)
    	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72)
    	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:192)
    	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:99)
    	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:81)
    	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:680)
    	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142)
    	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117)
    	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82)
    	at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:297)
    	at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:228)
    	at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:40)
    	at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:139)
    	at org.elasticsearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:151)
    	at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:420)
    	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246)
    	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:174)
    	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:131)
    	at java.lang.Thread.run(Thread.java:834)
    ], cause [org.elasticsearch.search.SearchContextMissingException: No search context found for id [7]
    	at org.elasticsearch.search.SearchService.findReaderContext(SearchService.java:648)
    	at org.elasticsearch.search.SearchService.getShard(SearchService.java:445)
    	at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:331)
    	at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$5(SearchTransportService.java:311)
    	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72)
    	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:192)
    	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:99)
    	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:81)
    	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:680)
    	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142)
    	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117)
    	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82)
    	at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:297)
    	at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:228)
    	at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:40)
    	at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:139)
    	at org.elasticsearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:151)
    	at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:420)
    	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246)
    	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:174)
    	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:131)
    	at java.lang.Thread.run(Thread.java:834)
    ]]
    Expected: <0>
         but: was <1>
        at __randomizedtesting.SeedInfo.seed([9D7169CD5CBEFE35:845FE8F49266CD52]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures(ElasticsearchAssertions.java:305)
        at org.elasticsearch.xpack.core.search.PointInTimeIT.testMultipleIndices(PointInTimeIT.java:137)
REPRODUCE WITH: ./gradlew ':x-pack:plugin:core:internalClusterTest' --tests "org.elasticsearch.xpack.core.search.PointInTimeIT.testMultipleIndices" -Dtests.seed=9D7169CD5CBEFE35 -Dtests.security.manager=true -Dtests.locale=en-MT -Dtests.timezone=Etc/GMT-11 -Druntime.java=11

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)