[CI] MultiClusterSearchWithSecurityYamlTestSuiteIT

[albertzaharovits]

The following Rest IntegTest is failing:

./gradlew :x-pack:qa:multi-cluster-search-security:mixedClusterTestRunner -Dtests.seed=986EC309467F6543 -Dtests.class=org.elasticsearch.xpack.security.MultiClusterSearchWithSecurityYamlTestSuiteIT -Dtests.method="test {yaml=multi_cluster/20_info/Add transient remote cluster based on the preset cluster and check remote info}" -Dtests.security.manager=true -Dtests.locale=gu -Dtests.timezone=Pacific/Fiji -Dtests.rest.suite=multi_cluster

with

MultiClusterSearchWithSecurityYamlTestSuiteIT.test {yaml=multi_cluster/20_info/Add transient remote cluster based on the preset cluster and check remote info} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failure at [multi_cluster/20_info:64]: expected [2xx] status code but api [search] returned [403 Forbidden] [{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/shards/search_shards] is unauthorized for user [_system]","stack_trace":"ElasticsearchSecurityException[action [indices:admin/shards/search_shards] is unauthorized for user [_system]]\n\tat org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:552)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:157)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:147)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:150)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)\n\tat 

and does not reproduce locally, looks like a race.

Indeed _system user somehow gets to handle a shard search which is not something he has privileges for and is rightfully rejected.

The problem is to see how the _system user assigned to a search request without any Authorization header: x-pack/qa/multi-cluster-search-security/src/test/resources/rest-api-spec/test/multi_cluster/20_info.yml:L64

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=virtual&&linux/124/consoleFull

Note: looks similar to #30565

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

Another identical failure, this time on 6.x:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=amazon/1132/console

17:43:08   1> [2018-06-20T05:43:05,769][INFO ][o.e.x.s.MultiClusterSearchWithSecurityYamlTestSuiteIT] Stash dump on test failure [{
17:43:08   1>   "stash" : {
17:43:08   1>     "remote_ip" : "[::1]:46125",
17:43:08   1>     "body" : {
17:43:08   1>       "error" : {
17:43:08   1>         "root_cause" : [
17:43:08   1>           {
17:43:08   1>             "type" : "security_exception",
17:43:08   1>             "reason" : "action [indices:admin/shards/search_shards] is unauthorized for user [_system]",
17:43:08   1>             "stack_trace" : "ElasticsearchSecurityException[action [indices:admin/shards/search_shards] is unauthorized for user [_system]]
17:43:08   1> 	at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:574)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:552)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:157)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:147)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:150)
17:43:08   1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:172)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:205)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:216)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:170)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:131)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:101)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:129)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:302)
17:43:08   1> 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
17:43:08   1> 	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1663)
17:43:08   1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
17:43:08   1> 	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135)
17:43:08   1> 	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1621)
17:43:08   1> 	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1485)
17:43:08   1> 	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
17:43:08   1> 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
17:43:08   1> 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
17:43:08   1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
17:43:08   1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
17:43:08   1> 	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
17:43:08   1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
17:43:08   1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
17:43:08   1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
17:43:08   1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
17:43:08   1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
17:43:08   1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
17:43:08   1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
17:43:08   1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
17:43:08   1> 	at java.lang.Thread.run(Thread.java:844)
17:43:08   1> "
17:43:08   1>           }
17:43:08   1>         ],
17:43:08   1>         "type" : "security_exception",
17:43:08   1>         "reason" : "action [indices:admin/shards/search_shards] is unauthorized for user [_system]",
17:43:08   1>         "stack_trace" : "ElasticsearchSecurityException[action [indices:admin/shards/search_shards] is unauthorized for user [_system]]
17:43:08   1> 	at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:574)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:552)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:157)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:147)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:150)
17:43:08   1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:172)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:205)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:216)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:170)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:131)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:101)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:129)

[jasontedor]

Maybe this was caused by #31241?

[jasontedor]

@ywelsch Would you take a look at this one?