[CI] org.elasticsearch.client.documentation.SecurityDocumentationIT.testGetUsers failing in master since Oct 22

[dliappis]

Example build failure

https://gradle-enterprise.elastic.co/s/glagjcu63cncy

Reproduction line

/gradlew ':client:rest-high-level:integTestRunner' --tests "org.elasticsearch.client.documentation.SecurityDocumentationIT.testGetUsers" \
  -Dtests.seed=42D796933B048528 \
  -Dtests.security.manager=true \
  -Dtests.locale=th \
  -Dtests.timezone=America/Resolute \
  -Dcompiler.java=12 \
  -Druntime.java=11

It does not reproduce locally or on an OracleLinux 7 CI worker (tried with 20 and 100 iterations).

Example relevant log:

Log excerpt

09:10:06 »    ↓ errors and warnings from /var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/oraclelinux-7&&immutable/client/rest-high-level/build/testclusters/integTest-0/logs/es.stdout.log ↓
09:10:06 » WARN ][o.e.b.BootstrapChecks    ] [integTest-0] HTTPS is required in order to use the API key service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the API key service using the [xpack.security.authc.api_key.enabled] setting
09:10:06 » WARN ][o.e.b.BootstrapChecks    ] [integTest-0] HTTPS is required in order to use the token service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the token service using the [xpack.security.authc.token.enabled] setting
09:10:06 » WARN ][o.e.d.FileBasedSeedHostsProvider] [integTest-0] expected, but did not find, a dynamic hosts list at [/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/oraclelinux-7&&immutable/client/rest-high-level/build/testclusters/integTest-0/config/unicast_hosts.txt]
09:10:06 » WARN ][o.e.c.m.TemplateUpgradeService] [integTest-0] Templates are still reported as out of date after the upgrade. The template upgrade will be retried.
09:10:06 »   ↑ repeated 31 times ↑
09:10:06 » WARN ][o.e.x.c.a.AutoFollowCoordinator] [integTest-0] skipping auto-follower coordination
09:10:06 »  org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [ccr]
09:10:06 »  	at org.elasticsearch.license.LicenseUtils.newComplianceException(LicenseUtils.java:24) ~[x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.updateAutoFollowers(AutoFollowCoordinator.java:204) [x-pack-ccr-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.clusterChanged(AutoFollowCoordinator.java:323) [x-pack-ccr-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$6(ClusterApplierService.java:521) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?]
09:10:06 »  	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) [?:?]
09:10:06 »  	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
09:10:06 »  	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:517) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:492) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:176) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:699) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
09:10:06 »  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
09:10:06 »  	at java.lang.Thread.run(Thread.java:834) [?:?]
09:10:06 »   ↑ repeated 2 times ↑
09:10:06 » WARN ][o.e.l.LicenseService     ] [integTest-0] LICENSE [EXPIRED] ON [THURSDAY, JANUARY 01, 1970].
09:10:06 »  # IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT. OTHERWISE, PLEASE REACH OUT TO
09:10:06 »  # YOUR SUPPORT CONTACT.
09:10:06 »  # 
09:10:06 »  # COMMERCIAL PLUGINS OPERATING WITH REDUCED FUNCTIONALITY
09:10:06 »  # - security
09:10:06 »  #  - Cluster health, cluster stats and indices stats operations are blocked
09:10:06 »  #  - All data operations (read and write) continue to work
09:10:06 »  # - watcher
09:10:06 »  #  - PUT / GET watch APIs are disabled, DELETE watch API continues to work
09:10:06 »  #  - Watches execute and write to the history
09:10:06 »  #  - The actions of the watches don't execute
09:10:06 »  # - monitoring
09:10:06 »  #  - The agent will stop collecting cluster and indices metrics
09:10:06 »  #  - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]
09:10:06 »  # - graph
09:10:06 »  #  - Graph explore APIs are disabled
09:10:06 »  # - ml
09:10:06 »  #  - Machine learning APIs are disabled
09:10:06 »  # - logstash
09:10:06 »  #  - Logstash will continue to poll centrally-managed pipelines
09:10:06 »  # - beats
09:10:06 »  #  - Beats will continue to poll centrally-managed configuration
09:10:06 »  # - deprecation
09:10:06 »  #  - Deprecation APIs are disabled
09:10:06 »  # - upgrade
09:10:06 »  #  - Upgrade API is disabled
09:10:06 »  # - sql
09:10:06 »  #  - SQL support is disabled
09:10:06 »  # - rollup
09:10:06 »  #  - Creating and Starting rollup jobs will no longer be allowed.
09:10:06 »  #  - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function.
09:10:06 »  # - transform
09:10:06 »  #  - Creating, starting, updating transforms will no longer be allowed.
09:10:06 »  #  - Stopping/Deleting existing transforms continue to function.
09:10:06 »  # - analytics
09:10:06 »  #  - Aggregations provided by Data Science plugin are no longer usable.
09:10:06 » WARN ][o.e.s.SnapshotShardsService] [integTest-0] [[.security-7][0]][test_repository:test_snapshot/nbwyF2cJSs2i-_0VMGCUww] failed to snapshot shard
09:10:06 »  org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: Aborted
09:10:06 »  	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotShard(BlobStoreRepository.java:1098) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:338) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.snapshots.SnapshotShardsService.lambda$startNewShards$1(SnapshotShardsService.java:286) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:699) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
09:10:06 »  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
09:10:06 »  	at java.lang.Thread.run(Thread.java:834) [?:?]
09:10:06 » ERROR][o.e.x.w.Watcher          ] [integTest-0] triggered watches could not be deleted [DljOFUpDeW_3cc13089-9113-47ce-8b98-83fbc25b29cb-2019-10-24T06:04:34.899534Z], failure [[.triggered_watches] org.elasticsearch.index.IndexNotFoundException: no such index [.triggered_watches]]
09:10:06 » ERROR][o.e.x.w.Watcher          ] [integTest-0] triggered watches could not be deleted [HQjyhEueVV_9ff73d86-0ef0-4be6-8725-c7e7b2a173d9-2019-10-24T06:04:37.024772Z], failure [[.triggered_watches] org.elasticsearch.index.IndexNotFoundException: no such index [.triggered_watches]]
09:10:06 » ERROR][o.e.x.i.IndexLifecycleRunner] [integTest-0] policy [my_policy] for index [my_index] failed on step [{"phase":"warm","action":"shrink","name":"shrink"}]. Moving to ERROR step
09:10:06 »  java.lang.IllegalArgumentException: the number of target shards [3] must be less that the number of source shards [2]
09:10:06 »  	at org.elasticsearch.cluster.metadata.IndexMetaData.selectShrinkShards(IndexMetaData.java:1537) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.admin.indices.shrink.TransportResizeAction.prepareCreateIndexRequest(TransportResizeAction.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.admin.indices.shrink.TransportResizeAction.lambda$masterOperation$2(TransportResizeAction.java:109) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:89) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:144) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:138) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:380) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onNodeResponse(TransportBroadcastByNodeAction.java:349) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:321) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:313) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1081) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1159) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1139) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:414) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:388) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:228) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:196) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:277) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:196) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.ArrayList.forEach(ArrayList.java:1540) [?:?]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:196) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.ArrayList.forEach(ArrayList.java:1540) [?:?]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:196) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.ArrayList.forEach(ArrayList.java:1540) [?:?]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:196) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.ArrayList.forEach(ArrayList.java:1540) [?:?]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:196) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.ArrayList.forEach(ArrayList.java:1540) [?:?]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:290) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:314) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:287) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:152) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:231) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:112) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:103) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:284) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:724) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:10:06 »  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
09:10:06 »  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
09:10:06 »  	at java.lang.Thread.run(Thread.java:834) [?:?]
09:10:06 »   ↑ repeated 2 times ↑

Frequency

3 times so far in CI

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-7d,mode:quick,to:now))&_a=(columns:!(_source),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'class:%22org.elasticsearch.client.documentation.SecurityDocumentationIT%22%20AND%20test:%22testGetUsers%22%20AND%20branch:%20%22master%22'),sort:!(time,desc))

[elasticmachine]

Pinging @elastic/es-security (:Security/Client)

[DaveCTurner]

https://gradle-enterprise.elastic.co/s/lwq5a4bcyecfi is a more recent failure. @jkakavas does this have the extra logging you need and if so should we now mute this test?

[jkakavas]

it does, we can mute it - thanks

Relevant log line for troubleshooting:

  1> [2019-10-25T05:17:38,204][INFO ][o.e.c.d.SecurityDocumentationIT] [testGetUsers] [
User[username=remote_monitoring_user,roles=[remote_monitoring_collector,remote_monitoring_agent],metadata={_reserved=true},fullName=null,email=null], 
User[username=beats_system,roles=[beats_system],metadata={_reserved=true},fullName=null,email=null], 
User[username=user3,roles=[user3],metadata={},fullName=null,email=null], 
User[username=kibana,roles=[kibana_system],metadata={_reserved=true},fullName=null,email=null], 
User[username=elastic,roles=[superuser],metadata={_reserved=true},fullName=null,email=null], 
User[username=apm_system,roles=[apm_system],metadata={_reserved=true},fullName=null,email=null], 
User[username=user2,roles=[user2],metadata={},fullName=null,email=null], 
User[username=user1,roles=[user1],metadata={},fullName=null,email=null], 
User[username=mz,roles=[RWa,nqp,LLx],metadata={string=null, string_list=[]},fullName=N,email=null], 
User[username=logstash_system,roles=[logstash_system],metadata={_reserved=true},fullName=null,email=null]]

need to figure out where this fellow

User[username=mz,roles=[RWa,nqp,LLx],metadata={string=null, string_list=[]},fullName=N,email=null], 

comes from

[tvernum]

need to figure out where this fellow
User[username=mz, roles=[RWa,nqp,LLx], metadata={string=null, string_list=[]}, fullName=N, email=null], comes from

It's SecurityIT.randomUser

Now we need to work out why it isn't cleaned up...