[CI] DocsClientYamlTestSuiteIT test {yaml=reference/ccr/apis/get-ccr-stats} #35496
Description
Failure:
DocsClientYamlTestSuiteIT.test {yaml=reference/ccr/apis/get-ccr-stats/line_88} <<< FAILURES!
> Throwable #1: java.lang.AssertionError: Failure at [reference/ccr/apis/get-ccr-stats:74]: $body didn't match expected value:
> $body:
> auto_follow_stats:
> number_of_failed_follow_indices: same [0]
> number_of_failed_remote_cluster_state_requests: expected [0] but was [1]
> number_of_successful_follow_indices: same [0]
> recent_auto_follow_errors:
> received [1] more entries than expected
> follow_stats:
> indices:
> 0:
> index: same [follower_index]
> shards:
> 0:
> bytes_read: same [0]
> failed_read_requests: same [0]
> failed_write_requests: same [0]
> follower_global_checkpoint: same [0]
> follower_index: same [follower_index]
> follower_mapping_version: same [0]
> follower_max_seq_no: same [0]
> follower_settings_version: same [0]
> last_requested_seq_no: same [0]
> leader_global_checkpoint: same [0]
> leader_index: same [leader_index]
> leader_max_seq_no: same [0]
> operations_read: same [0]
> operations_written: same [0]
> outstanding_read_requests: same [0]
> outstanding_write_requests: same [0]
> read_exceptions: same [empty list]
> remote_cluster: same [remote_cluster]
> shard_id: same [0]
> successful_read_requests: same [0]
> successful_write_requests: same [0]
> time_since_last_read_millis: same [-1]
> total_read_remote_exec_time_millis: same [0]
> total_read_time_millis: same [0]
> total_write_time_millis: same [0]
> write_buffer_operation_count: same [0]
> write_buffer_size_in_bytes: same [0]
> at __randomizedtesting.SeedInfo.seed([9F632EDCF44DF
Does not reproduce. Log is full with connection error. The AutoFollowCoordinator could fetch remote cluster state, because the license couldn't be determined and that was caused by a node disconnect exception:
[2018-11-13T06:31:16,896][WARN ][o.e.x.c.a.AutoFollowCoordinator] [node-0] failure occurred while fetching cluster state for auto follow pattern [my_auto_follow_pattern]
org.elasticsearch.ElasticsearchStatusException: can not fetch remote cluster state as the license state of the remote cluster [remote_cluster] could not be determined
at org.elasticsearch.xpack.ccr.CcrLicenseChecker.clusterStateUnknownRemoteLicense(CcrLicenseChecker.java:425) ~[x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.CcrLicenseChecker.lambda$checkRemoteClusterLicenseAndFetchClusterState$5(CcrLicenseChecker.java:171) ~[x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.CcrLicenseChecker$1.onFailure(CcrLicenseChecker.java:217) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onFailure(RemoteClusterLicenseChecker.java:180) [x-pack-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1127) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService.lambda$onConnectionClosed$8(TransportService.java:982) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.ElasticsearchException: could not determine the license type for cluster [remote_cluster]
... 9 more
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [node-0][127.0.0.1:59722][cluster:monitor/xpack/info] disconnected
... SKIPPED 56 LINES ...
The test failed because a the auto follow stats are not expected to report failures (number_of_failed_remote_cluster_state_requests should be 0 and recent_auto_follow_errors should be empty).
Returned auto follow stats:
{
1> "auto_follow_stats" : {
1> "number_of_failed_follow_indices" : 0,
1> "number_of_failed_remote_cluster_state_requests" : 1,
1> "number_of_successful_follow_indices" : 0,
1> "recent_auto_follow_errors" : [
1> {
1> "leader_index" : "my_auto_follow_pattern",
1> "auto_follow_exception" : {
1> "type" : "exception",
1> "reason" : "ElasticsearchStatusException[can not fetch remote cluster state as the license state of the remote cluster [remote_cluster] could not be determined]; nested: ElasticsearchException
[could not determine the license type for cluster [remote_cluster]]; nested: NodeDisconnectedException[[node-0][127.0.0.1:59722][cluster:monitor/xpack/info] disconnected];",
1> "caused_by" : {
1> "type" : "status_exception",
1> "reason" : "can not fetch remote cluster state as the license state of the remote cluster [remote_cluster] could not be determined",
1> "caused_by" : {
1> "type" : "exception",
1> "reason" : "could not determine the license type for cluster [remote_cluster]",
1> "caused_by" : {
1> "type" : "node_disconnected_exception",
1> "reason" : "[node-0][127.0.0.1:59722][cluster:monitor/xpack/info] disconnected"
1> }
1> }
1> }
1> }
1> }
1> ]
1> }