[CI] MixedClusterClientYamlTestSuiteIT timing out

[mark-vieira]

This has started happening quite often, there might be two separate issues here as it seems to be related to two separate BWC versions. For the failures against 6.8.15 clusters it's possible that it's related to #66837. For the 7.11 BWC timeouts, it's not clear what's happening.

Build scan:
6.8.15: https://gradle-enterprise.elastic.co/s/xoyrv3wc2lndw/console-log?task=:qa:mixed-cluster:v6.8.15%23mixedClusterTest
7.11.2: https://gradle-enterprise.elastic.co/s/pmdhdw3nyy5gg/console-log?task=:qa:mixed-cluster:v7.11.2%23mixedClusterTest*:

Applicable branches:
7.x, 7.12, 7.11

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=America/Los_Angeles&tests.container=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT&tests.disabledDistributions=WyJvdXRjb21lOnNraXBwZWQiXQ&tests.sortField=FAILED&tests.test=classMethod&tests.unstableOnly=true

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[mark-vieira]

Apologies, it's clear to me what area this distinctly lands in so I've added both core/features and distributed teams to this issue.

[jakelandis]

It is believed that #69147 (once fully back ported) will help here (possibly fix).

[mark-vieira]

@jakelandis as I understand it #69147 only addresses all the additional errors when a 6.8.15 node is involved so wouldn't specifically address the issue here with the timeouts occurring against 7.11.2 clusters?

[jakelandis]

@mark-vieira #69147 is applicable across both 7.x and master. The scenerio is for a mixed cluster where the master does not know about composable templates, but the upgraded node does and tries to install one.

I haven't fully looked into if this is the root cause of the timeout, but it does cause alot of log noise and possibly some retry logic. If we keep timing out after #69147 has landed all branches then we probably have a different problem to investigate.

[benwtrent]

Here is a new timeout: https://gradle-enterprise.elastic.co/s/az2nnpcpm5dd6

Looking through the logs, I couldn't see anything around component templates.

Though there were a bit of:

» [2021-03-01T17:19:34,928][WARN ][o.e.i.s.IndexShard ] [v7.11.2-1] [test][0] no index mapper found for field: [text._index_prefix] returning default postings format

But that may be desired given its a mixed cluster test.

Many of the thread stacks were identical, but this one stood out:

  2>   10) Thread[id=297, name=TEST-MixedClusterClientYamlTestSuiteIT.test-seed#[787B410092F1C71E], state=RUNNABLE, group=TGRP-MixedClusterClientYamlTestSuiteIT]
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3963)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0  2> .10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4254)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4941)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Ques.match(Pattern.java:4253)
  2>         at java.base@11.0.10/java.util.regex.Pattern$CharPropertyGreedy.match(Pattern.java:4306)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.matchInit(Pattern.java:4963)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Prolog.match(Pattern.java:4887)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3964)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BranchConn.match(Pattern.java:4713)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match0(Pattern.java:4417)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match(Pattern.java:4379)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4749)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BranchConn.match(Pattern.java:4713)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BranchConn.match(Pattern.java:4713)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3964)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4749)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4747)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4749)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BranchConn.match(Pattern.java:4713)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BranchConn.match(Pattern.java:4713)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3964)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4749)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4747)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4749)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BranchConn.match(Pattern.java:4713)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BranchConn.match(Pattern.java:4713)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3964)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4749)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4747)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4749)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Branch.match(Pattern.java:4747)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharPropertyGreedy.match(Pattern.java:4344)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match0(Pattern.java:4424)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match(Pattern.java:4379)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4953)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3964)
  2>         at ja  2> va.base@11.0.10/java.util.regex.Pattern$Curly.match0(Pattern.java:4424)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match(Pattern.java:4379)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4920)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3964)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match0(Pattern.java:4424)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match(Pattern.java:4379)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.match(Pattern.java:4920)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupTail.match(Pattern.java:4863)
  2>         at java.base@11.0.10/java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3964)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match0(Pattern.java:4417)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Curly.match(Pattern.java:4379)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.matchInit(Pattern.java:4963)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Prolog.match(Pattern.java:4887)
  2>         at java.base@11.0.10/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Loop.matchInit(Pattern.java:4963)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Prolog.match(Pattern.java:4887)
  2>         at java.base@11.0.10/java.util.regex.Pattern$Begin.match(Pattern.java:3683)
  2>         at java.base@11.0.10/java.util.regex.Matcher.search(Matcher.java:1729)
  2>         at java.base@11.0.10/java.util.regex.Matcher.find(Matcher.java:746)
  2>         at app//org.elasticsearch.test.hamcrest.RegexMatcher.matchesSafely(RegexMatcher.java:36)
  2>         at app//org.elasticsearch.test.hamcrest.RegexMatcher.matchesSafely(RegexMatcher.java:19)
  2>         at app//org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:67)
  2>         at app//org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10)
  2>         at app//org.junit.Assert.assertThat(Assert.java:956)
  2>         at app//org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:57)
  2>         at app//org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:65)
  2>         at app//org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:444)
  2>         at app//org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:428)
  2>         at jdk.internal.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
  2>         at java.base@11.0.10/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.10/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  2>         at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2>         at app//org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  2>         at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2>         at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  2>         at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2>         at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2>         at app//org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2>         at java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

[benwtrent]

another failure: https://gradle-enterprise.elastic.co/s/hvmjgbyzml24i

This may be just the yaml tests being parsed are taking a while (and thus both thread stacks talk about parsing the test matches) or it could be something more fundamentally wrong there.

Another similar thread stack:

  2> "TEST-MixedClusterClientYamlTestSuiteIT.test-seed#[3FAF59F1BA0C6220]" ID=294 RUNNABLE
  2> 	at java.util.regex.Pattern$CharProperty.match(Pattern.java:3790)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4241)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.matchInit(Pattern.java:4815)
  2> 	at java.util.regex.Pattern$Prolog.match(Pattern.java:4755)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4616)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4616)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4  2> 731)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4616)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4616)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4808)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4786)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4786)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.matchInit(Pattern.java:4815)
  2> 	at java.util.regex.Pattern$Prolog.match(Pattern.java:4755)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4808)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4196)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4799)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Ques.match(Pattern.java:4195)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.matchInit(Pattern.java:4815)
  2> 	at java.util.regex.Pattern$Prolog.match(Pattern.java:4755)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.jav  2> a:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4616)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4616)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4582)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4616)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4618)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Branch.match(Pattern.java:4616)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4808)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4786)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.match(Pattern.java:4786)
  2> 	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4731)
  2> 	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3812)
  2> 	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  2> 	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.matchInit(Pattern.java:4815)
  2> 	at java.util.regex.Pattern$Prolog.match(Pattern.java:4755)
  2> 	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  2> 	at java.util.regex.Pattern$Loop.matchInit(Pattern.java:4815)
  2> 	at java.util.regex.Pattern$Prolog.match(Pattern.java:4755)
  2> 	at java.util.regex.Pattern$Begin.match(Pattern.java:3539)
  2> 	at java.util.regex.Matcher.search(Matcher.java:1248)
  2> 	at java.util.regex.Matcher.find(Matcher.java:637)
  2> 	at org.elasticsearch.test.hamcrest.RegexMatcher.matchesSafely(RegexMatcher.java:36)
  2> 	at org.elasticsearch.test.hamcrest.RegexMatcher.matchesSafely(RegexMatcher.java:19)
  2> 	at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:67)
  2> 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10)
  2> 	at org.junit.Assert.assertThat(Assert.java:956)
  2> 	at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:57)
  2> 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:65)
  2> 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:444)
  2> 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:428)
  2> 	at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
  2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.lang.reflect.Method.invoke(Method.java:498)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleStoreClassNa  2> me$1.evaluate(TestRuleStoreClassName.java:41)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at java.lang.Thread.run(Thread.java:748)

[jakelandis]

I don't think this is directly related to #69147,it appears that a test is possibly getting stuck. There are not any telling errors in the logs, but all nodes get killed at about the same time after a very long pause (the logs are noisy with debug messages prior to this pause).

[2021-03-01T18:30:32,686][DEBUG][o.e.c.c.PublicationTransportHandler] [v7.11.2-0] received diff cluster state version [8073] with uuid [wlfOSsTyQFunRvt7NNAaoA], diff size [289]
[2021-03-01T19:05:34.020349348Z] [BUILD] Stopping node